lwac 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 39b62b54afaf2a83e20c9679e9565aa97e103053
+  data.tar.gz: 0d0aa8384bead19ef07dde68cefda240079f12c8
+SHA512:
+  metadata.gz: f89aa00d904cd2fa06d14eb703393a824f683cb2fd13a1425ec0a74836e8ba0395a4f2cb1cb4308749c1b5841de15cabfc628e6922fc58d5b0360cc89729d554
+  data.tar.gz: 4df133f013e0b0fe85ed911b32944b0edd633faf48077c98b3f10720cce57480e600f6bcb945e32861b76a2fc64377913402e4fd0b9f505a99d991fff3c47931

data/LICENSE

@@ -0,0 +1,70 @@
+LICENSE
+-------
+LWAC is governed by a creative-commons noncommercial sharealike license, v3, plus some extensions.  In addition to the below CC license, I apply the conditions that you must:
+
+ * Not profit from directly selling the code (including as part of something else).
+ * Not profit from selling corpora built using LWAC.  (Note that I will probably grant permission if contacted, I merely wish to stop people turning it into a commercial service)
+ * Credit LWAC and provide a link to http://stephenwattam.com/project/LWAC/ or http://ucrel.lancs.ac.uk/LWAC/ in any publications (You can also cite the paper pending for WaC8, once it's published).
+ * Don't use it to DDOS things.
+
+The text of the CC license is below.  It is also available from http://creativecommons.org/licenses/by-nc-sa/3.0/legalcode and is nicely explained at http://creativecommons.org/licenses/by-nc-sa/3.0/ .
+
+License
+
+THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.
+
+BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY BE CONSIDERED TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.
+
+1. Definitions
+
+"Adaptation" means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation, adaptation, derivative work, arrangement of music or other alterations of a literary or artistic work, or phonogram or performance and includes cinematographic adaptations or any other form in which the Work may be recast, transformed, or adapted including in any form recognizably derived from the original, except that a work that constitutes a Collection will not be considered an Adaptation for the purpose of this License. For the avoidance of doubt, where the Work is a musical work, performance or phonogram, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered an Adaptation for the purpose of this License.
+"Collection" means a collection of literary or artistic works, such as encyclopedias and anthologies, or performances, phonograms or broadcasts, or other works or subject matter other than works listed in Section 1(g) below, which, by reason of the selection and arrangement of their contents, constitute intellectual creations, in which the Work is included in its entirety in unmodified form along with one or more other contributions, each constituting separate and independent works in themselves, which together are assembled into a collective whole. A work that constitutes a Collection will not be considered an Adaptation (as defined above) for the purposes of this License.
+"Distribute" means to make available to the public the original and copies of the Work or Adaptation, as appropriate, through sale or other transfer of ownership.
+"License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, Noncommercial, ShareAlike.
+"Licensor" means the individual, individuals, entity or entities that offer(s) the Work under the terms of this License.
+"Original Author" means, in the case of a literary or artistic work, the individual, individuals, entity or entities who created the Work or if no individual or entity can be identified, the publisher; and in addition (i) in the case of a performance the actors, singers, musicians, dancers, and other persons who act, sing, deliver, declaim, play in, interpret or otherwise perform literary or artistic works or expressions of folklore; (ii) in the case of a phonogram the producer being the person or legal entity who first fixes the sounds of a performance or other sounds; and, (iii) in the case of broadcasts, the organization that transmits the broadcast.
+"Work" means the literary and/or artistic work offered under the terms of this License including without limitation any production in the literary, scientific and artistic domain, whatever may be the mode or form of its expression including digital form, such as a book, pamphlet and other writing; a lecture, address, sermon or other work of the same nature; a dramatic or dramatico-musical work; a choreographic work or entertainment in dumb show; a musical composition with or without words; a cinematographic work to which are assimilated works expressed by a process analogous to cinematography; a work of drawing, painting, architecture, sculpture, engraving or lithography; a photographic work to which are assimilated works expressed by a process analogous to photography; a work of applied art; an illustration, map, plan, sketch or three-dimensional work relative to geography, topography, architecture or science; a performance; a broadcast; a phonogram; a compilation of data to the extent it is protected as a copyrightable work; or a work performed by a variety or circus performer to the extent it is not otherwise considered a literary or artistic work.
+"You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
+"Publicly Perform" means to perform public recitations of the Work and to communicate to the public those public recitations, by any means or process, including by wire or wireless means or public digital performances; to make available to the public Works in such a way that members of the public may access these Works from a place and at a place individually chosen by them; to perform the Work to the public by any means or process and the communication to the public of the performances of the Work, including by public digital performance; to broadcast and rebroadcast the Work by any means including signs, sounds or images.
+"Reproduce" means to make copies of the Work by any means including without limitation by sound or visual recordings and the right of fixation and reproducing fixations of the Work, including storage of a protected performance or phonogram in digital form or other electronic medium.
+2. Fair Dealing Rights. Nothing in this License is intended to reduce, limit, or restrict any uses free from copyright or rights arising from limitations or exceptions that are provided for in connection with the copyright protection under copyright law or other applicable laws.
+
+3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:
+
+to Reproduce the Work, to incorporate the Work into one or more Collections, and to Reproduce the Work as incorporated in the Collections;
+to create and Reproduce Adaptations provided that any such Adaptation, including any translation in any medium, takes reasonable steps to clearly label, demarcate or otherwise identify that changes were made to the original Work. For example, a translation could be marked "The original work was translated from English to Spanish," or a modification could indicate "The original work has been modified.";
+to Distribute and Publicly Perform the Work including as incorporated in Collections; and,
+to Distribute and Publicly Perform Adaptations.
+The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. Subject to Section 8(f), all rights not expressly granted by Licensor are hereby reserved, including but not limited to the rights described in Section 4(e).
+
+4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:
+
+You may Distribute or Publicly Perform the Work only under the terms of this License. You must include a copy of, or the Uniform Resource Identifier (URI) for, this License with every copy of the Work You Distribute or Publicly Perform. You may not offer or impose any terms on the Work that restrict the terms of this License or the ability of the recipient of the Work to exercise the rights granted to that recipient under the terms of the License. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties with every copy of the Work You Distribute or Publicly Perform. When You Distribute or Publicly Perform the Work, You may not impose any effective technological measures on the Work that restrict the ability of a recipient of the Work from You to exercise the rights granted to that recipient under the terms of the License. This Section 4(a) applies to the Work as incorporated in a Collection, but this does not require the Collection apart from the Work itself to be made subject to the terms of this License. If You create a Collection, upon notice from any Licensor You must, to the extent practicable, remove from the Collection any credit as required by Section 4(d), as requested. If You create an Adaptation, upon notice from any Licensor You must, to the extent practicable, remove from the Adaptation any credit as required by Section 4(d), as requested.
+You may Distribute or Publicly Perform an Adaptation only under: (i) the terms of this License; (ii) a later version of this License with the same License Elements as this License; (iii) a Creative Commons jurisdiction license (either this or a later license version) that contains the same License Elements as this License (e.g., Attribution-NonCommercial-ShareAlike 3.0 US) ("Applicable License"). You must include a copy of, or the URI, for Applicable License with every copy of each Adaptation You Distribute or Publicly Perform. You may not offer or impose any terms on the Adaptation that restrict the terms of the Applicable License or the ability of the recipient of the Adaptation to exercise the rights granted to that recipient under the terms of the Applicable License. You must keep intact all notices that refer to the Applicable License and to the disclaimer of warranties with every copy of the Work as included in the Adaptation You Distribute or Publicly Perform. When You Distribute or Publicly Perform the Adaptation, You may not impose any effective technological measures on the Adaptation that restrict the ability of a recipient of the Adaptation from You to exercise the rights granted to that recipient under the terms of the Applicable License. This Section 4(b) applies to the Adaptation as incorporated in a Collection, but this does not require the Collection apart from the Adaptation itself to be made subject to the terms of the Applicable License.
+You may not exercise any of the rights granted to You in Section 3 above in any manner that is primarily intended for or directed toward commercial advantage or private monetary compensation. The exchange of the Work for other copyrighted works by means of digital file-sharing or otherwise shall not be considered to be intended for or directed toward commercial advantage or private monetary compensation, provided there is no payment of any monetary compensation in con-nection with the exchange of copyrighted works.
+If You Distribute, or Publicly Perform the Work or any Adaptations or Collections, You must, unless a request has been made pursuant to Section 4(a), keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or if the Original Author and/or Licensor designate another party or parties (e.g., a sponsor institute, publishing entity, journal) for attribution ("Attribution Parties") in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; (ii) the title of the Work if supplied; (iii) to the extent reasonably practicable, the URI, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and, (iv) consistent with Section 3(b), in the case of an Adaptation, a credit identifying the use of the Work in the Adaptation (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). The credit required by this Section 4(d) may be implemented in any reasonable manner; provided, however, that in the case of a Adaptation or Collection, at a minimum such credit will appear, if a credit for all contributing authors of the Adaptation or Collection appears, then as part of these credits and in a manner at least as prominent as the credits for the other contributing authors. For the avoidance of doubt, You may only use the credit required by this Section for the purpose of attribution in the manner set out above and, by exercising Your rights under this License, You may not implicitly or explicitly assert or imply any connection with, sponsorship or endorsement by the Original Author, Licensor and/or Attribution Parties, as appropriate, of You or Your use of the Work, without the separate, express prior written permission of the Original Author, Licensor and/or Attribution Parties.
+For the avoidance of doubt:
+
+Non-waivable Compulsory License Schemes. In those jurisdictions in which the right to collect royalties through any statutory or compulsory licensing scheme cannot be waived, the Licensor reserves the exclusive right to collect such royalties for any exercise by You of the rights granted under this License;
+Waivable Compulsory License Schemes. In those jurisdictions in which the right to collect royalties through any statutory or compulsory licensing scheme can be waived, the Licensor reserves the exclusive right to collect such royalties for any exercise by You of the rights granted under this License if Your exercise of such rights is for a purpose or use which is otherwise than noncommercial as permitted under Section 4(c) and otherwise waives the right to collect royalties through any statutory or compulsory licensing scheme; and,
+Voluntary License Schemes. The Licensor reserves the right to collect royalties, whether individually or, in the event that the Licensor is a member of a collecting society that administers voluntary licensing schemes, via that society, from any exercise by You of the rights granted under this License that is for a purpose or use which is otherwise than noncommercial as permitted under Section 4(c).
+Except as otherwise agreed in writing by the Licensor or as may be otherwise permitted by applicable law, if You Reproduce, Distribute or Publicly Perform the Work either by itself or as part of any Adaptations or Collections, You must not distort, mutilate, modify or take other derogatory action in relation to the Work which would be prejudicial to the Original Author's honor or reputation. Licensor agrees that in those jurisdictions (e.g. Japan), in which any exercise of the right granted in Section 3(b) of this License (the right to make Adaptations) would be deemed to be a distortion, mutilation, modification or other derogatory action prejudicial to the Original Author's honor and reputation, the Licensor will waive or not assert, as appropriate, this Section, to the fullest extent permitted by the applicable national law, to enable You to reasonably exercise Your right under Section 3(b) of this License (right to make Adaptations) but not otherwise.
+5. Representations, Warranties and Disclaimer
+
+UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING AND TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO THIS EXCLUSION MAY NOT APPLY TO YOU.
+
+6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+7. Termination
+
+This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Adaptations or Collections from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
+Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.
+8. Miscellaneous
+
+Each time You Distribute or Publicly Perform the Work or a Collection, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
+Each time You Distribute or Publicly Perform an Adaptation, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
+If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
+No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
+This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.
+The rights granted under, and the subject matter referenced, in this License were drafted utilizing the terminology of the Berne Convention for the Protection of Literary and Artistic Works (as amended on September 28, 1979), the Rome Convention of 1961, the WIPO Copyright Treaty of 1996, the WIPO Performances and Phonograms Treaty of 1996 and the Universal Copyright Convention (as revised on July 24, 1971). These rights and subject matter take effect in the relevant jurisdiction in which the License terms are sought to be enforced according to the corresponding provisions of the implementation of those treaty provisions in the applicable national law. If the standard suite of rights granted under applicable copyright law includes additional rights not granted under this License, such additional rights are deemed to be included in the License; this License is not intended to restrict the license of any rights under applicable law.
+

data/README.md

@@ -0,0 +1,31 @@
+LWAC Downloader
+===============
+The system comprises two parts: a server which manages sample consistency and data storage, and a client which makes requests to remote resources and reports the results.
+
+Clients and servers are not persistently connected during this process: a client will connect to a server, receive a job, disconnect, execute it, then reconnect to return the results.  Multiple clients thus compete for the single connection, and compete to consume links from the server.  Clients are expected to return to a server repeatedly until told to wait.  They will then back off until more work  is available.
+
+Time is not necessarily expected to be synchronised between server and client, but we recommend the use of NTP to esure that times reported in the results of clients are reliable.
+
+
+Dependencies
+------------
+The client and server components have slightly different dependencies: the client need not perform any database lookups or complex storage operations, but it must be capable of performing HTTP and FTP requests.
+
+### Common dependencies
+
+  * Ruby 1.9.1+ (String#encode is required)
+  * simplerpc
+
+### Client
+
+  * cURL Ruby bindings (gem install -r curb)
+  * The 'gethostname' syscall
+
+### Server
+
+  * SQLite3 and Ruby bindings (gem install -r sqlite3)
+
+
+Configuring
+-----------
+Please see the user documentation at docs/User for more information, as well as the sample documentation in configs/

data/bin/lwac

@@ -0,0 +1,132 @@
+#!/usr/bin/env ruby
+
+# -----------------------------------------------------------------------------
+# Load the launch tools and check gem prerequisites
+require 'lwac'
+require 'lwac/shared/launch_tools'
+require 'lwac/shared/identity'
+
+# Load config using launch_tools
+tool, config = LWAC.load_config 
+
+# Announce version
+LWAC::Identity::announce_version
+
+# Summarise logs
+$log.summarise_logging
+
+# -----------------------------------------------------------------------------
+# do stuff
+
+case tool
+  # ---------------------------------------------------------------------------
+  #  Server
+  # ---------------------------------------------------------------------------
+  when :server
+    require 'lwac/server'
+    require 'simplerpc/server'
+      
+    # Fire up the server
+    server  = LWAC::DownloadServer.new(config)
+    service = LWAC::DownloadService.new(server)
+    
+    # construct the rpc handler
+    rpc = SimpleRPC::Server.new( service, config[:server] )
+
+    # listen
+    $log.info "Starting server on #{config[:server][:hostname]}:#{config[:server][:port]}"
+    loop{
+      begin
+        rpc.listen
+      rescue StandardError => e
+        $log.error "Error: #{e}"
+        $log.debug "#{e.backtrace.join("\n")}"
+      rescue SignalException => e
+        $log.fatal "Caught Signal: #{e}"
+        # $log.debug "#{e.backtrace.join("\n")}"
+        break
+      end
+
+      $log.info "Restarting server after a short delay..."
+      sleep(1)
+    }
+
+    # Ensure we exit cleanly after EM's done
+    server.close
+
+
+
+  # ---------------------------------------------------------------------------
+  #  Client
+  # ---------------------------------------------------------------------------
+  when :client
+    require 'lwac/client'
+
+    begin
+      # Start the client going
+      dc = LWAC::DownloadClient.new(config)
+
+      # download
+      dc.work
+    rescue StandardError => e
+      $log.error "Error: #{e}"
+      $log.debug "#{e.backtrace.join("\n")}"
+    end
+
+
+  # ---------------------------------------------------------------------------
+  #  Import Tool 
+  # ---------------------------------------------------------------------------
+  when :import
+    file = nil
+    if ARGV[2] and File.exist?(ARGV[2]) and File.readable?(ARGV[2]) then
+      file = ARGV[2]
+    else
+      $log.fatal "Cannot read file to import: #{file}" if ARGV[2]
+      $log.fatal "Please provide a file to import!" if not ARGV[2]
+      exit(1)
+    end
+
+
+    require 'lwac/import'
+
+    begin
+      im = LWAC::Importer.new(config)
+      im.import(file)
+    rescue StandardError => e
+      $log.fatal "#{e.to_s}"
+      $log.debug "#{e.backtrace.join("\n")}"
+    end
+
+
+
+  # ---------------------------------------------------------------------------
+  #  Export Tool
+  # ---------------------------------------------------------------------------
+  when :export
+    require 'lwac/export'
+
+    begin
+      # Construct the exporter object and load stuff from disk
+      ex = LWAC::Exporter.new(config)
+
+      # Dump stuff back to disk
+      ex.export
+      
+    rescue StandardError => e
+      $log.error "Error: #{e}"
+      $log.debug "#{e.backtrace.join("\n")}"
+    end
+
+
+  # ---------------------------------------------------------------------------
+  #  Else...
+  # ---------------------------------------------------------------------------
+  else
+    $log.error "Unknown tool: #{tool}."
+    $log.error "The code should also never reach this statement unless something is wrong."
+    exit(1)
+end
+
+
+$log.info "Goodbye"

data/client_config.md

@@ -0,0 +1,71 @@
+Client Configuration
+====================
+The client is responsible for applying to the server and accessing the web in order to download small batches of links, which it then uploads back to the server for storage.  As such its config file is concerned with network access both to the server and to the web.
+
+
+
+Server
+------
+This section defines which server to connect to.  This section contains configuration for [SimpleRPC](http://stephenwattam.com/projects/simplerpc/), and supports all features it does.  Only the salient ones are documented here.
+
+ * `hostname`   --- The IP address or hostname at which to contact the server
+ * `port`       --- The port to use when contacting the server
+ * `password`   --- Optional.  The password to use for auth (must match server config)
+ * `secret`     --- Optional.  The encryption key to use when sending the password (must match server config)
+
+For example:
+
+    :server:
+      :hostname: "127.0.0.1"
+      :port: 27401
+      :password: lwacpass
+      :secret: egrniognhre89n34ifnui4n8gf490
+
+Network
+-------
+This section governs the manner in which clients attempt to contact the server, notably aggressiveness of retrying and polling for jobs.  Clients implement a linear backoff system to ensure they do not over-compete for server resources when failing to perform transactions.
+
+ * `connect_timeout`         --- How long we should give the socket to respond when connecting to the server
+ * `minimum_reconnect_time`  --- The minimum time we should wait before reconnecting
+ * `maximum_reconnect_time`  --- The maximum time we should wait before connecting, approached gradually from the minimum
+ * `connect_failure_penalty` --- The delay to add to the backoff time upon each failure, up to the `maximum_reconnect_time`
+
+For example:
+
+    :network:
+      :connect_timeout: 20
+      :minimum_reconnect_time: 1
+      :maximum_reconnect_time: 240
+      :connect_failure_penalty: 3
+
+Client
+------
+The client's limitations as a system are described here, as well as a way of identifying multiple clients run from a single host.  
+
+Clients check out batches of links, process them, then check in smaller batches (since the datapoints now have a large payload).  The ratio of these sizes should be tuned in accordance with the filesizes being uploaded on a regular basis, and the degree of data security one wishes to ensure.
+
+ * `announce_progress` --- Boolean.  Set to true to print worker status to the screen every half second during operation.
+ * `uuid_salt` --- A human-readable string to prepend the client UUID with.  Each client computes its ID from the hostname, and this is a way of making the IDs more human-readable (as well as running multiple clients on the same host).
+ * `batch_capacity` --- How many links to check out and download in one batch.  The client will receive up to this number of links to download each time it contacts the server.
+ * `check_in_size` --- How many datapoints to upload at once, in MB.  Set to the `cache_limit` to make uploads go fastest, or below it to split them.
+ * `cache_limit` --- The approximate size of the cache used by the client (in bytes).  After downloading this amount of data, the cache will be swapped out and uploaded in chunks to the server.
+ * `cache_dir` --- A directory to create file caches in.  Reduces client RAM requirements, as the cache will store web data before upload.  If you wish to use memory instead, leave this blank.  At most two caches will be active at any one time, meaning memory limits will be:
+   * If using memory caching, `2 * cache_limit + simultaneous_workers * max_body_size`
+   * If using disk caching, `check_in_size + simultaneous_workers * max_body_size`
+ * `simultaneous_workers` --- The number of workers to run in the same pool.  Given preferrable network conditions, this many connections to websites will be open at once, and this number must be chosen whilst bearing in mind the limitations of your kernel and netiquette (especially if you have many links pointing at the same servers).  Within each client, links are downloaded from servers by a series of workers, which consume links from the pending pool.  This has the distinct advantage of being capable of very high degrees of parallelism (beyond that where the kernel will start dropping connections) with relatively little overhead.
+
+For example:
+
+    :client:
+      :announce_progress: true
+      :monitor_rate: 0.5
+      :uuid_salt: "LOCAL"
+      :batch_capacity: 1000
+      :cache_limit: 209715200
+      :check_in_size: 209715200
+      :simultaneous_workers: 200
+      :cache_dir:                   # nil to use RAM cache
+  
+Logging
+-------
+The logging system is the same for all tools and shares a configuration format.  For details, see [configuring logging](log_config.html)

data/concepts.md

@@ -0,0 +1,70 @@
+LWAC Concepts
+=============
+This document describes the format and purpose of the corpus around which LWAC is based, and serves to describe how one would go about operationalising the data therein.
+
+
+Overview
+--------
+LWAC is based around a central longitudinal corpus, stored in an arbitrary directory (as defined in the server config) in a serialised object format (JSON, YAML, or ruby's native binary format).  This means the process of sampling is thus:
+
+ 1. Define a population of links
+ 2. Sample them with as small a time differential as possible
+ 3. Wait until the next sample time
+ 4. go to 2
+
+A `link` without web data attached is known as a `datapoint` in this documentation.  A `sample` is one cross-sectional attempt to download all links.
+
+Theoretically, this forms three levels at which we may access the data:
+
+ 1. Server level, describing the whole sample (all links, samples, and datapoints, a conventional longitudinal sample);
+ 2. Sample level, describing one attempt to download links (a conventional cross-sectional sample containing many datapoints);
+ 3. Datapoint level, describing one attempt to download a single link.
+
+Simply, a server has many samples, each of which has many datapoints.  Samples are temporally homogenous to the greatest extent possible, and datapoints refer to the same URI (for their id).  Since the cumulative time taken to download each sample applies some drift, links are downloaded as intensively as possible.
+
+The corpus, as stored on disk, consists of two types of storage:
+
+ 1. Metadata, stored in an SQLite database, which contains a table simply listing the links along with a unique ID.
+ 2. Corpus data, stored in a flatfile structure as serialised ruby DataPoint objects.
+
+The format of the corpus is described in greater detail in the rest of this document.
+
+
+The Corpus
+----------
+
+### Structure
+The corpus itself is structured as a root directory, containing a specific structure:
+
+    root/
+    root/database.db
+    root/state
+    root/files/sample_id/sample
+    root/files/sample_id/1/2/3/456
+
+
+The corpus includes:
+
+ * The metadata database (if using SQLite3)
+ * The state of the current sample.  This is stored as a serialised ruby object so that a sample may be resumed later if the server is stopped.
+ * A list of sample ID folders containing:
+   * A file describing the properties of this sample as a serialised ruby Sample object
+   * A structure of directories describing link IDs, each of which has up to N files within it (as defined in the server config).  This structure uses the first characters of the ID to nest directories in order to avoid filesystem limits on inode size and speed up random access, i.e.:
+      0/1/1
+      0/1/2
+      0/1/3
+      0/2/1
+      0/2/2
+      0/2/3
+      etc.
+
+
+### File Formats
+Each of the files within a corpus, with the exception of the metadata database, is a serialised ruby object, as defined in `/lib/shared/data_types.rb`.  These objects are serialised using one of three formats:
+ 
+ * `:marshal` --- Ruby's native binary format is very fast but cannot realistically be read from other languages
+ * `:json` --- JSON is widely used but slow
+ * `:yaml` --- YAML is also readable by other languages, but is slow (around 60 times slower than `:marshal`)
+
+Note that a corpus written using one serialisation system will be unreadable by a server using another.  I highly recommend using `:marshal` and using the export tool to extract your data into a more workable format later.
+

data/config_docs.md

@@ -0,0 +1,40 @@
+Reading Configuration Documentation
+===================================
+Configuration files in LWAC are valid YAML files, and typically follow a hash structure.  As such they resemble large trees, with occasional lists and many small textual elements.
+
+A first place to look when interpeting these will be [the YAML overview at Wikipedia](http://en.wikipedia.org/wiki/YAML), which will help familiarise you with the format.  The rest of this document is about how I refer to keys and values in this documentation.
+
+Keys
+----
+YAML keys will be specified as paths from root, loosely following XPATH notation: `/key1/key2/key3` will denote
+
+    ---
+    :key1:
+      :key2:
+         :key3: value
+
+
+Where YAML files contain lists, aspect specifiers will be used in a similar manner to C-like languages (0-base), i.e. `/key1/key2[2]` refers to `orange`:
+
+    ---
+    :key1:
+      :key2:
+        - apple
+        - banana
+        - orange
+
+Where YAML keys contain hashes, that must be retained as simple key-value pairs (i.e. for SQLite pragma settings), curly braces will be used as aspect specifiers, i.e. `/key1/key2{orange}` will refer to `bob`:
+
+    ---
+    :key1:
+      :key2:
+        apple: adolf
+        banana: martin
+        orange: bob
+
+
+Symbols and Data Formats
+------------------------
+Most, though not all, keys in LWAC configuration files are ruby symbols, and thus are prefixed with a colon (:).  Those that are specifically not symbols are noted as such in the text, as it is normally relevant to their use.
+
+Booleans and other binary options are also noted in the text.  It's worth noting that ruby considers `nil` to be false in boolean tests.

data/doc/compile.rb

@@ -0,0 +1,52 @@
+#!/usr/bin/env ruby
+
+# 
+# Compiles all nearby markdown into HTML, and pops it in ./docs
+#
+
+require 'markdown'
+require 'fileutils'
+require 'erb'
+
+input_dir = "./user/*"
+output_dir = "./html_docs"
+TEMPLATE = "template.rhtml"
+
+# for version
+require '../lib/lwac.rb'
+
+
+
+if File.exist?(output_dir) then
+    $stderr.puts "Output directory exists (#{output_dir}) --- please delete and run again."
+    exit(1)
+end
+
+
+def template(version, filename, content, pages)
+  return ERB.new(File.read(TEMPLATE)).result(binding)
+end
+
+# create output dir
+FileUtils.mkdir_p(output_dir)
+
+# create list of pages
+pages = Dir.glob(input_dir).to_a.delete_if{|f| File.extname(f)[1..-1] != "md" or File.directory?(f)}.map{|f| File.basename(f).to_s[0..-(File.extname(f).length + 1)]}
+
+puts "LIST: #{pages.join(', ')}"
+
+Dir.glob(input_dir){|f|
+  if not File.directory?(f) and File.extname(f) == ".md" then
+    puts "Compiling #{f}..."
+
+    File.open(File.join(output_dir, File.basename(f)[0..-(File.extname(f).length + 1)] + ".html"), 'w'){|of|      
+      of.write( template(LWAC::VERSION, File.basename(f), Markdown.new(File.read(f)).to_html, pages ))
+    }
+  elsif f != $0 and File.basename(f) != File.basename(output_dir)
+    puts "Copying #{f}..."
+    FileUtils.cp_r(f, File.join(output_dir, File.basename(f)))
+  end
+}
+
+puts "Done."
+