retreval 0.1

data/CHANGELOG

@@ -0,0 +1,7 @@
+CHANGELOG
+=========
+
+Version 0.1
+-----------
+
+- First public version released.

data/README.md

@@ -0,0 +1,321 @@
+README
+======
+
+This is a simple API to evaluate information retrieval results. It allows you to load ranked and unranked query results and calculate various evaluation metrics (precision, recall, MAP, kappa) against a previously loaded gold standard.
+
+Start this program from the command line with:
+
+    retreval -l <gold-standard-file> -q <query-results> -f <format> -o <output-prefix>
+
+The options are outlined when you pass no arguments and just call
+
+    retreval
+
+You will find further information in the RDOC documentation and the HOWTO section below.
+
+If you want to see an example, use this command:
+
+    retreval -l example/gold_standard.yml -q example/query_results.yml -f yaml -v
+
+
+INSTALLATION
+============
+
+You can manually download the sources and build the Gem from there by `cd`ing to the folder where this README is saved and calling
+
+    gem build retreval.gemspec
+
+This will create a file called `retreval` which you just have to install:
+
+    gem install retreval
+
+And you're done.
+
+
+HOWTO
+=====
+
+This API supports the following evaluation tasks:
+
+- Loading a Gold Standard that takes a set of documents, queries and corresponding judgements of relevancy (i.e. "Is this document relevant for this query?")
+- Calculation of the _kappa measure_ for the given gold standard
+
+- Loading ranked or unranked query results for a certain query
+- Calculation of _precision_ and _recall_ for each result
+- Calculation of the _F-measure_ for weighing precision and recall
+- Calculation of _mean average precision_ for multiple query results
+- Calculation of the _11-point precision_ and _average precision_ for ranked query results
+
+- Printing of summary tables and results
+
+Typically, you will want to use this Gem either standalone or within another application's context.
+
+Standalone Usage
+================
+
+Call parameters
+---------------
+
+After installing the Gem (see INSTALLATION), you can always call `retreval` from the commandline. The typical call is:
+
+    retreval -l <gold-standard-file> -q <query-results> -f <format> -o <output-prefix>
+
+Where you have to define the following options:
+
+- `gold-standard-file` is a file in a specified format that includes all the judgements
+- `query-results` is a file in a specified format that includes all the query results in a single file
+- `format` is the format that the files will use (either "yaml" or "plain")
+- `output-prefix` is the prefix of output files that will be created
+
+Formats
+-------
+
+Right now, we focus on the formats you can use to load data into the API. Currently, we support YAML files that must adhere to a special syntax. So, in order to load a gold standard, we need a file in the following format:
+
+ * "query"       denotes the query
+ * "documents"   these are the documents judged for this query
+ * "id"          the ID of the document (e.g. its filename, etc.)
+ * "judgements"  an array of judgements, each one with:
+ * "relevant"    a boolean value of the judgment (relevant or not)
+ * "user"        an optional identifier of the user
+
+Example file, with one query, two documents, and one judgement:
+
+        - query: 12th air force germany 1957
+          documents:
+          - id: g5701s.ict21311
+            judgements: []
+
+          - id: g5701s.ict21313
+            judgements: 
+            - relevant: false
+              user: 2
+
+So, when calling the program, specify the format as `yaml`.
+For the query results, a similar format is used. Note that it is necessary to specify whether the result sets are ranked or not, as this will heavily influence the calculations. You can specify the score for a document. By "score" we mean the score that your retrieval algorithm has given the document. But this is not necessary. The documents will always be ranked in the order of their appearance, regardless of their score. Thus in the following example, the document with "07" at the end is the first and "25" is the last, regardless of the score.
+
+        ---
+        query: 12th air force germany 1957
+        ranked: true
+        documents:
+        -   score: 0.44034874
+            document: g5701s.ict21307
+        -   score: 0.44034874
+            document: g5701s.ict21309
+        -   score: 0.44034874
+            document: g5701s.ict21311
+        -   score: 0.44034874
+            document: g5701s.ict21313
+        -   score: 0.44034874
+            document: g5701s.ict21315
+        -   score: 0.44034874
+            document: g5701s.ict21317
+        -   score: 0.44034874
+            document: g5701s.ict21319
+        -   score: 0.44034874
+            document: g5701s.ict21321
+        -   score: 0.44034874
+            document: g5701s.ict21323
+        -   score: 0.44034874
+            document: g5701s.ict21325
+        ---
+        query: 1612
+        ranked: true
+        documents:
+        -   score: 1.0174774
+            document: g3290.np000144
+        -   score: 0.763108
+            document: g3201b.ct000726
+        -   score: 0.763108
+            document: g3400.ct000886
+        -   score: 0.6359234
+            document: g3201s.ct000130
+        ---
+
+**Note**: You can also use the `plain` format, which will load the gold standard in a different way (but not the results):
+
+        my_query        my_document_1     false
+        my_query        my_document_2     true
+
+See that every query/document/relevancy pair is separated by a tabulator? You can also add the user's ID in the fourth column if necessary.
+
+Running the evaluation
+-----------------------
+
+After you have specified the input files and the format, you can run the program. If needed, the `-v` switch will turn on verbose messages, such as information on how many judgements, documents and users there are, but this shouldn't be necessary.
+
+The program will first load the gold standard and then calculate the statistics for each result set. The output files are automatically created and contain a YAML representation of the results.
+
+Calculations may take a while depending on the amount of judgements and documents. If there are a thousand judgements, always consider a few seconds for each result set.
+
+Interpreting the output files
+------------------------------
+
+Two output files will be created:
+
+- `output_avg_precision.yml`
+- `output_statistics.yml`
+
+The first lists the average precision for each query in the query result file. The second file lists all supported statistics for each query in the query results file.
+
+For example, for a ranked evaluation, the first two entries of such a query result statistic look like this:
+
+        --- 
+        12th air force germany 1957: 
+        - :precision: 0.0
+          :recall: 0.0
+          :false_negatives: 1
+          :false_positives: 1
+          :true_negatives: 2516
+          :true_positives: 0
+          :document: g5701s.ict21313
+          :relevant: false
+        - :precision: 0.0
+          :recall: 0.0
+          :false_negatives: 1
+          :false_positives: 2
+          :true_negatives: 2515
+          :true_positives: 0
+          :document: g5701s.ict21317
+          :relevant: false
+
+You can see the precision and recall for that specific point and also the number of documents for the contingency table (true/false positives/negatives). Also, the document identifier is given.
+
+API Usage
+=========
+
+Using this API in another ruby application is probably the more common use case. All you have to do is include the Gem in your Ruby or Ruby on Rails application. For details about available methods, please refer to the API documentation generated by RDoc.
+
+**Important**: For this implementation, we use the document ID, the query and the user ID as the primary keys for matching objects. This means that your documents and queries are identified by a string and thus the strings should be sanitized first.
+
+Loading the Gold Standard
+-------------------------
+
+Once you have loaded the Gem, you will probably start by creating a new gold standard.
+
+    gold_standard = GoldStandard.new
+
+Then, you can load judgements into this standard, either from a file, or manually:
+
+    gold_standard.load_from_yaml_file "my-file.yml"
+    gold_standard.add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John
+
+There is a nice shortcut for the `add_judgement` method. Both lines are essentially the same:
+
+    gold_standard.add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John
+    gold_standard << :document => doc_id, :query => query_string, :relevant => boolean, :user => John
+
+Note the usage of typical Rails hashes for better readability (also, this Gem was developed to be used in a Rails webapp).
+
+Now that you have loaded the gold standard, you can do things like:
+
+        gold_standard.contains_judgement? :document => "a document", :query => "the query"
+        gold_standard.relevant? :document => "a document", :query => "the query"
+
+
+Loading the Query Results
+-------------------------
+
+Now we want to create a new `QueryResultSet`. A query result set can contain more than one result, which is what we normally want. It is important that you specify the gold standard it belongs to.
+
+    query_result_set = QueryResultSet.new :gold_standard => gold_standard
+
+Just like the Gold Standard, you can read a query result set from a file:
+
+    query_result_set.load_from_yaml_file "my-results-file.yml"
+
+Alternatively, you can load the query results one by one. To do this, you have to create the results (either ranked or unranked) and then add documents:
+
+    my_result = RankedQueryResult.new :query => "the query"
+    my_result.add_document :document => "test_document 1", :score => 13
+    my_result.add_document :document => "test_document 2", :score => 11
+    my_result.add_document :document => "test_document 3", :score => 3
+
+This result would be ranked, obviously, and contain three documents. Documents can have a score, but this is optional. You can also create an Array of documents first and add them altogether:
+
+        documents = Array.new
+        documents << ResultDocument.new :id => "test_document 1", :score => 20
+        documents << ResultDocument.new :id => "test_document 2", :score => 21
+        my_result = RankedQueryResult.new :query => "the query", :documents => documents
+
+The same applies to `UnrankedQueryResult`s, obviously. The order of ranked documents is the same as the order in which they were added to the result.
+
+The `QueryResultSet` will now contain all the results. They are stored in an array called `query_results`, which you can access. So, to iterate over each result, you might want to use the following code:
+
+        query_result_set.query_results.each_with_index do |result, index|
+        # ...
+        end
+
+Or, more simply:
+
+        for result in query_result_set.query_results
+        # ...
+        end
+
+Calculating statistics
+----------------------
+
+Now to the interesting part: Calculating statistics. As mentioned before, there is a conceptual difference between ranked and unranked results. Unranked results are much easier to calculate and thus take less CPU time.
+
+No matter if unranked or ranked, you can get the most important statistics by just calling the `statistics` method.
+
+        statistics = my_result.statistics
+
+In the simple case of an unranked result, you will receive a hash with the following information:
+
+* `precision` - the precision of the results
+* `recall` - the recall of the results
+* `false_negatives` - number of not retrieved but relevant items
+* `false_positives` - number of retrieved but nonrelevant
+* `true_negatives` - number of not retrieved and nonrelevantv items
+* `true_positives` - number of retrieved and relevant items
+
+In case of a ranked result, you will receive an Array that consists of _n_ such Hashes, depending on the number of documents. Each Hash will give you the information at a certain rank, e.g. the following to lines return the recall at the fourth rank. 
+
+        statistics = my_ranked_result.statistics
+        statistics[3][:recall]
+
+In addition to the information mentioned above, you can also get for each rank:
+
+* `document` - the ID of the document that was returned at this rank
+* `relevant` - whether the document was relevant or not
+
+Calculating statistics with missing judgements
+----------------------------------------------
+
+Sometimes, you don't have judgements for all document/query pairs in the gold standard. If this happens, the results will be cleaned up first. This means that every document in the results that doesn't appear to have a judgement will be removed temporarily.
+
+As an example, take the following results:
+
+* A
+* B
+* C
+* D
+
+Our gold standard only contains judgements for A and C. The results will be cleaned up first, thus leading to:
+
+* A
+* C
+
+With this approach, we can still provide meaningful results (for precision and recall).
+
+Other statistics
+----------------
+
+There are several other statistics that can be calculated, for example the **F measure**. The F measure weighs precision and recall and has one parameter, either "alpha" or "beta". Get the F measure like so:
+
+        my_result.f_measure :beta => 1
+
+If you don't specify either alpha or beta, we will assume that beta = 1.
+
+Another interesting measure is **Cohen's Kappa**, which tells us about the inter-agreement of assessors. Get the kappa statistic like this:
+
+        gold_standard.kappa
+
+This will calculate the average kappa for each pairwise combination of users in the gold standard.
+
+For ranked results one might also want to calculate an **11-point precision**. Just call the following:
+
+        my_ranked_result.eleven_point_precision
+
+This will return a Hash that has indices at the 11 recall levels from 0 to 1 (with steps of 0.1) and the corresponding precision at that recall level.

data/TODO

@@ -0,0 +1,5 @@
+TODO
+====
+
+- Find a suitable plaintext format for retrieval results
+- Use Hashes instead of Arrays to store classes

data/bin/retreval

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require 'retreval/runner'
+
+runner = Retreval::Runner.new(ARGV)
+runner.run

data/example/gold_standard.yml

@@ -0,0 +1,48 @@
+- query: Example Query
+  documents: 
+  - id: ict21307
+    judgements:
+    - relevant: true
+      user: Bob
+  - id: ict21309
+    judgements:
+    - relevant: false
+      user: Bob
+  - id: ict21311
+    judgements:
+    - relevant: false
+      user: Bob
+  - id: ict21313
+    judgements: 
+    - relevant: false
+      user: Bob
+  - id: ict21315
+    judgements:
+    - relevant: true
+      user: Bob
+    - relevant: true
+      user: John
+  - id: ict21317
+    judgements: 
+    - relevant: false
+      user: Bob
+    - relevant: false
+      user: John
+  - id: ict21319
+    judgements: 
+    - relevant: false
+      user: Bob
+    - relevant: false
+      user: John
+  - id: ict21321
+    judgements: 
+    - relevant: false
+      user: John
+  - id: ict21323
+    judgements: 
+    - relevant: true
+      user: John
+  - id: ict21325
+    judgements: 
+    - relevant: true
+      user: John

data/example/query_results.yml

@@ -0,0 +1,23 @@
+- query: Example Query
+  ranked: true
+  documents:
+  -   score: 0.24921744
+      id: ict21307
+  -   score: 0.1623808
+      id: ict21309
+  -   score: 0.13997056
+      id: ict21311
+  -   score: 0.12525019
+      id: ict21313
+  -   score: 0.11482056
+      id: ict21315
+  -   score: 0.1131133
+      id: ict21317
+  -   score: 0.09897413
+      id: ict21319
+  -   score: 0.09897413
+      id: ict21321
+  -   score: 0.09742848
+      id: ict21323
+  -   score: 0.09742848
+      id: ict21325

data/lib/retreval/gold_standard.rb

@@ -0,0 +1,424 @@
+module Retreval
+
+  # A gold standard is composed of several judgements for the 
+  # cartesian product of documents and queries
+  class GoldStandard
+    
+    attr_reader :documents, :judgements, :queries, :users
+    
+    # Creates a new gold standard. One can optionally construct the gold
+    # standard with triples given. This would be a hash like:
+    #     triples = { 
+    #        :document => "Document ID", 
+    #        :query => "Some query", 
+    #        :relevant => "true" 
+    #      }
+    # 
+    # Called via:
+    #     GoldStandard.new :triples => an_array_of_triples
+    def initialize(args = {})
+      @documents = Hash.new
+      @queries = Array.new
+      @judgements = Array.new
+      @users = Hash.new
+      
+      # one can also construct a Gold Standard with everything already loaded
+      unless args[:triples].nil?
+        args[:triples].each do |triple|
+          add_judgement(triple)
+        end
+      end
+    end
+    
+        
+    # Parses a YAML file adhering to the following generic standard:
+    # 
+    # * "query"       denotes the query
+    # * "documents"   these are the documents judged for this query
+    # * "id"          the ID of the document (e.g. its filename, etc.)
+    # * "judgements"  an array of judgements, each one with:
+    # * "relevant"    a boolean value of the judgment (relevant or not)
+    # * "user"        an optional identifier of the user
+    #
+    # Example file:
+    #     - query: 12th air force germany 1957
+    #       documents:
+    #       - id: g5701s.ict21311
+    #         judgements: []
+    # 
+    #       - id: g5701s.ict21313
+    #         judgements: 
+    #         - relevant: false
+    #           user: 2
+    def load_from_yaml_file(file)
+      begin
+        ydoc = YAML.load(File.open(file, "r"))
+        ydoc.each do |entry|
+          
+          # The query is first in the hierarchy
+          query = entry["query"]
+
+          # Every query contains several documents
+          documents = entry["documents"]
+          documents.each do |doc|
+            
+            document = doc["id"]
+            
+            # Only count the map if it has judgements
+            if doc["judgements"].empty?  
+              add_judgement :document => document, :query => query, :relevant => nil, :user => nil
+            else
+              doc["judgements"].each do |judgement|
+                relevant = judgement["relevant"]
+                user = judgement["user"]
+                
+                add_judgement :document => document, :query => query, :relevant => relevant, :user => user
+              end
+            end
+            
+          end
+        end
+
+      rescue Exception => e
+        raise "Error while parsing the YAML document: " + e.message
+      end
+    end
+    
+    
+    # Parses a plaintext file adhering to the following standard:
+    # Every line of text should include a triple that designates the judgement.
+    # The symbols should be separated by a tabulator.
+    # E.g.
+    #   my_query        my_document_1     false
+    #   my_query        my_document_2     true
+    #
+    # You can also add the user's ID in the fourth column.
+    def load_from_plaintext_file(file)
+      begin
+        File.open(file).each do |line|
+          line.chomp!
+          info = line.split("\t")
+          if info.length == 3
+            add_judgement :query => info[0], :document => info[1], :relevant => info[2]
+          elsif info.length == 4
+            add_judgement :query => info[0], :document => info[1], :relevant => info[2], :user => info[3]
+          end
+        end
+      rescue Exception => e
+        raise "Error while parsing the document: " + e.message
+      end
+    end
+    
+    
+    # Adds a judgement (document, query, relevancy) to the gold standard.
+    # All of those are strings in the public interface.
+    # The user ID is an optional parameter that can be used to measure kappa later.
+    # Call this with:
+    #     add_judgement :document => doc_id, :query => query_string, :relevant => boolean, :user => John
+    def add_judgement(args)
+      document_id = args[:document]
+      query_string = args[:query]
+      relevant = args[:relevant]
+      user_id = args[:user]
+      
+      
+      unless document_id.nil? or query_string.nil?
+        document = Document.new :id => document_id
+        query = Query.new :querystring => query_string
+        
+        
+        # If the user exists, load it, otherwise create a new one
+        if @users.has_key?(user_id)
+          user = @users[user_id]
+        else
+          user = User.new :id => user_id unless user_id.nil?
+        end
+        
+        # If there is no judgement for this combination, just add the document/query pair
+        if relevant.nil?
+          # TODO: improve efficiency by introducing hashes !
+          @documents[document_id] = document
+          @queries << query unless @queries.include?(query)
+          return
+        end
+        
+        if user_id.nil?
+          judgement = Judgement.new :document => document, :query => query, :relevant => relevant
+        else
+          judgement = Judgement.new :document => document, :query => query, :relevant => relevant, :user => user
+          
+          user.add_judgement(judgement)
+          @users[user_id] = user
+        end
+        
+        @documents[document_id] = document
+        @queries << query unless @queries.include?(query)
+        @judgements << judgement
+      else
+        #TOOD I think there is somethink like an ArgumentExcpetion in Ruby; use that if applicable
+        raise "Need at least a Document, and a Query for creating the new entry."
+      end
+      
+    end
+    
+    # This is essentially the same as adding a Judgement, we can use this operator too.
+    def <<(args)
+      self.add_judgement args
+    end
+    
+    # Returns true if a Document is relevant for a Query, according to this GoldStandard.
+    # Called by:
+    #     relevant? :document => "document ID", :query => "query"
+    def relevant?(args)
+      query = Query.new :querystring => args[:query]
+      document = Document.new :id => args[:document]
+      
+      relevant_count = 0
+      nonrelevant_count = 0
+      
+      #TODO: looks quite inefficient. Would a hash with document-query-pairs as key help?
+      @judgements.each do |judgement|
+        if judgement.document == document and judgement.query == query
+          judgement.relevant ? relevant_count += 1 : nonrelevant_count += 1
+        end
+      end
+      
+      # If we didn't find any judgements, just leave it as false
+      if relevant_count == 0 and relevant_count == 0
+        false
+      else
+        relevant_count >= nonrelevant_count
+      end
+    end
+    
+    
+    # Returns true if this GoldStandard contains a Judgement for this Query / Document pair
+    # This is called by:
+    #     contains_judgement? :id => "the document ID", :querystring => "the query"
+    def contains_judgement?(args)
+      query = Query.new :querystring => args[:query]
+      document = Document.new :id => args[:document]
+      
+      #TODO: a hash could improve performance here as well
+      
+      @judgements.each { |judgement| return true if judgement.document == document and judgement.query == query }
+      
+      false
+    end
+    
+    
+    # Returns true if this GoldStandard contains this Document
+    # Called by:
+    #     contains_document? :id => "document ID"
+    def contains_document?(args)
+      document_id = args[:id]
+      @documents.key? document_id
+    end
+    
+    
+    # Returns true if this GoldStandard contains this Query string
+    # Called by:
+    #     contains_query? :querystring => "the query"
+    def contains_query?(args)
+      querystring = args[:querystring]
+      query = Query.new :querystring => querystring
+      @queries.include? query
+    end
+    
+    
+    # Returns true if this GoldStandard contains this User
+    # Called by:
+    #     contains_user? :id => "John Doe"
+    def contains_user?(args)
+      user_id = args[:id]
+      @users.key? user_id
+    end
+    
+    
+    # Calculates and returns the Kappa measure for this GoldStandard. It shows
+    # to which degree the judges agree in their decisions
+    # See: http://nlp.stanford.edu/IR-book/html/htmledition/assessing-relevance-1.html
+    def kappa
+      
+      # FIXME: This isn't very pretty, maybe there's a more ruby-esque way to do this?
+      sum = 0
+      count = 0
+      
+      # A repeated_combination yields all the pairwise combinations of
+      # users to generate the pairwise kappa statistic. Elements are also
+      # paired with themselves, so we need to remove those.
+      @users.values.repeated_combination(2) do |combination|
+        user1, user2 = combination[0], combination[1]
+        unless user1 == user2
+          kappa = pairwise_kappa(user1, user2)
+          unless kappa.nil?
+            puts "Kappa for User #{user1.id} and #{user2.id}: #{kappa}" if $verbose
+            sum += kappa unless kappa.nil?
+            count += 1
+          end
+        end
+      end
+      
+      @kappa = sum / count.to_f
+      puts "Average pairwise kappa: #{@kappa}" if $verbose
+      return @kappa
+    end
+    
+    private
+    
+    # Calculates the pairwise kappa statistic for two users.
+    # The two users objects need at least one Judgement in common.
+    # Note that the kappa statistic is not really meaningful when there are
+    # too little judgements in common!
+    def pairwise_kappa(user1, user2)
+      
+      user1_judgements = user1.judgements.reject { |judgement| not user2.judgements.include?(judgement) }
+      user2_judgements = user2.judgements.reject { |judgement| not user1.judgements.include?(judgement) }
+      
+      total_count = user1_judgements.count
+      
+      unless user1_judgements.empty? or user1_judgements.empty?
+        
+        positive_agreements = 0     # => when both judges agree positively (relevant)
+        negative_agreements = 0     # => when both judges agree negatively (nonrelevant)
+        negative_disagreements = 0  # => when the second judge disagrees by using "nonrelevant"
+        positive_disagreements = 0  # => when the second judge disagrees by using "relevant"
+        
+        for i in 0..(user1_judgements.count-1)
+          if user1_judgements[i].relevant == true
+            if user2_judgements[i].relevant == true
+              positive_agreements += 1
+            else
+              negative_disagreements += 1
+            end
+          elsif user1_judgements[i].relevant == false
+            if user2_judgements[i].relevant == false
+              negative_agreements += 1
+            else
+              positive_disagreements += 1
+            end
+          end
+        end
+        
+        # The proportion the judges agreed:
+        p_agreed = (positive_agreements + negative_agreements) / total_count.to_f
+        
+        # The pooled marginals:
+        p_nonrelevant = (positive_disagreements + negative_agreements * 2 + negative_disagreements) / (total_count.to_f * 2)
+        # This one is the opposite of P(nonrelevant):
+        # p_relevant = (positive_agreements * 2 + negative_disagreements + positive_disagreements) / (total_count.to_f * 2)
+        p_relevant = 1 - p_nonrelevant
+        
+        # The probability that the judges agreed by chance
+        p_agreement_by_chance = p_nonrelevant ** 2 + p_relevant ** 2
+        
+        
+        # Finally, the pairwise kappa value
+        # If there'd be a division by zero, we avoid it and return 0 right away
+        if p_agreed - p_agreement_by_chance == 0
+          return 0
+        # In any other case, the kappa value is correct and we can return it
+        else
+          kappa = (p_agreed - p_agreement_by_chance) / (1 - p_agreement_by_chance)
+          return kappa
+        end
+      end
+      
+      # If there are no common judgements, there is no kappa value to calculate
+      return nil
+    end
+    
+  end
+  
+  
+  # A Query is effectively a string that is used as its ID.
+  class Query
+    
+    attr_reader :querystring
+    
+    # Compares two Query objects according to their query string
+    def ==(query)
+      query.querystring == self.querystring
+    end
+
+    # Creates a new Query object with a specified string    
+    def initialize(args)
+      @querystring = args[:querystring].to_s
+      raise "Can not construct a Query with an empty query string" if @querystring.nil?
+    end
+    
+  end
+  
+  # A Document is a generic resource that is identified by its ID (which could be anything).
+  class Document
+    
+    attr_reader :id
+
+    # Compares two Document objects according to their id
+    def ==(document)
+      document.id == self.id
+    end
+
+    # Creates a new Document object with a specified id    
+    def initialize(args)
+      @id = args[:id].to_s
+      raise "Can not construct a Document with an empty identifier" if @id.nil?
+    end
+    
+  end
+  
+  # A Judgement references one query and one document as being relevant to each other or not.
+  # It also keeps track of the User who created the Judgement, if necessary.
+  class Judgement
+    
+    attr_reader :relevant, :document, :query, :user
+
+    # Creates a new Judgement that belongs to a Query, a Document, and optionally to a User
+    # Called by (note the usage of IDs, not objects):
+    #     Judgement.new :document => my_doc_id, :user => my_user_id, :query => query_string, :relevant => true
+    def initialize(args)
+      @relevant = args[:relevant]
+      @document = args[:document]
+      @query = args[:query]
+      @user = args[:user]
+    end
+    
+    
+    # A Judgement is considered equal to another when they are for the same Query or Document.
+    # This comparison happens regardless of the user, so it is easier to generate "unique" Judgements
+    # or calculate the kappa measure.
+    def ==(judgement)
+      self.document == judgement.document and self.query == judgement.query
+    end
+    
+  end
+  
+  # A User is optional for a Judgement, they are identified by their ID, which could be anything.
+  class User
+    
+    attr_reader :id, :judgements
+    
+    # Compares two User objects according to their id
+    def ==(user)
+      user.id == self.id
+    end
+    
+    
+    # Creates a new User object with a specified id
+    def initialize(args)
+      @id = args[:id]
+      @judgements = Array.new
+      raise "Can not construct a User with an empty identifier" if @id.nil? 
+    end
+    
+    
+    # Adds a reference to a Judgement to this User object, since this makes it 
+    # easier to calculate kappa later. Some users have multiple judgements for
+    # the same Document Query pair, which isn't really helpful. We therefore eliminate
+    # duplicates.
+    def add_judgement(judgement)
+      @judgements << judgement unless @judgements.include?(judgement)
+    end
+    
+  end
+end