math_metadata_lookup 0.1.4 → 0.2.0

data/README.md

@@ -2,9 +2,18 @@ About
 -----
 
 This utility search mathematical reviews sites and fetches metadata about articles.
-It returns results as one of text, xml, html, yaml or ruby formats.
+It returns results as one of text, xml, html, yaml, json or ruby formats.
 It can work with LaTeX accent notation.
 
+Sites
+=====
+
+* MathSciNet
+* Zentralblatt
+* Numdam
+* Cedram
+* DmlCZ
+
 
 Installation
 ------------
@@ -18,7 +27,7 @@ then you can install ``math_metadata_lookup`` using rubygems:
 Command line usage example
 --------------------------
 
-To get full help run it without any argument
+To get full help run it without any argument:
 
     math_metadata_lookup
 
@@ -26,10 +35,12 @@ Fetching metadata about an article:
 
     math_metadata_lookup.rb article -t "Sobolev embeddings with variable exponent. II"
 
-Returns list of articles:
-    bin/math_metadata_lookup.rb article -t "Sobolev embeddings" -a "Rakosnik, Jiri" -a "Edmunds, David" -f html
+Returns list of articles as html:
+
+    bin/math_metadata_lookup.rb article -t "Sobolev embeddings" -a "Rakosnik" -a "Edmunds" -f html
 
 Searching for authors:
+
     bin/math_metadata_lookup.rb author -a "Vesely, Jiri"
 
 
@@ -39,11 +50,13 @@ Usage from ruby
     require 'rubygems'
     require 'math_metadata_lookup'
     
-    # initialize search engine to look only to Mathematical Reviews database
+    # initialize search engine to look only into Mathematical Reviews database
     l = MathMetadata::Lookup.new( :sites => [:mr], :verbose => false )
     
-    # fetch result
+    # fetch first site from search result
     mr_result = l.article( :title => "Sobolev embeddings with variable exponent. II" ).first
+
+    # get first article from the :mr site
     article = mr_result[:result].first
     
     # print out article authors separated with semicolon
@@ -55,27 +68,28 @@ Resources
 
 Content of the resource directory:
 
-* **``math_metadata_lookup.js``**: contains function ``toggle_references( id )``. It can toggle visibility of references in html document. By default are all references visible. If you set in css class references attribute display to none it will be hidden by default.
+* ``math_metadata_lookup.js``: contains function ``toggle_references( id )``. It can toggle visibility of references in html document. By default are all references visible. If you set in css class references attribute display to none it will be hidden by default.
 
 
-Function reference
-------------------
+Function reference for MathMetadata::Lookup class
+-------------------------------------------------
 
-#Lookup#article( hash )
+#article( hash )
 
 Hash arguments are:
 
 * article id is known
-  * **:id** String
+  * **:id**      String
+
 * article id is unknown 
-  * **:title** String
+  * **:title**   String
   * **:authors** Array of Strings
-  * **:year** String
+  * **:year**    String
 
 Returns instance of class Result.
 
 
-#Lookup#author( hash )
+#author( hash )
 
 Search for authors "name forms".
 
@@ -86,27 +100,27 @@ Hash arguments are:
 Returns instance of class Result.
 
 
-#Lookup#heuristic( hash )
+#heuristic( hash )
 
 Returns only one best match from each site where similarity is higher then threshold.
-It runs article searh with first two words from title and only surnames from author names.
+It runs article search with first two words from title and only surnames from author names.
 The result of search is sorted by similarity and articles with similarity less then threshold are deleted.
 Similarity is count as weighted average from title, authors and year using Levenshtein distance method.
 The Levenshtein distance function is run on full given title and full given names.
 
 Hash arguments are:
 
-* **:title**   String
-* **:author**  Array of Strings
-* **:year**    String
+* **:title**     String
+* **:authors**   Array of Strings
+* **:year**      String
 * **:threshold** Float. Range: 0.0...1.0. Default: 0.6
 
 Returns instance of class Result.
 
 
-#Lookup#reference( hash )
+#reference( hash )
 
-Parse reference string and run heuristic.
+Parse reference string and run heuristic. It expects that authors are separated by colon.
 
 Hash arguments are:
 

data/Rakefile

@@ -2,16 +2,23 @@
 # vi: fenc=utf-8:expandtab:ts=2:sw=2:sts=2
 # 
 # @author: Petr Kovar <pejuko@gmail.com>
-$KCODE='UTF8'
+$KCODE='UTF8' if RUBY_VERSION < "1.9"
 
 require 'rake/gempackagetask'
 require 'rake/clean'
 
-CLEAN << "coverage" << "pkg" << "README.html" << "CHANGELOG.html" << '*.rbc'
+CLEAN << "coverage" << "pkg" << "README.html" << "CHANGELOG.html" << '*.rbc' << "html/" << "yardoc/"
 
 task :default => [:doc, :gem]
 
-Rake::GemPackageTask.new(eval(File.read("math_metadata_lookup.gemspec"))) {|pkg|}
+task :gem do |t|
+  load 'math_metadata_lookup.gemspec'
+  builder = Gem::Builder.new @spec
+  builder.build
+end
+
+
+docs = []
 
 begin
   require 'bluecloth'
@@ -27,11 +34,44 @@ begin
     File.open(htmlfile, "w") { |f| f << md.to_html }
   end
 
-  task :doc => [:readme]
 
   task :readme do |t|
     build_document("README.md")
   end
 
+  docs << :readme
+
 rescue LoadError
 end
+
+
+begin
+
+  require 'rake/rdoctask'
+
+  Rake::RDocTask.new do |rd|
+    rd.main = "README.md"
+    rd.rdoc_files.include("README.md", "lib/**/*.rb", "bin/*")
+  end
+
+  docs << :rdoc
+
+rescue LoadError
+end
+
+
+begin
+
+  require 'yard'
+
+  YARD::Rake::YardocTask.new do |t|
+    t.files    = ['README.md', 'lib/**/*.rb', 'bin/*']   # optional
+    t.options  = ['--output-dir=yardoc'] # optional
+  end
+
+  docs << :yard
+
+rescue LoadError
+end
+
+task :doc => docs

data/TODO

@@ -10,8 +10,8 @@ Completed
 
 * year argument
 * support for references
-* xml output
+* xml, yaml and json output
 * heuristic
   x joining results
   * guess relevant article if possible
-
+* sites: MathSciNet, Zentralblatt, Cedram, Numdam, DmlCZ

data/bin/math_metadata_lookup

@@ -39,22 +39,22 @@ def print_help
     --threshold, -d <0.0...1.0> -- default: 0.6
 
 
-  reference -- parse reference string and run heuristic
+  reference -- parse reference string and run heuristic; it expects that authors are separated by colon
 
     --reference, -r <string with reference>  -- parse the string to get title, authors and year
     --threshold, -d <0.0...1.0>              -- default: 0.6
 
 
   common options:
-    --site, -s <mr,zbl>                      -- repeatable, sites to search on, default: all
-    --format, -f <text|html|xml|ruby|yaml>   -- output format, default: text
+    --site, -s <mr,zbl,dmlcz,cedram,numdam>       -- repeatable, sites to search on, default: all
+    --format, -f <text|html|xml|ruby|yaml|json>   -- output format, default: text
     --verbose, -v
 
 
   Examples:
 
   #{$0} article -t \"Sobolev embeddings with variable exponent. II\"
-  #{$0} article -t \"Sobolev embeddings\" -a \"Rakosnik, Jiri\" -a \"Edmunds, David\" -f html
+  #{$0} article -t \"Sobolev embeddings\" -a Rakosnik -a Edmunds -f html
   #{$0} author -a \"Vesely, Jiri\"
   #{$0} reference -r \"Kufner, A., John, O., and Fučík, S.: Function Spaces, Noordhoff, Leyden, and Academia, Prague, 1977\" -d 0.4
 "
@@ -129,6 +129,6 @@ end
 case $options[:format].to_sym
 when :ruby
   pp result
-when :yaml, :html, :xml, :text
+when :yaml, :html, :xml, :text, :json
   puts result.format($options[:format])
 end

data/lib/math_metadata_lookup/article.rb

@@ -1,5 +1,19 @@
 module MathMetadata
 
+  # == Attributes
+  #
+  # * :id [String]
+  # * :similarity [String]
+  # * :publication [String]
+  # * :title [String]
+  # * :authors [Array of Strings]
+  # * :year [String]
+  # * :language [String]
+  # * :msc [Array of Strings]
+  # * :pages [String]
+  # * :issn [Array of Strings]
+  # * :keywords [Array of Strings]
+  # * :references [Array of MathMetadata::Reference]
   class Article < Entity
 
     def ==(article)
@@ -27,6 +41,11 @@ module MathMetadata
     end
 
 
+    def to_json(*args)
+      @metadata.to_json(*args)
+    end
+
+
     def to_text
       result = ""
       result += %~Id: #{@metadata[:id]}
@@ -39,7 +58,8 @@ Language: #{@metadata[:language]}
 MSC: #{[@metadata[:msc]].flatten.join("; ")}
 Pages: #{@metadata[:range]}
 ISSN: #{@metadata[:issn].join('; ')}
-Keywords: #{@metadata[:keywords].join('; ')}~
+Keywords: #{@metadata[:keywords].join('; ')}
+Publisher: #{@metadata[:publisher]}~
       @metadata[:references].to_a.each_with_index do |ref, idx|
         a = ref.article
         result += %~
@@ -79,6 +99,7 @@ Ref.: #{idx+1}. #{[a[:authors]].flatten.join("; ")}: #{a[:title]}~
             <keyword>#{::CGI.escapeHTML keyword.to_s}</keyword>~
         end
         result += %~
+            <publisher>#{::CGI.escapeHTML @metadata[:publisher].to_s}</publisher>
             <references>
 ~
       @metadata[:references].to_a.each_with_index do |reference, idx|
@@ -97,6 +118,7 @@ Ref.: #{idx+1}. #{[a[:authors]].flatten.join("; ")}: #{a[:title]}~
                     <publication>#{::CGI.escapeHTML ref[:publication].to_s}</publication>
                     <publisher>#{::CGI.escapeHTML ref[:publisher].to_s}</publisher>
                     <year>#{::CGI.escapeHTML ref[:year].to_s}</year>
+                    <range>#{::CGI.escapeHTML ref[:range].to_s}</range>
                 </reference>
 ~
       end
@@ -121,6 +143,7 @@ Ref.: #{idx+1}. #{[a[:authors]].flatten.join("; ")}: #{a[:title]}~
         <span class="label">Pages:</span> <span class="pages">#{::CGI.escapeHTML @metadata[:range].to_s}</span><br />
         <span class="label">ISSN:</span> <span class="issn">#{::CGI.escapeHTML @metadata[:issn].to_a.join('; ')}</span><br />
         <span class="label">Keywords:</span> <span class="keywords">#{::CGI.escapeHTML @metadata[:keywords].to_a.join('; ')}</span><br />
+        <span class="label">Publisher:</span> <span class="publisher">#{::CGI.escapeHTML @metadata[:publisher].to_s}</span><br />
 ~
       if @metadata[:references].to_a.size > 0
         result += %~
@@ -137,6 +160,7 @@ Ref.: #{idx+1}. #{[a[:authors]].flatten.join("; ")}: #{a[:title]}~
                 <span class="label">Publication:</span> <span class="publication">#{::CGI.escapeHTML ref[:publication].to_s}</span><br />
                 <span class="label">Publisher:</span> <span class="publisher">#{::CGI.escapeHTML ref[:publisher].to_s}</span><br />
                 <span class="label">Year:</span> <span class="year">#{::CGI.escapeHTML ref[:year].to_s}</span><br />
+                <span class="label">Pages:</span> <span class="range">#{::CGI.escapeHTML ref[:range].to_s}</span><br />
                 <span class="label">Id:</span> <span class="id">#{::CGI.escapeHTML ref[:id].to_s}</span><br />
             </div>
 ~

data/lib/math_metadata_lookup/author.rb

@@ -1,5 +1,10 @@
 module MathMetadata
 
+  # == Attributes
+  #
+  # * :id [String] an id of the article in the database
+  # * :preferred [String] a preferred form of name in the database
+  # * :forms [Array of Strings] other name forms
   class Author < Entity
 
     def to_text

data/lib/math_metadata_lookup/entity.rb

@@ -27,7 +27,7 @@ module MathMetadata
       result = self
 
       case f.to_sym
-      when :text, :html, :xml
+      when :text, :html, :xml, :json
         result = self.send("to_#{f}")
       end
 

data/lib/math_metadata_lookup/lookup.rb

@@ -36,7 +36,7 @@ module MathMetadata
     end
 
 
-    # try to decide what is best result for query and combine results from all sites to one article response
+    # returns best result for each site
     def heuristic( args={} )
       opts = {:threshold => 0.6, :authors => []}.merge(args)
       result = Result.new

data/lib/math_metadata_lookup/reference.rb

@@ -3,6 +3,10 @@
 
 module MathMetadata
 
+  # == Attributes
+  #
+  # * :source [String] original string
+  # * :article [MathMetadata::Article] parsed metadata
   class Reference
 
     # 1=authors, 2=title, 3=publication, 4=year, 5=range
@@ -27,6 +31,8 @@ module MathMetadata
     ARTICLE_REFERENCE_10_RE = %r{([^:]+):\s*(.*?),\s*(.*?)\s*.*?}mi
     # 1=authors, 2=title, 3=place, 4=year
     ARTICLE_REFERENCE_11_RE = %r{([^:]+):\s*(.*),(.*?)\s+(\d{4})}mi
+    # 1=authors, 2=title
+#    ARTICLE_REFERENCE_12_RE = %r{([^:]+):\s*([^,]+),.*}mi
 
 
     attr_accessor :source, :article, :suffix, :number, :reg
@@ -43,7 +49,20 @@ module MathMetadata
     end
 
 
-    def self.parse( str )
+    def to_json(*args)
+      {
+        :number => @number,
+        :source => @source,
+        :article => @article
+      }.to_json(*args)
+    end
+
+
+    def self.parse( ref_str )
+      str = ref_str.dup
+      if ref_str =~ %r~^\s*(?:\[.*?\]|\(.*?\)|\d+[\.]?)\s*(.*)~mi
+        str = $1
+      end
       article = Article.new
       rnumber = 0
       suffix = nil
@@ -86,6 +105,9 @@ module MathMetadata
           when 11
             # 1=authors, 2=title, 3=place, 4=year
             found = [$1, $2, nil, $4, nil, nil, $3]
+#          when 12
+#            # 1=authors, 2=title, 3=place, 4=year
+#            found = [$1, $2, nil, nil, nil, nil, nil]
           end
           rnumber = j
           break

data/lib/math_metadata_lookup/result.rb

@@ -2,13 +2,37 @@
 # vi: fenc=utf-8:expandtab:ts=2:sw=2:sts=2
 
 require 'ya2yaml'
+require 'json'
 
 module MathMetadata
 
+  # == Example of Result class
+  #
+  #   # initialize search engine
+  #   l = MathMetadata::Lookup.new(:verbose => false)
+  #
+  #   # result containing articles (l.article, l.heuristic, l.reference)
+  #   result = l.article(:title => 'sobolev')
+  #   result.each do |site|
+  #     next unless site[:result]
+  #     site[:result].each do |article|
+  #       puts "#{site[:name]}: #{article[:title]} (#{article[:authors].join('; ')})"
+  #     end
+  #   end
+  #
+  #   # result containing authors (l.author)
+  #   result = l.author(:name => 'Vesely, Jiri')
+  #   result.each do |site|
+  #     next unless site[:result]
+  #     site[:result].each do |author|
+  #       puts "#{site[:name]}: #{author[:preferred]} (#{author[:forms].join('; ')})"
+  #     end
+  #   end
+
   class Result
     include Enumerable
 
-    FORMATS = [:ruby, :yaml, :xml, :html, :text]
+    FORMATS = [:ruby, :yaml, :json, :xml, :html, :text]
 
     def initialize( meta=[] )
       @metadata = meta
@@ -75,6 +99,11 @@ module MathMetadata
     end
 
 
+    def to_json
+      @metadata.to_json
+    end
+
+
     def to_array
       @metadata
     end

data/lib/math_metadata_lookup/site.rb

@@ -11,7 +11,7 @@ module MathMetadata
   SITES = []
 
   # Abstract class. Inherit in your sites definition.
-  class Site 
+  class Site
 
     def initialize( opts={} )
       @options = { :verbose => true }.merge(opts)
@@ -45,7 +45,7 @@ module MathMetadata
       page = fetch_article(opts)
       articles = []
   
-      return metadata unless page
+      return articles unless page
 
       if list_of_articles?(page)
         articles = get_article_list(page)
@@ -54,10 +54,13 @@ module MathMetadata
         articles << a unless a[:title].to_s.strip.empty?
       end
   
-      return nil if articles.size == 0
+      #return [] if articles.size == 0
       articles
     end
 
+    def to_json
+      ""
+    end
 
   protected
 
@@ -112,6 +115,7 @@ module MathMetadata
     def author_name_forms( name )
       forms = []
 
+      return forms if self.class::AUTHOR_URL.empty?
       page = fetch_author name
       forms = get_author_m page, 2, 1
 
@@ -145,16 +149,18 @@ module MathMetadata
     def get_article( page, opts={} )
       a = Article.new( {
         :id => get_article_id(page),
+        :language => get_article_language(page),
         :authors => get_article_author_s(page),
         :msc => get_article_msc(page),
         :publication => get_article_publication(page),
+        :publisher => get_article_publisher(page),
         :range => MathMetadata.normalize_range(get_article_range(page)),
         :year => get_article_year(page),
         :keywords => get_article_keyword_s(page),
         :issn => get_article_issn_s(page)
       } )
 
-      a.title, a.language = get_article_title(page, 2)
+      a.title = get_article_title(page)
       a.title = a.title.to_s.gsub(/<\/span>/,'')
       a.references = get_article_references(page) if opts[:references]
 
@@ -166,23 +172,50 @@ module MathMetadata
       articles = []
       page.scan(self.class::ARTICLE_ENTRY_RE).each do |match|
         a = article(:id => match[0]).first
+        next unless a
         articles << a unless a[:title].to_s.strip.empty?
       end
       articles
     end
 
 
+    # select first n words from string which are longer then 2 characters
     def nwords(s)
-      s.split(" ")[0...@options[:nwords].to_i].join(" ")
+      n = @options[:nwords].to_i
+      words = 0
+      s.split(" ").inject("") do |res,word|
+        if words < n
+          res << " " << word
+          words += 1 if word.size > 2
+        end
+        res.strip
+      end
     end
 
 
-    def fetch_page( url, args={} )
+    def normalize_page(page, args={})
       opts = {:entities => true}.merge(args)
-  
+
+      if page =~ %r{<meta.*charset=([^"\s]+)}i
+        if page.respond_to?(:force_encoding)
+          page.force_encoding($1)
+          page.encode!("utf-8")
+        else
+          page = Iconv.new('utf-8', $1).iconv(page)
+        end
+      end
+      if page and opts[:entities]
+        coder = HTMLEntities.new
+        page = coder.decode(page)
+      end
+
+    end
+
+
+    def fetch_page( url, args={} )
       puts "fetching #{url}" if @options[:verbose]
       page = URI.parse(url).read
-      page = HTMLEntities.decode_entities(page) if page and opts[:entities]
+      page = normalize_page page, args
     
       page
     end
@@ -197,7 +230,7 @@ module MathMetadata
   
 
     def join_article_authors( authors )
-      authors.collect { |author| URI.escape MathMetadata.normalize_name(author) }.join('; ') || ''
+      authors.collect { |author| URI.escape MathMetadata.normalize_name(author) }.join(';%20') || ''
     end
   
     def fetch_article( args={} )

data/lib/math_metadata_lookup/sites/cedram.rb

@@ -0,0 +1,158 @@
+# -*-: coding: utf-8 -*-
+# vi: fenc=utf-8:expandtab:ts=2:sw=2:sts=2
+
+require 'net/http'
+
+module MathMetadata
+
+  # NUMDAM
+  # http://numdam.org/
+  class CEDRAM < Site
+    ID = :cedram
+    NAME = "CEDRAM"
+    URL = "http://cedram.org/"
+
+
+    # AUTHOR_URL % "Author, Name"
+    AUTHOR_URL = %~~
+
+    AUTHORS_RE = %r{}mi
+    AUTHOR_RE = %r{}mi
+
+
+    ARTICLE_ID_URL = "http://aif.cedram.org/aif-bin/item?id=%s"
+    ARTICLE_ID_AFST_URL = "http://afst.cedram.org/afst-bin/item?id=%s"
+    ARTICLE_URL = "http://www.cedram.org/cedram-bin/search?ti=%s&au=%s&py1=%s&lang=en"
+#    ARTICLE_URL = "http://www.cedram.org/cedram-bin/search"
+
+    LIST_OF_ARTICLES_RE = %r{matches(.*?)</td>}mi
+    ARTICLE_ENTRY_RE = %r{<a href=".*?id=([^"]+)">Details</a>}mi
+    ARTICLE_ID_RE = %r{<input type="hidden" name="id" value="([^"]+)" />}mi
+    ARTICLE_TITLE_RE = %r{<span class="atitle">(.*?)</span>}mi
+    ARTICLE_LANGUAGE_RE = %r{xxxxxxxxxxxxxxx}mi
+    ARTICLE_AUTHORS_RE = %r{<hr />(<a.*?)<br />}mi
+    ARTICLE_AUTHOR_RE = %r{<a.*?>(.*?)</a>}mi
+    ARTICLE_MSCS_RE = %r{Class. Math.:(.*?)<br}mi
+    ARTICLE_MSC_RE = %r{([^,]+),?\s*}mi
+    ARTICLE_PUBLICATION_RE = %r{(<span class="jtitle">.*?</span>,\s*<a href="http://aif.cedram.org:80/aif-bin/get\?id=[^"]+">\d+</a>\s*no\.\s*<a href="http://aif.cedram.org:80/aif-bin/browse\?id=[^"]+">\d+</a>\s*\(<a href="http://aif.cedram.org:80/aif-bin/get\?id=[^"]+">\d+</a>\))}mi
+    ARTICLE_PUBLISHER_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_RANGE_RE = %r{(\d+\-\d+)\s*<br\s*/>\s*Article}mi
+    ARTICLE_YEAR_RE = %r{<span class="jtitle">.*?</span>,\s*<a href="http://aif.cedram.org:80/aif-bin/get\?id=[^"]+">\d+</a>\s*no\.\s*<a href="http://aif.cedram.org:80/aif-bin/browse\?id=[^"]+">\d+</a>\s*\(<a href="http://aif.cedram.org:80/aif-bin/get\?id=[^"]+">(\d+)</a>\)}mi
+    ARTICLE_ISSNS_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_ISSN_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_KEYWORDS_RE = %r{Keywords:(.*?)<div}mi
+    ARTICLE_KEYWORD_RE = %r{([^,]+),?\s*}mi
+    ARTICLE_REFERENCES_RE = %r{<P>\s*<B>\s*Bibliography\s*</B>\s*</P>\s*</DIV>\s*(.*?)\s*</td>}mi
+    ARTICLE_REFERENCE_RE = %r{\[\d+\]\s*(.*?)(?:(?:<BR>)|(?:<br\s*/>)|(?:</div>))}mi
+
+    def get_article_publication( page )
+      page =~ ARTICLE_PUBLICATION_RE
+      return nil unless $1
+      $1.gsub(/<.*?>/, '')
+    end
+
+    def get_article_references( page )
+      references = []
+
+      refs = get_article_reference_s page
+
+      i = 0;
+      refs.each do |r|
+        i+=1
+        r.gsub!(/\r|\n/, ' ')
+        r.gsub!(/ +/,' ')
+        ref = Reference.new nil, i
+        ref.source = r
+        ref.article = Article.new
+
+        r =~ %r{<span class="(?:atitle|btitre)">(.*?)</span>}im
+        ref.article.title = $1.to_s.strip
+
+        ref.article.authors = []
+        r.split(%r{<span class="bauteur">\s?(.*?)\s*</span>}).each do |a|
+          next if a.strip.empty? or a.strip == "-" or [',', '&'].include?(a.strip[0,1])
+          author = a.gsub /<.*?>/, ''
+          ref.article.authors << author
+        end
+
+        r =~ %r{<span class="(?:brevue|bjournal)">(.*?)</span>}mi
+        ref.article.publication = $1.strip if $1
+
+=begin
+        r =~ %r{<bediteur>(.*?)</bediteur>}mi
+        ref.article.publisher = $1.strip if $1
+
+        r =~ %r{<blieued>(.*?)</blieued>}mi
+        ref.article.place = $1.strip if $1
+=end
+        r =~ %r{(\d\d\d\d)}mi
+        ref.article.year = $1.strip if $1
+
+        r =~ %r{(\d+)\-(\d+)}mi
+        ref.article.range = "#{$1.strip}-#{$2.strip}" if $1
+
+        references << ref
+      end
+
+      references
+    end
+
+
+    def fetch_article( args={} )
+      opts = {:id => nil, :title => "", :year => "", :authors => []}.merge(args)
+      url = ""
+      unless opts[:id].to_s.strip.empty?
+        pattern = self.class::ARTICLE_ID_URL
+        pattern = self.class::ARTICLE_ID_AFST_URL if opts[:id] =~ /^AFST/mi
+        url = pattern % URI.escape(opts[:id].to_s.strip)
+      end
+      form = {'submit' => " Start search "}
+      if opts[:id].to_s.strip.empty?
+        author = join_article_authors opts[:authors]
+        title = opts[:title]
+        title = '' if not title.kind_of?(String)
+        title = MathMetadata.normalize_text(title)
+        title = nwords(title) if @options[:nwords]
+
+        form['ti'] = title
+        form['au'] = author unless author.empty?
+        form['py1'] = opts[:year].to_s
+        form['py2'] = ""
+        form['pages'] = ""
+        form['bibitems_text'] = ""
+        form['maxdocs'] = "300"
+        form['format'] = "short"
+        form['ti_op'] = "and"
+        form["au_op"] = "and"
+        form["bibitems.text_op"] = "and"
+
+        url = self.class::ARTICLE_URL % [URI.escape(title), author, opts[:year].to_s]
+#        url = self.class::ARTICLE_URL
+      else
+        return fetch_page(url, opts)
+      end
+      
+      uri = URI.parse(url)
+      puts uri if opts[:verbose]
+      req = Net::HTTP::Post.new(uri.path, {
+        'Host' => "www.cedram.org",
+        'User-Agent'=> "Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20110429 Firefox/4.0.1",
+        'Accept' => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+        'Accept-Language' => "cs,en;q=0.7,en-us;q=0.3",
+        'Accept-Encoding' => "gzip, deflated",
+        'Accept-Charset' => "UTF-8,*",
+        'Keep-Alive' => "115",
+        'Connection' => "keep-alive",
+        'Referer' => "http://www.cedram.org/cedram-bin/search",
+        'Content-Type' => "application/x-www-form-urlencoded",
+      })
+      req.set_form_data(form)
+      http = Net::HTTP.new(uri.host, uri.port)
+      resp = http.request(req)
+      page = normalize_page resp.body
+      page
+    end
+
+  end # MRev
+
+end

data/lib/math_metadata_lookup/sites/dmlcz.rb

@@ -0,0 +1,57 @@
+# -*-: coding: utf-8 -*-
+# vi: fenc=utf-8:expandtab:ts=2:sw=2:sts=2
+
+module MathMetadata
+
+  # Czech Digital Mathematics Library
+  # http://dml.cz/
+  # does not support author search
+  class DMLCZ < Site
+    ID = :dmlcz
+    NAME = "DMLCZ"
+    URL = "http://dml.cz/"
+
+
+    # AUTHOR_URL % "Author, Name"
+    AUTHOR_URL = %~~
+
+    AUTHORS_RE = %r{}mi
+    AUTHOR_RE = %r{}mi
+
+
+    ARTICLE_ID_URL = "http://dml.cz/handle/10338.dmlcz/%s?show=full"
+    ARTICLE_URL = "http://dml.cz/advanced-search?num_search_field=10&results_per_page=100&scope=%%2F&field1=title&query1=%s%s&conjunction2=AND&field2=year&query2=%s&submit=Go"
+
+    LIST_OF_ARTICLES_RE = %r{<ul class="bibliolist">(.*?)</ul>}mi
+    ARTICLE_ENTRY_RE = %r{<li>.*?href="/handle/10338.dmlcz/(\d+)".*?</li>}mi
+    #ARTICLE_ENTRY_RE = %r{<div class="headlineText">\s*<a href="/mathscinet/search/publdoc.html[^"]+">\s*<strong>\s*([^< ]+)\s*</strong>\s*<strong>}mi
+
+    ARTICLE_ID_RE = %r{<meta\s*name="citation_id"\s*content="(\d+)"\s*/>}mi
+    ARTICLE_TITLE_RE = %r{<meta\s*name="dc.Title"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_LANGUAGE_RE = %r{<meta\s*name="citation_language"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_AUTHORS_RE = %r{<meta\s*name="citation_authors"\s*content="([^"]+)" />}mi
+    ARTICLE_AUTHOR_RE = %r{([^;]+);?\s*}mi
+    ARTICLE_MSCS_RE = %r{<table\s*xmlns:fn="http://www.w3.org/2003/11/xpath-functions"\s*class="dml_detail_view">(.*?)</table>}mi
+    ARTICLE_MSC_RE = %r{<tr>\s*<td\s*class="label">\s*MSC:\s*</td>\s*<td\s*class="value">\s*([^< ]+)\s*</td>\s*</tr>}mi
+    ARTICLE_PUBLICATION_RE = %r{<meta\s*name="citation_journal_title"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_PUBLISHER_RE = %r{<meta\s*name="citation_publisher"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_RANGE_RE = %r{<tr>\s*<td class="label">\s*Pages:\s*</td>\s*<td\s*class="value">([^ <]+)</td>\s*</tr>}mi
+    ARTICLE_YEAR_RE = %r{<meta\s*name="citation_year"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_ISSNS_RE = %r{<head>(.*?)</head>}mi
+    ARTICLE_ISSN_RE = %r{<meta\s*name="citation_issn"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_KEYWORDS_RE = %r{<head>(.*?)</head>}mi
+    ARTICLE_KEYWORD_RE = %r{<meta\s*name="citation_keywords"\s*content="([^"]+)"\s*/>}mi
+    ARTICLE_REFERENCES_RE = %r{<table\s*xmlns:fn="http://www.w3.org/2003/11/xpath-functions"\s*class="dml_detail_view">(.*?)</table>}mi
+    ARTICLE_REFERENCE_RE = %r{<tr>\s*<td class="label">Reference:\s*</td>\s*<td class="value">\s*\[[^\]]+\]\s*([^<]+)</td>\s*</tr>}mi
+
+    def join_article_authors( authors )
+      i = 2
+      authors.collect { |author| 
+        i += 1
+        "&conjunction#{i}=AND&field#{i}=author&query#{i}=#{URI.escape MathMetadata.normalize_name(author)}"
+      }.join("&")
+    end
+
+  end # MRev
+
+end

data/lib/math_metadata_lookup/sites/mr.rb

@@ -27,12 +27,14 @@ module MathMetadata
     ARTICLE_ENTRY_RE = %r{<div class="headlineText">\s*<a href="/mathscinet/search/publdoc.html[^"]+">\s*<strong>\s*([^< ]+)\s*</strong>\s*<strong>}mi
 
     ARTICLE_ID_RE = %r{<strong>(.*?)</strong>}mi
-    ARTICLE_TITLE_RE = %r{<span class="title">(?:<span class="searchHighlight">)?(.*?)</span>\s*(?:<span class="sumlang">\(?(.*?)\)?</span>)?}mi
+    ARTICLE_TITLE_RE = %r{<span class="title">(?:<span class="searchHighlight">)?(.*?)</span>}mi
+    ARTICLE_LANGUAGE_RE = %r{<span class="sumlang">\s*\(?(.*?)\)?\s*</span>?}mi
     ARTICLE_AUTHORS_RE = %r{<br />(<a href="/mathscinet/search/publications.html[^"]*">.*?</a>)<br />}mi
     ARTICLE_AUTHOR_RE = %r{<a href="/mathscinet/search/publications.html[^"]*">(.*?)</a>}mi
     ARTICLE_MSCS_RE = %r{<a href="/mathscinet/search/mscdoc.html\?code=[^"]*">(.*?)</a>}mi
     ARTICLE_MSC_RE = %r{([^, ]+)}mi
     ARTICLE_PUBLICATION_RE = %r{<a href="/mathscinet/search/journaldoc\.html\?cn=[^"]*">\s*<em>(.*?)</em>\s*</a>}mi
+    ARTICLE_PUBLISHER_RE = %r{xxxxxxxxxxxxxxxxxx}mi
     ARTICLE_RANGE_RE = %r{(\d+–\d+)}mi
     ARTICLE_YEAR_RE = %r{<a href="/mathscinet/search/publications\.html[^"]*">\s*\(?(\d{4})\)?, </a>}mi
     ARTICLE_ISSNS_RE = %r{(ISSN.*?)<br>}mi

data/lib/math_metadata_lookup/sites/numdam.rb

@@ -0,0 +1,93 @@
+# -*-: coding: utf-8 -*-
+# vi: fenc=utf-8:expandtab:ts=2:sw=2:sts=2
+
+require 'rexml/document'
+
+module MathMetadata
+
+  # NUMDAM
+  # http://numdam.org/
+  class NUMDAM < Site
+    ID = :numdam
+    NAME = "NUMDAM"
+    URL = "http://numdam.org/"
+
+
+    # AUTHOR_URL % "Author, Name"
+    AUTHOR_URL = %~~
+
+    AUTHORS_RE = %r{}mi
+    AUTHOR_RE = %r{}mi
+
+
+    ARTICLE_ID_URL = "http://www.numdam.org/numdam-bin/item?id=%s"
+    ARTICLE_URL = "http://www.numdam.org/numdam-bin/search?bibitems.au_op=and&bibitems.text_op=and&ti=%s&au=%s&ti_op=and&Index1.y=0&Index1.x=0&bibitems.ti_op=and&au_op=and&py1=%s"
+
+    LIST_OF_ARTICLES_RE = %r{<P>\s*<DIV\s+align="center">.*?</DIV>\s*</P>\s*(.*?)\s*<P>\s*<DIV\s+align="center">.*?</DIV>\s*</P>}mi
+    ARTICLE_ENTRY_RE = %r{<a href="http://www.numdam.org:80/numdam-bin/item\?id=([^"]+)">Full entry</a>}mi
+
+    ARTICLE_ID_RE = %r{<P>stable URL: http://www.numdam.org/item\?id=([^<]+)</P>}mi
+    ARTICLE_TITLE_RE = %r{<SPAN class="atitle">(.*?)</SPAN>}mi
+    ARTICLE_LANGUAGE_RE = %r{xxxxxxxxxxxxxxx}mi
+    ARTICLE_AUTHORS_RE = %r{<head>\s*(.*?)\s*</head>}mi
+    ARTICLE_AUTHOR_RE = %r{<meta content="([^"]+)" name="DC.creator">}mi
+    ARTICLE_MSCS_RE = %r{Class\. Math\.:\s*(.*?)\s*<br>}mi
+    ARTICLE_MSC_RE = %r{([^,]+)(?:,\s*)?}mi
+    ARTICLE_PUBLICATION_RE = %r{<SPAN class="jtitle">(.*?)</SPAN>}mi
+    ARTICLE_PUBLISHER_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_RANGE_RE = %r{(\d+\-\d+)\s*<BR>\s*Full text}mi
+    ARTICLE_YEAR_RE = %r{py=(\d+)}mi
+    ARTICLE_ISSNS_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_ISSN_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_KEYWORDS_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_KEYWORD_RE = %r{xxxxxxxxxxxxxxxxx}mi
+    ARTICLE_REFERENCES_RE = %r{<P>\s*<B>\s*Bibliography\s*</B>\s*</P>\s*</DIV>\s*(.*?\s*</td>)}mi
+    ARTICLE_REFERENCE_RE = %r{\[\d+\](.*?)(?:<BR>|</td>)}mi
+
+    def get_article_references( page )
+      references = []
+
+      refs = get_article_reference_s page
+
+      i = 0;
+      refs.each do |r|
+        i+=1
+        ref = Reference.new nil, i
+        ref.source = r.gsub(/  +/,' ')
+        ref.article = Article.new
+
+        r =~ %r{<span class="atitle">(.*?)</span>}im
+        ref.article.title = $1.to_s.strip
+
+        ref.article.authors = []
+        r.split(%r{<span class="bauteur">\s?(.*?)\s*</span>}).each do |a|
+          a.strip!
+          next if a.empty? or a.== "-" or [',', '&'].include?(a[0,1]) or a.downcase=='and'
+          author = a.gsub /<.*?>/, ''
+          ref.article.authors << author
+        end
+
+        r =~ %r{<span class="brevue">(.*?)</span>}mi
+        ref.article.publication = $1.strip if $1
+
+        r =~ %r{<bediteur>(.*?)</bediteur>}mi
+        ref.article.publisher = $1.strip if $1
+
+        r =~ %r{<blieued>(.*?)</blieued>}mi
+        ref.article.place = $1.strip if $1
+
+        r =~ %r{<bannee>(.*?)</bannee>}mi
+        ref.article.year = $1.strip if $1
+
+        r =~ %r{<bpagedeb>(\d+)</bpagedeb>-<bpagefin>(\d+)</bpagefin>}mi
+        ref.article.range = "#{$1.strip}-#{$2.strip}" if $1
+
+        references << ref
+      end
+
+      references
+    end
+
+  end # MRev
+
+end

data/lib/math_metadata_lookup/sites/zbl.rb

@@ -26,7 +26,8 @@ module MathMetadata
     ARTICLE_ENTRY_RE = %r{<span[^>]*?>\s*<a href="\?q=an:([^\&]+)\&format=complete">[^<]+</a>\s*<b>}mi
 
     ARTICLE_ID_RE = %r{<a href="\?q=an:.*?complete">(.*?)</a>}mi
-    ARTICLE_TITLE_RE = %r{</a><br>(.*?)\.</b>\s*\((.*?)\)<br>}mi
+    ARTICLE_TITLE_RE = %r{</a><br>(.*?)\.</b>\s*\(.*?\)<br>}mi
+    ARTICLE_LANGUAGE_RE = %r{</a><br>.*?\.</b>\s*\((.*?)\)<br>}mi
     ARTICLE_AUTHORS_RE = %r{<br><b>(<a href="\?q=[^"]*">.*?</a>)<br>}mi
     ARTICLE_AUTHOR_RE = %r{<a href="\?q=[^"]*">(.*?)</a>}mi
     ARTICLE_MSCS_RE = %r{<dd>(.*?)</dd>}mi

data/lib/math_metadata_lookup/tools.rb

@@ -44,7 +44,7 @@ module MathMetadata
 
 
     def normalize_mscs( mscs )
-      mscs.map{|m| m.split(/,|;/) }.flatten.map{|m| m =~ /\s*\(?([^\s\)\(]+)\)?\s*/; $1}
+      mscs.map{|m| m.split(/,|;/) }.flatten.map{|m| m.gsub(/<.*?>/,'')}.map{|m| m =~ /\s*\(?([^\s\)\(]+)\)?\s*/; $1}
     end
 
 
@@ -80,6 +80,7 @@ module MathMetadata
       str = remove_punctuation(str)
       str.gsub!(%r{\W+}, ' ')
       str.gsub!(%r{(?:the|a|of|)\s+}i, ' ')
+      str.gsub!(%r{\s+}, ' ')
       str.strip
     end
 

data/math_metadata_lookup.gemspec

@@ -6,25 +6,26 @@
 require 'rubygems'
 require 'find'
 
-spec = Gem::Specification.new do |s|
+@spec = Gem::Specification.new do |s|
   s.platform = Gem::Platform::RUBY
   s.summary = "Search mathematical reviews sites and fetches metadata about articles."
   s.homepage = "http://github.com/pejuko/math_metadata_lookup"
   s.email = "pejuko@gmail.com"
   s.authors = ["Petr Kovar"]
   s.name = 'math_metadata_lookup'
-  s.version = '0.1.4'
+  s.version = '0.2.0'
   s.date = Time.now.strftime("%Y-%m-%d")
   s.add_dependency('unicode')
   s.add_dependency('unidecoder')
   s.add_dependency('ya2yaml')
+  s.add_dependency('json')
   s.require_path = 'lib'
   s.files = ["bin/math_metadata_lookup", "README.md", "math_metadata_lookup.gemspec", "TODO", "Rakefile"]
   s.files += Dir["lib/**/*.rb", "resources/*"]
   s.executables = ["math_metadata_lookup"]
   s.description = <<EOF
 This utility/library search mathematical reviews sites and fetches metadata about articles.
-It can return results as one of text, xml, html, yaml or ruby formats.
+It can return results as one of text, xml, html, yaml, json or ruby formats.
 EOF
 end
 

metadata

@@ -1,13 +1,12 @@
 --- !ruby/object:Gem::Specification 
 name: math_metadata_lookup
 version: !ruby/object:Gem::Version 
-  hash: 19
-  prerelease: 
+  prerelease: false
   segments: 
   - 0
-  - 1
-  - 4
-  version: 0.1.4
+  - 2
+  - 0
+  version: 0.2.0
 platform: ruby
 authors: 
 - Petr Kovar
@@ -15,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-01-27 00:00:00 +01:00
+date: 2011-06-10 00:00:00 +02:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -26,7 +25,6 @@ dependencies:
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        hash: 3
         segments: 
         - 0
         version: "0"
@@ -40,7 +38,6 @@ dependencies:
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        hash: 3
         segments: 
         - 0
         version: "0"
@@ -54,15 +51,27 @@ dependencies:
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        hash: 3
         segments: 
         - 0
         version: "0"
   type: :runtime
   version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: json
+  prerelease: false
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        version: "0"
+  type: :runtime
+  version_requirements: *id004
 description: |
   This utility/library search mathematical reviews sites and fetches metadata about articles.
-  It can return results as one of text, xml, html, yaml or ruby formats.
+  It can return results as one of text, xml, html, yaml, json or ruby formats.
 
 email: pejuko@gmail.com
 executables: 
@@ -83,8 +92,11 @@ files:
 - lib/math_metadata_lookup/tools.rb
 - lib/math_metadata_lookup/lookup.rb
 - lib/math_metadata_lookup/result.rb
+- lib/math_metadata_lookup/sites/numdam.rb
 - lib/math_metadata_lookup/sites/mr.rb
 - lib/math_metadata_lookup/sites/zbl.rb
+- lib/math_metadata_lookup/sites/cedram.rb
+- lib/math_metadata_lookup/sites/dmlcz.rb
 - lib/math_metadata_lookup/reference.rb
 - lib/math_metadata_lookup/entity.rb
 - lib/math_metadata_lookup/author.rb
@@ -103,7 +115,6 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      hash: 3
       segments: 
       - 0
       version: "0"
@@ -112,14 +123,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      hash: 3
       segments: 
       - 0
       version: "0"
 requirements: []
 
 rubyforge_project: 
-rubygems_version: 1.4.1
+rubygems_version: 1.3.7
 signing_key: 
 specification_version: 3
 summary: Search mathematical reviews sites and fetches metadata about articles.