infoboxer 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 23b2c1e2b3e0953dcab1ee64ea145cefc02f2174
-  data.tar.gz: e8747e93e3bafcddff783bc08efb6df6ed32703b
+  metadata.gz: 37db410a52fa9f9755b70ff3a1284c6a427509b8
+  data.tar.gz: cd9952485ffffb6b3eb3d3cb69cb771aa2c9610a
 SHA512:
-  metadata.gz: 28c1f2fb24a4831d4ca5af047e8ca3c7b573e1747aed0ec152468d64e5710c2c51e342de39178b2938f4af49d0f31a2d5afe0c8a37def863040a3803d9aa4aa0
-  data.tar.gz: 1296e07dd28bd64435856dbbf8bdf5cc6bb65be135dd51483aa9a1637e5dc78403b221780b4cbe2ad85d8ac89dac0907309c96f0a921634b88636f9e0473bee2
+  metadata.gz: 2800426e4a055f838a89b50bd9388364f7a2499d2736816c686a5d9bb2cc727d2bbdfe485565835d844cefe75c8d56f510f2fd2823411e4b7a37c4d0fd3122bf
+  data.tar.gz: 6dc878f4957c468051ecff57eee57ca66ac2ed19e157cb6f62f993113ba2c8bd220f87926eda4f51aa5917d097ca2e8c13d94e2acd9dff8b8f82036412845245

data/.yardopts

@@ -1 +1,2 @@
 --markup=markdown
+--no-private

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Infoboxer's change log
+
+## 0.1.1 (2015-08-11)
+
+Basically, preparing for wider release!
+
+* Small refactorings;
+* Documentation fixes;
+
+## 0.1.0 (2015-08-07)
+
+Initial (ok, I know it's typically called 0.1.1, but here's work of
+three monthes, numerous documentations and examples and so on... so, let
+it be 0.1.0).

data/CONTRIBUTING.md

@@ -0,0 +1,62 @@
+# Contributing to Infoboxer
+
+_(Also duplicated in [wiki](https://github.com/molybdenum-99/infoboxer/wiki/Contributing).)_
+
+## Contributing via test cases
+
+If you are assured that Infoboxer takes some page wrong, please create an
+[issue](https://github.com/molybdenum-99/infoboxer/issues) with link
+to page (or raw wikitext) and description of a problem.
+
+## Contributing via localizations and templates describing
+
+Look at [en.wikipedia.org](https://github.com/molybdenum-99/infoboxer/blob/master/lib/infoboxer/definitions/en.wikipedia.org.rb)
+template definitions. It can be extended. Also, similar definitions
+can/should be created for other language wikipedias and other popular
+wikis.
+
+You can do pull requests with your own definitions, or create an
+[issue](https://github.com/molybdenum-99/infoboxer/issues) describing
+which template definitions should be added to Infoboxer.
+
+## Contributing via code
+
+If you want to fix some bug or implement some feature, please just
+follow the standard process for github opensource: fork, fix, push,
+make pull request.
+
+Some (scanty) information below.
+
+### Understanding the code
+
+* Infoboxer is splitted in several modules (which are clearly visible in
+  API docs and folders structure).
+* Most of "easy features"
+  can be added to [Navigation](http://www.rubydoc.info/gems/infoboxer/Infoboxer/Navigation)
+  module and its submodules: enchancing of navigational experience and
+  implement clever shortcuts (like "converting table to dataframe/list of
+  hashes", for ex.).
+* Most of potential bugs can seat in
+  [Parser](http://www.rubydoc.info/gems/infoboxer/Infoboxer/Parser) class
+  and its modules; MediaWiki markup IS tricky and tightly coupled and
+  ambigous; there's also some non-implemented features, like `<source>`
+  tag parsing and template definition pages (which, possibly, is not
+  target of Infoboxer anyways).
+* Most of underfeatured area is in
+  [MediaWiki](http://www.rubydoc.info/gems/infoboxer/Infoboxer/MediaWiki)
+  -- seems reasonable for information extraction purposes to have more
+  features from MediaWiki API, like "page list generators", search,
+  "what links here" and similar functionality.
+* Most of clarification and documentation is required for 
+  [Templates](http://www.rubydoc.info/gems/infoboxer/Infoboxer/Templates)
+  module, which is still underloved heart of Infoboxer.
+
+### Parser: quick, not clever
+
+Whether you'd want to put your hands on Parser: please remember, that
+it's hand-crafted and thoroughly optimized. The first thought you may
+have that it needs more OO decompozition, a class for each case; or more
+ideomatic Ruby, or ... Trust me, I've tried it all. But when you are
+dealing with hundreds of thousands of parsing operations and tens of
+thousands of resulting nodes, it turns out even simplest things like
+`Object#tap` have performance penalty on large number of calls.

data/README.md

@@ -16,7 +16,7 @@ obvious structure, you can navigate that tree easily, and you have a
 bunch of hi-level helpers method, so typical information extraction
 tasks should be super-easy, one-liners in best cases.
 
-_(For those already thinkind "Why should you do this, we already have
+_(For those already thinking "Why should you do this, we already have
 DBPedia?" -- please, read "[Reasons](https://github.com/molybdenum-99/infoboxer/wiki/Reasons)"
 page in our wiki.)_
 

data/infoboxer.gemspec

@@ -5,7 +5,7 @@ Gem::Specification.new do |s|
   s.version  = Infoboxer::VERSION
   s.authors  = ['Victor Shepelev']
   s.email    = 'zverok.offline@gmail.com'
-  s.homepage = 'https://github.com/zverok/infoboxer'
+  s.homepage = 'https://github.com/molybdenum-99/infoboxer'
 
   s.summary = 'MediaWiki client and parser, targeting information extraction.'
   s.description = <<-EOF
@@ -40,4 +40,5 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'ruby-prof'
   s.add_development_dependency 'vcr'
   s.add_development_dependency 'webmock'
+  s.add_development_dependency 'inch'
 end

data/lib/infoboxer/media_wiki/traits.rb

@@ -1,8 +1,20 @@
 # encoding: utf-8
 module Infoboxer
   class MediaWiki
+    # DSL for defining "traits" for some site.
+    #
+    # More docs (and possible refactoring) to follow.
+    #
+    # You can look at current
+    # [English Wikipedia traits](https://github.com/molybdenum-99/infoboxer/blob/master/lib/infoboxer/definitions/en.wikipedia.org.rb)
+    # definitions in Infoboxer's repo.
     class Traits
       class << self
+        # Define set of templates for current site's traits.
+        #
+        # See {Templates::Set} for longer (yet insufficient) explanation.
+        #
+        # Expected to be used inside Traits definition block.
         def templates(&definition)
           @templates ||= Templates::Set.new
 
@@ -11,35 +23,49 @@ module Infoboxer
           @templates.define(&definition)
         end
 
-        # NB: explicitly store all domains in base Traits class
+        # @private
         def domain(d)
+          # NB: explicitly store all domains in base Traits class
           Traits.domains.key?(d) and
             fail(ArgumentError, "Domain binding redefinition: #{Traits.domains[d]}")
 
           Traits.domains[d] = self
         end
 
+        # @private
         def get(domain, options = {})
           cls = Traits.domains[domain]
           cls ? cls.new(options) : Traits.new(options)
         end
 
+        # @private
         def domains
           @domains ||= {}
         end
 
+        # Define traits for some  domain. Use it like:
+        #
+        # ```ruby
+        # MediaWiki::Traits.for 'ru.wikipedia.org' do
+        #   templates do
+        #     template '...' do
+        #       # some template definition
+        #     end
+        #   end
+        # end
+        # ```
+        #
+        # Again, you can look at current
+        # [English Wikipedia traits](https://github.com/molybdenum-99/infoboxer/blob/master/lib/infoboxer/definitions/en.wikipedia.org.rb)
+        # for example implementation.
         def for(domain, &block)
           Class.new(self, &block).domain(domain)
         end
 
+        # @private
         alias_method :default, :new
       end
 
-      DEFAULTS = {
-        file_prefix: 'File',
-        category_prefix: 'Category'
-      }
-
       def initialize(options = {})
         @options = options
         @file_prefix = [DEFAULTS[:file_prefix], options.delete(:file_prefix)].
@@ -48,13 +74,21 @@ module Infoboxer
           flatten.compact.uniq
       end
 
+      # @private
       attr_reader :file_prefix, :category_prefix
 
-      #attr_accessor :re
-
+      # @private
       def templates
         self.class.templates
       end
+
+      private
+
+      DEFAULTS = {
+        file_prefix: 'File',
+        category_prefix: 'Category'
+      }
+
     end
   end
 end

data/lib/infoboxer/templates.rb

@@ -1,7 +1,42 @@
 module Infoboxer
-  # Templates are cool, powerful and undocumented. Sorry :(
+  # This module covers advanced MediaWiki templates usage.
+  #
+  # It is seriously adviced to read [Wikipedia docs](https://en.wikipedia.org/wiki/Help:Template)
+  # or at least look through it (and have it opened while reading further).
+  #
+  # If you just have a page with templates and want some variable value
+  # (like "page about country - infobox - total population"), you should
+  # be totally happy with {Tree::Template} and its features.
+  #
+  # What this module does is, basically, two things:
+  # * allow you to define for arbitrary templates how they are converted
+  #   to text; by default, templates are totally excluded from text, which
+  #   is not most reasonable behavior for many formatting templates;
+  # * allow you to define additional functionality for arbitrary templates;
+  #   many of them containing pretty complicated logic (see, for ex.,
+  #   [Template:Convert](https://en.wikipedia.org/wiki/Template:Convert),
+  #   and it seems reasonable to extend instances of such a template.
+  #
+  # Infoboxer allows you to define {Templates::Set} of template-specific
+  # classes for some site/domain.
+  # There is already defined set of most commonly used templates at
+  # en.wikipedia.org (so, most of English Wikipedia texts will be rendered
+  # correctly, and also some advanced functionality is provided).
+  # You can take a look at
+  # [lib/infoboxer/definitions/en.wikipedia.org.rb](https://github.com/molybdenum-99/infoboxer/blob/master/lib/infoboxer/definitions/en.wikipedia.org.rb)
+  # to feel it (and also see a couple of TODOs and FIXMEs and other
+  # considerations).
+  #
+  # From Infoboxer's point-of-view, templates are the most complex part
+  # of Wikipedia, and we are currently trying hard to do the most reasonable
+  # things about them.
+  #
+  # Future versions also should:
+  # * define more of common English Wikipedia templates;
+  # * define templates for other popular wikis;
+  # * allow to add template definitions on-the-fly, while loading some
+  #   page.
   #
-  # I do my best.
   module Templates
     %w[base set].each do |tmpl|
       require_relative "templates/#{tmpl}"

data/lib/infoboxer/templates/base.rb

@@ -15,29 +15,6 @@ module Infoboxer
         end
       end
 
-      def unnamed_variables
-        variables.select{|v| v.name =~ /^\d+$/}
-      end
-
-      def fetch(*patterns)
-        Nodes[*patterns.map{|p| variables.find(name: p)}.flatten]
-      end
-
-      def fetch_hash(*patterns)
-        fetch(*patterns).map{|v| [v.name, v]}.to_h
-      end
-
-      def fetch_date(*patterns)
-        components = fetch(*patterns)
-        components.pop while components.last.nil? && !components.empty?
-        
-        if components.empty?
-          nil
-        else
-          Date.new(*components.map{|v| v.to_s.to_i})
-        end
-      end
-
       def ==(other)
         other.kind_of?(Tree::Template) && _eq(other)
       end
@@ -54,7 +31,9 @@ module Infoboxer
     end
 
     # Renders all of its unnamed variables as space-separated text
-    # Also allows in-template navigation
+    # Also allows in-template navigation.
+    #
+    # Used for {Set} definitions.
     class Show < Base
       alias_method :children, :unnamed_variables
 
@@ -65,6 +44,9 @@ module Infoboxer
       end
     end
 
+    # Replaces template with replacement, while rendering.
+    #
+    # Used for {Set} definitions.
     class Replace < Base
       def replace
         fail(NotImplementedError, "Descendants should define :replace")
@@ -75,6 +57,9 @@ module Infoboxer
       end
     end
 
+    # Replaces template with its name, while rendering.
+    #
+    # Used for {Set} definitions.
     class Literal < Base
       alias_method :text, :name
     end

data/lib/infoboxer/templates/set.rb

@@ -1,31 +1,103 @@
 # encoding: utf-8
 module Infoboxer
   module Templates
+    # Base class for defining set of templates, used for some site/domain.
+    #
+    # Currently only can be plugged in via {MediaWiki::Traits.templates}.
+    #
+    # Template set defines a DSL for creating new template definitions --
+    # also simplest ones and very complicated.
+    #
+    # You can look at implementation of English Wikipedia
+    # [common templates set](https://github.com/molybdenum-99/infoboxer/blob/master/lib/infoboxer/definitions/en.wikipedia.org.rb)
+    # in Infoboxer's repo.
+    #
     class Set
       def initialize(&definitions)
         @templates = []
         define(&definitions) if definitions
       end
-      
+
+      # @private
       def find(name)
         _, template = @templates.detect{|m, t| m === name.downcase}
         template || Base
       end
 
+      # @private
       def define(&definitions)
         instance_eval(&definitions)
       end
 
+      # @private
       def clear
         @templates.clear
       end
 
-      private
-
+      # Most common form of template definition.
+      #
+      # Can be used like:
+      #
+      # ```ruby
+      # template 'Age' do
+      #   def from
+      #     fetch_date('1', '2', '3')
+      #   end
+      # 
+      #   def to
+      #     fetch_date('4', '5', '6') || Date.today
+      #   end
+      # 
+      #   def value
+      #     (to - from).to_i / 365 # FIXME: obviously
+      #   end
+      # 
+      #   def text
+      #     "#{value} years"
+      #   end
+      # end
+      # ```
+      #
+      # @param name Definition name.
+      # @param options Definition options.
+      #   Currently recognized options are:
+      #   * `:match` -- regexp or string, which matches template name to
+      #     add this definition to (if not provided, `name` param used
+      #     to match relevant templates);
+      #   * `:base` -- name of template definition to use as a base class;
+      #     for example you can do things like:
+      #
+      #   ```ruby
+      #   # ...inside template set definition...
+      #   template 'Infobox', match: /^Infobox/ do
+      #     # implementation
+      #   end
+      #
+      #   template 'Infobox cheese', base: 'Infobox' do
+      #   end
+      #   ```
+      #
+      # Expected to be used inside Set definition block.
       def template(name, options = {}, &definition)
         setup_class(name, Base, options, &definition)
       end
 
+      # Define list of "replacements": templates, which text should be replaced
+      # with arbitrary value.
+      #
+      # Example:
+      #
+      # ```ruby
+      # # ...inside template set definition...
+      # replace(
+      #   '!!' => '||',
+      #   '!(' => '['
+      # )
+      # ```
+      # Now, all templates with name `!!` will render as `||` when you
+      # call their (or their parents') {Tree::Node#text}.
+      #
+      # Expected to be used inside Set definition block.
       def replace(*replacements)
         case
         when replacements.count == 2 && replacements.all?{|r| r.is_a?(String)}
@@ -44,18 +116,50 @@ module Infoboxer
         end
       end
 
+      # Define list of "show children" templates. Those ones, when rendered
+      # as text, just provide join of their children text (space-separated).
+      #
+      # Example:
+      #
+      # ```ruby
+      # #...in template set definition...
+      # show 'Small'
+      # ```
+      # Now, wikitext paragraph looking like...
+      #
+      # ```
+      # This is {{small|text}} in template
+      # ```
+      # ...before this template definition had rendered like
+      # `"This is  in template"` (template contents ommitted), and after
+      # this definition it will render like `"This is text in template"`
+      # (template contents rendered as is).
+      #
+      # Expected to be used inside Set definition block.
       def show(*names)
         names.each do |name|
           setup_class(name, Show)
         end
       end
 
+      # Define list of "literally rendered templates". It means, when
+      # rendering text, template is replaced with just its name.
+      #
+      # Explanation: in
+      # MediaWiki, there are contexts (deeply in other templates and
+      # tables), when you can't just type something like `","` and not
+      # have it interpreted. So, wikis oftenly define wrappers around
+      # those templates, looking like `{{,}}` -- so, while rendering texts,
+      # such templates can be replaced with their names.
+      #
+      # Expected to be used inside Set definition block.
       def literal(*names)
         names.each do |name|
           setup_class(name, Literal)
         end
       end
 
+      # @private
       def setup_class(name, base_class, options = {}, &definition)
         match = options.fetch(:match, name.downcase)
         base = options.fetch(:base, base_class)

data/lib/infoboxer/tree/compound.rb

@@ -21,6 +21,7 @@ module Infoboxer
         children.index(child)
       end
 
+      # @private
       # Internal, used by {Parser}
       def push_children(*nodes)
         nodes.each{|c| c.parent = self}.each do |n|
@@ -45,21 +46,25 @@ module Infoboxer
 
       # Kinda "private" methods, used by Parser only -------------------
       
+      # @private
       # Internal, used by {Parser}
       def can_merge?(other)
         false
       end
 
+      # @private
       # Internal, used by {Parser}
       def closed!
         @closed = true
       end
 
+      # @private
       # Internal, used by {Parser}
       def closed?
         @closed
       end
 
+      # @private
       # Internal, used by {Parser}
       def empty?
         children.empty?

data/lib/infoboxer/tree/html.rb

@@ -21,6 +21,7 @@ module Infoboxer
 
       include HTMLTagCommons
 
+      # @private
       # Internal, used by {Parser}.
       def empty?
         # even empty tag, for ex., <br>, should not be dropped!

data/lib/infoboxer/tree/linkable.rb

@@ -0,0 +1,42 @@
+module Infoboxer
+  module Tree
+    # Module included into everything, that can be treated as
+    # link to some MediaWiki page, despite of behavior. Namely,
+    # {Wikilink} and {Template}.
+    module Linkable
+      # Extracts wiki page by this link and returns it parsed (or nil,
+      # if page not found).
+      #
+      # About template "following" see also {Template#follow} docs.
+      #
+      # @return {MediaWiki::Page}
+      #
+      # **See also**:
+      # * {Tree::Nodes#follow} for extracting multiple links at once;
+      # * {MediaWiki#get} for basic information on page extraction.
+      def follow
+        client.get(link)
+      end
+
+      # Human-readable page URL
+      #
+      # @return [String]
+      def url
+        # FIXME: fragile as hell.
+        page.url.sub(/[^\/]+$/, link.gsub(' ', '_'))
+      end
+
+      protected
+
+      def page
+        page = lookup_parents(MediaWiki::Page).first or
+          fail("Not in a page from real source")
+      end
+
+      def client
+        page.client or fail("MediaWiki client not set")
+      end
+
+    end
+  end
+end

data/lib/infoboxer/tree/list.rb

@@ -3,12 +3,14 @@ module Infoboxer
   module Tree
     # Represents item of ordered or unordered list. 
     class ListItem < BaseParagraph
+      # @private
       # Internal, used by {Parser}
       def can_merge?(other)
         other.class == self.class &&
           other.children.first.kind_of?(List)
       end
 
+      # @private
       # Internal, used by {Parser}
       def merge!(other)
         ochildren = other.children.dup
@@ -111,6 +113,7 @@ module Infoboxer
     class List < Compound
       include Mergeable
 
+      # @private
       # Internal, used by {Parser}
       def merge!(other)
         ochildren = other.children.dup
@@ -123,6 +126,7 @@ module Infoboxer
         push_children(*ochildren)
       end
 
+      # @private
       # Internal, used by {Parser}
       def self.construct(marker, nodes)
         m = marker.shift

data/lib/infoboxer/tree/node.rb

@@ -61,11 +61,13 @@ module Infoboxer
         Nodes[] # redefined in descendants
       end
 
+      # @private
       # Used only during tree construction in {Parser}.
       def can_merge?(other)
         false
       end
 
+      # @private
       # Whether node is empty (definition of "empty" varies for different
       # kinds of nodes). Used mainly in {Parser}.
       def empty?

data/lib/infoboxer/tree/nodes.rb

@@ -146,6 +146,7 @@ module Infoboxer
         page.client.get(*links)
       end
 
+      # @private
       # Internal, used by {Parser}
       def <<(node)
         if node.kind_of?(Array)
@@ -159,6 +160,7 @@ module Infoboxer
         end
       end
 
+      # @private
       # Internal, used by {Parser}
       def strip
         res = dup
@@ -167,6 +169,7 @@ module Infoboxer
         res
       end
 
+      # @private
       # Internal, used by {Parser}
       def flow_templates
         make_nodes map{|n| n.is_a?(Paragraph) ? n.to_templates? : n}

data/lib/infoboxer/tree/paragraphs.rb

@@ -15,7 +15,6 @@ module Infoboxer
     end
 
     # @private
-    # Internal! Nothing to see here! Just YARD `@private` tag not working at class level
     class EmptyParagraph < Node
       def initialize(text)
         @text = text
@@ -30,7 +29,6 @@ module Infoboxer
     end
 
     # @private
-    # Internal! Nothing to see here! Just YARD `@private` tag not working at class level
     module Mergeable
       def can_merge?(other)
         !closed? && self.class == other.class
@@ -49,7 +47,6 @@ module Infoboxer
     end
     
     # @private
-    # Internal! Nothing to see here! Just YARD `@private` tag not working at class level
     class MergeableParagraph < BaseParagraph
       include Mergeable
 
@@ -61,21 +58,25 @@ module Infoboxer
 
     # Represents plain text paragraph.
     class Paragraph < MergeableParagraph
+      # @private
       # Internal, used by {Parser} for merging
       def splitter
         Text.new(' ')
       end
 
+      # @private
       # Internal, used by {Parser}
       def templates_only?
         children.all?{|c| c.is_a?(Template) || c.is_a?(Text) && c.raw_text.strip.empty?}
       end
 
+      # @private
       # Internal, used by {Parser}
       def to_templates
         children.select(&filter(itself: Template))
       end
 
+      # @private
       # Internal, used by {Parser}
       def to_templates?
         templates_only? ? to_templates : self
@@ -104,6 +105,7 @@ module Infoboxer
     #
     # Paragraph-level thing, can contain many lines of text.
     class Pre < MergeableParagraph
+      # @private
       # Internal, used by {Parser}
       def merge!(other)
         if other.is_a?(EmptyParagraph) && !other.text.empty?
@@ -113,6 +115,7 @@ module Infoboxer
         end
       end
 
+      # @private
       # Internal, used by {Parser} for merging
       def splitter
         Text.new("\n")

data/lib/infoboxer/tree/ref.rb

@@ -18,6 +18,7 @@ module Infoboxer
       # @!attribute [r] name
       def_readers :name
 
+      # @private
       # Internal, used by {Parser}
       def empty?
         # even empty tag should not be dropped!

data/lib/infoboxer/tree/template.rb

@@ -1,4 +1,6 @@
 # encoding: utf-8
+require_relative 'linkable'
+
 module Infoboxer
   module Tree
     # Template variable.
@@ -29,15 +31,79 @@ module Infoboxer
       end
     end
 
-    # Wikipedia template.
+    # Represents MediaWiki **template**.
+    #
+    # [**Template**](https://en.wikipedia.org/wiki/Wikipedia:Templates)
+    # is basically a thing with name, some variables and their
+    # values. When pages are displayed in browser, templates are rendered in
+    # something different by wiki engine; yet, when extracting information
+    # with Infoboxer, you are working with original templates.
+    #
+    # It requires some mastering and understanding, yet allows to do
+    # very poweful things. There are many kinds of them, from pure 
+    # formatting-related (which are typically not more than small bells
+    # and whistles for page outlook, and should be rendered as a text)
+    # to very information-heavy ones, like
+    # [**infoboxes**](https://en.wikipedia.org/wiki/Help:Infobox), from
+    # which Infoboxer borrows its name!
+    #
+    # Basically, for information extraction from template you'll list
+    # its {#variables}, and then use {#fetch} method
+    # (and its variants: {#fetch_hash}/#{fetch_date}) to extract their
+    # values.
+    #
+    # ### On variables naming
+    # MediaWiki templates can contain _named_ and _unnamed_ variables.
+    # Example:
+    #
+    # ```
+    # {{birth date and age|1953|2|19|df=y}}
+    # ```
     #
-    # Templates are complicated! Also, they are useful.
+    # This is template with name "birth date and age", three unnamed
+    # variables with values "1953", "2" and "19", and one named variable
+    # with name "df" and value "y".
     #
-    # You'd need to understand them from [Wikipedia docs](https://en.wikipedia.org/wiki/Wikipedia:Templates)
-    # and then use much of Infoboxer's goodness provided with {Templates}
-    # separate module.
+    # For consistency, Infoboxer treats unnamed variables _exactly_ the
+    # same way MediaWiki does: they considered to have numeric names,
+    # which are _started from 1_ and _stored as a strings_. So, for
+    # template shown above, the following is correct:
+    #
+    # ```ruby
+    # template.fetch('1').text == '1953'
+    # template.fetch('2').text == '2'
+    # template.fetch('3').text == '19'
+    # template.fetch('df').text == 'y'
+    # ```
+    #
+    # Note also, that _named variables with simple text values_ are
+    # duplicated as a template node {Node#params}, so, the following is
+    # correct also:
+    #
+    # ```ruby
+    # template.params['df'] == 'y'
+    # template.params.has_key?('1') == false
+    # ```
+    #
+    # For more advanced topics, like subclassing templates by names and
+    # converting them to inline text, please read {Templates} module's
+    # documentation.
     class Template < Compound
-      attr_reader :name, :variables
+      # Template name, designating its contents structure.
+      #
+      # See also {Linkable#url #url}, which you can navigate to read template's
+      # definition (and, in Wikipedia and many other projects, its
+      # documentation).
+      #
+      # @return [String]
+      attr_reader :name
+
+      # Template variables list.
+      #
+      # See {Var} class to understand what you can do with them.
+      #
+      # @return [Nodes<Var>]
+      attr_reader :variables
 
       def initialize(name, variables = Nodes[])
         super(Nodes[], extract_params(variables))
@@ -51,6 +117,97 @@ module Infoboxer
           variables.map{|var| var.to_tree(level+1)}.join
       end
 
+      # Returns list of template variables with numeric names (which
+      # are treated as "unnamed" variables by MediaWiki templates, see
+      # {Template class docs} for explanation).
+      #
+      # @return [Nodes<Var>]
+      def unnamed_variables
+        variables.find(name: /^\d+$/)
+      end
+
+      # Fetches template variable(s) by name(s) or patterns.
+      #
+      # Usage:
+      #
+      # ```ruby
+      # argentina.infobox.fetch('leader_title_1')   # => one Var node
+      # argentina.infobox.fetch('leader_title_1',
+      #                         'leader_name_1')    # => two Var nodes
+      # argentina.infobox.fetch(/leader_title_\d+/) # => several Var nodes
+      # ```
+      #
+      # @return [Nodes<Var>]
+      def fetch(*patterns)
+        Nodes[*patterns.map{|p| variables.find(name: p)}.flatten]
+      end
+
+      # Fetches hash `{name => variable}`, by same patterns as {#fetch}.
+      #
+      # @return [Hash<String => Var>]
+      def fetch_hash(*patterns)
+        fetch(*patterns).map{|v| [v.name, v]}.to_h
+      end
+
+      # Fetches date by list of variable names containing date components.
+      #
+      # _(Experimental, subject to change or enchance.)_
+      #
+      # Explanation: if you have template like
+      # ```
+      # {{birth date and age|1953|2|19|df=y}}
+      # ```
+      # ...there is a short way to obtain date from it:
+      # ```ruby
+      # template.fetch_date('1', '2', '3') # => Date.new(1953,2,19)
+      # ```
+      #
+      # @return [Date]
+      def fetch_date(*patterns)
+        components = fetch(*patterns)
+        components.pop while components.last.nil? && !components.empty?
+        
+        if components.empty?
+          nil
+        else
+          Date.new(*components.map{|v| v.to_s.to_i})
+        end
+      end
+
+      include Linkable
+
+      # @!method follow
+      # Extracts template source and returns it parsed (or nil,
+      # if template not found).
+      #
+      # **NB**: Infoboxer does NO variable substitution or other template
+      # evaluation actions. Moreover, it will almost certainly NOT parse
+      # template definitions correctly. You should use this method ONLY
+      # for "transclusion" templates (parts of content, which are
+      # included into other pages "as is").
+      #
+      # Look for example at [this page's](https://en.wikipedia.org/wiki/Tropical_and_subtropical_coniferous_forests)
+      # [source](https://en.wikipedia.org/w/index.php?title=Tropical_and_subtropical_coniferous_forests&action=edit):
+      # each subtable about some region is just a transclusion of
+      # template. This can be processed like:
+      #
+      # ```ruby
+      # Infoboxer.wp.get('Tropical and subtropical coniferous forests').
+      #   templates(name: /forests^/).
+      #   follow.tables #.and_so_on
+      # ```
+      #
+      # @return {MediaWiki::Page}
+      #
+      # **See also** {Linkable#follow} for general notes on the following links.
+
+      # Wikilink name of this template's source.
+      def link
+        # FIXME: super-naive for now, doesn't thinks about subpages and stuff.
+        "Template:#{name}"
+      end
+
+      # @private
       # Internal, used by {Parser}.
       def empty?
         false

data/lib/infoboxer/tree/text.rb

@@ -29,11 +29,13 @@ module Infoboxer
         "#{indent(level)}#{text} <#{descr}>\n"
       end
 
+      # @private
       # Internal, used by {Parser}
       def can_merge?(other)
         other.is_a?(String) || other.is_a?(Text)
       end
 
+      # @private
       # Internal, used by {Parser}
       def merge!(other)
         if other.is_a?(String)
@@ -45,6 +47,7 @@ module Infoboxer
         end
       end
 
+      # @private
       # Internal, used by {Parser}
       def empty?
         raw_text.empty?

data/lib/infoboxer/tree/wikilink.rb

@@ -1,4 +1,6 @@
 # encoding: utf-8
+require_relative 'linkable'
+
 module Infoboxer
   module Tree
     # Internal MediaWiki link class.
@@ -6,6 +8,8 @@ module Infoboxer
     # See [Wikipedia docs](https://en.wikipedia.org/wiki/Help:Link#Wikilinks)
     # for extensive explanation of Wikilink concept.
     #
+    # Note, that Wikilink is {Linkable}, so you can {Linkable#follow #follow}
+    # it to obtain linked pages.
     class Wikilink < Link
       def initialize(*)
         super
@@ -37,20 +41,7 @@ module Infoboxer
       # See {#topic} for explanation.
       attr_reader :refinement
 
-      # Extracts wiki page by this link and returns it parsed (or nil,
-      # if page not found).
-      #
-      # @return {MediaWiki::Page}
-      #
-      # **See also**:
-      # * {Tree::Nodes#follow} for extracting multiple links at once;
-      # * {MediaWiki#get} for basic information on page extraction.
-      def follow
-        page = lookup_parents(MediaWiki::Page).first or
-          fail("Not in a page from real source")
-        page.client or fail("MediaWiki client not set")
-        page.client.get(link)
-      end
+      include Linkable
 
       private
 

data/lib/infoboxer/version.rb

@@ -1,4 +1,4 @@
 # encoding: utf-8
 module Infoboxer
-  VERSION = '0.1.0'
+  VERSION = '0.1.1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: infoboxer
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Victor Shepelev
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-08-07 00:00:00.000000000 Z
+date: 2015-08-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: htmlentities
@@ -178,6 +178,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: inch
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: |2
       Infoboxer is library targeting use of Wikipedia (or any other
       MediaWiki-based wiki) as a rich powerful data source.
@@ -188,6 +202,8 @@ extra_rdoc_files: []
 files:
 - ".dokaz"
 - ".yardopts"
+- CHANGELOG.md
+- CONTRIBUTING.md
 - LICENSE.txt
 - Parsing.md
 - README.md
@@ -225,6 +241,7 @@ files:
 - lib/infoboxer/tree/html.rb
 - lib/infoboxer/tree/image.rb
 - lib/infoboxer/tree/inline.rb
+- lib/infoboxer/tree/linkable.rb
 - lib/infoboxer/tree/list.rb
 - lib/infoboxer/tree/node.rb
 - lib/infoboxer/tree/nodes.rb
@@ -245,7 +262,7 @@ files:
 - regression/pages/south_america.wiki
 - regression/pages/ukraine.wiki
 - regression/pages/usa.wiki
-homepage: https://github.com/zverok/infoboxer
+homepage: https://github.com/molybdenum-99/infoboxer
 licenses:
 - MIT
 metadata: {}