whatis 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 0bc1cdd23a45a493b8ef361682d38309bcc8758f
+  data.tar.gz: 9d6b933c9d46756ed745b569cf6c40db265befd5
+SHA512:
+  metadata.gz: b64b7110f7d840a12e350523712655d138848efcef71932fae5bfc0e5e54b7ee37c4f14fced08a32cfb7caafd42a78206ed9ab5d02dd1822e3d5bffa51229dea
+  data.tar.gz: 9d9e5e4bf040acd0b4d0f58b1d2d443f6f6020b7fd19fc68ba0a99519e0906493774ccb75ff7a6394d4a9915f3c1c63c0d6021a1cd26f2224d51b34ee9d4a10f

data/LICENSE.txt

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2014-15 Victor 'Zverok' Shepelev
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,149 @@
+# WhatIs.this
+
+[![Gem Version](https://badge.fury.io/rb/whatis.svg)](http://badge.fury.io/rb/whatis)
+[![Build Status](https://travis-ci.org/molybdenum-99/whatis.svg?branch=master)](https://travis-ci.org/molybdenum-99/whatis)
+
+**WhatIs.this** is a quick probe for the meaning and metadata of concepts through Wikipedia.
+
+## Showcase
+
+```ruby
+require 'whatis'
+
+sparta = WhatIs.this('Sparta')
+# => #<ThisIs Sparta [img] {37.081944,22.423611}>
+sparta.coordinates
+# => #<Geo::Coord 37.081944,22.423611>
+sparta.image
+# => "https://upload.wikimedia.org/wikipedia/commons/6/6c/Sparta_territory.jpg"
+
+sparta.describe
+# => Sparta
+#            title: "Sparta"
+#      description: "city-state in ancient Greece"
+#      coordinates: #<Geo::Coord 37.081944,22.423611>
+#          extract: "Sparta (Doric Greek: ; Attic Greek: ) was a prominent city-state in ancient Greece."
+#            image: "https://upload.wikimedia.org/wikipedia/commons/6/6c/Sparta_territory.jpg"
+
+# Fetch additional information: categories & translations:
+sparta = WhatIs.this('Sparta', categories: true, languages: 'el')
+# => #<ThisIs Sparta/Αρχαία Σπάρτη, 7 categories [img] {37.081944,22.423611}>
+sparta.describe
+# => Sparta
+#            title: "Sparta"
+#      description: "city-state in ancient Greece"
+#      coordinates: #<Geo::Coord 37.081944,22.423611>
+#       categories: ["Former countries in Europe", "Former populated places in Greece", "Locations in Greek mythology", "Populated places in Laconia", "Sparta", "States and territories disestablished in the 2nd century BC", "States and territories established in the 11th century BC"]
+#        languages: {"el"=>#<ThisIs::Link el:Αρχαία Σπάρτη>}
+#          extract: "Sparta (Doric Greek: ; Attic Greek: ) was a prominent city-state in ancient Greece."
+#            image: "https://upload.wikimedia.org/wikipedia/commons/6/6c/Sparta_territory.jpg"
+
+sparta.languages['el'].resolve
+# => #<ThisIs Αρχαία Σπάρτη [img]>
+
+# Multiple entities at once:
+WhatIs.these('Paris', 'Berlin', 'Rome', 'Athens')
+# => {
+#   "Paris"=>#<ThisIs Paris [img] {48.856700,2.350800}>,
+#   "Berlin"=>#<ThisIs Berlin [img] {52.516667,13.388889}>,
+#   "Rome"=>#<ThisIs Rome [img] {41.900000,12.500000}>,
+#   "Athens"=>#<ThisIs Athens [img] {37.983972,23.727806}>
+# }
+```
+## Applications
+
+The gem is intended to be a simple tool for entities resolution/normalization. Possible usages:
+
+* You have a lot of user-entered answers to "What city are you from". Through `WhatIs.these` it is
+  pretty easy to resolve them to "canonical" city name (e.g. "Warsaw", "Warszawa", "Warsaw, Poland" =>
+  "Warsaw") and map locations;
+* Quick check on user-entered cultural objects, "what is it";
+* Canonical Wikipedia-powered translations of toponyms, movie titles and historical people;
+* ...and so-on.
+
+## Features/problems
+
+* Fetches Wikipedia data by entity names: canonical title, geographical coordinates, main page image,
+  the first phrase, short entity description from Wikidata;
+* Optionally fetches links to other Wikipedia languages and list of page categories;
+* Fetches any number of Wikipedia pages in minimal number of API requests (50-page batches);
+  * Note that despite this optimization, Wikipedia API responses are not very small, so resolving,
+    say, 1000 entities, will errrm _take some time_;
+* Works with any language version of Wikipedia:
+```ruby
+WhatIs[:de].this('München')
+# => #<ThisIs München [img] {48.137222,11.575556}>
+```
+* Handles not found pages and allows to search them in place:
+```ruby
+g = WhatIs.this('Guardians Of The Galaxy') # Wikipedia pages is case-sensitive
+# => #<ThisIs::NotFound Guardians Of The Galaxy>
+g.search(3)
+# => [#<ThisIs::Ambigous Guardians of the Galaxy (11 options)>, #<ThisIs Guardians of the Galaxy (film)>, #<ThisIs Guardians of the Galaxy Vol. 2>]
+```
+* Handles disambiguation pages:
+```ruby
+g = WhatIs.this('Guardians of the Galaxy')
+# => #<ThisIs::Ambigous Guardians of the Galaxy (11 options)>
+g.describe
+# => Guardians of the Galaxy: ambigous (11 options)
+#      #<ThisIs::Link Marvel Comics teams/Guardians of the Galaxy (1969 team)>: Guardians of the Galaxy (1969 team), the original 31st-century team from an alternative timeline of the Marvel Universe (Earth-691)
+#      #<ThisIs::Link Marvel Comics teams/Guardians of the Galaxy (2008 team)>: Guardians of the Galaxy (2008 team), the modern version of the team formed in the aftermath of Annihilation: Conquest
+#    <...skip...>
+#      Usage: .variants[0].resolve, .resolve_all
+g.variants[1].resolve(categories: true)
+# => #<ThisIs Guardians of the Galaxy (2008 team), 13 categories>
+```
+* Provides command-line tool:
+```
+$ whatis Paris Berlin Rome
+Paris: Paris {48.856700,2.350800} - capital city of France
+Berlin: Berlin {52.516667,13.388889} - capital city of Germany
+Rome: Rome {41.900000,12.500000} - capital city of Italy
+
+$ whatis --help
+Usage: `whatis [options] title1, title2, title3
+
+Options:
+    -l, --language CODE              Which language Wikipedia to ask, 2-letter code. "en" by default
+    -t, --languages [CODE]           Without argument, fetches all translations for entity.
+                                     With argument (two-letter code) fetches only one translation.
+                                     By default, no translations are fetched.
+        --categories                 Whether to fetch entity categories
+    -f, --format FORMAT              Output format: one line per entity ("short"), several lines per
+                                     entity ("long"), or "json". Default is "short".
+    -h, --help                       Show this message
+```
+
+### Note on disambiguation pages
+
+Unfortunately, Wikipedia does not provide a consistent way to tell disambiguation pages from others,
+the only way is to know is to see the page's categories (different for different languages). Therefore,
+currently, disambiguation works currently for English, Ukrainian, Russian and Belorussian. Feel free
+to contribute disambiguation categories for your language versions!
+
+## Usage
+
+`gem install whatis` or add `gem "whatis"` to your `Gemfile`.
+
+Then use it as library (see docs for [WhatIs](www.rubydoc.info/gems/whatis/WhatIs) and its methods)
+or command-line tool (try `$ whatis --help`).
+
+## How it works
+
+`WhatIs.this` is a small brother of large [reality](https://github.com/molybdenum-99/reality). Under
+the hood, it uses [infoboxer](https://github.com/molybdenum-99/infoboxer) semantic Wikipedia client.
+
+Most of the information is taken from API response metadata, but for some features (ambiguities
+resolution), Wikipedia page is actually parsed.
+
+Unlike `reality` (which tries to be _comprehensive_), `WhatIs.this` tries to be as simple yet useful,
+as possible.
+
+## Author
+
+[Victor Shepelev](http://zverok.github.io)
+
+## License
+
+MIT

data/exe/whatis

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+require_relative '../lib/whatis'
+require_relative '../lib/whatis/cli'
+require 'optparse'
+require 'ostruct'
+
+options = OpenStruct.new
+options.language = 'en'
+options.format = 'short'
+
+parser = OptionParser.new do |opts|
+  opts.banner = 'Usage: `whatis [options] title1, title2, title3'
+
+  opts.separator ''
+  opts.separator 'Options:'
+
+  opts.on('-l', '--language CODE', 'Which language Wikipedia to ask, 2-letter code. "en" by default') do |lang|
+    options.language = lang
+  end
+
+  opts.on('-t', '--languages [CODE]', 'Without argument, fetches all translations for entity. With argument (two-letter code) fetches only one translation. By default, no translations are fetched.') do |langs|
+    options.languages = langs || true
+  end
+
+  opts.on('--categories', 'Whether to fetch entity categories') do
+    options.categories = true
+  end
+
+  opts.on('-f', '--format FORMAT', %w[short long json], 'Output format: one line per entity ("short"), several lines per entity ("long"), "json". Default is "short"') do |format|
+    options.format = format
+  end
+
+  opts.on_tail('-h', '--help', 'Show this message') do
+    puts opts
+    exit
+  end
+end
+
+parser.parse!(ARGV)
+
+if ARGV.empty?
+  puts parser
+  exit
+end
+
+puts WhatIs::CLI.new(ARGV, options).run

data/lib/whatis.rb

@@ -0,0 +1,209 @@
+require 'infoboxer'
+require 'geo/coord'
+require 'backports/2.4.0/hash/transform_values'
+
+# `WhatIs` is a simple entity resolver through Wikipedia.
+#
+# @example
+#   # Simplest usage
+#   WhatIs.this('Sparta') # => #<ThisIs Sparta [img] {37.081944,22.423611}>
+#
+#   # Additional options
+#   WhatIs.this('Sparta', languages: :el, categories: true)
+#   # => #<ThisIs Sparta/Αρχαία Σπάρτη, 7 categories [img] {37.081944,22.423611}>
+#
+#   # Several pages at once (in batch requests to Wikipedia API)
+#   WhatIs.these('Paris', 'Athens', 'Rome')
+#   # => {"Paris"=>#<ThisIs Paris [img] {48.856700,2.350800}>, "Athens"=>#<ThisIs Athens [img] {37.983972,23.727806}>, "Rome"=>#<ThisIs Rome [img] {41.900000,12.500000}>}
+#
+#   # Other language Wikipedia
+#    WhatIs[:ru].this('Спарта') # => #<ThisIs Спарта [img]>
+#
+# See {#this} and {#these} methods docs for details on call sequence and options, and response classes:
+#
+# * {ThisIs} -- normal response object;
+# * {ThisIs::Ambigous} -- response object representing disambiguation page;
+# * {ThisIs::NotFound} -- response object for not found page, includes search for term service.
+#
+class WhatIs
+  # This constant lists Wikipedia ambiguity categories per Wikipedia language. For {ThisIs::Ambigous}
+  # feature to work for your language, this list should include it.
+  AMBIGOUS_CATEGORIES = {
+    be: ['Катэгорыя:Неадназначнасці'],
+    en: ['Category:All disambiguation pages', 'Category:All set index articles'],
+    ru: ['Категория:Страницы значений по алфавиту'],
+    uk: ['Категорія:Всі статті визначеного індексу', 'Категорія:Всі сторінки неоднозначності статей']
+  }.freeze
+
+  # String-like class, with the only difference for how its #inspect is represented.
+  #
+  # Used for {ThisIs#describe} method for its answer to be readable in Ruby console (IRB or Pry)
+  #
+  # @example
+  #   "foo\nbar"
+  #   # => "foo\nbar"
+  #   Description.new("foo\nbar")
+  #   # => foo
+  #   # bar
+  #
+  class Description < String
+    alias inspect to_s # Allows pretty inspect of multi-line descriptions
+  end
+
+  class << self
+    # @param lang [Symbol, String] Wikipedia version language code, usually two-letter ("en", "fr"),
+    #   but not for all languages (for example, "be-x-old" or "zh-classical").
+    #
+    # @return [WhatIs]
+    def [](lang)
+      all[lang.to_s]
+    end
+
+    # Shortcut for `WhatIs[:en].these`, see {#these} for details.
+    # @param titles [Array<String>] Titles of entities to resolve.
+    # @option options [true, String, Symbol] :languages If `true`, fetches all titles of languages versions
+    #   of entity resolved; if some language code (like "ru" or "zh-classical") is passed, fetches only
+    #   this language's title, the latter is faster.
+    # @option options [true, false] :categories Fetch entity's categories. Due to how Wikipedia's API
+    #   work, this may lead to additional API calls, so use with caution.
+    # @return [Hash{String => ThisIs, ThisIs::Ambigous, ThisIs::NotFound}] Hash keys are original
+    #   title requested, so it is easy to find how particular title was resolved.
+    def these(*titles, **options)
+      self[:en].these(*titles, **options)
+    end
+
+    # Shortcut for `WhatIs[:en].this`, see {#this} for details.
+    # @param title [String] Title of the entity to resolve.
+    # @param options [Hash]
+    # @option options [true, String, Symbol] :languages If `true`, fetches all titles of languages versions
+    #   of entity resolved; if some language code (like "ru" or "zh-classical") is passed, fetches only
+    #   this language's title, the latter is faster.
+    # @option options [true, false] :categories Fetch entity's categories. Due to how Wikipedia's API
+    #   work, this may lead to additional API calls, so use with caution.
+    # @return [ThisIs, ThisIs::Ambigous, ThisIs::NotFound]
+    def this(title, **options)
+      self[:en].this(title, **options)
+    end
+
+    private
+
+    def all
+      @all ||= Hash.new { |h, lang| h[lang] = WhatIs.new(lang) }
+    end
+  end
+
+  # @private
+  attr_reader :language
+
+  # @private
+  def initialize(language = :en)
+    @language = language
+    @infoboxer = Infoboxer.wikipedia(language)
+  end
+
+  # Batch resolving of several entities. Wikipedia API allows batch fetching of pages, so there would
+  # be roughly 1 API call for each 50 entities. Number of API calls could be larger if additional
+  # information (languages, categories) is requested.
+  #
+  # @param titles [Array<String>] Titles of entities to resolve.
+  # @option options [true, String, Symbol] :languages If `true`, fetches all titles of languages versions
+  #   of entity resolved; if some language code (like "ru" or "zh-classical") is passed, fetches only
+  #   this language's title, the latter is faster.
+  # @option options [true, false] :categories Fetch entity's categories. Due to how Wikipedia's API
+  #   work, this may lead to additional API calls, so use with caution.
+  # @return [Hash{String => ThisIs, ThisIs::Ambigous, ThisIs::NotFound}] Hash keys are original
+  #   title requested, so it is easy to find how particular title was resolved.
+  # @see #this #this: singular entity resolution.
+  # @example
+  #   WhatIs.these('Warszawa', 'Warsaw', 'Berlin', 'Kharkov', languages: :fr)
+  #   # => {
+  #   #  "Warszawa" => #<ThisIs Warsaw/Varsovie [img] {52.233333,21.016667}>,
+  #   #  "Warsaw" => #<ThisIs Warsaw/Varsovie [img] {52.233333,21.016667}>,
+  #   #  "Berlin" => #<ThisIs Berlin/Berlin [img] {52.516667,13.388889}>,
+  #   #  "Kharkov" => #<ThisIs Kharkiv/Kharkiv [img] {50.004444,36.231389}>,
+  #   #  "Bela Crkva" => #<ThisIs::Ambigous Bela Crkva (6 options)>,
+  #   #  "Hrmpf" => #<ThisIs::NotFound Hrmpf>
+  #   # }
+  def these(*titles, **options)
+    titles.any? or
+      fail(ArgumentError, "Usage: `these('Title 1', 'Title 2', ..., **options). At least one title is required.")
+    @infoboxer
+      .get_h(*titles) { |req| setup_request(req, **options) }
+      .map { |title, page| [title, ThisIs.create(self, title, page)] }.to_h
+  end
+
+  # Resolves one entity through Wikipedia API.
+  #
+  # @param title [String] Title of the entity to resolve.
+  # @param options [Hash]
+  # @option options [true, String, Symbol] :languages If `true`, fetches all titles of languages versions
+  #   of entity resolved; if some language code (like "ru" or "zh-classical") is passed, fetches only
+  #   this language's title, the latter is faster.
+  # @option options [true, false] :categories Fetch entity's categories. Due to how Wikipedia's API
+  #   work, this may lead to additional API calls, so use with caution.
+  # @return [ThisIs, ThisIs::Ambigous, ThisIs::NotFound]
+  # @note Special {ThisIs::Ambigous} wrapper for disambiguation pages can be created only for those
+  #   language Wikipedias which `WhatIs` knows "disambiguation" category name, see {AMBIGOUS_CATEGORIES}.
+  # @see #these #these: batch fetching several entities.
+  # @example
+  #   WhatIs.this('Warszawa', languages: :fr)
+  #   # => #<ThisIs Warsaw/Varsovie [img] {52.233333,21.016667}>
+  #   WhatIs.this('Bela Crkva', languages: :fr)
+  #   # => #<ThisIs::Ambigous Bela Crkva (6 options)>
+  #   WhatIs.this('Hrmpf', languages: :fr)
+  #   # => #<ThisIs::NotFound Hrmpf>
+  def this(title, **options)
+    these(title, **options).values.first
+  end
+
+  # @return [String]
+  def inspect
+    "#<WhatIs(#{language}). Usage: .this(*pages, **options)>"
+  end
+
+  # @private
+  # Used by {ThisIs::NotFound#search}
+  def search(title, limit = 5, **options)
+    @infoboxer
+      .search(title, limit: limit) { |req| setup_request(req, **options) }
+      .map { |page| ThisIs.create(self, page.title, page) }
+  end
+
+  # @private
+  def ambigous_categories
+    @ambigous_categories = AMBIGOUS_CATEGORIES.fetch(language.to_sym, [])
+  end
+
+  private
+
+  def setup_request(request, categories: false, languages: false, **) # rubocop:disable Metrics/MethodLength
+    request = request
+      .prop(:coordinates)
+      .prop(:extracts).sentences(1)
+      .prop(:pageimages).prop(:original)
+      .prop(:pageterms)
+
+    request = setup_categories(request, categories)
+    if languages
+      request = request.prop(:langlinks)
+      request = request.lang(languages) unless languages == true
+    end
+    request
+  end
+
+  def setup_categories(request, categories_requested)
+    if categories_requested
+      # Fetch all categories, include "hidden" flag to filter out internal
+      request.prop(:categories).prop(:hidden)
+    elsif !ambigous_categories.empty?
+      # Fetch only "ambigous" categories, to tell ambigous pages out
+      request.prop(:categories).categories(*ambigous_categories)
+    else
+      request
+    end
+  end
+end
+
+require_relative 'whatis/refinements'
+require_relative 'whatis/thisis'
+require_relative 'whatis/formatter'

data/lib/whatis/cli.rb

@@ -0,0 +1,38 @@
+class WhatIs
+  # @private
+  class CLI
+    def initialize(titles, options)
+      @whatis = WhatIs[options.language]
+      @titles = titles
+      @options = {categories: options.categories, languages: options.languages}
+      @format = options.format
+    end
+
+    def run
+      __send__("#{@format}_format", @whatis.these(*@titles, **@options))
+    end
+
+    private
+
+    def short_format(objects)
+      formatter = Formatter.new
+      objects.map { |title, o| formatter.call(title, o) }.join("\n")
+    end
+
+    def long_format(objects)
+      objects.flat_map { |title, o|
+        [
+          '',
+          title,
+          '-' * title.length,
+          o.describe(help: false)
+        ]
+      }.join("\n")
+    end
+
+    def json_format(objects)
+      require 'json'
+      JSON.pretty_generate(objects)
+    end
+  end
+end