mediawiktory 0.0.3 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 653255269bbd0a100edd618e44a25050ccb9b0cc
-  data.tar.gz: 072d290adc899957ec46bdf787b30eee5cd6ed32
+  metadata.gz: f4acc4a7df0501374b8379ed81b1bea96e8fb10f
+  data.tar.gz: 5be2d52f4efadfe7cb8e6fb1158ed742a67cb80e
 SHA512:
-  metadata.gz: 3f681541924daaac250a2ba34775fb4fcbf1312217d31199545d4bfcaed7cf2f428a0961728136153d0793d13c1ec9efbf37dfbc9e6bf3670151e88c8c7a5440
-  data.tar.gz: bcce8fcff0eb4d3f18839b38d3fd00e6f1dee1bc421ca4b061319c5d05caed70b7e3dc7d391abf53905a76288d8b97817e28c1e7cecc402ab8ade3ecc1f97de0
+  metadata.gz: 2214253635d370f907cc5a137e5f0e6136125a2cee3798aff806579127ec25e3f930ccb8428e0f3c9acd33a000cdb16c96ed6503eae7c4621bfa6c1ab24f29dc
+  data.tar.gz: c3bcdbf656c22b2b63ea3587fbe3814f93837c1fcb8cbd3fa251c30358b95cf98f79046aa2af61bd9b002099f833fe754e9d47434150fcf9f44921aa512ab275

data/.dokaz

@@ -0,0 +1 @@
+--require ./spec/dokaz_helpers.rb

data/.rubocop_todo.yml

@@ -0,0 +1,16 @@
+# This configuration was generated by
+# `rubocop --auto-gen-config`
+# on 2017-05-06 16:11:16 +0300 using RuboCop version 0.48.1.
+# The point is for the user to remove these configuration records
+# one by one as the offenses are removed from the code base.
+# Note that changes in the inspected code, or installation of new
+# versions of RuboCop, may require this file to be generated again.
+
+# Offense count: 5
+# Cop supports --auto-correct.
+# Configuration parameters: EnforcedStyle, SupportedStyles, IndentationWidth.
+# SupportedStyles: aligned, indented, indented_relative_to_receiver
+#Style/MultilineMethodCallIndentation:
+  #Exclude:
+    #- 'lib/mediawiktory/generator/api.rb'
+    #- 'lib/mediawiktory/generator/renderable.rb'

data/.yardopts

@@ -0,0 +1,2 @@
+--markup=markdown
+--no-private

data/CHANGELOG.md

@@ -1,11 +1,17 @@
 # MediaWiktory Changelog
 
-# 0.0.3 (18.03.2017)
+# 0.1.0 (2017-05-10)
+
+* First mature release;
+* Change of API wrapper approach and allow to generate several different API wrappers;
+* Specs & docs.
+
+# 0.0.3 (2017-03-18)
 
 * Regenerated Wikipedia wrapper in correspondence to current API version;
 * Preparing to independent wrappers concept.
 
-# 0.0.2 (21.11.2015)
+# 0.0.2 (2015-12-21)
 
 * `Page` class for grouping records in `Query::Response`;
 * `User-Agent` header support;
@@ -13,10 +19,10 @@
 * Correct behavior on empty pagelist when using generator (@smostovoy);
 * Raise on errors in response (@smostovoy);
 * Access to warnings in response (@smostovoy);
-* Domain redirects transparent handling, 
+* Domain redirects transparent handling,
   http://en.wikipedia.org -> https://en.wikipedia.org (@smostovoy).
 
-# 0.0.1 (30.11.2015)
+# 0.0.1 (2015-11-30)
 
 Initial!
 

data/CONTRIBUTING.md

@@ -1,16 +1,13 @@
 # Contributing to MediaWiktory
 
-Currently, MediaWiktory at early stages of development (aka "young yet
-useful"). Currenty problems and roadmap can be found at
-[GitHub Issues](https://github.com/molybdenum-99/mediawiktory/issues) and
+Currently, MediaWiktory at early stages of development (aka "young yet useful"). Currenty problems
+and roadmap can be found at [GitHub Issues](https://github.com/molybdenum-99/mediawiktory/issues) and
 are grouped into milestones.
 
 Therefore, the big picture of "what next" is:
-* more documentation for generated MediaWiki wrappers (both YARD and
-  on-the-fly accessible from code);
 * more validations and cleaner errors;
-* simplification of frequent and common tasks.
+* gfaceful handling of logging in, cookies and tokens;
+* support of "old" docs format (think wikia.com).
 
-You can join MediaWiktory development by usual fork-commit-pull request
-process, or by providing your own cases and problems you'd like to solve
-with it.
+You can join MediaWiktory development by usual fork-commit-pull request process, or by providing
+your own cases and problems you'd like to solve with it.

data/README.md

@@ -1,102 +1,183 @@
 # MediaWiktory, The MediaWiki Client
 
-**MediaWiktory** is MediaWiki API client that doesn't suck. No, seriously.
+[![Gem Version](https://badge.fury.io/rb/mediawiktory.svg)](http://badge.fury.io/rb/mediawiktory)
+[![Build Status](https://travis-ci.org/molybdenum-99/mediawiktory.svg?branch=master)](https://travis-ci.org/molybdenum-99/mediawiktory)
 
-[MediaWiki API](https://www.mediawiki.org/wiki/API:Main_page)
-currently is very powerful and full-featured (thought not very easy to use).
-Things like "fetch first 50 pages from that category alongside with their
-revision history and interwiki links and mediafile stats" are typically done
+**MediaWiktory** is a MediaWiki (think Wikipedia, Wiktionary and others) API client. It is the only
+client that allows (almost) full access to MediaWiki API powers without loosing of Ruby powers.
+
+No, seriously.
+
+[MediaWiki API](https://www.mediawiki.org/wiki/API:Main_page) currently is very powerful and
+full-featured (thought not very easy to use). Things like "fetch first 50 pages from that category
+alongside with their revision history and interwiki links and mediafile stats" are typically done
 with one carefully constructed request and return lots of useful information.
 
 Yes, there already are [several](https://www.mediawiki.org/wiki/API:Client_code#Ruby)
 API clients for Ruby, including ["official" one](https://github.com/wikimedia/mediawiki-ruby-api).
-Typical approach for all of them is thick wrapper around some functionality
-(like "login and edit pages" or "search and analyze pages"), and leave
-all the other cool things for generic `query` method (at best), or
-without any coverage at all.
-
-MediaWiktory, on the contrary, does this:
-* wraps all the functionality of all the modules of modern MediaWiki,
-  as per [this documentation](https://en.wikipedia.org/w/api.php?action=help&modules=main);
-  * and, for the record, provides sparse documentation for all modules
-    and possibilities;
-* makes the wrapper reasonably thin, so you can always inspect raw responses
-  and raw request MediaWiktory will provide to the API;
-* makes the wrapper reasonably smart, so things like "fetch next page for
-  this request" or "login with a token" are done with ease (but without
-  unnecessary "auto-magic", like fetching 500 pages of data without any
-  chance to control this);
-* validates requests and simplifies parameters passing, while still
-  preserving abilitity to use any existing parameter and feature;
-* it manages cookies and tokens for you, but can expose underlying
-  Faraday web client so the middleware can be added (like caching).
-
-**Warning: Above is IDEAL state of the library. Current state, at
-version 0.0.1 is far less impressive (yet already useful).**
-
-## Structure and usage
-
-MediaWiktory tries to resemble original API structure, but make its calls
-more Ruby-ish. So, [Arel](https://github.com/rails/arel)-alike "chainable
-query" approach is implemented.
-
-Fast example:
-
-**Goal**: fetch content for first 30 pages from category "Countries in
-South America".
-
-**Raw request** (split in lines for readability):
+Typical approach for all of them is thick wrapper around some functionality (like "login and edit
+pages" or "search and analyze pages"), and leave all the other cool things for generic `action` method
+(at best), or without any coverage at all.
+
+MediaWiktory, to the contrary is:
+
+* **thin** wrapper...
+* around **all** MediaWiki API features...
+* making access to them available through idiomatic Ruby code, easy to use and clearly documented.
 
+## Examples
+
+**Example 1.** Fetching page's text and metadata:
+
+```ruby
+api = MediaWiktory::Wikipedia::Api.new
+response = api.query.       # "query" action is a basis for all pages/categories/meta receiving
+  titles('Argentina').      # query page titles: Argentina
+  prop(:info, :revisions).  # query page properties: info, revisions
+  prop(:url, :content).     # query those properties subproperties: full URL (from info) and content (from revisions)
+  response                  # perform query and parse it!
+
+page = response['pages'].values.first
+puts page['title']
+# Prints:
+#  Argentina
+puts page['fullurl']
+# Prints:
+#  https://en.wikipedia.org/wiki/Argentina
+puts page['revisions'].first['*'].slice(0..200) # first 200 chars of page contents
+# Prints:
+#  {{other uses}}
+#  {{pp-semi|small=yes}}
+#  {{Use dmy dates|date=March 2017}}
+#  {{Coord|34|S|64|W|display=title}}
+#  {{Infobox country
+#  |coordinates = {{Coord|34|36|S|58|23|W|type:city}}
+#  |conventional_long_name = A
 ```
-https://en.wikipedia.org/w/api.php?
-  action=query&
-  generator=categorymembers&
-  gcmtitle=Category:Countries_in_South_America&
-  gcmlimit=30&
-  prop=revisions&
-  rvprop=content
+
+Note, that for using MediaWiktory API wrapper you need to understand the underlying API. While previous
+experience might make you expect something like `api.page('Argentina').text`, in fact you should
+use the `query` action, request page title 'Argentina', its `:revisions` property, its `:content`
+subproperty—and voila, you have a _1-element list of revisions_ for the page and last revisions `'*'`
+key has page's text.
+
+The good news is all methods are documented at [RubyDoc.info](http://www.rubydoc.info/gems/mediawiktory).
+Most of the time, the documentation has enough details, so you don't need to refer to MediaWiki
+official docs.
+
+**Example 2:** Editing the page (we are editing [Sandbox](https://en.wikipedia.org/wiki/Wikipedia:Sandbox)
+here, which is safe, but be careful while experimenting, this code **really** replaces page's text!):
+
+```ruby
+token = api.query.meta(:tokens).response.dig('tokens', 'csrftoken')
+response = api.edit.title('Wikipedia:Sandbox').text("Test '''me''', MediaWiktory!").token(token).response
+response.to_h
+# => {"result"=>"Success", "pageid"=>16283969, "title"=>"Wikipedia:Sandbox", "contentmodel"=>"wikitext", "oldrevid"=>779502714, "newrevid"=>779502729, "newtimestamp"=>"2017-05-09T08:24:26Z"}
+
+# This, without token, will raise:
+api.edit.title('Wikipedia:Sandbox').text("Test '''me''', MediaWiktory without token!").response
+# MediaWiktory::Wikipedia::Response::Error: The "token" parameter must be set.
 ```
 
-**MediaWiktory request**—almost the same, but structured and validated
-on-the-fly, and without `gcm` and `rv` prefixes black magic:
+**Example 3:** Fetching all "main" page images for the pages of category:
 
 ```ruby
-client = MediaWiktory::Client.new('https://en.wikipedia.org/w/api.php')
+response = api.query.                   # "query" action again
+  generator(:categorymembers).          # instead of listing titles, we use "page list generator": all members of a category
+  title('Category:1960s_automobiles').  # ...of this category
+  prop(:pageimages).prop(:thumbnail).   # and fetch "pageimages" property, its "thumbnail" sub-property
+  limit('max').                         # limit to maximum number of pages available in one response
+  response
+
+# You can fetch ALL of them with, it will be a lot:
+# response = response.continue while response.continue?
+
+response.to_h['pages'].values.each do |page|
+  puts "#{page['title']}: #{page.dig('thumbnail', 'source')}"
+end
+# AC Cobra: https://upload.wikimedia.org/wikipedia/commons/thumb/e/e8/Shelby_AC_427_Cobra_vl_blue.jpg/50px-Shelby_AC_427_Cobra_vl_blue.jpg
+# Acadian (automobile):
+# Alfa Romeo 33 Stradale: https://upload.wikimedia.org/wikipedia/commons/thumb/e/eb/1968_Alfa_Romeo_Tipo_33_Stradale.jpg/50px-1968_Alfa_Romeo_Tipo_33_Stradale.jpg
+# Alfa Romeo 105/115 Series Coupés: https://upload.wikimedia.org/wikipedia/commons/thumb/8/81/Alfa_Romeo_GT_1300_Junior.jpg/50px-Alfa_Romeo_GT_1300_Junior.jpg
+# Alfa Romeo 1750 Berlina: https://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Alfa_Romeo_1750_berlina_grey-front.JPG/50px-Alfa_Romeo_1750_berlina_grey-front.JPG
+# Alfa Romeo 2000: https://upload.wikimedia.org/wikipedia/commons/thumb/f/f6/Alfa_2000_touring_spider.JPG/50px-Alfa_2000_touring_spider.JPG
+# Alfa Romeo 2600: https://upload.wikimedia.org/wikipedia/commons/thumb/6/6b/Alfa-Romeo_2600-Spider-Touring.JPG/50px-Alfa-Romeo_2600-Spider-Touring.JPG
+# ...
+```
+
+## Usage
 
-response = client.
-  query.
-  generator(categorymembers: {title: 'Category:Countries_in_South_America', limit: 30}).
-  prop(revisions: {prop: :content}).
-  perform
+```
+gem install mediawiktory
+```
+
+There are a lot of popular installations of MediaWiki besides Wikipedia. All of them are having
+different versions installed with different features enabled and custom extensions turned on.
+
+To catch with this multitude of features, MediaWiktory provides two ways of usage.
 
-# MediaWiktory handles "next page fetching" for you, if you want
-response.continue! while response.can_continue?
+### 1. Use default wrapper, generated from English Wikipedia:
 
-# MediaWiktory parses response and provides smart shortcuts
-p response.pages.map(&:title)
+```ruby
+require 'mediawiktory'
+api = MediaWiktory::Wikipedia::Api.new # => English Wikipedia
+# or
+api = MediaWiktory::Wikipedia::Api.new('http://some.site/w/api.php') # => any other MediaWiki
 ```
 
-The same approach works for ANY `action` described in [docs](https://en.wikipedia.org/w/api.php),
-and all of its "modules" and "submodules".
+...and wonder through docs of [MediaWiktory::Wikipedia::Api](http://www.rubydoc.info/gems/mediawiktory/MediaWiktory/Wikipedia/Api)
+class to understand what you can do.
+
+### 2. Custom wrapper generation.
+```
+mediawiktory-gen -u http://some.site/w/api.php --path lib/path/to/wrapper --namespace My::Wrapper
+```
+This will generate `My::Wrapper::Api` class and a lot of other classes wrapping all actions and
+modules of target APIs. The generated code is **independent** of MediaWiktory (so you can exclude it
+from your runtime), and depends only on `addressable`, `faraday` and `faraday_middleware` gems.
 
-**NB**: its 0.0.1, ok? Domain is complex and is still investigated. For
-example, above statement definitely **wants** to look like
+The usage of custom wrapper is basically the same:
 
 ```ruby
-response = client.
-  query.
-  generator(categorymembers: 'Category:Countries_in_South_America').
-  limit(30).
-  prop(revisions: :content).
-  perform
+require 'path/to/wrapper/api'
+api = My::Wrapper::Api.new
+api.query # .and.so.on
 ```
 
-...but it should be done without flexibility loss.
+You need custom wrapper if:
+
+* you want to have the exact list of features your site has: for example, with Wikia sites, most of
+  generic functionality (like query and edit) will work, but most of fancy modern Wikipedia actions
+  will fail with "unknown action";
+* your target site has some custom actions and modules: for example, most informative Wikidata actions
+  are custom ones, like [wbgetentities](https://www.wikidata.org/w/api.php?action=help&modules=wbgetentities),
+  they are not present in default wrapper;
+* you want to catch up with some edge Wikipedia features; Wikipedia wrapper is generated on gem
+  release, but Wikipedia's API changes everyday with new small and large exerimental features.
+
+**Generator limitations:** Wrapper is generated from [HTML docs of API](en.wikipedia.org/w/api.php),
+but currently generator can't process old MediaWiki versions ASCII docs format, which, unfortunately,
+is stil in use on [Wikia](https://marvel.wikia.com/api.php), for example. It is subject to further
+development, as some "old" installations of MediaWiki provide pretty useful content and a lot of
+custom modules.
+
+If you integrate wrapper generated by MediaWiktory into some other library, you should note that:
+
+* All generated code is documented in YARD format, Markdown markup flavour;
+* If you use Rubocop, you will find some "good code" practices broken in generated code, because it
+  is hard to follow them in large code generation.
+
+## Roadmap
+
+* Expose underlying Faraday client for fine-tuning;
+* Handle cookies automatically (for logging in);
+* Handle file uploads (should be done as multipart, use appropriate Faraday middleware);
+* Add parser for outdated ASCII docs.
 
 ## Authors
 
 * [Victor Shepelev](https://zverok.github.io) [@zverok](https://github.com/zverok);
-* Sergey Mostovoy [@smostovoy](https://github.com/smostovoy).
+* Serhiy Mostovyi [@smostovoy](https://github.com/smostovoy).
 
 ## License
 

data/bin/mediawiktory-gen

@@ -0,0 +1,35 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+
+options = {}
+OptionParser.new do |opts|
+  opts.banner =
+    "Usage: mediawiktory-gen [options]\n\n"\
+    "Generates MediaWiki API wrapper Ruby code for specified site in specified folder and namespace.\n"\
+    "Note that all previous content of the folder will be removed.\n\n"\
+
+  opts.on("-uURL", "--url=URL", "Source URL for option parsing. Typically it is http://some.domain/w/api.php.") do |u|
+    options[:url] = u
+  end
+
+  opts.on("-nNAMESPACE", "--name=NAMESPACE", "Namespace which should include all the newly-generated classes and modules.") do |n|
+    options[:namespace] = n
+  end
+
+  opts.on("-pPATH", "--path=PATH", "Path where to store new modules and classes. Note that there would be dozens of them, and that previous content of the folder will be removed.") do |p|
+    options[:path] = p
+  end
+
+  opts.on("-h", "--help", "Prints this help") do
+    puts opts
+    exit
+  end
+end.parse!
+
+$:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require_relative '../lib/mediawiktory/generator'
+
+MediaWiktory::Generator
+  .new(options[:url])
+  .generate(path: options[:path], namespace: options[:namespace])

data/lib/media_wiktory.rb

@@ -1 +1,3 @@
+# frozen_string_literal: true
+
 require_relative './mediawiktory'

data/lib/mediawiktory.rb

@@ -1,17 +1,16 @@
-require 'addressable/uri'
-require 'faraday'
-require 'faraday_middleware'
+# frozen_string_literal: true
 
+# MediaWiktory is wrapper and wrapper generator for [MediaWiki API](https://www.mediawiki.org/wiki/API:Main_page).
+#
+# There are two ways of using MediaWiktory:
+#
+# 1. Either to generate your own wrapper (see {file:README.md} for explanations why and how to do it), or
+# 2. By using default wrapper, generated from recent Wikipedia docs, see {MediaWiktory::Wikipedia::Api}.
+#    Note, that most of basic features (like "query some pages") of default wrapper could be used
+#    with **any** MediaWiki installation, but more advanced features could work differently. Please
+#    refer to {file:README.md} for more detailed explanation of those matters.
+#
 module MediaWiktory
-  %w[version client mw_module action].each do |mod|
-    require_relative "mediawiktory/#{mod}"
-  end
-
-  Dir[File.expand_path('../mediawiktory/generated/*.rb', __FILE__)].each do |gen|
-    require gen
-  end
-
-  %w[query].each do |mod|
-    require_relative "mediawiktory/#{mod}"
-  end
+  require_relative 'mediawiktory/version'
+  require_relative 'mediawiktory/wikipedia/api'
 end

data/lib/mediawiktory/generator.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+require 'nokogiri' # FIXME: catch if unavailable
+require 'nokogiri/more'
+require 'hashie'
+require 'addressable'
+require 'fileutils'
+require 'pp'
+require 'mediawiktory/version'
+
+require_relative 'generator/renderable'
+
+# @private
+class Object
+  # Just a loosy implementation of Ruby 2.3's `&.` operator
+  # It is used only in generator and will not pollute your scopes.
+  def _n
+    self
+  end
+end
+
+# @private
+class NilClass
+  class BlackHole
+    def method_missing(*) # rubocop:disable Style/MethodMissing
+      nil
+    end
+  end
+
+  def _n
+    BlackHole.new
+  end
+end
+
+module MediaWiktory
+  # @private
+  # Used only by mediawiktory-gen, should not be necessary to read :)
+  class Generator
+    class BaseAction
+      def initialize(api)
+        @api = api
+      end
+
+      include Renderable
+
+      def main_template
+        'action.rb'
+      end
+
+      def main
+        @api.main
+      end
+    end
+
+    class Client
+      include Renderable
+
+      def main_template
+        'client.rb'
+      end
+    end
+
+    class Response
+      include Renderable
+
+      def main_template
+        'response.rb'
+      end
+    end
+
+    def initialize(url)
+      @api = Api.from_url(url)
+      @url = url
+    end
+
+    def generate(path:, namespace:)
+      opts = {
+        namespace: namespace,
+        version: VERSION,
+        source: @url,
+        friendly_date: Time.now.strftime('%B %d, %Y')
+      }
+      FileUtils.rm_rf path
+      FileUtils.mkdir_p path
+
+      BaseAction.new(@api).render_to(File.join(path, 'actions/base.rb'), opts)
+      Client.new.render_to(File.join(path, 'client.rb'), opts)
+      Response.new.render_to(File.join(path, 'response.rb'), opts)
+
+      @api.render_to(File.join(path, 'api.rb'), opts)
+      @api.render_to(File.join(path, 'actions.rb'), opts.merge(template: 'actions.rb'))
+      @api.actions.each do |a|
+        a.render_to(File.join(path, "actions/#{a.name}.rb"), opts)
+      end
+      @api.non_actions.each do |a|
+        a.render_to(
+          File.join(path, "modules/#{a.name}.rb"),
+          opts.merge(template: 'module_class.rb')
+        )
+      end
+    end
+  end
+end
+
+require_relative 'generator/api'
+require_relative 'generator/param'
+require_relative 'generator/module'