elexis-wiki-interface 0.4

data/README.textile

@@ -0,0 +1,104 @@
+h1. Elexis::Wiki::Interface
+
+A Ruby Gem to allow pulling and pushing between Elexis source code repositories and the mediawiki installation
+
+Use it at your own risk, as it is just a small utility gem to help for some release engineering for the Elexis-RCP. See http://elexis.info.
+
+But I have an open ear for suggestions for improvements, especially if they come with a patch including specs.
+
+License: GPLv3 or later
+
+Copyright: 2014 (c) by Niklaus Giger <niklaus.giger@member.fsf.org>
+
+h2. Installation
+
+Ensure that you use Ruby 1.9.2 or higher (mediawiki-gateway does not support Ruby 1.8.7).
+
+Add this line to your application's Gemfile:
+
+bc. $ gem 'elexis-wiki-interface'
+
+And then execute:
+
+bc. $ bundle
+
+Or install it yourself as:
+
+bc. $ gem install elexis-wiki-interface
+
+h2. Usage
+
+h3. using some built-in helpers (mainly for the jenkins-CI)
+
+Each of the helper script below should work if you have a ruby installation >= 1.9.2 and executed @gem install bundler && bundle install@ in directory where
+
+bc. wiki_pull
+  wiki_pull_daily
+  wiki_push
+
+h3. in a ruby script
+
+bc. require 'elexis/wiki/interface/workspace'
+require 'pp'
+workspace =  Elexis::Wiki::Interface::Workspace.new('.')
+workspace.pull # get actual mediawiki content and place it under doc inside the plugin
+workspace.info.show
+pp workspace.views_missing_documentation            # returns an array of plugin-ids
+pp workspace.perspectives_missing_documentation     # returns an array of plugin-ids
+pp workspace.plugins_missing_documentation          # returns an array of plugis
+workspace.push # push doc/*.mediwiki files to the wiki
+
+Or you can inspire you from the spec/*_spec.rb files to see tested examples.
+
+h3. Username/password to access your wiki
+
+The username and password is fetched from a config.yml, which may be either
+* /etc/elexis-wiki-interface/config.yml
+* <current workdir>/config.yml
+
+Here is an example:
+
+bc. ---
+  wiki: http://wiki.elexis.info/api.php
+  password: topsecrect
+  user: elexis
+
+This user must have the privilege to access the wiki via the API interface.
+
+h3. Conventions
+
+* You must create a directory config with a file hosts.yml containing the mediawiki-gateway configuration (host, username, password)
+
+* The push/pull methods use the convention
+** in page names all '.' are removed
+** a plugin plugin.name has a page plugin.name
+** a view view.name with a localized name localized_name has a page <plugin.name>Views<localized_name>
+** a perspective perspective.name with a localized name localized_name has a page  P_<localized_name>
+** png files should start with their plugin/feature-name followed by '_' by a unique name
+** png files resides inside the 'doc' sub-directory of each plugin/feature
+
+h3. Limitations/TODO
+
+* Upload all perspectives/view images from a jubula run
+** View-images were in doc_de/images/views/Artikelauswahl.png (localized_name) -> http://wiki.elexis.info/Datei:Ch.elexis:Artikelauswahl.png
+** https://srv.elexis.info/jenkins/view/3.0/job/Elexis-3.0-Jubula/118/artifact/test-results/screenshots/Other/https://srv.elexis.info/jenkins/view/3.0/job/Elexis-3.0-Jubula/118/artifact/test-results/screenshots/Other/Fixmedikation.png
+** Bug ch.elexis -> ch.elexis.core splitted. now you find it here
+*** ch.elexis.core.application/plugin_de.properties:elexis.articleSelectorView=Artikelauswahl
+*** ch.elexis.core.ui.laboratory/plugin_de.properties:elexis.articleSelectorView   = Artikelauswahl
+*** ch.elexis.core.ui/plugin_de.properties:elexis.articleSelectorView = Artikelauswahl
+*** ch.elexis.core.application/plugin.xml:            name="%elexis.articleSelectorView">
+** Missing line [[Datei:Ch.elexis:Artikelauswahl.png]] in http://wiki.elexis.info/ChElexisViewsArtikelArtikelselektor
+* Perspektives: were not uploaded to doc_de
+* No images are pulled/pushed at the moment.
+* Check whether a link to the view exists on the wiki.elexis.info/Hauptseite
+* No support for features at the moment
+* No multilingual support at the moment
+
+
+h2. Contributing
+
+# Fork it ( http://github.com/ngiger/elexis-wiki-interface/fork )
+# Create your feature branch (`git checkout -b my-new-feature`)
+# Commit your changes (`git commit -am 'Add some feature'`)
+# Push to the branch (`git push origin my-new-feature`)
+# Create new Pull Request

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/wiki_pull

@@ -0,0 +1,6 @@
+#!/bin/env ruby
+require 'elexis/wiki/interface/workspace'
+
+workspace =  Elexis::Wiki::Interface::Workspace.new(Dir.pwd)
+workspace.pull
+workspace.show_missing(true)

data/bin/wiki_pull_daily

@@ -0,0 +1,25 @@
+#!/bin/env ruby
+
+require 'elexis/wiki/interface/workspace'
+
+workspace =  Elexis::Wiki::Interface::Workspace.new(Dir.pwd)
+workspace.pull
+workspace.show_missing(true)
+files = Dir.glob('doc_??') +
+    Dir.glob('*/doc/*.mediawiki') +
+    Dir.glob('*/doc/*.png')
+    Dir.glob('*/doc/*.jpg')
+    Dir.glob('*/doc/*.gif')
+files.each{
+  |file|
+  if /:/.match(File.basename(file))
+      puts "You may not add a file containg ':' or it will break git for Windows. Remove/rename #{file}"
+      exit 2
+  end
+}
+system("git add -f #{files.join(' ')}")
+system("git status")
+system("git commit --all -m '#{File.basename(__FILE__)}: Added mediawiki content from #{workspace.wiki}'")
+system("git status")
+system("git log -1")
+# system("git push")

data/bin/wiki_push

@@ -0,0 +1,5 @@
+#!/bin/env ruby
+require 'elexis/wiki/interface/workspace'
+workspace =  Elexis::Wiki::Interface::Workspace.new(Dir.pwd)
+workspace.push
+

data/elexis-wiki-interface.gemspec

@@ -0,0 +1,28 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'elexis/wiki/interface/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "elexis-wiki-interface"
+  spec.version       = Elexis::Wiki::Interface::VERSION
+  spec.authors       = ["Niklaus Giger"]
+  spec.email         = ["niklaus.giger@member.fsf.org"]
+  spec.summary       = "Interface between elexis source and wiki"
+  spec.description   = "Support for pulling/pushing wiki content from source repository to elexis.wiki"
+  spec.homepage      = "http://wiki.elexis.info"
+  spec.license       = "GPLv3"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+  spec.add_dependency 'rubyzip', '< 1.0.0'
+  spec.add_dependency 'unicode'
+  spec.add_dependency 'mediawiki-gateway'
+  spec.add_dependency 'eclipse-plugin', '>= 0.1'
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec"
+  spec.add_development_dependency "simplecov"
+end

data/lib/elexis/wiki/interface.rb

@@ -0,0 +1,8 @@
+require "elexis/wiki/interface/version"
+
+module Elexis
+  module Wiki
+    module Interface
+    end
+  end
+end

data/lib/elexis/wiki/interface/version.rb

@@ -0,0 +1,7 @@
+module Elexis
+  module Wiki
+    module Interface
+      VERSION = "0.4"
+    end
+  end
+end

data/lib/elexis/wiki/interface/workspace.rb

@@ -0,0 +1,298 @@
+#encoding: utf-8
+
+require 'eclipse/plugin'
+require 'media_wiki' # TODO: port it to require 'mediawiki_api'
+require 'fileutils'
+require 'open-uri'
+require 'time'
+require 'yaml'
+
+module Elexis
+  module Wiki
+    module Interface
+      class Workspace
+        attr_reader :info, :mw, :wiki, :views_missing_documentation, :perspectives_missing_documentation, :plugins_missing_documentation, :features_missing_documentation,
+            :doc_project, :features, :info
+        def initialize(dir, wiki = 'http://wiki.elexis.info/api.php')
+          @wiki = wiki
+          @mw = MediaWiki::Gateway.new(@wiki)
+          @info = Eclipse::Workspace.new(dir)
+          @doc_projects = Dir.glob(File.join(dir, "doc_??", ".project"))
+          @info.parse_sub_dirs
+          @info.show if $VERBOSE
+          @views_missing_documentation        =[]
+          @perspectives_missing_documentation =[]
+          @plugins_missing_documentation      =[]
+          @features_missing_documentation     =[]
+        end
+        def show_missing(details = false)
+          if views_missing_documentation.size and
+              plugins_missing_documentation.size == 0 and
+              features_missing_documentation.size == 0 and
+              perspectives_missing_documentation.size == 0
+            puts "Eclipse-Workspace #{@info.workspace_dir} seems to have documented all views, features, plugins and perspectives"
+          else
+            puts "Eclipse-Workspace #{@info.workspace_dir} needs documenting "
+            if views_missing_documentation.size > 0
+              puts "  #{views_missing_documentation.size} views"
+              puts "    #{views_missing_documentation.inspect}" if details
+            end
+            if plugins_missing_documentation.size > 0
+              puts "  #{plugins_missing_documentation.size} plugins"
+              puts "    #{plugins_missing_documentation.inspect}" if details
+            end
+            if features_missing_documentation.size > 0
+              puts "  #{features_missing_documentation.size} features"
+              puts "    #{features_missing_documentation.inspect}" if details
+            end
+            if perspectives_missing_documentation.size > 0
+              puts "  #{perspectives_missing_documentation.size} perspectives"
+              puts "    #{perspectives_missing_documentation.inspect}" if details
+            end
+          end
+        end
+        def push
+          check_config_file
+          raise "must define wiki with user and password in #{@config_yml}" unless @user and @password and @wiki
+          @mw.login(@user, @password)
+          @doc_projects.each{
+            |prj|
+            dir = File.dirname(prj)
+          }
+          @info.plugins.each{
+            |id,plugin|
+              to_push = Dir.glob("#{plugin.jar_or_src}/doc/*.mediawiki")
+              to_push.each{
+                           |file|
+                            # verify that locally committed file is newer than the page in the wiki
+                            # verify that the content after the push matches the local content
+                            my_new_content = File.new(file).read
+                            to_verify = my_new_content.gsub(/\n+/,"\n").chomp
+                            pagename = File.basename(file, '.mediawiki').capitalize
+                            last_wiki_modification = get_page_modification_time(pagename)
+                            last_git_modification = get_git_modification(file)
+                            unless last_wiki_modification
+                              puts "first upload #{File.basename(file)} last_git_modification is #{last_git_modification}" if $VERBOSE
+                              @mw.create(pagename, my_new_content,{:overwrite => true, :summary => "pushed by #{File.basename(__FILE__)}" })
+                            else
+                              got = @mw.get(pagename).gsub(/\n+/,"\n")
+                              if got == to_verify
+                                puts "No changes to push for #{file}"
+                                next
+                              end
+                              @mw.edit(pagename, to_verify,{:overwrite => true, :summary => "pushed by #{File.basename(__FILE__)}" })
+                              puts "Uploaded #{file} to #{pagename}" if $VERBOSE
+                            end
+                        }
+              if to_push.size > 0 # then upload also all *.png files
+                images_to_push = Dir.glob("#{plugin.jar_or_src}/doc/*.png")
+                images_to_push.each{
+                                 |image|
+                                if /:/.match(File.basename(image))
+                                   puts "You may not add a file containg ':' or it will break git for Windows. Remove/rename #{image}"
+                                   exit
+                                end
+
+                                git_mod   = get_git_modification(image)
+                                wiki_mod  = get_image_modification_name(image)
+
+                                if wiki_mod == nil
+                                  puts "first upload #{File.basename(image)} as last_git_modification is #{git_mod}" if $VERBOSE
+                                else
+                                  to_verify = File.new(image, 'rb').read
+                                  got       = @mw.get(File.basename(image))
+                                  if got == to_verify
+                                    puts "nothing to upload for #{image}" if $VERBOSE
+                                    next
+                                  end
+                                end
+                                begin
+                                  res = @mw.upload(image, 'filename' => File.basename(image))
+                                  puts "res für #{image}  exists? #{File.exists?(image)} ist #{res.to_s}"
+                                  puts "If you received API error: code 'verification-error', info 'This file did not pass file verification'"
+                                  puts "this means that the file type and content do not match, e.g. you have a *png file but in reality it is a JPEG file."
+                                  puts "In this case convert file.png file.png fixes this problem"
+                                rescue MediaWiki::APIError => e
+                                  puts "rescue für #{image} #{e}" #  if $VERBOSE
+                                end
+                }
+            end
+          }
+        end
+
+        def get_git_modification(file)
+          return nil unless File.exists?(file)
+          git_time = `git log -1 --pretty=format:%ai #{file}`
+          return nil  unless git_time.length > 8
+          Time.parse(git_time.chomp).utc
+        end
+
+        def get_page_modification_time(pagename)
+          json_url = "#{@wiki}?action=query&format=json&prop=revisions&titles=#{pagename}&rvprop=timestamp"
+          json = RestClient.get(json_url)
+          wiki_json_timestamp_to_time(json, pagename)
+        end
+
+        def pull
+          @doc_projects.each{
+            |prj|
+            dir = File.dirname(prj)
+            get_content_from_wiki(dir, File.basename(dir))
+          }
+          @info.plugins.each{
+            |id, info|
+              puts "Pulling for plugin #{id}" if $VERBOSE
+              pull_docs_views(info)
+              pull_docs_plugins(info)
+              pull_docs_perspectives(info)
+          }
+          @info.features.each{
+            |id, info|
+              puts "Pulling for feature #{id}" if $VERBOSE
+              pull_docs_features(info)
+          }
+          saved = Dir.pwd
+        end
+
+        def perspectiveToPageName(perspective)
+          # http://wiki.elexis.info/P_Abrechnungen
+          name = 'P_'+ perspective.id.gsub(' ', '')
+          puts "perspectiveToPageName for #{perspective.inspect} is '#{name}'" if $VERBOSE
+          name
+        end
+        def viewToPageName(plugin_id, view)
+          # für ch.elexis.agenda.views.TagesView (= view.id)
+          # http://wiki.elexis.info/ChElexisAgendaViewsTagesview
+          # wurde unter http://wiki.elexis.info/Hauptseite ein Link Agenda (= view.name) angelegt.
+          # evtl. sollten wir testen, ob dieser Link vorhanden ist
+          # http://wiki.elexis.info/ChElexisIcpcViewsEpisodesview
+          comps = view.id.split('.')
+          pageName = comps[0..-2].collect{|x| x.capitalize}.join + 'Views'+view.id.split('.').last.capitalize
+          puts "viewToPageName for #{plugin_id}/#{view.id} is #{pageName}" if $VERBOSE
+          pageName
+        end
+
+        private
+        def wiki_json_timestamp_to_time(json, page_or_img)
+          return nil unless json
+          begin
+            m = json.match(/timestamp['"]:['"]([^'"]+)/)
+            return Time.parse(m[1]) if m
+          end
+          nil
+        end
+
+        def check_config_file
+          possibleCfgs = ['/etc/elexis-wiki-interface/config.yml', File.join(Dir.pwd, 'config.yml'), ]
+          possibleCfgs.each{ |cfg| @config_yml = cfg; break if File.exists?(cfg) }
+          raise "need a config file #{possibleCfgs.join(' or ')} for wiki with user/password" unless File.exists?(@config_yml)
+          yaml = YAML.load_file(@config_yml)
+          @user = yaml['user']
+          @password = yaml['password']
+          @wiki = yaml['wiki']
+          puts "MediWiki #{@wiki} user #{@user} with password #{@password}" if $VERBOSE
+        end
+
+        def shorten_wiki_image(image)
+          return File.basename(image) unless File.basename(image).index(':')
+          File.basename(image).split(':')[1..-1].join(':').gsub(':', '_')
+        end
+
+        # http://wiki.elexis.info/api.php?action=query&format=json&list=allimages&ailimit=5&aiprop=timestamp&aiprefix=Ch.elexis.notes:config.png&*
+        def get_image_modification_name(image)
+          json_url = "#{@wiki}?action=query&format=json&list=allimages&ailimit=5&aiprop=timestamp&aiprefix=#{shorten_wiki_image(image)}"
+          json = RestClient.get(json_url)
+          wiki_json_timestamp_to_time(json, image)
+        end
+
+        # helper function, as mediawiki-gateway does not handle this situation correctly
+        def download_image_file(image, downloaded_image)
+          short_image = shorten_wiki_image(image)
+          unless File.exist? downloaded_image
+            json_url = "#{@wiki}?action=query&format=json&list=allimages&ailimit=5&aiprop=url&aiprefix=#{short_image}"
+            json = RestClient.get(json_url)
+            unless json
+              puts "JSON: Could not fetch for image #{image} using #{json_url}"
+              return
+            end
+            begin
+              answer = JSON.parse(json)
+              image_url = nil
+              image_url = answer['query'].first[1].first['url'] if answer['query'] and answer['query'].size >= 1 and answer['query'].first[1].size > 0
+              if image_url
+                      File.open(downloaded_image, 'w') do |file|
+                        file.write(open(image_url).read)
+                      end
+              else
+                puts "skipping image #{image}"
+              end
+              rescue => e
+                puts "JSON: Could not fetch for image #{image} using #{json_url}"
+                puts "      was '#{json}'"
+                puts "      error was #{e.inspect}"
+            end
+          end
+          puts "Downloaded image #{downloaded_image} #{File.size(downloaded_image)} bytes" if $VERBOSE
+        end
+
+        def get_content_from_wiki(out_dir, pageName)
+          puts "get_content_from_wiki page #{pageName} -> #{out_dir}" if $VERBOSE
+          out_name = File.join(out_dir, pageName + '.mediawiki')
+          FileUtils.makedirs(out_dir) unless File.directory?(out_dir)
+          begin
+            content = @mw.get(pageName)
+          rescue MediaWiki::Gateway::Exception => e
+            puts "Unable to get #{pageName} for #{out_dir} from #{File.dirname(@mw.wiki_url)}"
+            return nil
+          end
+          if content
+            ausgabe = File.open(out_name, 'w+') { |f| f.write content.gsub(/(Image:[\w\.]+):(\w+.png)/, '\1_\2') }
+            @mw.images(pageName).each{
+              |image|
+                image = image.gsub(/[^\w\.:]/, '_')
+                downloaded_image = File.join(out_dir, shorten_wiki_image(image))
+                download_image_file(image, downloaded_image.sub(':', ''))
+                break if defined?(RSpec) # speed up rspec
+            }
+          else
+            puts "Could not fetch #{pageName} from #{@mw}" if $VERBOSE
+          end
+          content
+        end
+        def pull_docs_views(plugin)
+          id = plugin.symbolicName
+          plugin.views.each{
+            |id, view|
+            pageName = viewToPageName(plugin.symbolicName, view)
+            content = get_content_from_wiki(File.join(@info.workspace_dir, File.basename(plugin.jar_or_src), 'doc'), pageName)
+            @views_missing_documentation << pageName unless content
+          }
+        end
+       def pull_docs_perspectives(plugin)
+          id = plugin.symbolicName
+          plugin.perspectives.each{
+            |id, perspective|
+            pageName = perspectiveToPageName(perspective)
+            content = get_content_from_wiki(File.join(@info.workspace_dir, File.basename(plugin.jar_or_src), 'doc'), pageName)
+            @perspectives_missing_documentation << pageName unless content
+          }
+        end
+        def pull_docs_plugins(plugin)
+          id = plugin.symbolicName
+          pageName = id.capitalize
+          content = get_content_from_wiki(File.join(@info.workspace_dir, File.basename(plugin.jar_or_src), 'doc'), pageName)
+          @perspectives_missing_documentation << pageName unless content
+        end
+        def pull_docs_features(feature)
+          id = feature.symbolicName
+          pageName = id.capitalize
+          content = get_content_from_wiki(File.join(@info.workspace_dir, id, 'doc'), pageName)
+          unless content
+            content = get_content_from_wiki(File.join(@info.workspace_dir, id, 'doc'), pageName.sub(/feature$/, 'feature.feature.group'))
+            puts "pull_docs_features failed #{id} #{pageName}" unless content
+          end
+        end
+      end
+    end
+  end
+end