iqvoc 4.7.0 → 4.8.0

data/README.md

@@ -4,30 +4,34 @@
 [![Build Status](https://secure.travis-ci.org/innoq/iqvoc.png)](http://travis-ci.org/innoq/iqvoc)
 [![Code Climate](https://codeclimate.com/github/innoq/iqvoc.png)](https://codeclimate.com/github/innoq/iqvoc)
 
-iQvoc is a vocabulary management tool that combines easy-to-use human interfaces with Semantic Web interoperability.
+iQvoc is a vocabulary management tool that combines easy-to-use human interfaces
+with Semantic Web interoperability.
 
-iQvoc supports vocabularies that are common to many knowledge organization systems, such as:
+iQvoc supports vocabularies that are common to many knowledge organization
+systems, such as:
 
 * Thesauri
 * Taxonomies
 * Classification schemes
 * Subject heading systems
 
-iQvoc provides comprehensive functionality for all aspects of managing such vocabularies:
+iQvoc provides comprehensive functionality for all aspects of managing such
+vocabularies:
 
 * import of existing vocabularies from a SKOS representation
 * multilingual display and navigation in any Web browser
 * editorial features for registered users
 * publishing the vocabulary in the Semantic Web
 
-iQvoc is built with state-of-the-art technology and can be easily customized according to user's needs.
-
+iQvoc is built with state-of-the-art technology and can be easily customized
+according to user's needs.
 ## Setup
 
 ### Heroku
 
-You can easily setup your iQvoc instance in under 5 minutes, we wanted to make this process really easy.
-In order to deploy to heroku you need to have an account and [heroku toolbelt](https://toolbelt.heroku.com) installed.
+You can easily setup your iQvoc instance in under 5 minutes, we wanted to make
+this process really easy. In order to deploy to heroku you need to have an
+account and [heroku toolbelt](https://toolbelt.heroku.com) installed.
 
 ```
 $ bundle install
@@ -46,23 +50,26 @@ Remember to visit the Users section and change the default passwords!
 ### Custom
 
 We recommend running [iQvoc as a Rails engine](https://github.com/innoq/iqvoc/wiki/iQvoc-as-a-Rails-Engine).
-Running the cloned source code is possible but any modifications would require a fork.
+Running the cloned source code is possible but any modifications would require a
+fork.
 
-1. Configure your database via `config/database.template.yml`. Don't forget to rename it to `database.yml`
+1. Configure your database via `config/database.template.yml`.
+   Don't forget to rename it to `database.yml`
 2. Run `bundle install`
 3. Run `bundle exec rake db:create` to create the database
 4. Create the necessary tables by running `rake db:migrate`
 5. Load some base data by running `rake db:seed`
 6. Make sure you have got `config/secrets.yml` in place
-7. Boot up the app using `bundle exec rails s` (or `passenger start` if you use passenger)
+7. Boot up the app using `bundle exec rails s` (or `passenger start`
+   if you use passenger)
 8. Log in with "admin@iqvoc" / "admin" or "demo@iqvoc" / "cooluri" (cf. step #5)
 9. Visit the Users section and change the default passwords
 
 ## Background Jobs
 
-Note that some features like "Import" and "Export" exposed in the Web UI store their workload
-as jobs. You can either issue a job worker that runs continuously and watches
-for new jobs via
+Note that some features like "Import" and "Export" exposed in the Web UI store
+their workload as jobs. You can either issue a job worker that runs continuously
+and watches for new jobs via
 
 ```
 $ rake jobs:work
@@ -76,13 +83,14 @@ $ rake jobs:workoff
 
 ## Compatibility
 
-iQvoc is fully compatible with Ruby 1.9, 2.0, 2.1 and JRuby 1.7.
+iQvoc is fully compatible with Ruby 1.9, 2.0, 2.1 and JRuby 1.7
+(in Ruby 1.9-mode).
 
 ## Customization
 
-There are many hooks providing support for your own classes and configuration. The core app
-also works as a Rails Engine. The config residing in `lib/iqvoc.rb` provides a basic
-overview of the possibilities.
+There are many hooks providing support for your own classes and configuration.
+The core app also works as a Rails Engine. The config residing in `lib/iqvoc.rb`
+provides a basic overview of the possibilities.
 
 ## Documentation
 
@@ -100,13 +108,19 @@ For more information on SemVer, visit http://semver.org/.
 
 If you want to help out there are several options:
 
-* Found a bug? Just create an issue on the [GitHub Issue tracker](https://github.com/innoq/iqvoc/issues) and/or submit a patch by initiating a pull request
-* You're welcome to fix bugs listed under [Issues](https://github.com/innoq/iqvoc/issues)
-* Proposal, discussion and implementation of new features on our mailing list iqvoc@lists.innoq.com or on the issue tracker
+- Found a bug? Just create an issue on the
+  [GitHub Issue tracker](https://github.com/innoq/iqvoc/issues) and/or submit a
+  patch by initiating a pull request
+- You're welcome to fix bugs listed under
+  [Issues](https://github.com/innoq/iqvoc/issues)
+- Proposal, discussion and implementation of new features on our mailing list
+  [iqvoc@lists.innoq.com] or on the issue tracker
 
-If you make changes to existing code please make sure that the test suite stays green. Please include tests to your additional contributions.
+If you make changes to existing code please make sure that the test suite stays
+green. Please include tests to your additional contributions.
 
-Tests can be run via `bundle exec rake test`. We're using capybara-webkit for integration tests with JavaScript support.
+Tests can be run via `bundle exec rake test`. We're using Poltergeist for
+integration tests with JavaScript support.
 
 ## Maintainer & Contributors
 
@@ -115,9 +129,10 @@ iQvoc was originally created and is being maintained by [innoQ Deutschland GmbH]
 * Robert Glaser ([mrreynolds](http://github.com/mrreynolds))
 * Till Schulte-Coerne ([tillsc](http://github.com/tillsc))
 * Frederik Dohr ([FND](http://github.com/FND))
+* Marc Jansing ([mjansing](http://github.com/mjansing))
 
 ## License
 
-Copyright 2014 innoQ Deutschland GmbH
+Copyright 2014 [innoQ Deutschland GmbH](https://www.innoq.com).
 
-Licensed under the Apache License, Version 2.0
+Licensed under the Apache License, Version 2.0.

data/{test/performance/browsing_test.rb → app/aides/inline_data_helper.rb}

@@ -14,12 +14,29 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-require File.join(File.expand_path(File.dirname(__FILE__)), '../test_helper')
-require 'rails/performance_test_help'
+require 'csv'
 
-# Profiling results for each test method are written to tmp/performance.
-class BrowsingTest < ActionDispatch::PerformanceTest
-  def test_homepage
-    get '/'
+class InlineDataHelper
+  JOINER = ', '
+  SPLITTER = /[,\n] */
+
+  CSV_OPTIONS = {
+    col_sep: ', ',
+    quote_char: '"'
+  }
+
+  def self.parse_inline_values(inline_values)
+    options = CSV_OPTIONS.clone
+    options[:col_sep] = options[:col_sep].strip
+    begin
+      values = inline_values.parse_csv(options)
+    rescue CSV::MalformedCSVError => exc
+      values = inline_values.parse_csv(CSV_OPTIONS)
+    end
+    values ? values.map(&:strip) : []
+  end
+
+  def self.generate_inline_values(values)
+    values.to_csv(CSV_OPTIONS).strip
   end
 end

data/app/aides/maker.rb

@@ -0,0 +1,139 @@
+module Maker
+  # labels:
+  # -
+  #   value: <string>
+  #   <attribute>: <value>
+  #   inflectionals: [<string>, ...]
+  #   components: [<label>, ...]
+  #
+  # concepts:
+  # -
+  #   <attribute>: <value>
+  #   pref_labels: [<label>, ...]
+  #   alt_labels: [<label>, ...]
+  #   broader: <concept>
+  #   narrower: <concept>
+  #   related: [<concept>, ...]
+  #
+  # NB:
+  # * <label> and <concept> can either be strings referencing a previously
+  #   declared entity or objects representing a new entity
+  #   XXX: The latter is not currently supported yet!
+  # * order of concepts matters when referencing relations
+  # * order of labels matters when referencing components
+  def self.from_yaml(yml)
+    data = YAML.load(yml)
+
+    labels = {}
+    data['labels'].each { |label| # XXX: use omap to simplify format (making `value` the key instead of an attribute)?
+      term = label.delete('value')
+
+      components = label.delete('components').map { |term|
+        labels[term]
+      } if label['components']
+
+      options = {
+        inflectionals: label.delete('inflectionals'),
+        components: components,
+        label_attributes: label
+      }
+
+      labels[term] = self.label(term, options)
+    } if data['labels']
+
+    concepts = {}
+    data['concepts'].each { |concept| # XXX: use omap to simplify format (using a single pref_label as key)?
+      relations = {}
+      ['broader', 'narrower'].each { |type| # TODO: missing related, support for poly-hierarchies
+        relations[type] = concepts[concept.delete(type)] if concept[type]
+      }
+
+      lbls = {} # TODO: rename
+      ['pref', 'alt'].each { |type| # TODO: missing hidden
+        key = "#{type}_labels"
+        lbls[type] = concept.delete(key).map { |term|
+          labels[term]
+        } if concept[key]
+      }
+
+      options = {
+        pref_labels: lbls['pref'],
+        alt_labels: lbls['alt'],
+        concept_attributes: concept
+      }
+
+      identifier = options[:pref_labels].first.value
+      concepts[identifier] = self.concept(options)
+      concepts[identifier].send(Iqvoc::Concept.broader_relation_class.name.to_relation_name).
+          create_with_reverse_relation(relations['broader']) if relations['broader']
+      concepts[identifier].send(Iqvoc::Concept.broader_relation_class.narrower_class.name.to_relation_name).
+          create_with_reverse_relation(relations['narrower']) if relations['narrower']
+    } if data['concepts']
+
+    return { concepts: concepts, labels: labels }
+  end
+
+  # optional arguments:
+  # concept_attributes for custom concept attributes
+  # pref_labels is an array of strings or label instances to be used as prefLabels
+  # alt_labels is an array of strings or label instances to be used as altLabels
+  def self.concept(options={})
+    attributes = options[:concept_attributes] || {}
+    pref_labels = options[:pref_labels] || []
+    alt_labels = options[:alt_labels] || []
+
+    defaults = { # NB: must use strings, not symbols as keys due to YAML
+      'published_at' => 3.days.ago
+    }
+    attributes = defaults.merge(attributes)
+
+    concept = Iqvoc::Concept.base_class.create!(attributes)
+
+    pref_labels.each { |term|
+      label = term.is_a?(String) ? self.label(term) : term
+      Iqvoc::Concept.pref_labeling_class.
+          create!(owner: concept, target: label)
+    }
+    alt_labels.each { |term|
+      label = term.is_a?(String) ? self.label(term) : term
+      Iqvoc::Concept.further_labeling_classes.first.first.
+          create!(owner: concept, target: label)
+    }
+
+    return concept
+  end
+
+  # optional arguments:
+  # label_attributes for custom label attributes
+  # inflectionals is an array of strings to be used as inflectionals
+  # components is an array of labels to be used as compound form contents
+  def self.label(value, options={}) # FIXME: move into SKOS-XL extension
+    attributes = options[:label_attributes] || {}
+    inflectionals = options[:inflectionals] || []
+    components = options[:components] || []
+
+    defaults = { # NB: must use strings, not symbols as keys due to YAML
+      value: value, # intentionally not a string; symbol takes precedence
+      'origin' => Origin.new(value).to_s,
+      'language' => Iqvoc::Concept.pref_labeling_languages.first,
+      'published_at' => 2.days.ago
+    }
+    attributes = defaults.merge(attributes)
+
+    klass = Iqvoc::XLLabel rescue Iqvoc::Label # FIXME: breaks encapsulation (hard-coded iqvoc_skosxl dependency)
+    label = klass.base_class.create!(attributes)
+
+    inflectionals.each { |inf|
+      label.inflectionals.create!(value: inf)
+    }
+
+    if components.length > 0
+      compound_form_contents = components.each_with_index.map { |label, i|
+        CompoundForm::Content::Base.new(label: label, order: i)
+      }
+      label.compound_forms.create!(compound_form_contents: compound_form_contents)
+    end
+
+    return label
+  end
+end

data/app/aides/origin.rb

@@ -0,0 +1,47 @@
+# encoding: UTF-8
+
+# Copyright 2011-2013 innoQ Deutschland GmbH
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class Origin
+  attr_accessor :initial_value, :value
+
+  def initialize(value = nil)
+    self.initial_value = value
+    self.value = "_#{SecureRandom.hex(4)}"
+  end
+
+  def valid?
+    valid = true
+
+    if blank_node = initial_value.match(RDFAPI::BLANK_NODE_REGEXP)
+      # blank node validation, should not contain special chars
+      valid = false if CGI.escape(blank_node[1]) != blank_node[1]
+    else
+      # regular subject validation
+
+      # should not start with a number
+      valid = false if initial_value.match(/^\d.*/)
+
+      # should not contain special chars
+      valid = false if CGI.escape(initial_value) != initial_value
+    end
+
+    valid
+  end
+
+  def to_s
+    value
+  end
+end

data/app/aides/rdfapi.rb

@@ -0,0 +1,59 @@
+# encoding: UTF-8
+
+# Copyright 2011-2013 innoQ Deutschland GmbH
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class RDFAPI
+  FIRST_LEVEL_OBJECT_CLASSES  = [Iqvoc::Concept.base_class, Iqvoc::Collection.base_class]
+  SECOND_LEVEL_OBJECT_CLASSES = Iqvoc::Concept.labeling_classes.keys +
+  Iqvoc::Concept.note_classes +
+  Iqvoc::Concept.relation_classes +
+  Iqvoc::Concept.match_classes +
+  Iqvoc::Concept.notation_classes +
+  Iqvoc::Concept.additional_association_classes.keys +
+  [Iqvoc::Collection.member_class]
+
+  OBJECT_DICTIONARY = FIRST_LEVEL_OBJECT_CLASSES.inject({}) do |hash, klass|
+    hash["#{klass.rdf_namespace}:#{klass.rdf_class}"] = klass
+    hash
+  end
+
+  PREDICATE_DICTIONARY = SECOND_LEVEL_OBJECT_CLASSES.inject({}) do |hash, klass|
+    hash["#{klass.rdf_namespace}:#{klass.rdf_predicate}"] = klass
+    hash
+  end
+
+  URI_REGEXP = /^https?:\/\/[^\s]+$/
+  LITERAL_REGEXP = /^"(.+)"(@(.+))?$/
+  BLANK_NODE_REGEXP = /^_:(.+)/
+
+  def self.devour(rdf_subject, rdf_predicate, rdf_object)
+    case rdf_predicate
+    when 'a', 'rdf:type'
+      case rdf_object
+      when String
+        target = OBJECT_DICTIONARY[rdf_object] || rdf_object.constantize
+      else
+        target = rdf_object
+      end
+      target.find_or_initialize_by(origin: rdf_subject)
+    when String
+      # dictionary lookup
+      target = PREDICATE_DICTIONARY[rdf_predicate] || rdf_predicate.constantize
+      target.build_from_rdf(rdf_subject, target, rdf_object)
+    else # is a class
+      rdf_predicate.build_from_rdf(rdf_subject, rdf_predicate, rdf_object)
+    end
+  end
+end

data/app/aides/skos_exporter.rb

@@ -0,0 +1,151 @@
+require 'iq_rdf'
+require 'uri'
+require 'fileutils'
+
+class SkosExporter
+  include RdfHelper # necessary to use render_concept helper
+  include RdfNamespacesHelper
+  include Rails.application.routes.url_helpers
+
+  def initialize(file_path, type, default_namespace_url, logger = Rails.logger)
+    default_url_options[:port] = URI.parse(default_namespace_url).port
+    default_url_options[:host] = URI.parse(default_namespace_url).to_s.gsub(/\/$/, '')
+
+    @file_path = file_path
+    @type = type
+    @logger = logger
+    @document = IqRdf::Document.new
+
+    unless ['ttl', 'nt', 'xml'].include? @type
+      raise "SkosExporter: Unknown rdf serialization. Parameter 'type' should be 'ttl' (Turtle), 'nt' (N-Triples) or 'xml' (RDF-XML)."
+    end
+
+    unless @file_path.is_a?(String)
+      raise "SkosExporter#export: Parameter 'file' should be a String."
+    end
+  end
+
+  def run
+    export
+  end
+
+  private
+
+  def export
+    ActiveSupport.run_load_hooks(:rdf_export_before, self)
+
+    start = Time.now
+    @logger.info 'Starting export...'
+    @logger.info "file_path = #{@file_path}"
+    @logger.info "type = #{@type}"
+
+    # add export data
+    add_namespaces(@document)
+    add_collections(@document)
+    add_concepts(@document)
+
+    ActiveSupport.run_load_hooks(:rdf_export_before_save, self)
+
+    # saving export to disk
+    save_file(@file_path, @type, @document)
+
+    done = Time.now
+    @logger.info "Export Job finished in #{(done - start).to_i} seconds."
+
+    ActiveSupport.run_load_hooks(:rdf_export_after, self)
+  end
+
+  def add_namespaces(document)
+    @logger.info 'Exporting namespaces...'
+
+    RdfNamespacesHelper.instance_methods.each do |meth|
+      namespaces = send(meth)
+      document.namespaces(namespaces) if namespaces.is_a?(Hash)
+    end
+
+    @logger.info 'Finished exporting namespaces.'
+  end
+
+  def add_collections(document)
+    @logger.info 'Exporting collections...'
+
+    offset = 0
+    while true
+      collections = Iqvoc::Collection.base_class.published.order('id').limit(100).offset(offset)
+      limit = collections.size < 100 ? collections.size : 100
+      break if collections.size == 0
+
+      # Todo: Preloading???
+      collections.each do |collection|
+        render_collection(document, collection)
+      end
+
+      @logger.info "Collections #{offset+1}-#{offset+limit} exported."
+      offset += collections.size # Size is important!
+    end
+
+    @logger.info "Finished exporting collections (#{offset} collections exported)."
+  end
+
+  def add_concepts(document)
+    @logger.info 'Exporting concepts...'
+
+    offset = 0
+    while true
+      concepts = Iqvoc::Concept.base_class.published.order('id').limit(100).offset(offset)
+      limit = concepts.size < 100 ? concepts.size : 100
+      break if concepts.size == 0
+
+      # When in single query mode, AR handles ALL includes to be loaded by that
+      # one query. We don't want that! So let's do it manually :-)
+      ActiveRecord::Associations::Preloader.new.preload(concepts,
+      Iqvoc::Concept.base_class.default_includes + [
+        :matches,
+        :collection_members,
+        :notations,
+        { relations: :target, labelings: :target, notes: :annotations }
+        ])
+
+        concepts.each do |concept|
+          render_concept(document, concept, true)
+        end
+
+        @logger.info "Concepts #{offset+1}-#{offset+limit} exported."
+        offset += concepts.size # Size is important!
+      end
+
+      @logger.info "Finished exporting concepts (#{offset} concepts exported)."
+    end
+
+    def save_file(file_path, type, content)
+      begin
+        @logger.info "Saving export to '#{@file_path}'"
+        create_directory(@file_path)
+        file = File.open(@file_path, 'w')
+        content = serialize_rdf(content, type)
+        file.write(content)
+      rescue IOError => e
+        # some error occur
+        # e.g not writable
+      ensure
+        file.close unless file == nil
+      end
+    end
+
+    def create_directory(file_path)
+      dirname = File.dirname(file_path)
+      unless File.directory?(dirname)
+        FileUtils.mkdir_p(dirname)
+      end
+    end
+
+    def serialize_rdf(document, type)
+      if type == 'xml'
+        document.to_xml
+      elsif type == 'ttl'
+        document.to_turtle
+      else
+        document.to_ntriples
+      end
+    end
+  end