rdfa_parser 0.1.0

data/.gitignore

@@ -0,0 +1,7 @@
+rdoc/
+pkg/
+doc/
+.loadpath
+.project
+*.gemspec
+*.DS_Store

data/History.txt

@@ -0,0 +1,47 @@
+=== 0.1.0 2009-10-17
+* First gem release, all XHTML1 approved test cases pass at http://rdfa.digitalbazaar.com/test-suite/
+
+=== 0.0.5 2009-10-17
+* Fixed generation of RDF/XML for XML Literals.
+* Update Specs for style
+* Add eql? and hash methods to BNode and URIRef to allow for uniq
+* Added test based on 0017 to find BNode XML output problems.
+* Allow empty BNodes to be considered to be named
+* Add xml:lang to XMLLiterals as necessary
+* Don't add existing namespace definition to XML Literal
+* Special output for XML Literal to treat xml:lang within string. Add on create.
+* Added Namespace#xmlns_hash
+* Use XML Parser, not HTML (this should probably vary on doc type). HTML parser was removing namespace declarations from XMLLiterals.
+* Added _profile_ option to RdfaParser#parse to set parsetype to xhtml1, html4, or html5. Right now only xhtml1 is used.
+* Use to_html when re-serializing XMLLiterals, whitespace was being destroid otherwise.
+* All specs pass!
+
+=== 0.0.4 2009-10-16
+* Fixed namespace bug in RDF/XML generation from Graph
+* Fixed URI generation from namespace, which failed when suffix was a defined method of Namespace (use ns + suffix)
+
+=== 0.0.3 2009-10-12
+* Added --quiet option to executable
+* BNode RDF/XML output
+* Output all default namespaces in RDF/XML
+=== 0.0.2 2009-10-10
+* Switch to jeweler from newgem/hoe
+* RdfaParser.new just creates parser, allowing a different graph to be substituted.
+* parser.parse() performs parse.
+* Added Graph.to_rdfxml to output graph in XML serialization
+
+=== 0.0.1 2009-09-26
+
+* Previous versions based on original work by Ben Adida.
+* Spec tests use Redland/Rasqal to check results.
+  * Version 0.9.16 of Rasqal does not support UNION, so those tests are compared against NTriples
+* Passes most XHTML+RDFa tests. Of those that don't pass:
+  * NTriple BNode production differences.
+  * XMLLiteral attribute/namespace ordering differences (requires deep XML compare, not in Nokogiri).
+* Unexplained failures:
+  * Test0102: XMLLiteral with explicit namespace and xml:lang; not overwriting existing langs
+  * Test0103: XMLLiteral with explicit namespace; not overwriting local namespaces
+  * Test0105: inner @rel neither CURIE nor LinkType
+  * Test0106: chaining with empty value in inner @rel
+  * Test0121: "[]" is a valid safe CURIE
+    * RDFa parser seems to be working correctly, but Nokogiri parses embedded <p> incorrectly

data/README.rdoc

@@ -0,0 +1,61 @@
+= rdfa_parser
+
+* http://github.com/gkellogg/rdfa_parser
+
+== DESCRIPTION:
+
+RDFa parser written in pure Ruby. Yields each triple, or generate in-memory graph
+
+== FEATURES/PROBLEMS:
+
+* Most, but not all tests pass.
+* Tests performed by comparison to expected NTriples, not based on SPARQL query
+* Ultimately, this should be merged in with the Reddy gem to become part of a comprehensive Ruby RDF implementation
+  * Support libraries (Graph, Triple, and URIRef are used substantially intact, to facilitate a future merger)
+
+== SYNOPSIS:
+
+  Based on original work of Ben Adida.
+  
+  Basic usage:
+  
+  require 'rdfa_parser'
+  parser = RdfaParser::RdfaParser.new(stream, base_uri) do |triple|
+    puts triple.to_ntriple
+  end
+
+== REQUIREMENTS:
+
+* addressable
+* nokogiri
+* redland (for testing only)
+
+== INSTALL:
+
+* sudo gem install rdfa_parser
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2009 Gregg Kellogg
+Copyright (c) 2008 Tom Morris and contributors (Portions of Reddy utilities)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,52 @@
+require 'rubygems'
+begin
+  gem 'jeweler'
+  require 'jeweler'
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name = "rdfa_parser"
+    gemspec.summary = "RDFa parser written in pure Ruby."
+    gemspec.description = " Yields each triple, or generate in-memory graph"
+    gemspec.email = "gregg@kellogg-assoc.com"
+    gemspec.homepage = "http://github.com/gkellogg/rdfa_parser"
+    gemspec.authors = ["Gregg Kellogg"]
+    gemspec.add_dependency('addressable', '>= 2.0.0')
+    gemspec.add_dependency('nokogiri', '>= 1.3.3')
+    gemspec.add_dependency('builder', '>= 2.1.2')
+    gemspec.add_development_dependency('rspec')
+    gemspec.add_development_dependency('activesupport', '>= 2.3.0')
+    gemspec.extra_rdoc_files     = %w(README.rdoc History.txt)
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'spec/rake/spectask'
+Spec::Rake::SpecTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :spec => :check_dependencies
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION')
+    version = File.read('VERSION')
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "rdfa_parser #{version}"
+  rdoc.rdoc_files.include('README*', "History.txt")
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/bin/rdfa_parser

@@ -0,0 +1,55 @@
+#!/usr/bin/env ruby -s
+require 'rubygems'
+require File.expand_path(File.dirname(__FILE__) + "/../lib/rdfa_parser")
+require 'getoptlong'
+
+class Parse
+  def parse(file, base_uri)
+    puts "Parse: #{file}" if $quiet
+    parser = RdfaParser::RdfaParser.new
+    parser.parse(File.read(file), base_uri)
+    output = case $format
+    when "xml"
+      parser.graph.to_rdfxml
+    else
+      parser.graph.to_ntriples
+    end
+    puts output unless $quiet
+
+    puts parser.debug.join("\n\t") if $verbose
+  rescue RdfaParser::ParserException => e
+    puts "Parse failure: #{e.message}"
+    puts parser.debug if $verbose && parser
+    #raise
+  rescue Exception => e
+    puts "Parser fault: #{e.message}"
+    puts parser.debug if parser && !$quiet
+    raise
+  end
+end
+
+$verbose = false
+base_uri  = "http://example.com"
+
+opts = GetoptLong.new(
+  ["--verbose", GetoptLong::NO_ARGUMENT],
+  ["--quiet", GetoptLong::NO_ARGUMENT],
+  ["--debug", GetoptLong::NO_ARGUMENT],
+  ["--format", GetoptLong::REQUIRED_ARGUMENT],
+  ["--uri", GetoptLong::REQUIRED_ARGUMENT]
+)
+opts.each do |opt, arg|
+  case opt
+  when '--verbose' then $verbose = true
+  when '--quiet' then $quiet = true
+  when '--debug' then $DEBUG = true
+  when '--format' then $format = arg
+  when '--uri' then base_uri = arg
+  end
+end
+
+x = Parse.new
+ARGV.each do |test_file|
+  x.parse(test_file, base_uri)
+end
+

data/lib/rdfa_parser.rb

@@ -0,0 +1,35 @@
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+Dir.glob(File.join(File.dirname(__FILE__), 'rdfa_parser/**.rb')).each { |f| require f }
+
+begin
+  require 'nokogiri'
+  require 'addressable/uri'
+  require 'builder'
+rescue LoadError
+  require 'rubygems' unless ENV['NO_RUBYGEMS']
+  gem 'nokogiri'
+  gem 'addressable'
+  require 'nokogiri'
+  require 'addressable/uri'
+  require 'builder'
+end
+
+module RdfaParser
+  LINK_TYPES = %w(
+    alternate appendix bookmark cite chapter contents copyright first glossary
+    help icon index last license meta next p3pv1 prev role section stylesheet subsection
+    start top up
+  )
+
+  RDF_TYPE    = URIRef.new("http://www.w3.org/1999/02/22-rdf-syntax-ns#type")
+  XML_LITERAL = Literal::Encoding.xmlliteral
+
+  RDF_NS      = Namespace.new("http://www.w3.org/1999/02/22-rdf-syntax-ns#", "rdf")
+  RDFS_NS     = Namespace.new("http://www.w3.org/2000/01/rdf-schema#", "rdfs")
+  XHV_NS      = Namespace.new("http://www.w3.org/1999/xhtml/vocab#", "xhv")
+  XML_NS      = Namespace.new("http://www.w3.org/XML/1998/namespace", "xml")
+
+  XH_MAPPING  = {"" => Namespace.new("http://www.w3.org/1999/xhtml/vocab\#", nil)}
+end

data/lib/rdfa_parser/bnode.rb

@@ -0,0 +1,92 @@
+module RdfaParser
+  # The BNode class creates RDF blank nodes.
+  class BNode
+    @@next_generated = "a"
+    @@named_nodes = {}
+    
+    # Create a new BNode, optionally accept a identifier for the BNode. Otherwise, generated sequentially
+    def initialize(identifier = nil)
+      if identifier != nil && self.valid_id?(identifier)
+        # Generate a name if it's blank
+        @identifier = (@@named_nodes[identifier] ||= identifier.to_s.length > 0 ? identifier : self )
+      else
+        # Don't actually allocate the name until it's used, to save generation space
+        # (and make checking test cases easier)
+        @identifier = self
+      end
+    end
+
+    def to_s
+      return self.identifier.to_s
+    end
+
+    ## 
+    # Exports the BNode in N-Triples form.
+    #
+    # ==== Example
+    #   b = BNode.new; b.to_n3  # => returns a string of the BNode in n3 form
+    #
+    # ==== Returns
+    # @return [String] The BNode in n3.
+    #
+    # @author Tom Morris
+
+    def to_n3
+      "_:#{self.identifier}"
+    end
+
+    ## 
+    # Exports the BNode in N-Triples form.
+    #
+    # ==== Example
+    #   b = BNode.new; b.to_ntriples  # => returns a string of the BNode in N-Triples form
+    #
+    # ==== Returns
+    # @return [String] The BNode in N-Triples.
+    #
+    # @author Tom Morris
+
+    def to_ntriples
+      self.to_n3
+    end
+
+    # Output URI as resource reference for RDF/XML
+    def xml_args
+      [{"rdf:nodeID" => self.identifier}]
+    end
+    
+    # The identifier used used for this BNode. Not evaluated until this is called, which means
+    # that BNodes that are never used in a triple won't polute the sequence.
+    def identifier
+      return @identifier unless @identifier.is_a?(BNode)
+      if @identifier.equal?(self)
+        # Generate from the sequence a..zzz, unless already taken
+        @@next_generated = @@next_generated.succ while @@named_nodes.has_key?(@@next_generated)
+        @identifier, @@next_generated = @@next_generated, @@next_generated.succ
+      else
+        # Previously allocated node
+        @identifier = @identifier.identifier
+      end
+      @identifier
+    end
+    
+    def eql?(other)
+      other.is_a?(self.class) && self.identifier == other.identifier
+    end
+    alias_method :==, :eql?
+    
+    # Needed for uniq
+    def hash; to_s.hash; end
+    
+    # Start _identifier_ sequence from scratch
+    def self.reset(init = "a")
+      @@next_generated = init
+      @@named_nodes = {}
+    end
+
+    protected
+    def valid_id?(name)
+      name =~ /^[a-zA-Z_][a-zA-Z0-9]*$/ || name.empty?
+    end
+  end
+end

data/lib/rdfa_parser/exceptions.rb

@@ -0,0 +1,11 @@
+module RdfaParser
+  class RdfException < RuntimeError; end
+  class ParserException < RdfException; end
+  class GraphException < RdfException; end
+  class TypeError < RdfException; end
+  class AboutEachException < ParserException; end
+  class UriRelativeException < RdfException; end
+  class InvalidPredicate < RdfException; end
+  class InvalidSubject < RdfException; end
+  class InvalidObject < RdfException; end
+end

data/lib/rdfa_parser/graph.rb

@@ -0,0 +1,222 @@
+module RdfaParser
+  # A simple graph to hold triples (from Reddy)
+  class Graph
+    attr_accessor :triples, :nsbinding
+
+    def initialize
+      @triples = []
+      @nsbinding = {}
+    end
+
+    def self.load (uri)
+      RdfXmlParser.new(open(uri)).graph
+    end
+
+    def size
+      @triples.size
+    end
+
+    def each
+      @triples.each { |value| yield value }
+    end
+    
+    def subjects
+      @triples.map {|t| t.subject}.uniq
+    end
+    
+    def [] (item)
+      @triples[item]
+    end
+
+    def each_with_subject(subject)
+      @triples.each do |value|
+        yield value if value.subject == subject
+      end
+    end
+
+    def get_resource(subject)
+      @triples.find_all { |i| true if i.subject == subject}
+    end
+
+    ## 
+    # Adds a triple to a graph directly from the intended subject, predicate, and object.
+    #
+    # ==== Example
+    #   g = Graph.new; g.add_triple(BNode.new, URIRef.new("http://xmlns.com/foaf/0.1/knows"), BNode.new) # => results in the triple being added to g; returns an array of g's triples
+    #
+    # @param [URIRef, BNode] s the subject of the triple
+    # @param [URIRef] p the predicate of the triple
+    # @param [URIRef, BNode, Literal, TypedLiteral] o the object of the triple
+    #
+    # ==== Returns
+    # @return [Array] An array of the triples (leaky abstraction? consider returning the graph instead)
+    #
+    # @raise [Error] Checks parameter types and raises if they are incorrect.
+    # @author Tom Morris
+    def add_triple(s, p, o)
+      @triples += [ Triple.new(s, p, o) ]
+    end
+
+
+    ## 
+    # Adds an extant triple to a graph
+    #
+    # ==== Example
+    #   g = Graph.new; t = Triple.new(BNode.new, URIRef.new("http://xmlns.com/foaf/0.1/knows"), BNode.new); g << t) # => results in the triple being added to g; returns an array of g's triples
+    #
+    # @param [Triple] t the triple to be added to the graph
+    #
+    # ==== Returns
+    # @return [Array] An array of the triples (leaky abstraction? consider returning the graph instead)
+    #
+    # @author Tom Morris
+    def << (triple)
+  #    self.add_triple(s, p, o)
+      @triples += [ triple ]
+    end
+    
+    ## 
+    # Exports the graph to RDF in N-Triples form.
+    #
+    # ==== Example
+    #   g = Graph.new; g.add_triple(BNode.new, URIRef.new("http://xmlns.com/foaf/0.1/knows"), BNode.new); g.to_ntriples  # => returns a string of the graph in N-Triples form
+    #
+    # ==== Returns
+    # @return [String] The graph in N-Triples.
+    #
+    # @author Tom Morris
+
+    def to_ntriples
+      @triples.collect do |t|
+        t.to_ntriples
+      end * "\n" + "\n"
+    end
+    
+    # Output graph using to_ntriples
+    def to_s; self.to_ntriples; end
+
+    # Dump model to RDF/XML
+    def to_rdfxml
+      replace_text = {}
+      rdfxml = ""
+      xml = builder = Builder::XmlMarkup.new(:target => rdfxml, :indent => 2)
+
+      extended_bindings = nsbinding.merge(
+        "rdf"   => RDF_NS,
+        "rdfs"  => RDFS_NS,
+        "xhv"   => XHV_NS,
+        "xml"   => XML_NS
+      )
+      rdf_attrs = extended_bindings.values.inject({}) { |hash, ns| hash.merge(ns.xmlns_attr => ns.uri.to_s)}
+      uri_bindings = extended_bindings.values.inject({}) { |hash, ns| hash.merge(ns.uri.to_s => ns.short)}
+      xml.instruct!
+      xml.rdf(:RDF, rdf_attrs) do
+        # Add statements for each subject
+        subjects.each do |s|
+          xml.rdf(:Description, (s.is_a?(BNode) ? "rdf:nodeID" : "rdf:about") => s) do
+            each_with_subject(s) do |triple|
+              xml_args = triple.object.xml_args
+              if triple.object.is_a?(Literal) && triple.object.xmlliteral?
+                replace_text["__replace_with_#{triple.object.object_id}__"] = xml_args[0]
+                xml_args[0] = "__replace_with_#{triple.object.object_id}__"
+              end
+              xml.tag!(triple.predicate.to_qname(uri_bindings), *xml_args)
+            end
+          end
+        end
+      end
+
+      # Perform literal substitutions
+      replace_text.each_pair do |match, value|
+        rdfxml.sub!(match, value)
+      end
+      
+      rdfxml
+    end
+    
+    ## 
+    # Creates a new namespace given a URI and the short name and binds it to the graph.
+    #
+    # ==== Example
+    #   g = Graph.new; g.namespace("http://xmlns.com/foaf/0.1/", "foaf") # => binds the Foaf namespace to g
+    #
+    # @param [String] uri the URI of the namespace
+    # @param [String] short the short name of the namespace
+    #
+    # ==== Returns
+    # @return [Namespace] The newly created namespace.
+    #
+    # @raise [Error] Checks validity of the desired shortname and raises if it is incorrect.
+    # @raise [Error] Checks that the newly created Namespace is of type Namespace and raises if it is incorrect.
+    # @author Tom Morris
+
+    def namespace(uri, short)
+      self.bind Namespace.new(uri, short)
+    end
+
+    def bind(namespace)
+      if namespace.class == Namespace
+        @nsbinding["#{namespace.short}"] = namespace
+      else
+        raise GraphException, "Can't bind #{namespace.inspect} as namespace"
+      end
+    end
+
+    def has_bnode_identifier?(bnodeid)
+      temp_bnode = BNode.new(bnodeid)
+      returnval = false
+      @triples.each { |triple|
+        if triple.subject.eql?(temp_bnode)
+          returnval = true
+          break
+        end
+        if triple.object.eql?(temp_bnode)
+          returnval = true
+          break
+        end
+      }
+      return returnval
+    end
+
+    def get_bnode_by_identifier(bnodeid)
+      temp_bnode = BNode.new(bnodeid)
+      each do |triple|
+        if triple.subject == temp_bnode
+          return triple.subject
+        end
+        if triple.object == temp_bnode
+          return triple.object
+        end
+      end
+      return false
+    end
+    
+    def get_by_type(object)
+      out = []
+      each do |t|
+        next unless t.is_type?
+        next unless case object
+                    when String
+                      object == t.object.to_s
+                    when Regexp
+                      object.match(t.object.to_s)
+                    else
+                      object == t.object
+                    end
+        out << t.subject
+      end
+      return out
+    end
+    
+    def join(graph)
+      if graph.class == Graph
+        graph.each { |t| 
+          self << t
+        }
+      else
+        raise GraphException, "join requires you provide a graph object"
+      end
+    end
+
+  end
+end