jruby-boilerpipe 0.2.0.rc2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 0ffb112f79745540183e4e4c3c4b8ea1948aeffa
-  data.tar.gz: 91b3e83d47ff60a9ca37bac2810600b158d90f6e
+SHA256:
+  metadata.gz: 2b25faefd03910a9b1e5e88e733b35f373a7b71a241e8a742e43e0419080fd70
+  data.tar.gz: 21f97fe1fb05465f3a4ffe9751054e423854c6028a356152b42a0582d28fc260
 SHA512:
-  metadata.gz: d52bafba68cedd42c72c1a0d12cc5f65530968337a4f787bc8bb10a207d034437597d98613ccc51b58cddb7c7e3d4e61e73b1aca901b9c17a64ac5af3a7d443a
-  data.tar.gz: 2a79ea8aa79047ee60bd9b14cee607dc88fb7e3d3d033991e17ad797d2faf5d92ee7561ca35cdd4357b9515911de1d1463a9391f8482b6b47adc0abda90b55ad
+  metadata.gz: a8c7ba7de6d49ce2479121e5dbbf783ebbc8dfbfe7e17236a9523d1eed47cbac2f5d549597c7cb4ac1d34aa03e048efa349599cfe9d86bf5b9b5e316b11aff02
+  data.tar.gz: 0df70ee51cbce0b1299f8b24027e115bc6dc0a4824a4dcea0c25716077e93dbeb12da476f1681e69317c41fa3d7c6a42ebd4670bf1db22700381ce7c39d4f699

data/README.md

@@ -1,13 +1,19 @@
 # Boilerpipe
 
-I saw other gems wrapping boilerpipe but they seemed to be outdated, hit the free api or I couldn't get them to work because of dependency issues. I went directly to the original author's github https://github.com/kohlschutter/boilerpipe and forked that code base here https://github.com/gregors/boilerpipe .  I made one notible change that is to add a user agent so article extractor wouldn't return a 403 for a vareity of web servers. I compiled all dependencies into the jar using the maven-assembly-plugin using Java 8. Until the original author releases a proper 2.0 release I'll be appending rcx to reflect the latest snapshots.
+I saw other gems wrapping boilerpipe but they seemed to be outdated, hit the free api or I couldn't get them to work because of dependency issues. I went directly to the original author's github https://github.com/kohlschutter/boilerpipe and forked that code base here https://github.com/gregors/boilerpipe .  I made one notible change that is to add a user agent so article extractor wouldn't return a 403 for a variety of web servers. I compiled all dependencies into the jar using the maven-assembly-plugin using Java 8.
+
+Extractors added so far:
+
+* ArticleExtractor
+* DefaultExtractor
+
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'jruby-boilerpipe'
+gem 'jruby-boilerpipe', require: 'boilerpipe'
 ```
 
 And then execute:
@@ -16,16 +22,22 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install boilerpipe
+    $ gem install jruby-boilerpipe
 
 ## Usage
-
-    jruby-1.7.24 :001 > require 'boilerpipe'
-     => true 
-    jruby-1.7.24 :002 > Boilerpipe::ArticleExtractor.text("https://github.com/jruby/jruby/wiki/AboutJRuby")
-     => "Clone this wiki locally\nAbout JRuby\nJRuby is a 100% Java implementation of the Ruby programming language. It is Ruby for the JVM.\nJRuby provides a complete set of core \"builtin\" classes and syntax for the Ruby language, as well as most of the Ruby Standard Libraries. The standard libraries are mostly Ruby's own complement of .rb files, but a few that depend on C language-based extensions have been reimplemented. Some are still missing, but we hope to implement as many as is feasible.\nSee Differences Between MRI And JRuby for more information on potential incompatibilities between JRuby and the C implementation of Ruby.\nDevelopment Team\nJRuby's current core development team consists of nine developers:\nCharles Oliver Nutter (Red Hat) aka headius\nThomas Enebo (Red Hat) aka enebo\nNick Sieger (LivingSocial)\n" 
-
-
+    You can feed Boilerpipe:ArticleExractor either a valid url or html content.
+
+    jruby-9.1.7.0 :001 >  require 'boilerpipe'
+    => true
+    jruby-9.1.7.0 :003 > Boilerpipe::ArticleExtractor.text("https://github.com/jruby/jruby/wiki/AboutJRuby")
+    => "Clone this wiki locally\nAbout JRuby\nJRuby is a 100% Java implementation of the Ruby programming language. It is Ruby for the JVM.\nJRuby provides a complete set of core \"builtin\" classes and syntax for the Ruby language, as well as most of the Ruby Standard Libraries. The standard libraries are mostly Ruby's own complement of .rb files, but a few that depend on C language-based extensions have been reimplemented. Some are still missing, but we hope to implement as many as is feasible.\nSee Differences Between MRI And JRuby for more information on potential incompatibilities between JRuby and the C implementation of Ruby.\nDevelopment Team\nJRuby's current core development team consists of nine developers:\nCharles Oliver Nutter (Red Hat) aka headius\nThomas Enebo (Red Hat) aka enebo\nNick Sieger (LivingSocial)\nThere are also many past contributors who still help out from time to time:\nOla Bini (Thoughtworks)\nMany more...check out the JRuby commit logs!\nLinks\n"
+
+     jruby-9.1.7.0 :002 > require 'open-uri'
+    => true
+    jruby-9.1.7.0 :004 > contents = open("https://github.com/jruby/jruby/wiki/AboutJRuby").read
+    jruby-9.1.7.0 :005 > Boilerpipe::ArticleExtractor.text(contents)
+    => "Clone this wiki locally\nAbout JRuby\nJRuby is a 100% Java implementation of the Ruby programming language. It is Ruby for the JVM.\nJRuby provides a complete set of core \"builtin\" classes and syntax for the Ruby language, as well as most of the Ruby Standard Libraries. The standard libraries are mostly Ruby's own complement of .rb files, but a few that depend on C language-based extensions have been reimplemented. Some are still missing, but we hope to implement as many as is feasible.\nSee Differences Between MRI And JRuby for more information on potential incompatibilities between JRuby and the C implementation of Ruby.\nDevelopment Team\nJRuby's current core development team consists of nine developers:\nCharles Oliver Nutter (Red Hat) aka headius\nThomas Enebo (Red Hat) aka enebo\nNick Sieger (LivingSocial)\nThere are also many past contributors who still help out from time to time:\nOla Bini (Thoughtworks)\nMany more...check out the JRuby commit logs!\nLinks\n"
+    
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -34,5 +46,5 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/boilerpipe.
+Bug reports and pull requests are welcome on GitHub at https://github.com/gregors/boilerpipe.
 

data/jruby-boilerpipe.gemspec

@@ -21,4 +21,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "bundler", "~> 1.10"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec"
+  spec.add_development_dependency "rickshaw"
 end

data/lib/boilerpipe.rb

@@ -1,18 +1,9 @@
-require 'boilerpipe/version'
 require_relative 'boilerpipe-common-2.0-SNAPSHOT-jar-with-dependencies.jar'
-
-module Boilerpipe
-  java_import 'com.kohlschutter.boilerpipe.extractors.ArticleExtractor'
-  java_import java.net.URL
-
-  class ArticleExtractor
-    def  self.get_text(s)
-      url = URL.new(s)
-      ArticleExtractor::INSTANCE.get_text(url)
-    end
-
-    class <<self  
-      alias_method :text, :get_text
-    end 
-  end
-end
+require 'boilerpipe/version'
+require 'boilerpipe/sax/boilerpipe_html_parser'
+require 'boilerpipe/document/document'
+require 'boilerpipe/extractors/article_extractor'
+require 'boilerpipe/extractors/default_extractor'
+require 'boilerpipe/extractors/largest_content_extractor'
+require 'boilerpipe/filters/filters'
+require 'boilerpipe/labels/labels'

data/lib/boilerpipe/document/document.rb

@@ -0,0 +1,16 @@
+module Boilerpipe
+  module Document
+    java_import 'com.kohlschutter.boilerpipe.document.TextDocument'
+    java_import 'com.kohlschutter.boilerpipe.document.TextBlock'
+
+    class TextBlock
+      # Adding a mapping from ruby symbols to the format string used on the java side
+      # e.g. de.l3s.boilerpipe/INDICATES_END_OF_TEXT is not the same as INDICATES_END_OF_TEXT
+      # This is only for when we do TextBlock#has_label? from jruby
+      def has_label?(l)
+        l = "de.l3s.boilerpipe/#{l.to_s}" if l.is_a?(Symbol)
+        self.hasLabel(l)
+      end
+    end
+  end
+end

data/lib/boilerpipe/extractors/article_extractor.rb

@@ -0,0 +1,39 @@
+module Boilerpipe
+  java_import java.net.URL
+
+  module Extractors
+    class ArticleExtractor
+      java_import 'com.kohlschutter.boilerpipe.extractors.ArticleExtractor'
+
+      def self.process(doc)
+        ArticleExtractor::INSTANCE.process doc
+      end
+
+      def  self.get_text(s)
+        url = nil
+
+        begin
+          url = Java::JavaNet::URL.new(s)
+        rescue Java::JavaNet::MalformedURLException => e
+          # not a URL
+        end
+        input = url ? url : s
+        ArticleExtractor::INSTANCE.get_text(input)
+      end
+
+      class <<self
+        alias_method :text, :get_text
+      end
+    end
+  end
+
+  class ArticleExtractor
+    def self.get_text(s)
+      Extractors::ArticleExtractor.get_text s
+    end
+
+    class <<self
+      alias_method :text, :get_text
+    end
+  end
+end

data/lib/boilerpipe/extractors/default_extractor.rb

@@ -0,0 +1,13 @@
+module Boilerpipe::Extractors
+  class DefaultExtractor
+    java_import 'com.kohlschutter.boilerpipe.extractors.DefaultExtractor'
+
+    def  self.get_text(s)
+      DefaultExtractor::INSTANCE.get_text s
+    end
+
+    class <<self
+      alias_method :text, :get_text
+    end
+  end
+end

data/lib/boilerpipe/extractors/largest_content_extractor.rb

@@ -0,0 +1,13 @@
+module Boilerpipe::Extractors
+  class LargestContentExtractor
+    java_import 'com.kohlschutter.boilerpipe.extractors.LargestContentExtractor'
+
+    def  self.get_text(s)
+      LargestContentExtractor::INSTANCE.get_text s
+    end
+
+    class <<self
+      alias_method :text, :get_text
+    end
+  end
+end

data/lib/boilerpipe/filters/filters.rb

@@ -0,0 +1,75 @@
+module Boilerpipe
+  module Filters
+    java_import 'com.kohlschutter.boilerpipe.filters.english.DensityRulesClassifier'
+    java_import 'com.kohlschutter.boilerpipe.filters.english.IgnoreBlocksAfterContentFilter'
+    java_import 'com.kohlschutter.boilerpipe.filters.english.TerminatingBlocksFinder'
+    java_import 'com.kohlschutter.boilerpipe.filters.english.NumWordsRulesClassifier'
+    java_import 'com.kohlschutter.boilerpipe.filters.english.HeuristicFilterBase'
+
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.BlockProximityFusion'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.DocumentTitleMatchClassifier'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.ExpandTitleToContentFilter'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.KeepLargestBlockFilter'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.LargeBlockSameTagLevelToContentFilter'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.ListAtEndFilter'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.SimpleBlockFusionProcessor'
+    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.TrailingHeadlineToBoilerplateFilter'
+
+    java_import 'com.kohlschutter.boilerpipe.filters.simple.BoilerplateBlockFilter'
+
+    class DensityRulesClassifier
+      def self.process(doc)
+        new.process(doc)
+      end
+    end
+
+    class ExpandTitleToContentFilter
+      def self.process(doc)
+        new.process(doc)
+      end
+    end
+
+    class IgnoreBlocksAfterContentFilter
+      def self.process(doc)
+        DEFAULT_INSTANCE.process(doc)
+      end
+    end
+
+    class ListAtEndFilter
+      def self.process(doc)
+        INSTANCE.process(doc)
+      end
+    end
+
+    class LargeBlockSameTagLevelToContentFilter
+      def self.process(doc)
+        INSTANCE.process(doc)
+      end
+    end
+
+    class SimpleBlockFusionProcessor
+      def self.process(doc)
+        new.process(doc)
+      end
+    end
+
+    class TerminatingBlocksFinder
+      def self.process(doc)
+        new.process(doc)
+      end
+    end
+
+    class TrailingHeadlineToBoilerplateFilter
+      def self.process(doc)
+        new.process(doc)
+      end
+    end
+
+    class NumWordsRulesClassifier
+      def self.process(doc)
+        new.process(doc)
+      end
+    end
+
+  end
+end

data/lib/boilerpipe/labels/labels.rb

@@ -0,0 +1,3 @@
+module Boilerpipe::Labels
+  java_import 'com.kohlschutter.boilerpipe.labels.DefaultLabels'
+end

data/lib/boilerpipe/sax/boilerpipe_html_parser.rb

@@ -0,0 +1,17 @@
+module Boilerpipe
+  module SAX
+    java_import 'com.kohlschutter.boilerpipe.sax.BoilerpipeHTMLParser'
+    java_import 'org.xml.sax.InputSource'
+    java_import java.io.StringReader
+
+    class BoilerpipeHTMLParser
+      def self.parse(text)
+        parser = BoilerpipeHTMLParser.new
+        string_reader = StringReader.new(text)
+        is = InputSource.new(string_reader)
+        parser.parse(is)
+        parser.to_text_document
+      end
+    end
+  end
+end

data/lib/boilerpipe/version.rb

@@ -1,3 +1,3 @@
 module Boilerpipe
-  VERSION = '0.2.0.rc2'
+  VERSION = '0.2.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jruby-boilerpipe
 version: !ruby/object:Gem::Version
-  version: 0.2.0.rc2
+  version: 0.2.0
 platform: ruby
 authors:
 - Gregory Ostermayr
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2016-03-13 00:00:00.000000000 Z
+date: 2017-09-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: rickshaw
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Java8 compiled - latest boilerpipe-2.0-SNAPSHOT - including all dependencies
 email:
 - "<gregory.ostermayr@gmail.com>"
@@ -69,6 +83,13 @@ files:
 - jruby-boilerpipe.gemspec
 - lib/boilerpipe-common-2.0-SNAPSHOT-jar-with-dependencies.jar
 - lib/boilerpipe.rb
+- lib/boilerpipe/document/document.rb
+- lib/boilerpipe/extractors/article_extractor.rb
+- lib/boilerpipe/extractors/default_extractor.rb
+- lib/boilerpipe/extractors/largest_content_extractor.rb
+- lib/boilerpipe/filters/filters.rb
+- lib/boilerpipe/labels/labels.rb
+- lib/boilerpipe/sax/boilerpipe_html_parser.rb
 - lib/boilerpipe/version.rb
 homepage: https://github.com/gregors/jruby-boilerpipe
 licenses: []
@@ -84,12 +105,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubyforge_project:
-rubygems_version: 2.4.8
+rubygems_version: 2.6.11
 signing_key:
 specification_version: 4
 summary: Ruby wrapper around boilerpipe java library