jruby-boilerpipe 0.1.0 → 0.2.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA256:
-  metadata.gz: 03a8f58cd513e79cfe0b2de46a694fc174311a51ab26bc283b1eb0b6095623a9
-  data.tar.gz: 8ac91071568e525d647c92b01ddda2caabb6c5bec0e0e3201adb3c93eee3d77f
+SHA1:
+  metadata.gz: 1a4b3ef0fc314be4616e53a70dcaa6cb689542f3
+  data.tar.gz: 94f1c2a283a255bf7c7e5aaa941807d4ceb9cf94
 SHA512:
-  metadata.gz: fabda6c210c3d56c8785b4e1369e65682644cdda3b6670124128fe1333c08d65b67cbe72ab254060621d467692263b0ac2c195d761d1400d4e9ac7c7679b96dc
-  data.tar.gz: 29acf923aea8178af93e0e195af72d133b6eae92cb9ce45b5e0e9454d0af9c27343c2096533275ca746fb2105d71e575e3f4c100a1323ed6aa2d5ac4842b1bf1
+  metadata.gz: 3720fab10515e2a6794bfe79042789cd4df4299192742498035001d5cb135555eb7b26576e6ce212b104ac23a5c4e975cbcc4dacd895a767bb98a6947d06add3
+  data.tar.gz: 9eebfc825da6725cdf8e253b5bbfe80097b1b7913f30f6465a0c2963e14a1a0444777f78887d538c60d9801b98efab6dfbaa4df6d5f5c4817f297b37977ad43d

data/README.md

@@ -1,19 +1,13 @@
 # Boilerpipe
 
-I saw other gems wrapping boilerpipe but they seemed to be outdated, hit the free api or I couldn't get them to work because of dependency issues. I went directly to the original author's github https://github.com/kohlschutter/boilerpipe and forked that code base here https://github.com/gregors/boilerpipe .  I made one notible change that is to add a user agent so article extractor wouldn't return a 403 for a variety of web servers. I compiled all dependencies into the jar using the maven-assembly-plugin using Java 8.
-
-Extractors added so far:
-
-* ArticleExtractor
-* DefaultExtractor
-
+I saw other gems wrapping boilerpipe but they seemed to be outdated, hit the free api or I couldn't get them to work because of dependency issues. I went directly to the original author's github https://github.com/kohlschutter/boilerpipe and forked that code base here https://github.com/gregors/boilerpipe .  I made one notible change that is to add a user agent so article extractor wouldn't return a 403 for a vareity of web servers. I compiled all dependencies into the jar using the maven-assembly-plugin using Java 8. Until the original author releases a proper 2.0 release I'll be appending rcx to reflect the latest snapshots.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'jruby-boilerpipe', require: 'boilerpipe'
+gem 'boilerpipe'
 ```
 
 And then execute:
@@ -22,22 +16,16 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install jruby-boilerpipe
+    $ gem install boilerpipe
 
 ## Usage
-    You can feed Boilerpipe:ArticleExractor either a valid url or html content.
-
-    jruby-9.1.7.0 :001 >  require 'boilerpipe'
-    => true
-    jruby-9.1.7.0 :003 > Boilerpipe::ArticleExtractor.text("https://github.com/jruby/jruby/wiki/AboutJRuby")
-    => "Clone this wiki locally\nAbout JRuby\nJRuby is a 100% Java implementation of the Ruby programming language. It is Ruby for the JVM.\nJRuby provides a complete set of core \"builtin\" classes and syntax for the Ruby language, as well as most of the Ruby Standard Libraries. The standard libraries are mostly Ruby's own complement of .rb files, but a few that depend on C language-based extensions have been reimplemented. Some are still missing, but we hope to implement as many as is feasible.\nSee Differences Between MRI And JRuby for more information on potential incompatibilities between JRuby and the C implementation of Ruby.\nDevelopment Team\nJRuby's current core development team consists of nine developers:\nCharles Oliver Nutter (Red Hat) aka headius\nThomas Enebo (Red Hat) aka enebo\nNick Sieger (LivingSocial)\nThere are also many past contributors who still help out from time to time:\nOla Bini (Thoughtworks)\nMany more...check out the JRuby commit logs!\nLinks\n"
-
-     jruby-9.1.7.0 :002 > require 'open-uri'
-    => true
-    jruby-9.1.7.0 :004 > contents = open("https://github.com/jruby/jruby/wiki/AboutJRuby").read
-    jruby-9.1.7.0 :005 > Boilerpipe::ArticleExtractor.text(contents)
-    => "Clone this wiki locally\nAbout JRuby\nJRuby is a 100% Java implementation of the Ruby programming language. It is Ruby for the JVM.\nJRuby provides a complete set of core \"builtin\" classes and syntax for the Ruby language, as well as most of the Ruby Standard Libraries. The standard libraries are mostly Ruby's own complement of .rb files, but a few that depend on C language-based extensions have been reimplemented. Some are still missing, but we hope to implement as many as is feasible.\nSee Differences Between MRI And JRuby for more information on potential incompatibilities between JRuby and the C implementation of Ruby.\nDevelopment Team\nJRuby's current core development team consists of nine developers:\nCharles Oliver Nutter (Red Hat) aka headius\nThomas Enebo (Red Hat) aka enebo\nNick Sieger (LivingSocial)\nThere are also many past contributors who still help out from time to time:\nOla Bini (Thoughtworks)\nMany more...check out the JRuby commit logs!\nLinks\n"
-    
+
+    jruby-1.7.24 :001 > require 'boilerpipe'
+     => true 
+    jruby-1.7.24 :002 > Boilerpipe::ArticleExtractor.text("https://github.com/jruby/jruby/wiki/AboutJRuby")
+     => "Clone this wiki locally\nAbout JRuby\nJRuby is a 100% Java implementation of the Ruby programming language. It is Ruby for the JVM.\nJRuby provides a complete set of core \"builtin\" classes and syntax for the Ruby language, as well as most of the Ruby Standard Libraries. The standard libraries are mostly Ruby's own complement of .rb files, but a few that depend on C language-based extensions have been reimplemented. Some are still missing, but we hope to implement as many as is feasible.\nSee Differences Between MRI And JRuby for more information on potential incompatibilities between JRuby and the C implementation of Ruby.\nDevelopment Team\nJRuby's current core development team consists of nine developers:\nCharles Oliver Nutter (Red Hat) aka headius\nThomas Enebo (Red Hat) aka enebo\nNick Sieger (LivingSocial)\n" 
+
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -46,5 +34,5 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/gregors/boilerpipe.
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/boilerpipe.
 

data/lib/boilerpipe.rb

@@ -1,8 +1,18 @@
-require_relative 'boilerpipe-common-2.0-SNAPSHOT-jar-with-dependencies.jar'
 require 'boilerpipe/version'
-require 'boilerpipe/sax/boilerpipe_html_parser'
-require 'boilerpipe/document/document'
-require 'boilerpipe/extractors/article_extractor'
-require 'boilerpipe/extractors/default_extractor'
-require 'boilerpipe/filters/filters'
-require 'boilerpipe/labels/labels'
+require_relative 'boilerpipe-common-2.0-SNAPSHOT-jar-with-dependencies.jar'
+
+module Boilerpipe
+  java_import 'com.kohlschutter.boilerpipe.extractors.ArticleExtractor'
+  java_import java.net.URL
+
+  class ArticleExtractor
+    def  self.get_text(s)
+      url = URL.new(s)
+      ArticleExtractor::INSTANCE.get_text(url)
+    end
+
+    class <<self  
+      alias_method :text, :get_text
+    end 
+  end
+end

data/lib/boilerpipe/version.rb

@@ -1,3 +1,3 @@
 module Boilerpipe
-  VERSION = '0.1.0'
+  VERSION = '0.2.0.rc1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jruby-boilerpipe
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0.rc1
 platform: ruby
 authors:
 - Gregory Ostermayr
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2017-09-07 00:00:00.000000000 Z
+date: 2016-03-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -69,12 +69,6 @@ files:
 - jruby-boilerpipe.gemspec
 - lib/boilerpipe-common-2.0-SNAPSHOT-jar-with-dependencies.jar
 - lib/boilerpipe.rb
-- lib/boilerpipe/document/document.rb
-- lib/boilerpipe/extractors/article_extractor.rb
-- lib/boilerpipe/extractors/default_extractor.rb
-- lib/boilerpipe/filters/filters.rb
-- lib/boilerpipe/labels/labels.rb
-- lib/boilerpipe/sax/boilerpipe_html_parser.rb
 - lib/boilerpipe/version.rb
 homepage: https://github.com/gregors/jruby-boilerpipe
 licenses: []
@@ -90,12 +84,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project:
-rubygems_version: 2.6.11
+rubygems_version: 2.4.8
 signing_key:
 specification_version: 4
 summary: Ruby wrapper around boilerpipe java library

data/lib/boilerpipe/document/document.rb

@@ -1,16 +0,0 @@
-module Boilerpipe
-  module Document
-    java_import 'com.kohlschutter.boilerpipe.document.TextDocument'
-    java_import 'com.kohlschutter.boilerpipe.document.TextBlock'
-
-    class TextBlock
-      # Adding a mapping from ruby symbols to the format string used on the java side
-      # e.g. de.l3s.boilerpipe/INDICATES_END_OF_TEXT is not the same as INDICATES_END_OF_TEXT
-      # This is only for when we do TextBlock#has_label? from jruby
-      def has_label?(l)
-        l = "de.l3s.boilerpipe/#{l.to_s}" if l.is_a?(Symbol)
-        self.hasLabel(l)
-      end
-    end
-  end
-end

data/lib/boilerpipe/extractors/article_extractor.rb

@@ -1,22 +0,0 @@
-module Boilerpipe
-  java_import 'com.kohlschutter.boilerpipe.extractors.ArticleExtractor'
-  java_import java.net.URL
-
-  class ArticleExtractor
-    def  self.get_text(s)
-      url = nil
-
-      begin
-        url = Java::JavaNet::URL.new(s)
-      rescue Java::JavaNet::MalformedURLException => e
-        # not a URL
-      end
-      input = url ? url : s
-      ArticleExtractor::INSTANCE.get_text(input)
-    end
-
-    class <<self  
-      alias_method :text, :get_text
-    end 
-  end
-end

data/lib/boilerpipe/extractors/default_extractor.rb

@@ -1,13 +0,0 @@
-module Boilerpipe
-  java_import 'com.kohlschutter.boilerpipe.extractors.DefaultExtractor'
-
-  class DefaultExtractor
-    def  self.get_text(s)
-      DefaultExtractor::INSTANCE.get_text s
-    end
-
-    class <<self
-      alias_method :text, :get_text
-    end
-  end
-end

data/lib/boilerpipe/filters/filters.rb

@@ -1,53 +0,0 @@
-module Boilerpipe
-  module Filters
-    java_import 'com.kohlschutter.boilerpipe.filters.english.IgnoreBlocksAfterContentFilter'
-    java_import 'com.kohlschutter.boilerpipe.filters.english.TerminatingBlocksFinder'
-    java_import 'com.kohlschutter.boilerpipe.filters.english.NumWordsRulesClassifier'
-    java_import 'com.kohlschutter.boilerpipe.filters.english.HeuristicFilterBase'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.BlockProximityFusion'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.DocumentTitleMatchClassifier'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.ExpandTitleToContentFilter'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.KeepLargestBlockFilter'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.LargeBlockSameTagLevelToContentFilter'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.ListAtEndFilter'
-    java_import 'com.kohlschutter.boilerpipe.filters.heuristics.TrailingHeadlineToBoilerplateFilter'
-    java_import 'com.kohlschutter.boilerpipe.filters.simple.BoilerplateBlockFilter'
-
-    class ExpandTitleToContentFilter
-      def self.process(doc)
-        new.process(doc)
-      end
-    end
-
-    class IgnoreBlocksAfterContentFilter
-      def self.process(doc)
-        DEFAULT_INSTANCE.process(doc)
-      end
-    end
-
-    class LargeBlockSameTagLevelToContentFilter
-      def self.process(doc)
-        INSTANCE.process(doc)
-      end
-    end
-
-    class TerminatingBlocksFinder
-      def self.process(doc)
-        new.process(doc)
-      end
-    end
-
-    class TrailingHeadlineToBoilerplateFilter
-      def self.process(doc)
-        new.process(doc)
-      end
-    end
-
-    class NumWordsRulesClassifier
-      def self.process(doc)
-        new.process(doc)
-      end
-    end
-
-  end
-end

data/lib/boilerpipe/labels/labels.rb

@@ -1,3 +0,0 @@
-module Boilerpipe::Labels
-  java_import 'com.kohlschutter.boilerpipe.labels.DefaultLabels'
-end

data/lib/boilerpipe/sax/boilerpipe_html_parser.rb

@@ -1,17 +0,0 @@
-module Boilerpipe
-  module SAX
-    java_import 'com.kohlschutter.boilerpipe.sax.BoilerpipeHTMLParser'
-    java_import 'org.xml.sax.InputSource'
-    java_import java.io.StringReader
-
-    class BoilerpipeHTMLParser
-      def self.parse(text)
-        parser = BoilerpipeHTMLParser.new
-        string_reader = StringReader.new(text)
-        is = InputSource.new(string_reader)
-        parser.parse(is)
-        parser.to_text_document
-      end
-    end
-  end
-end