hscrubber 0.0.1

data/.document

@@ -0,0 +1,4 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+LICENSE

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*

data/CHANGES.md

@@ -0,0 +1,8 @@
+Changes in hscrubber
+====================
+
+Changes in hscrubber reha filter releases are listed here.
+
+v0.0.1
+------
+- Scrubs an HTML code with a reha configuration filter

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in priehlazx.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Malo Skrylevo
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,70 @@
+# HScrubber
+
+HScrubber есть движокъ для прорѣшиванія HTML-документа. Онъ позволяетъ процѣдить содержимое входного потока очистивъ его отъ ненужныхъ предмѣтовъ на основѣ рѣхи, являющейся YAML-документомъ, по опредѣлённымъ правиламъ состаленнымъ.
+
+HScrubber is HTML reha engine, and it allows filtering an input flow according to the special reha template that is formed as YAML-document.
+
+# Рѣха (Reha)
+## Объясненіе рѣхи (Description of reha filter)
+
+Рѣха задаётся въ видѣ YAML-документа. На самомъ верхнемъ уровнѣ описываются мѣты (HTML tags), допустимыя въ выходномъ документѣ. Слѣдующій уровень задаётъ допустимыя свойства (attributes) для опредѣлённой мѣты, а также ключи, управляющія мѣтою и её содержимымъ. Возможныя ключи и их значенія суть такія:
+ * '%' содержимое мѣты будетъ очищено, если оно подпадаетъ подъ заданное въ значеніи ключа правило;
+ * '-' мѣта удаляется въ томъ случаѣ, если её содержимое подпадаетъ подъ заданное въ значеніи ключа правило;
+ * '^' содержимое мѣты добавляется къ содержимому родителькой мѣты въ томъ случаѣ, если содержимое сей мѣты подпадаетъ подъ правило, или если правило не задано.
+Ключи здѣ расположены въ порядкѣ первичности ихъ провѣрки. Каджый изъ нихъ обязательно предваряется символомъ '@'
+
+Reha is set up as an YAML-document. The allowed in an output flow HTML tags is described at the top level of the document. The following level described allowed attributes of the specified tag, and also rule keys that controls the tag and ots containment. The keys, and its values are the following:
+ * '%' declares that the containment of the tag will be cleaned up, if it matches to the specified rule;
+ * '-' a tag will be removed, if its containment matches to the specified rule;
+ * '^' containment of a tag will be added to containment of the parent tag, if containment of the tag matches to the specified rule, or if the rule isn't defined.
+The keys are ranged according to priority their analysing. The '@' symbol necessarily outruns each of the keys.
+
+## Примѣръ (Sample)
+Примѣрный шаблонъ файла рѣхи представленъ нижѣ:
+Sample reha template is described as follows:
+
+    ---
+    html:
+    body:
+    p:
+    i:
+      @-: ^[.,:;!?\s]*$
+    font:
+      face:
+      size:
+      @-: ^\s+$
+      @%: ^[.,:!?#]+$
+    span:
+      @^:
+      @-: ^[.,:;!?\s]*$
+    
+Поясненія:
+ * Мѣта 'i' не имѣетъ допустимыхъ ключей, и они будутъ удалены изъ входного потока. Въ случаѣ, если содержимое мѣты удовлѣтворяетъ правилу удаления, на выходѣ сія мѣта будетъ отсутствовать.
+  Примѣры:
+  * <i id="i_id">Text</i> -> <i>Text</i>
+  * <font>Text<i>?</i></font> -> <font>Text</font>
+
+ * Допустимыми ключами для мѣты 'font' являются 'face' и 'size'. Въ случаѣ, если содержимое мѣты удовлѣтворяетъ правилу удаления, на выходѣ сія мѣта будетъ отсутствовать, а если правилу очищенія, то её содержимое станетъ порожнимъ.
+  Примѣры:
+  * <font size="5" color="blue">Text</font> -> <font size="5">Text</font>
+  * <i>Text<font>  </font></i> -> <i>Text</i>
+  * <i>Text<font>??</font></i> -> <i>Text<font></font></i>
+
+ * Допустимыя ключи для мѣты 'span' отсутствуютъ, и въ случаѣ ихъ обнаруженія въ входномъ потокѣ они будутъ вырѣзаны изъ него. Если содержимое мѣты удовлѣтворяетъ правилу удаления, на выходѣ сія мѣта будетъ отсутствовать какъ таковая. Въ остальныхъ же случаяхъ её содержимое будетъ добавлено къ мѣтѣ родительской.
+  Примѣры:
+  * <span id="span_id">Text</span> -> <span>Text</span>
+  * <i>Text<span>?</span></i> -> <i>Text</i>
+
+Descriptions:
+ * Tag 'i' hasn't allowable attributes, so them will be removed from an output stream. In case, the tag containment meets a remove rule, the tag will be absent in the output;
+ * Allowable attributes for the 'font' tag are 'face', and 'size'. In case, if the tag containment meets a remove rule, the tag will be absent in the output, and if meets a cleanup rule, the containment will be purged;
+ * Tag 'span' hasn't allowable attributes, so them will be removed from an output stream. In case, the tag containment meets a remove rule, the tag will be absent in the output as it is. In other cases, its containment will be added to a parent tag.
+
+# Права (Copyright)
+
+Авторскія и исключительныя права (а) 2011 Малъ Скрылевъ
+Зри LICENSE за подробностями.
+
+Copyright (c) 2011 Malo Skrylevo
+See LICENSE for details.
+

data/Rakefile

@@ -0,0 +1,3 @@
+require 'rubygems'
+require 'bundler'
+Bundler::GemHelper.install_tasks

data/bin/hscrub

@@ -0,0 +1,78 @@
+#!/usr/bin/ruby -KU
+#<Encoding:UTF-8>
+
+require 'optparse'
+# check if hscrubber is available as not-a-gem
+begin
+  $: << '../lib' << './lib'
+  require 'hscrubber'
+rescue LoadError
+  begin require 'rubygems'; rescue LoadError; end
+  retry
+end
+
+begin
+    dest = File.expand_path(File.dirname($0))
+    content = ofile = nil
+    reha = '.реха.yml'
+
+    ARGV.options do |o|
+    script_name = File.basename($0)
+
+        o.set_summary_indent('  ')
+        o.banner =    "Usage: #{script_name} [OPTIONS] files"
+        o.separator   ""
+        o.separator   "Mandatory arguments to long options are mandatory for " +
+                "short options too."
+
+        o.on("-o", "--output-target=val", String,
+                "Output file or folder to store a result")  { |i| ofile = i }
+        o.on("-r", "--reha-filter-config=val", String,
+                "Reha filter configuration file")	    { |i| reha = i }
+
+        o.separator ""
+
+        o.on_tail("-h", "--help", "Show this help message") { $stderr.puts o; exit }
+
+        o.parse!
+
+    end
+
+    $hs = HScrubber.new( IO.read(reha) )
+
+    def scrub(content, of)
+      content = content.gsub(/\r/, '')
+      of.puts $hs.scrub_html(content)
+    end
+
+    if ofile
+      if ARGV.empty?
+	ofile = File.join(ofile, 'stdin') if File.exist?(ofile) and File.directory?(ofile)
+	File.open(ofile, 'w+') do |of| scrub( $stdin.read, of ) end
+      else
+	if File.exist?(ofile) and File.directory?(ofile)
+	  ARGV.each do |fn|
+	    content = IO.read(fn)
+	    File.open(File.join(ofile, File.basename(fn)), 'w+') do |of|
+	      scrub( content, of )
+	    end
+	  end
+	else
+	  File.open(ofile, 'w+') do |of|
+	    ARGV.each do |fn| scrub( IO.read(fn), of ) end
+	  end
+	end
+      end
+    else
+      if ARGV.empty?
+	scrub( $stdin.read, $stdout )
+      else
+	ARGV.each do |fn| scrub( IO.read(fn), $stdout ) end
+      end
+    end
+rescue
+    puts $!.to_s + "\n\t#{$@.join("\n\t")}"
+    exit 1
+end
+
+

data/hscrubber.gemspec

@@ -0,0 +1,29 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "hscrubber/version"
+
+Gem::Specification.new do |s|
+  s.name        = "hscrubber"
+  s.version     = HScrubber::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.authors      = [ 'Малъ Скрылёвъ (Malo Skrylevo)' ]
+  s.email        = [ '3aHyga@gmail.com' ]
+  s.homepage     = 'https://github.com/3aHyga/hscrubber'
+  s.summary      = 'hscrubber is HTML scrubber'
+  s.description  = 'hscrubber is HTML scrubber based on a HTML reha filter'
+
+  s.executables  = [ 'hscrub' ]
+  s.rubyforge_project = "hscrubber"
+
+  s.required_rubygems_version = '>= 1.6.0'
+
+  s.add_dependency 'hpricot', ">= 0.8.4"
+
+  s.add_development_dependency("bundler", ">= 1.0.0")
+  s.add_development_dependency("rspec", "~> 2.0.1")
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+end

data/lib/hscrubber/version.rb

@@ -0,0 +1,3 @@
+class HScrubber
+  VERSION = "0.0.1"
+end

data/lib/hscrubber.rb

@@ -0,0 +1,243 @@
+#!/usr/bin/ruby -KU
+# encoding: utf-8
+
+require 'yaml'
+require 'hpricot'
+require 'hscrubber/version'
+
+class HScrubber
+  def self.fix(str)
+      str.unpack('C*').pack('U*') # Workaround to fix HPricot error
+  end
+
+  def self.scrub_special(elem)
+      elem.each_child do |sub|
+	  if sub.class == Hpricot::Text
+	      sub.content = sub.content.gsub(/\x1F+/x, '')
+	  end
+      end
+      false
+  end
+
+  def self.scrub_follower(elem)
+      chnaged = false
+      if elem.children and elem.children.size == 1
+	  sub = elem.children[0]
+	  if sub.class == Hpricot::Elem and elem.name == sub.name
+	      html = sub.inner_html
+	      if elem.raw_attributes
+		  if sub.raw_attributes
+		      elem.raw_attributes.merge! sub.raw_attributes
+		  end
+	      else
+		  elem.raw_attributes = sub.raw_attributes
+	      end
+	      elem.children.delete(sub)
+	      elem.inner_html += html
+	      changed = true
+	  end
+      end
+      changed
+  end
+
+  def self.scrub_children(elem, verility)
+      changed = false
+      old = nil
+      elem.children.delete_if do |sub|
+	  if sub.class == Hpricot::Elem and sub.parent == elem
+	      self.scrub_elem(sub, verility)
+	      if old and old.name == sub.name and
+		      (old.raw_attributes.class == sub.raw_attributes.class and
+		      old.raw_attributes.class == NilClass or
+		      (old.raw_attributes.class == Hash and
+		      old.raw_attributes == sub.raw_attributes))
+		  sub_ch = sub.children
+		  idx = elem.children.index(sub)
+		  
+		  sub_ch.each do |x| x.parent = old end
+		  if not old.children
+		      old.children = sub_ch
+		  elsif old.children.empty?
+		      old.children.replace sub_ch
+		  else
+		      old.children.concat sub_ch
+		  end
+		  changed = true
+		  true
+	      elsif sub.children and sub.children.size != 0
+		  idx = elem.children.index(sub)
+		  old = sub
+		  false
+	      else
+		  idx = elem.children.index(sub)
+		  changed = true
+		  true
+	      end
+	  else
+	      old = nil
+	      false
+	  end
+      end if elem.children
+
+      changed
+  end
+
+  def self.scrub_remove_by_rule(elem, verility)
+      changed = nil
+      repeat = true
+
+      res = []
+
+      if elem.children and not elem.children.empty?
+	  elem.children.each do |sub|
+	  repeat = false
+	  sub_ch = nil
+	  case sub.class.to_s
+	  when 'Hpricot::Elem'
+	      if verility.key? sub.name
+		  new_attrs = {}
+		  strip_tags = []
+		  verility[sub.name].keys.sort do |x,y|
+		      x == '@-' ? -1 : y == '@-' ? 1 : x <=> y
+		  end.each do |key|
+		      value = verility[sub.name][key]
+		      # TODO match value to as regexp to attr value
+		      case key
+		      when '@-'
+			  # delete elem if match to re
+			  inner = sub.inner_html.gsub(/(\r\n|\n)/,' ')
+			  if self.fix(inner) =~ /#{value}/
+			      if elem.children.index(sub)
+				  repeat = true; changed = true
+			      end
+			  end
+		      when '@^'
+			  inner = sub.inner_html.gsub(/(\r\n|\n)/,' ')
+			  unless value and self.fix(inner) !~ /#{value}/
+			      sub_ch = sub.children
+			      idx = elem.children.index(sub)
+			      if idx and not sub_ch.empty?
+				  sub_ch.each do |x| x.parent = elem end
+				  elem.children[idx] = sub_ch
+			      else
+				  repeat = true; changed = true
+			      end
+			  end
+		      when '@%'
+			  # replace elem if match to re
+			  inner = sub.inner_html.gsub(/(\r\n|\n)/,' ')
+			  if self.fix(inner) =~ /#{value}/
+			      new = Hpricot::Text.new("\x1F")
+			      new.parent = sub
+			      sub.children.replace [ new ]
+			      changed = true
+			  end
+		      else
+			  attr = sub.get_attribute(key)
+			  new_attrs[key] = attr if attr
+		      end
+		  end if verility[sub.name]
+		  if sub.raw_attributes != new_attrs
+		      sub.raw_attributes = new_attrs
+		      changed = true
+		  end
+	      else
+		  repeat = true; changed = true
+	      end
+	  when 'Hpricot::Text'
+	  else
+	      repeat = true; changed = true
+	  end if sub.parent == elem
+	  if not repeat
+	      if sub_ch
+		  res.concat sub_ch
+	      else
+		  res << sub
+	      end
+	  end
+	  end
+
+	  elem.children.replace res
+      end
+
+      changed
+  end
+
+  def self.scrub_replace_by_rule(elem, verility)
+      changed = nil
+      repeat = true
+
+      res = []
+
+      elem.children.each do |sub|
+	  repeat = false
+	  sub_ch = nil
+	  case sub.class.to_s
+	  when 'Hpricot::Elem'
+	      if sub.children and verility.key? sub.name
+		  strip_tags = []
+		  verility[sub.name].each do |key, value|
+		      # TODO match value to as regexp to attr value
+		      case key
+		      when '@%'
+			  # replace elem if match to re
+			  inner = sub.inner_html.gsub(/(\r\n|\n)/,' ')
+			  if self.fix(inner) =~ /#{value}/
+			      new = Hpricot::Text.new("\x1F")
+			      new.parent = sub
+			      sub.children.replace [ new ]
+			      changed = true
+			  end
+		      end
+		  end if verility[sub.name]
+	      else
+		  repeat = true; changed = true
+	      end
+	  when 'Hpricot::Text'
+	  else
+	      repeat = true; changed = true
+	  end if sub.parent == elem
+	  if not repeat
+	      if sub_ch
+		  res.concat sub_ch
+	      else
+		  res << sub
+	      end
+	  end
+      end if elem.children and not elem.children.empty?
+
+      elem.children.replace res
+
+      changed
+  end
+
+  def self.scrub_elem(elem, verility)
+      while (
+	  self.scrub_children(elem, verility) ||
+	  self.scrub_remove_by_rule(elem, verility) ||
+	  self.scrub_follower(elem) ||
+	  self.scrub_special(elem))
+      end
+  end
+
+  def self.scrub_html(content, реха)
+      return content unless реха
+      реха = YAML.load( StringIO.new реха ) if реха.class == String
+      doc = Hpricot(content)
+      self.scrub_elem(doc, реха)
+      doc.inner_html
+  end
+
+  def initialize(реха)
+      @реха = YAML.load( StringIO.new реха ) if реха.class == String
+  end
+
+  def scrub_html(content)
+      return content unless @реха
+      doc = Hpricot(content)
+      self.class.scrub_elem(doc, @реха)
+      doc.inner_html
+  end
+
+end
+

metadata

@@ -0,0 +1,129 @@
+--- !ruby/object:Gem::Specification 
+name: hscrubber
+version: !ruby/object:Gem::Version 
+  hash: 29
+  prerelease: 
+  segments: 
+  - 0
+  - 0
+  - 1
+  version: 0.0.1
+platform: ruby
+authors: 
+- !binary |
+  0JzQsNC70Yog0KHQutGA0YvQu9GR0LLRiiAoTWFsbyBTa3J5bGV2byk=
+
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-04-05 00:00:00 +04:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: hpricot
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 55
+        segments: 
+        - 0
+        - 8
+        - 4
+        version: 0.8.4
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: bundler
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 23
+        segments: 
+        - 1
+        - 0
+        - 0
+        version: 1.0.0
+  type: :development
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 13
+        segments: 
+        - 2
+        - 0
+        - 1
+        version: 2.0.1
+  type: :development
+  version_requirements: *id003
+description: hscrubber is HTML scrubber based on a HTML reha filter
+email: 
+- 3aHyga@gmail.com
+executables: 
+- hscrub
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- .document
+- .gitignore
+- CHANGES.md
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/hscrub
+- hscrubber.gemspec
+- lib/hscrubber.rb
+- lib/hscrubber/version.rb
+has_rdoc: true
+homepage: https://github.com/3aHyga/hscrubber
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 15
+      segments: 
+      - 1
+      - 6
+      - 0
+      version: 1.6.0
+requirements: []
+
+rubyforge_project: hscrubber
+rubygems_version: 1.6.2
+signing_key: 
+specification_version: 3
+summary: hscrubber is HTML scrubber
+test_files: []
+