proselytism 0.0.1 → 0.0.2

data/README.md

@@ -1,13 +1,14 @@
 # Proselytism
 
-Document converter, text and image extractor using OpenOffice headless server, pdf_tools and net_pbm
+Document converter, text and image extractor using OpenOffice headless server (JOD or PYOD converter), pdf_tools and net_pbm
+
+Handled formats for document conversion : odt, doc, rtf, sxw, docx, txt, html, htm, wps, pdf
 
 ## Note
 
-This gem has been originally written for as a RoR 3.2 engine running on Ruby 1.8.7.
-It should be framework agnostic and has been tested on Ubuntu and MacOSX.
+This gem has been originally written as a RoR 3.2 engine running on Ruby 1.8.7. 
 
-Due to its dependency to system_timer it doesn't work with ruby 1.9.x
+It is framework agnostic and has been tested on Ubuntu and MacOSX.
 
 ## Installation
 
@@ -16,28 +17,45 @@ Install the required external librairies :
     # aptitude install netpbm
     # aptitude install xpdf
     # aptitude install libreoffice
-
+    
 Add this line to your application's Gemfile:
 
-    gem 'proselytism', :git => "git://github.com/itkin/proselytism.git"
+    gem 'proselytism'
+
+Note : for ruby 1.9 use the branch 1.9
+
+    gem 'proselytism', :git => "git://github.com/itkin/proselytism.git", :branch => "1.9"
 
 And then execute:
 
     $ bundle
 
-Generate the config file or / and an initializer
+Configure the gem:
 
-    $ rails g proselytism:config
-    $ rails g proselytism:initializer
+  - With a YAML config file:
 
-As an engine, Proselytism automatically load and autoconfig with /config/proselytism.yml if it exists
-You can override these configurations params with an initializer. This is especially usefull when you want a custom log file
-    
 ```ruby
-#/config/initializers/proselytism.rb
-Proselytism.config do |config|
-  config.logger = ActiveSupport::BufferedLogger.new(File.join(Rails.root, 'log', 'proselytism.log'))
-end
+  $ rails g proselytism:config
+```
+
+  As an engine, Proselytism automatically load /config/proselytism.yml (if the file exists) and set its config params depending on the current rails env.
+
+  - With an initializer (optional for Rails App) :
+
+  You can override the configuration file params by adding a custom initializer to /config/initializers .
+  By default Proselytism will log in a separate log file, if you want to use the rails logger
+
+```ruby
+  #/config/initializers/proselytism.rb
+  Proselytism.config do |config|
+    config.logger = Rails.logger
+  end
+```
+
+  To generate a full config initializer:
+
+```ruby
+  $ rails g proselytism:initializer
 ```
 
 ## Usage
@@ -54,32 +72,38 @@ Proselytism.extract_images source_file_path do |image_files_paths|
 end
 ```
 
-Proselytism create its converted files in temporary folders.
-  - If you pass a block to the method the folders are automatically deleted after the block is yield, so use or copy the file content within the block
-  - If you don't pass a block, don't forget to safely remove the temp folder
+Proselytism creates its converted files in temporary folders.
+  - If you pass a block to the method above the folders are automatically deleted after the block is yield, so use or copy the file content within the block
+  - If you don't pass a block, the mentioned folder and its content remains permanently, so don't forget to safely remove it yourself
 
 ```ruby
 pdf_file_path = Proselytism.convert source_file_path, :to => :pdf
+#my code
 FileUtils.remove_entry_secure File.dirname(pdf_file_path)
 ```
     
-## Add your own converter
+## Add your own converters
 
 Add your own converter by extending Proselytism::Converters::Base
-  - Your converter will be automatically selected and used related to the form and to extensions list
+  - Your converter will be automatically selected and used related to the params given to the :from and :to methods
   - Add a perform method which
-    - define a text command
-    - call execute
-    - return the converted file(s) path
+    - calls the execute method with your custom command
+    - returns the converted file(s) path(s)
+
+Proselytism::Converters::Base takes care of 
+  - raising error (if the command execution fail) 
+  - logging the command output
 
 ```ruby
 class MyConverter < Proselytism::Converters::Base
+  class Error < parent::Base::Error; end
+  
   form :ext1, :ext2
   to :ext3, :ext4
 
   def perform(origin, options={})
     destination = destination_file_path(origin, options)
-    command = "pdftotext #{origin} #{destination} 2>&1"
+    command = "mycommand #{origin} #{destination} 2>&1"
     execute command
     destination
   end

data/lib/generators/proselytism/templates/config.yml

@@ -8,6 +8,7 @@ test:
   oo_conversion_max_tries: 2
   oo_conversion_max_time: 4 #seconds
 
+
 development:
   open_office_path: "/Applications/OpenOffice.org.app/Contents/MacOS/soffice"
   oo_server_bridge: "JOD"

data/lib/generators/proselytism/templates/initializer.rb

@@ -25,6 +25,13 @@ Proselytism.config do |config|
   #Path where conversion are done by default system temp dir
   #config.tmp_path = File.expand_path("../tmp",  __FILE__)
 
-  #Logger (otherwhise rails logger)
-  #config.logger = ActiveSupport::BufferedLogger.new("your/log/path")
+  #Log level: By default env log level
+  #config.log_level  = Rails.logger.level
+
+  #Log path :
+  #config.log_path   = File.join(Rails.root, 'log', "proselytism.log")
+
+  #Logger instance
+  #config.logger     = Proselytism::BufferedLogger.new Proselytism.config.log_path, Proselytism.config.log_level
+
 end

data/lib/proselytism.rb

@@ -1,7 +1,7 @@
 require "active_support/core_ext"
 
 require "proselytism/version"
-require "proselytism/shared"
+require "proselytism/logger"
 require "proselytism/proselytism"
 require "proselytism/converter"
 

data/lib/proselytism/converter.rb

@@ -6,14 +6,13 @@ module Proselytism
   module Converters
     class Base
       include ::Singleton
-      include Proselytism::Shared
       class_attribute :from, :to, :subclasses
 
       class Error < Exception; end
 
-      def config
-        Proselytism.config
-      end
+
+      delegate :config, :log, :to => Proselytism
+
 
       def destination_file_path(origin, options={})
         if options[:dest]
@@ -25,11 +24,11 @@ module Proselytism
 
       #call perform logging duration and potential errors
       def convert(file_path, options={})
-        log :debug, "convert #{file_path} to :#{options[:to]}" do
+        log :debug, "#{self.class.name} converted #{file_path} to :#{options[:to]}" do
           begin
             perform(file_path, options)
           rescue Error => e
-            log :error, e.message
+            log :error, "#{e.class.name} #{e.message}\n#{e.backtrace}\n"
             raise e
           end
         end

data/lib/proselytism/converters/open_office.rb

@@ -28,16 +28,21 @@ class Proselytism::Converters::OpenOffice < Proselytism::Converters::Base
     destination
   end
 
-
-  # HACK pour contourner un comportement ?trange d'OpenOffice, normalement les enregistrements
-  # se font en UTF-8, mais parfois pour une raison obscure les fichiers texte sont en ISO-8859-1
-  # donc on rajoute un test pour re-convertir dans l'encodage qu'on attend
-  def convert_txt_to_utf8(file_path)
-    if `file #{file_path}` =~ /ISO/
-      system("iconv --from-code ISO-8859-1 --to-code UTF-8 #{file_path} > tmp_iconv.txt && mv tmp_iconv.txt #{file_path}")
+  # For unknown reason sometimes OpenOffice converts in ISO-8859-1,
+  # post process to ensure a conversion in UTF-8 when :to => :txt
+  def perform_with_ensure_utf8(origin, options={})
+    destination = perform_without_ensure_utf8(origin, options)
+    if options[:to].to_s == "txt" and `file #{destination}` =~ /ISO/
+      #lookup_on = Iconv.new('ASCII//TRANSLIT','UTF-8').iconv(str).upcase.strip.gsub(/'/, " ")
+      #log :warn, "***OOO has converted file in "
+      tmp_iconv_file = "#{destination}-tmp_iconv.txt"
+      execute("iconv --from-code ISO-8859-1 --to-code UTF-8 #{destination} > #{tmp_iconv_file} && mv #{tmp_iconv_file} #{destination}")
     end
+    destination
   end
 
+  alias_method_chain :perform, :ensure_utf8
+
   def server
     Server.instance
   end
@@ -45,12 +50,9 @@ class Proselytism::Converters::OpenOffice < Proselytism::Converters::Base
 
   class Server
     include Singleton
-    include Proselytism::Shared
     class Error < Proselytism::Converters::OpenOffice::Error; end
 
-    def config
-      Proselytism.config
-    end
+    delegate :config, :log, :to => Proselytism
 
     # Run a block with a timeout and retry if the first execution fails
     def perform(&block)
@@ -108,9 +110,9 @@ class Proselytism::Converters::OpenOffice < Proselytism::Converters::Base
       begin
         Timeout::timeout(3) do
           loop do
-            system("killall -9 soffice && killall -9 soffice.bin > /dev/null 2>&1")
+            system("killall -9 soffice > /dev/null 2>&1")
+            system("killall -9 soffice.bin > /dev/null 2>&1")
             break unless running?
-            sleep(0.2)
           end
         end
       rescue Timeout::Error

data/lib/proselytism/engine.rb

@@ -11,13 +11,14 @@ module Proselytism
           params[Rails.env].each do |k, v|
             config.send "#{k}=", v
           end
-          Proselytism.config.logger = nil
         end
       end
     end
 
     ActiveSupport.on_load :after_initialize do |app|
-      Proselytism.config.logger ||= Rails.logger
+      Proselytism.config.log_level  ||= Rails.logger.level
+      Proselytism.config.log_path   ||= File.join(Rails.root, 'log', "proselytism.log")
+      Proselytism.config.logger     ||= Proselytism::BufferedLogger.new Proselytism.config.log_path, Proselytism.config.log_level
     end
 
   end

data/lib/proselytism/logger.rb

@@ -0,0 +1,31 @@
+module Proselytism
+
+  class Logger < ActiveSupport::BufferedLogger
+    class Formatter
+      def call(severity, time, progname, msg)
+        formatted_time = time.strftime("%Y-%m-%d %H:%M:%S.") << time.usec.to_s[0..2].rjust(3)
+        "#{formatted_time} [#{severity}][pid:#{$$}] #{msg.strip}\n"
+      end
+    end
+
+    def initialize(log, level = DEBUG)
+      super(log, level)
+      @log.formatter = Formatter.new
+    end
+
+  end
+
+  def self.log(severity, message = nil, &block)
+    if config.logger
+      start_time = Time.now
+      if block_given?
+        result = yield
+        config.logger.send(severity, "#{message} in #{((Time.now - start_time)*1000).to_i} ms")
+      else
+        config.logger.send(severity, message.strip)
+      end
+      block_given? ? result : true
+    end
+  end
+
+end

data/lib/proselytism/proselytism.rb

@@ -1,7 +1,6 @@
 require "active_support/core_ext/module/attribute_accessors"
 
 module Proselytism
-  extend Shared
   mattr_accessor :config
 
   def self.config(&block)
@@ -25,6 +24,9 @@ module Proselytism
     end
   end
 
+
+
+
   # Finds the relevant converter
   def self.get_converter(origin, destination)
     Converters::Base.subclasses.detect do |converter|

data/lib/proselytism/version.rb

@@ -1,3 +1,3 @@
 module Proselytism
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/spec/fixtures/001-latin.txt

@@ -0,0 +1,63 @@
+K�BIR BRAHIM
+
+1 rue Pierre Bonnard 
+8 Pavillon Beethoven 62300 Lens
+N� le�: 29/08/1973 � Calais
+Nationalit�: Fran�aise
+T�l�: 03/62/90/95/31  Portable�: 06/61/10/48/63
+Email�: brahim62100@hotmail.fr
+Vie maritale 3 enfants
+
+
+
+	EXPERIENCES PROFESSIONNELLES
+
+
+Novembre 2008-Avril 2009�    : Atlantique Automatisme Incendie
+                               Monteur tuyauterie (Manpower)
+
+Juin 2008-Octobre 2008       : Soci�t� d?Etude Fabrication Montage
+                               Monteur (Manpower)
+
+Avril 2008-Mai 2008          : Ateliers Bois
+                               Monteur Charpentes m�talliques (Inter 5)
+
+D�cembre 2007-F�vrier 2008�  : Soci�t� d?Etude Fabrication Montage
+                               Monteur (Manpower)
+
+Aout 1998-Novembre 2007      : France Montage Fabrication
+                               Monteur
+
+Octobre 1996-Juillet 1998�   : France Montage Fabrication
+                               Monteur (Adia Interim)
+
+D�cembre 1995-Septembre 1996�: Service Militaire
+
+
+
+
+                       FORMATIONS ET DIPLOMES
+
+
+2005�: Littoral Formation (Dunkerque)
+       CACES 3 Chariot El�vateur
+       CACES 9 Engins de Manutention
+       CACES 3B Cat�gorie de PEMP
+
+1995�: Lyc�e Pierre de Coubertin � Calais
+       BTS Transport et Logistique (niveau)
+
+1993�: Lyc�e Pierre de Coubertin � Calais
+       Bac G2 Techniques Quantitatives de Gestion
+
+1988�: Coll�ge Lucien Vadez � Calais
+       BEPC
+
+
+
+                       INFORMATIONS COMPLEMENTAIRES
+
+
+Permis B + voiture personnel
+Loisirs�: Lecture,sport,sorties p�destres
+Maitrise de l?outil informatique�: Internet,Word,Excel

data/spec/open_office_spec.rb

@@ -55,6 +55,16 @@ describe Proselytism::Converters::OpenOffice.instance do
         subject.perform fixture_path("002.doc"), :dir => tmp_dir, :to => :txt
       }.should change(self, :tmp_dir_file_count).by 1
     end
+    it "should ensure destination file encoding is utf8" do
+      FileUtils.cp(fixture_path("001-latin.txt"), tmp_path("001-latin.txt"))
+      subject.should_receive(:perform_without_ensure_utf8).
+          with(fixture_path("001.doc"), :dir => tmp_dir, :to => :txt).
+          and_return(tmp_path("001-latin.txt"))
+      subject.perform fixture_path("001.doc"), :dir => tmp_dir, :to => :txt do |converted_file|
+        `file #{converted_file}`.should match 'UTF-8'
+        File.read(converted_file).should match('é')
+      end
+    end
 
     it "should not freeze" do
       3.times do |j|

data/spec/spec_helper.rb

@@ -15,7 +15,7 @@ Proselytism.config do |config|
   config.oo_conversion_max_time = 4 #seconds
 
   config.tmp_path = File.expand_path("../tmp",  __FILE__)
-  config.logger = ActiveSupport::BufferedLogger.new(File.expand_path("../tmp/log",  __FILE__))
+  config.logger = Proselytism::Logger.new(File.expand_path("../tmp/log",  __FILE__), 0)
 end
 
 

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: proselytism
 version: !ruby/object:Gem::Version 
-  hash: 29
+  hash: 27
   prerelease: 
   segments: 
   - 0
   - 0
-  - 1
-  version: 0.0.1
+  - 2
+  version: 0.0.2
 platform: ruby
 authors: 
 - Itkin
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2013-03-06 00:00:00 Z
+date: 2013-03-08 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: activesupport
@@ -125,12 +125,13 @@ files:
 - lib/proselytism/converters/pdf_to_text.rb
 - lib/proselytism/converters/ppm_to_jpeg.rb
 - lib/proselytism/engine.rb
+- lib/proselytism/logger.rb
 - lib/proselytism/proselytism.rb
-- lib/proselytism/shared.rb
 - lib/proselytism/version.rb
 - proselytism.gemspec
 - spec/.DS_Store
 - spec/base_converter_spec.rb
+- spec/fixtures/001-latin.txt
 - spec/fixtures/001.doc
 - spec/fixtures/001.pdf
 - spec/fixtures/001.txt
@@ -142,7 +143,6 @@ files:
 - spec/pdf_images_spec.rb
 - spec/pdf_to_text_spec.rb
 - spec/proselytism_spec.rb
-- spec/shared_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/itkin/proselytism.git
 licenses: []
@@ -180,6 +180,7 @@ summary: document converter and plain text extractor
 test_files: 
 - spec/.DS_Store
 - spec/base_converter_spec.rb
+- spec/fixtures/001-latin.txt
 - spec/fixtures/001.doc
 - spec/fixtures/001.pdf
 - spec/fixtures/001.txt
@@ -191,5 +192,4 @@ test_files:
 - spec/pdf_images_spec.rb
 - spec/pdf_to_text_spec.rb
 - spec/proselytism_spec.rb
-- spec/shared_spec.rb
 - spec/spec_helper.rb

data/lib/proselytism/shared.rb

@@ -1,22 +0,0 @@
-module Proselytism
-  module Shared
-
-    def log(severity, message = nil)
-      config.logger.send(severity, message) if config.logger
-    end
-
-    def log_with_time(severity, message = nil, &block)
-      start_time = Time.now
-      delay = nil
-      if block_given?
-        result = yield
-        delay = "(#{((Time.now - start_time)*1000).to_i} ms) "
-      end
-      message= "** Proselytism #{start_time.strftime("%Y-%m-%d %H:%M:%S")} #{delay}: " + message.to_s
-      log_without_time(severity, message)
-      block_given? ? result : true
-    end
-    alias_method_chain :log, :time
-
-  end
-end

data/spec/shared_spec.rb

@@ -1,26 +0,0 @@
-require "spec_helper"
-
-describe Proselytism do
-  context "log" do
-    it "should log with class and time data" do
-      subject.config.logger.should_receive(:debug).with do |message|
-        message.should match /Proselytism/
-        message.should match /io/
-        message.should match Time.now.strftime("%Y-%m-%d")
-      end
-      subject.log(:debug, 'io').should be_true
-    end
-    it "should log delay when a block is passed" do
-      subject.config.logger.should_receive(:debug).with do |message|
-        message.should match /Proselytism/
-        message.should match /io/
-        message.should match /([\d:]+)/
-      end
-      subject.log :debug , 'io' do
-        sleep(0.5)
-        false
-      end.should be_false
-    end
-
-  end
-end