iguvium 0.8.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 50748a9a17471a1f0f58a108ee632344cafad4c0ee6ab6096dbdb297c53b1381
+  data.tar.gz: 6e676513f8aaee937f2dbb28316452a0dcd38f850afdc31a5ef6685200189814
+SHA512:
+  metadata.gz: 4b0c5fc257ae4e8b667f0611f708e744c806de1f319e016e42df3553273c0cda445fe23f29e76ed8ef980eaf4680c21ecd7dccdf4602b721e5731981eace3097
+  data.tar.gz: 563368859fa3684b8baa54f25e788c91f76d46c71b56bfa271fd9d6db2b9e6f74ad4d5937488b42c5f0912fdcc77e09432c214fde7d5c76dca85884e594912cf

data/.gitignore

@@ -0,0 +1,22 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+.idea/
+
+Gemfile.lock
+
+*.gem
+prof*
+
+.DS_Store
+._.DS_Store
+**/.DS_Store
+**/._.DS_Store
+
+# rspec failure tracking
+.rspec_status

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,7 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.5.1
+  - 2.4.5
+  - 2.3.8
+before_install: gem install bundler && sudo apt-get install ghostscript

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in iguvium.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 adworse
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,123 @@
+# Iguvium
+[![Build Status](https://travis-ci.com/adworse/iguvium.svg?token=pKH4s9rC7sLLfFxdq8b6&branch=master)](https://travis-ci.com/adworse/iguvium)
+
+Iguvium extracts tables from PDF file in a structured form. It works like this.
+
+Take this PDF file:
+
+![PDF Table](https://user-images.githubusercontent.com/8277078/48663021-ba81e580-ea92-11e8-8ca6-53c5cd5c7b1b.png)
+
+Use this code:
+
+```
+pages = Iguvium.read('filename.pdf')
+tables = pages[1].extract_tables!
+csv = tables.first.to_a.map(&:to_csv).join 
+```
+
+Get this table:
+
+![Spreadsheet](https://user-images.githubusercontent.com/8277078/48663073-822ed700-ea93-11e8-8924-9974ab5da27b.png)
+
+## Features/Limitations: 
+* Iguvium renders pdf into an image, looks for table-like graphic structure and tries to place characters into detected cells.
+
+* Characters extraction is done by [PDF::Reader gem](https://github.com/yob/pdf-reader). Some PDFs are so messed up it can't extract meaningful text from them. If so, so does Iguvium.
+
+* Current version extracts regular (with constant number of rows per column and vise versa) tables with explicit lines formatting, like this:
+
+```
+.__________________.
+|____|_______|_____|
+|____|_______|_____|
+|____|_______|_____|
+```
+  Merged cells content is split as if cells were not merged.
+
+* Performance: considering the fact it has computer vision under the hood, the gem is reasonably fast. Full page extraction takes up to 1 second on modern CPUs and up to 2 seconds on the older ones.
+
+
+## Installation
+
+Make sure you have Ghostscript installed. 
+
+Linux: `sudo apt-get install ghostscript`
+
+Mac: `brew install ghostscript`
+
+Windows: download installer from the official [download page](https://www.ghostscript.com/download/gsdnld.html).
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'iguvium'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install iguvium
+
+## Usage
+
+#### Get all the tables in 2D text array format
+```
+pages = Iguvium.read('filename.pdf') #=> [Array<Iguvium::Page>]
+tables = pages.flat_map { |page| page.extract_tables! } #=> [Array<Iguvium::Table>]
+tables.map(&:to_a)
+```
+#### Get first table from the page 8
+```
+pages = Iguvium.read('filename.pdf')
+tables = pages[7].extract_tables!
+tables.first.to_a
+```
+
+## CLI
+
+Gem installation adds a command-line utility to the system. It's a simple wrapper:
+
+```
+iguvium filename.pdf [options]
+    -p, --pages     page numbers, comma-separated, no spaces
+    -i, --images    use pictures in pdf (usually a bad idea)
+    -n, --newlines  keep newlines
+    --verbose       verbose output
+```
+
+Given a filename, it generates CSV files for the tables detected 
+
+## Implementation details
+There are usually no actual tables in PDFs, only characters with coordinates,
+and some fancy lines. Human eye interprets this as a table. Iguvium behaves quite similarly. It prints PDF to an image file with GhostScript, then analyses the image.
+
+Long enough continuous edges are interpreted as possible cell borders. Gaussian blur is applied beforehand to get rid of possible inconsistencies and style features.
+
+Initially inspired by [camelot](https://github.com/socialcopsdev/camelot/) idea of image analysis to detect table structure. Besides this idea, is an independent work. Image recognition is written in Ruby, no OpenCV or other heavy computer vision libraries are used. Line detection algorithms are different too. The functionality of Camelot is significantly broader.
+
+## Roadmap
+
+The next version will deal with open-edged tables like
+
+```
+__|____|_______|_____|
+__|____|_______|_____|
+__|____|_______|_____|
+```
+
+It also will keep open-edged rows metadata ('floorless' and 'roofless') for the needs of multipage tables merger.
+
+The final one will recognize tables with merged cells.
+
+There are at the moment no plans to design recognition of whitespace-separated tables.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Name
+
+Just a place (ancient) where some [tables](https://en.wikipedia.org/wiki/Iguvine_Tablets) (incredibly cool ones) were found.

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.fail_on_error = true
+end
+
+task default: :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "iguvium"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/exe/iguvium

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'csv'
+require 'iguvium'
+require 'slop'
+
+opts = Slop.parse { |o|
+  o.array '-p', '--pages', 'page numbers, comma-separated, no spaces'
+  o.bool '-i', '--images', 'use pictures in pdf (usually a bad idea)'
+  o.bool '-n', '--newlines', 'keep newlines'
+  o.bool '--verbose', 'verbose output'
+  o.on '--version', 'print the version' do
+    puts Iguvium::VERSION
+    exit
+  end
+  o.on '-h', '--help', 'show help' do
+    puts o.to_s.gsub(/(usage:).+(iguvium)/, '\1 \2 filename.pdf')
+    exit
+  end
+}
+
+path = opts.args.first
+
+abort('No file name given') unless path
+
+opts = opts.to_hash
+page_numbers = opts.delete(:pages).map { |number| number.to_i - 1 }
+
+pages = Iguvium.read(path, opts)
+
+Iguvium.logger.level = Logger::INFO if opts[:verbose]
+
+page_numbers = pages.count.times.to_a if page_numbers.empty?
+
+# puts page_numbers.inspect
+# puts opts.to_hash.inspect
+
+page_numbers.each do |number|
+  print "Extracting page #{number + 1}... "
+  tables = pages[number].extract_tables!(images: opts[:images])
+  tables.empty? ? puts('no tables found') : puts
+
+  tables.each_with_index do |table, i|
+    csv = table.to_a(newlines: opts[:newlines]).map(&:to_csv).join
+    next if csv.empty?
+    csv_file = "#{path.gsub(/\.pdf$/, '')}_page_#{number + 1}_table_#{i}.csv"
+    puts "Saving #{File.expand_path(csv_file, __dir__)}"
+    File.write csv_file, csv
+  end
+end

data/iguvium.gemspec

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+lib = File.expand_path('lib', __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'iguvium/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'iguvium'
+  spec.version       = Iguvium::VERSION
+  spec.authors       = ['Dima Ermilov']
+  spec.email         = ['dima@scriptangle.com']
+
+  spec.summary       = 'Extract tables from PDF as a structured info'
+  spec.description   = 'Extract tables from PDF as a structured info. Uses ghostscript to print pdf to image, \
+then recognizes table separators optically. No OpenCV or other heavy dependencies'
+  spec.homepage      = 'https://github.com/adworse/iguvium'
+  spec.license       = 'MIT'
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files         = Dir.chdir(File.expand_path(__dir__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  end
+
+  spec.bindir        = 'exe'
+  # spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+
+  spec.executables   = ['iguvium']
+
+  spec.require_paths = ['lib']
+
+  spec.add_dependency 'pdf-reader', '~> 2.1'
+  spec.add_dependency 'convolver-light', '~> 0.3.1'
+  spec.add_dependency 'oily_png', '~> 1.2'
+  spec.add_dependency 'slop', '~> 4.2'
+
+  spec.add_development_dependency 'bundler', '~> 1.16'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+end

data/lib/iguvium/cv.rb

@@ -0,0 +1,194 @@
+# frozen_string_literal: true
+
+module Iguvium
+  GAUSS = NArray[
+      [0.0125786, 0.0251572, 0.0314465, 0.0251572, 0.0125786],
+      [0.0251572, 0.0566038, 0.0754717, 0.0566038, 0.0251572],
+      [0.0314465, 0.0754717, 0.0943396, 0.0754717, 0.0314465],
+      [0.0251572, 0.0566038, 0.0754717, 0.0566038, 0.0251572],
+      [0.0125786, 0.0251572, 0.0314465, 0.0251572, 0.0125786]
+  ]
+
+  HORIZONTAL = NArray[
+      [-1, -1, -1],
+      [2, 2, 2],
+      [-1, -1, -1]
+  ]
+
+  VERTICAL = NArray[
+      [-1, 2, -1],
+      [-1, 2, -1],
+      [-1, 2, -1]
+  ]
+
+  private_constant :GAUSS, :HORIZONTAL, :VERTICAL
+
+  # Performs all the computer vision job except table composition.
+  # Edge detection is performed using simplified two-directional version of
+  # Canny edge detection operator applied to rows and columns as integer vectors
+  class CV
+    # @!attribute lines
+    #   @return [Hash] :vertical and :horizontal lines. Horizontal lines are [Array<Range, Integer>],
+    #     vertical have [Integer] on x position
+    # @!attribute boxes
+    #   @return [Array<Range, Range>] X range, Y range
+
+    # Keeps recognized data
+    Recognized = Struct.new(:lines, :boxes)
+
+    # Prepares image for recognition: initial blur
+    # @param image [ChunkyPNG::Image] from {Iguvium::Image.read}
+    def initialize(image)
+      @image = blur image
+    end
+
+    # @return [Array] 8-bit representation of an image
+    attr_reader :image
+
+    # @return [Recognized]
+    #   lines most probably forming table cells and tables' outer borders as boxes
+    def recognize
+      Recognized.new(lines, boxes)
+      # {
+      #   lines: lines,
+      #   boxes: boxes
+      # }
+    end
+
+    private
+
+    def lines
+      @lines ||=
+        {
+          vertical: Labeler.new(verticals)
+                           .lines
+                           .map { |line| flip_line line }
+                           .sort_by { |x, yrange| [yrange.begin, x] },
+          horizontal: Labeler.new(horizontals).lines.map { |line| flip_line line }.sort_by { |_, y| [y] }
+        }
+    end
+
+    def boxes
+      return @boxes if @boxes
+
+      brightest = image.flatten.max
+      @boxes = Labeler.new(
+        # image.map { |row| row.map { |pix| 255 - pix } }
+        image.map { |row| row.map { |pix| pix < brightest } }
+      ).clusters.map { |cluster| box cluster }.sort_by { |xrange, yrange| [yrange.begin, xrange.begin] }
+    end
+
+    def verticals(threshold = 3)
+      Matrix
+        .rows(convolve(NArray[*horizontal_scan(image)], VERTICAL, 0).to_a)
+        .map { |pix| pix < threshold ? nil : pix }
+        .to_a
+    end
+
+    def horizontals(threshold = 3)
+      Matrix
+        .rows(convolve(NArray[*vertical_scan(image)], HORIZONTAL, 0).to_a)
+        .map { |pix| pix < threshold ? nil : pix }
+        .to_a
+    end
+
+    # START OF FLIPPER CODE
+    def flip_y(coord)
+      @height ||= image.count
+      @height - coord - 1
+    end
+
+    def flip_range(range)
+      flip_y(range.end)..flip_y(range.begin)
+    end
+
+    def flip_line(line)
+      y = line.last
+      y = if y.is_a?(Numeric)
+            flip_y y
+          elsif y.is_a?(Range)
+            flip_range y
+          else
+            raise ArgumentError, 'WTF?!'
+          end
+
+      [line.first, y]
+    end
+
+    # END OF FLIPPER CODE
+
+    def blur(image)
+      convolve(to_narray(image), GAUSS).to_a
+    end
+
+    def convolve(narray, conv, border_value = 255)
+      narray = border(narray, conv.shape.first / 2, border_value)
+      Convolver.convolve(narray, conv).ceil
+    end
+
+    def array_border(array, width, value = 0)
+      hl = Array.new width, Array.new(array.first.count + width * 2, value)
+      hl + array.map { |row| [value] * width + row + [value] * width } + hl
+    end
+
+    def border(narray, width, value = 0)
+      NArray[*array_border(narray.to_a, width, value)]
+    end
+
+    def to_narray(image)
+      palette = image.pixels.uniq
+      # Precalculation looks stupid but spares up to 0.35 seconds on calculation depending on colorspace width
+      dict = palette.zip(
+        palette.map { |color| ChunkyPNG::Color.grayscale_teint ChunkyPNG::Color.compose(color, 0xffffffff) }
+      ).to_h
+      NArray[
+        image.pixels.map { |color| dict[color] }
+      ].reshape(image.width, image.height)
+    end
+
+    # def minimums_old(ary)
+    #   ary.each_cons(2)
+    #      .each_with_index
+    #      .map { |(a, b), i| [i + 1, a <=> b] }
+    #      .slice_when { |a, b| a.last != -1 && b.last == -1 }
+    #      .to_a
+    #      .map { |seq| seq.reverse.detect do |a| a.last == 1 end&.first }
+    #      .compact
+    # end
+
+    def minimums(ary)
+      # This ugly piece of code takes ~200 ms per page scan to run vs ~700 ms for the prettier old one
+      i = 0
+      mins = []
+      local = 0
+      while i + 2 < ary.length
+        local = i + 1 if ary[i] > ary[i + 1]
+        mins << local if ary[i] >= ary[i + 1] && ary[i + 1] < ary[i + 2]
+        i += 1
+      end
+      mins.uniq
+    end
+
+    def edges(vector)
+      Array
+        .new(vector.count)
+        .tap { |ary| minimums(vector).each { |i| ary[i] = 1 } }
+    end
+
+    def horizontal_scan(image)
+      image.map { |row| edges row }
+    end
+
+    def vertical_scan(image)
+      image.transpose.map { |row| edges row }.transpose
+    end
+
+    def box(coord_array)
+      ax, bx = coord_array.map(&:last).minmax
+      ay, by = coord_array.map(&:first).minmax
+      # additional pixels removed from the box definition
+      # [ax - 1..bx + 1, ay - 1..by + 1]
+      [ax..bx, flip_range(ay..by)]
+    end
+  end
+end

data/lib/iguvium/image.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Iguvium
+  # PDF to image converter
+  class Image
+    # Prints single page without text to .rgb file and reads it back to memory
+    #
+    # @param path [String] path to PDF file to be read
+    # @param pagenumber [Integer] number of page, first page is 1, not 0
+    #
+    # @option opts [Boolean] :images (false) consider pictures in PDF as possible table separators
+    # @option opts [String] :gspath (nil) explicit path to the GhostScript executable. Use it in case of
+    #   non-standard gs executable placement. If not specified, gem tries standard options
+    #   like C:/Program Files/gs/gs*/bin/gswin??c.exe on Windows or just gs on Mac and Linux
+    #
+    # @return [ChunkyPNG::Image]
+    def self.read(path, pagenumber = 1, **opts)
+      rgb = path.gsub(/\.pdf$/, '.rgb')
+      Iguvium.logger.info `#{opts[:gspath]} -dSAFER -dBATCH -dNOPAUSE -sDEVICE=pnggray -dGraphicsAlphaBits=4 \
+    -r72 -dFirstPage=#{pagenumber} -dLastPage=#{pagenumber} \
+    -dFILTERTEXT #{'-dFILTERIMAGE' unless opts[:images]} -sOutputFile=#{rgb} #{path} 2>&1`
+
+      image = ChunkyPNG::Image.from_file(rgb)
+      File.delete rgb
+      image
+    end
+  end
+end

data/lib/iguvium/labeler.rb

@@ -0,0 +1,132 @@
+# frozen_string_literal: true
+
+module Iguvium
+  NEIGHBORS = [[0, -1], [-1, -1], [-1, 0], [-1, 1]].freeze
+  FLAT_THRESHOLD = 0.2
+
+  private_constant :NEIGHBORS, :FLAT_THRESHOLD
+
+  # Clusterizes connected pixels using two-pass connected component labelling algorithm (Hoshen-Kopelman),
+  # 8-connectivity is used. Line-like groups are then flattened using simplified dispersion ratio
+  class Labeler
+    # @param image [Array<Boolean>] should be an Array, binarized image
+    #    w/ falsy elements as background and anything truthy as pixels
+    #
+    def initialize(image)
+      @image = image
+      rows = image.count
+      cols = image.first.count
+      @labels = Array.new(rows) { Array.new(cols) }
+      @equalities = []
+    end
+
+    # @return [Hash] vertical and horizontal lines detected
+    def lines
+      clusters.map { |cluster| flatten_cluster cluster }.compact
+    end
+
+    # @return [Array<Array>] coordinates of connected pixels grouped together
+    def clusters
+      accumulator = Hash.new { |h, k| h[k] = [] }
+      label.each_index do |row|
+        labels[row].each_index do |column|
+          pix = labels[row][column]
+          next unless pix
+
+          accumulator[pix] << [row, column]
+        end
+      end
+      accumulator.values
+    end
+
+    private
+
+    attr_reader :image, :labels
+
+    def label
+      pass_one
+      @equalities = @equalities.map { |a| resolve a }
+      image.each_index do |row|
+        image[row].each_index do |column|
+          next unless labels[row][column]
+
+          @labels[row][column] = @equalities[labels[row][column]]
+        end
+      end
+      labels
+    end
+
+    def flatten_cluster(cluster)
+      xs = cluster.map(&:last)
+      ys = cluster.map(&:first)
+
+      if xs.uniq.count / xs.count.to_f < FLAT_THRESHOLD
+        [xs.max_by { |i| xs.count i }, ys.min..ys.max]
+      elsif ys.uniq.count / ys.count.to_f < FLAT_THRESHOLD
+        [xs.min..xs.max, ys.max_by { |i| ys.count i }]
+      else
+        Iguvium.logger.warn "NonFlattable, #{cluster.inspect}"
+        nil
+      end
+    end
+
+    def resolve(num)
+      resolved = @equalities[num]
+      resolved == num ? resolved : resolve(resolved)
+    end
+
+    def neighbors(row, col)
+      NEIGHBORS.map { |roffset, coffset|
+        r = row + roffset
+        c = col + coffset
+        labels.dig(r, c) unless [r, c].min < 0
+      }.compact
+    end
+
+    def neighbors2(row, col)
+      neighbors = []
+      NEIGHBORS.each do |roffset, coffset|
+        r = row + roffset
+        c = col + coffset
+        next if r < 0 || c < 0
+
+        label = labels[r][c]
+        neighbors << label if label
+      end
+      neighbors
+    end
+
+    def pass_one
+      next_label = 0
+      image.each_index do |row|
+        image[row].each_index do |column|
+          next unless image[row][column]
+
+          neighbors = neighbors row, column
+
+          if neighbors.empty?
+            @equalities[next_label] = next_label
+            @labels[row][column] = next_label
+            next_label += 1
+          else
+            neighbors.uniq!
+            neighbors.sort!
+            min = neighbors.shift
+            @labels[row][column] = min
+            count = neighbors.length
+            next if count == 0
+
+            if count == 1
+              @equalities[neighbors[0]] = min
+            elsif count > 1
+              neighbors.each do |neighbor|
+                @equalities[neighbor] = min
+              end
+            end
+          end
+        end
+      end
+      self
+    end
+  end
+end

data/lib/iguvium/page.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+module Iguvium
+
+  # It's document page, you can extract tables from here. to do so, use {Iguvium::Page#extract_tables!}.
+  #
+  # {Iguvium::Page#text} method is handy in order to pre-analyze whether you need this page.
+  #
+  # @example
+  #   pages = Iguvium.read('nixon.pdf', gspath: '/usr/bin/gs')
+  #   pages = pages.select { |page| page.text.match?(/[Tt]able.+\d+/) }
+  #   tables = pages.map(&:extract_tables!)
+  class Page
+    # @param page [PDF::Reader::Page]
+    # @param (see Iguvium.read)
+    # Typically you don't need it, prefer {Page} creation from {Iguvium.read}
+    def initialize(page, path, **opts)
+      @opts = opts
+      @reader_page = page
+      @path = path
+    end
+
+    # @!visibility private
+    # @return (see Iguvium::CV#lines)
+    attr_reader :lines
+
+    # This method does all the heavy lifting which include optical recognition of table borders.
+    # It returns an array of {Iguvium::Table}
+    # or an empty array if it fails to recognize any. To get structured data from parsed {Iguvium::Table},
+    # just call {Iguvium::Table#to_a}.
+    #
+    # @todo Further speed improvements should be done, expecting at least 30% speedup on multicore systems
+    #
+    # Due to the nature of PDF document which is generally a collection of independent pages,
+    # {Iguvium::Page#extract_tables!} is suitable for parallel processing. Concurrent processing
+    # (think fork as parallel vs. thread as concurrent) on the other hand would be not a great idea,
+    # because it's a CPU-intensive task.
+    #
+    # On some older CPUs it takes up to 2 seconds per page for it to work
+    # (up to 1 second on more modern ones), so use it wisely.
+    #
+    # @example extract tables using pictures as possible borders
+    #   tables = page.extract_tables! images: true #=> [Array<Iguvium::Table>]
+    # @return [Array<Iguvium::Table>]
+    def extract_tables!(images: @opts[:images])
+      return @tables if @tables
+
+      @opts[:images] = images
+      recognize!
+      @tables
+    end
+
+    # @return [String] rendered page text, result of underlying PDF::Reader::Page#text call
+    # It takes ~150 ms for it to work, so it's handy
+    # for picking up pages before trying to extract tables, which is an expensive operation
+    def text
+      @text ||= @reader_page.text
+    end
+
+    # @!visibility private
+    # @return [Array<PDF::Reader::TextRun>] array of characters on page. Each character has its coordinates,
+    #   size, and width
+    def characters
+      return @characters if @characters
+
+      receiver = PDF::Reader::PageTextReceiver.new
+      @reader_page.send(:walk, receiver)
+      @characters = receiver.instance_variable_get('@characters')
+    end
+
+    private
+
+    def recognize!
+      image = Image.read(@path, @reader_page.number, @opts)
+      recognized = CV.new(image).recognize
+      @lines = recognized[:lines]
+      @boxes = recognized[:boxes].reject { |box| box_empty?(box) }
+      @tables = @boxes.map { |box| Table.new(box, self) }.reverse
+      self
+    end
+
+    def box_empty?(box)
+      characters.select { |character|
+        box.first.cover?(character.x) && box.last.cover?(character.y)
+      }.empty?
+    end
+  end
+end

data/lib/iguvium/table.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+module Iguvium
+  # Represents single table from the [Iguvium::Page]:
+  # * table outer borders aka box,
+  # * set of detected horizontal and vertical lines to form table's grid,
+  # * set of characters with its coordinates to fill the grid.
+  #
+  # Additional functionality like an option to detect an open table grid at the end or at the beginning
+  # of the page will be added later
+  #
+  # To render table into 2D text array, call {#to_a}
+  class Table
+    # @api private
+    def initialize(box, page)
+      @box = box
+      @lines = page.lines
+      @page = page
+    end
+
+    # Renders the table into an array of strings.
+    #
+    # Newlines in PDF have usually no semantic value, and are replaced with spaces by default.
+    # Sometimes you may need to keep them; in this case use `newlines: true` option.
+    #
+    # @param  [Boolean] newlines keep newlines inside table cells, false by default
+    # @return [Array] 2D array of strings (content of table's cells)
+    #
+    def to_a(newlines: false)
+      grid[:rows]
+        .reverse
+        .map { |row| grid[:columns].map { |column| render(chars_inside(column, row), newlines: newlines) } }
+    end
+
+    private
+
+    attr_reader :page, :lines, :box
+
+    def enhancer(grid)
+      # @todo write grid enhancer to detect cells between outer grid lines and box borders
+    end
+
+    def characters
+      xrange = box.first
+      yrange = box.last
+      @characters ||=
+        page
+        .characters
+        .select { |character| xrange.cover?(character.x) && yrange.cover?(character.y) }
+    end
+
+    def grid
+      @grid ||=
+        {
+          rows: lines_to_ranges(lines[:horizontal]),
+          columns: lines_to_ranges(lines[:vertical])
+        }
+    end
+
+    def lines_to_ranges(lines)
+      lines.select { |line| line_in_box?(line, box) }
+           .map { |line| line.first.is_a?(Numeric) ? line.first : line.last }
+           .sort
+           .uniq
+           .each_cons(2)
+           .map { |a, b| a...b }
+    end
+
+    def line_in_box?(line, box)
+      line = line.map { |coord| coord.is_a?(Range) ? coord.to_a.minmax : [coord] }
+      (
+      line.first.map { |coord| box.first.cover?(coord) } +
+          line.last.map { |coord| box.last.cover?(coord) }
+    ).all?
+    end
+
+    def chars_inside(xrange, yrange)
+      characters.select { |character|
+        xrange.cover?(character.x) && yrange.cover?(character.y)
+      }
+    end
+
+    def render(characters, newlines: false)
+      separator = newlines ? "\n" : ' '
+      characters
+        .sort
+        .chunk_while { |a, b| a.mergable?(b) }
+        .map { |chunk| chunk.inject(:+).to_s.strip.gsub(/[\s|\p{Z}]+/, ' ') }
+        .join(separator)
+        .gsub(/ +/, ' ')
+    end
+  end
+end

data/lib/iguvium/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Iguvium
+  VERSION = '0.8.0'
+end

data/lib/iguvium.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+require 'convolver-light'
+require 'fileutils'
+require 'logger'
+require 'matrix'
+require 'oily_png'
+require 'pdf-reader'
+require 'rbconfig'
+
+require_relative 'iguvium/labeler'
+require_relative 'iguvium/cv'
+require_relative 'iguvium/image'
+require_relative 'iguvium/page'
+require_relative 'iguvium/table'
+require_relative 'iguvium/version'
+
+# PDF tables extractor.
+# @example Get all the tables in 2D text array format
+#   pages = Iguvium.read('filename.pdf') #=> [Array<Iguvium::Page>]
+#   tables = pages.flat_map { |page| page.extract_tables! } #=> [Array<Iguvium::Table>]
+#   tables.map(&:to_a)
+# @example Get first table from the page 8
+#   pages = Iguvium.read('filename.pdf')
+#   tables = pages[7].extract_tables!
+#   tables.first.to_a
+# For more details please look {Iguvium.read} and {Iguvium::Page#extract_tables!}
+# @author Dima Ermilov <wlaer@wlaer.com>
+#
+module Iguvium
+  class << self
+    # It's main method. Usually this is where you start.
+    #
+    # It returns an array of {Iguvium::Page}.
+    #
+    # Tables on those pages are neither extracted nor detected yet,
+    # all the heavy lifting is done in {Iguvium::Page#extract_tables!} method.
+    #
+    # @param path [String] path to PDF file to be read
+    # @option opts [String] :gspath (nil) explicit path to the GhostScript executable. Use it in case of
+    #   non-standard gs executable placement. If not specified, gem tries standard options
+    #   like `C:\\Program Files\\gs\\gs*\\bin\\gswin??c.exe` on Windows or just `gs` on Mac and Linux
+    # @option opts [Logger::Level] :loglevel level like Logger::INFO, default is Logger::ERROR
+    # @return [Array <Iguvium::Page>]
+    #
+    # @example prepare pages, consider images meaningful
+    #   pages = Iguvium.read('filename.pdf', images: true)
+    #
+    # @example set nonstandard gs path, get pages starting with the one which contains keyword
+    #   pages = Iguvium.read('nixon.pdf', gspath: '/usr/bin/gs')
+    #   pages = pages.drop_while { |page| !page.text.match?(/Watergate/) }
+    #   # {Iguvium::Page#text} does not require optical page scan and thus is relatively cheap.
+    #   # It uses an underlying PDF::Reader::Page#text which is fast but not completely free though.
+    #
+    # @option opts [Boolean] :images (false) consider pictures in PDF as possible table separators.
+    # This typically makes sense in a rare case when table grid in your pdf is filled with
+    # rasterized texture or is actually a background picture. Usually you don't want to use it.
+    #
+    def read(path, **opts)
+      if windows?
+        unless opts[:gspath]
+          gspath = Dir.glob('C:/Program Files/gs/gs*/bin/gswin??c.exe').first.tr('/', '\\')
+          opts[:gspath] = "\"#{gspath}\""
+        end
+
+        if opts[:gspath].empty?
+          puts "There's no gs utility in your $PATH.
+  Please install GhostScript: https://www.ghostscript.com/download/gsdnld.html"
+          exit
+        end
+      else
+        opts[:gspath] ||= gs_nix?
+      end
+
+      PDF::Reader.new(path, opts).pages.map { |page| Page.new(page, path, opts) }
+    end
+
+    # Creates and gives access to Ruby Logger. Default [Logger::Level] is Logger::ERROR.
+    #
+    # To set another level call `Iguvium.logger.level = Logger::INFO` or some other standard logger level
+    #
+    #
+    # It is possible to redefine Iguvium's logger, for example to replace it with a global one like
+    # `Iguvium.logger = Rails.logger`
+    # @return [Logger]
+    def logger
+      return @logger if @logger
+
+      @logger = Logger.new(STDOUT)
+      @logger.formatter = proc do |severity, _, _, msg|
+        "#{severity}: #{msg}\n"
+      end
+      @logger.level = Logger::ERROR
+      @logger
+    end
+    def logger=(new_logger)
+      @logger = new_logger
+    end
+
+    private
+
+    def gs_nix?
+      if `which gs`.empty?
+        puts "There's no gs utility in your $PATH.
+  Please install GhostScript with `brew install ghostscript` on Mac
+  or download it here: https://www.ghostscript.com/download/gsdnld.html"
+        exit
+      end
+      'gs'
+    end
+
+    def windows?
+      RbConfig::CONFIG['host_os'].match(/mswin|mingw|cygwin/)
+    end
+  end
+end
+
+# TODO: 4) Add options like maybe image thresholding
+#
+# TODO: 6) 0.9 - version capable of reading tables with open outer cells, like this:
+# __|____|_______|_____|
+# __|____|_______|_____|
+# __|____|_______|_____|
+#
+# TODO: 7) 1.0 - in addition it should deal with merged cells (move result to the upper left cell).

metadata

@@ -0,0 +1,163 @@
+--- !ruby/object:Gem::Specification
+name: iguvium
+version: !ruby/object:Gem::Version
+  version: 0.8.0
+platform: ruby
+authors:
+- Dima Ermilov
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2018-11-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: pdf-reader
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.1'
+- !ruby/object:Gem::Dependency
+  name: convolver-light
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+- !ruby/object:Gem::Dependency
+  name: oily_png
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.2'
+- !ruby/object:Gem::Dependency
+  name: slop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.2'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.16'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: |-
+  Extract tables from PDF as a structured info. Uses ghostscript to print pdf to image, \
+  then recognizes table separators optically. No OpenCV or other heavy dependencies
+email:
+- dima@scriptangle.com
+executables:
+- iguvium
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- exe/iguvium
+- iguvium.gemspec
+- lib/iguvium.rb
+- lib/iguvium/cv.rb
+- lib/iguvium/image.rb
+- lib/iguvium/labeler.rb
+- lib/iguvium/page.rb
+- lib/iguvium/table.rb
+- lib/iguvium/version.rb
+homepage: https://github.com/adworse/iguvium
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: Extract tables from PDF as a structured info
+test_files: []