traject_sequel_writer 0.9.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7783d67f45b23e824e2fbf85c66b6cfd6b24cdbd
+  data.tar.gz: 619072aaeaa9e74e5470b2e8e97610fa9931dd31
+SHA512:
+  metadata.gz: a8b8ae0fb6691de8f43365652b858a0c6adb9da6c755362c9d3e7a2ace98ff381aa3d0aed31eb087a0108eed0c20a1f7acba7275015ddf4dcccec3d23b50f5b6
+  data.tar.gz: 46a744c5574a998dd440aa8db3c966cedcc8c115eeb632142dae8ef8a170d05df25c152943720cecadd9e1b140a048bee82cbaf46f13cefee9b51cf6a3c9cc13

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Jonathan Rochkind
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,114 @@
+[![Build Status](https://travis-ci.org/traject/traject_sequel_writer.svg)](https://travis-ci.org/traject/traject_sequel_writer)
+
+# TrajectSequelWriter
+
+A `Writer` plugin for [traject](https://github.com/traject/traject), that writes to an rdbms using [Sequel](https://github.com/jeremyevans/sequel).
+
+The writer can be used as standard, as the destination of your indexing pipeline. 
+
+It was actually written for a use case where it's used as a "side effect" in a traject `each_record`, writing different data out to an rdbms on the side, while the main indexing is to Solr. This ends up a bit hacky at present but works.  
+
+
+## Installation
+
+We recommend using bundler with a traject project that has dependencies.
+Add this line to your traject project's Gemfile:
+
+```ruby
+gem 'traject_sequel_writer', "~> 1.0"
+```
+
+And then execute:
+
+    $ bundle
+
+Or if you don't use bundler with your traject project, install directly like: 
+
+    $ gem install traject_sequel_writer
+
+## Usage
+
+As a standard traject writer, you can just set a few settings:
+
+~~~ruby
+settings do
+  provide "writer_class_name", "Traject::SequelWriter"
+  # ...
+~~~
+
+You can set up the connection with a [Sequel connection string](http://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html). Here's
+an example for JDBC mysql (under JRuby). Note we discovered the `&characterEncoding=utf8` arg to the JDBC adapter was important. 
+Connection string parameters may vary for platform (MRI vs JDBC in Jruby) and database. 
+
+~~~ruby
+  provide "sequel_writer.connection_string", "jdbc:mysql://dbhost.example.com:3306/database_name?characterEncoding=utf8&user=user&password=password"
+  # You also need to tell the writer what table to write to; the table should already exist
+  provide "sequel_writer.table_name", "my_table"
+~~~
+
+By default, the writer will try to write to every non-pk column defined in your table -- if your Traject::Context output_hash's is missing
+a value for a column, null will be inserted for that column. Or you can explicitly define which columns to use:
+
+~~~ruby
+  provide "sequel_writer.columns", ["column1", "column2"]
+~~~~
+
+Still, your Context output_hash's must provide output key/values for every column mentioned, or else
+null will be inserted for that column. Keys in the output_hash that don't match output columns
+will be ignored. 
+
+### All settings
+
+* `sequel_writer.connection_string` : [Sequel connection string](http://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html)
+* `sequel_writer.database`: As an alternative to `sequel_connection_string`, pass in an already instantiated Sequel::Database object, as in from `Sequel.connect`
+* `sequel_writer.table_name`: Required, what table to write to. 
+* `sequel_writer.column_names` Which columns to write to, by default all non-pk columns in the table. Since we use multi-row import statements,
+  column_names not present in the Traject::Context#output_hash will end up with SQL `null` inserted. 
+* `sequel_writer.thread_pool_size` Number of threads to use for writing to DB. Default 1, should be good. 
+* `sequel_writer.batch_size` Count of records to batch together in a single multi-row SQL `INSERT`. Default 100. Should be good. 
+
+### Using as a side-channel additional output 
+
+In one project, we wanted to index to Solr. But we wanted to calculate completely different output to send
+as a side-channel to an RDBMS table. Here's a little bit hacky way to do that, that would really work
+for any traject writer. 
+
+~~~ruby
+sequel_writer = Traject::SequelWriter.new(
+  'sequel_writer.connection_string' => conn_str, 
+  'sequel_writer.table_name' => 'my_table')
+
+each_record do |record, context|
+  # imagine this returns an array of hashes, each of which represents a row
+  # you want to insert into the table. 
+
+  rows_to_insert = make_rows_to_insert(record)
+
+  rows_to_insert.each do |row|
+     # Don't re-use the variable name `context`, can cause accidental shared concurrent state
+     sequel_writer.put( Traject::Context.new(:output_hash => row, :source_record => record) )
+   end
+end
+
+# Don't forget to close our side-channel sequel writer, to make
+# sure anything queued gets written
+after_processing do
+  sequel_writer.close
+end
+~~~
+
+
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release` to create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/traject_sequel_writer/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+
+require 'rake'
+require 'rake/testtask'
+
+task :default => [:test]
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/test*.rb']
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "traject_sequel_writer"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/traject/sequel_writer.rb

@@ -0,0 +1,142 @@
+require 'traject'
+require 'traject/util'
+require 'traject/indexer/settings'
+
+require 'sequel'
+
+require 'thread'
+
+module Traject
+  class SequelWriter
+    # Sequel db connection object
+    attr_reader :sequel_db
+
+    # Sequel table/relation object
+    attr_reader :db_table
+
+    def initialize(argSettings)
+      @settings = Traject::Indexer::Settings.new(argSettings)
+
+      unless (!! @settings["sequel_writer.connection_string"]) ^ (!! @settings["sequel_writer.database"]) 
+        raise ArgumentError, "Exactly one of either setting `sequel_writer.connection_string` or `sequel_writer.database` is required"
+      end
+      unless @settings["sequel_writer.table_name"]
+        raise ArgumentError, "setting `sequel_writer.table_name` is required"
+      end
+
+      @sequel_db = @settings["sequel_writer.database"] || Sequel.connect(@settings["sequel_writer.connection_string"])
+      @db_table  = @sequel_db[  @settings["sequel_writer.table_name"].to_sym ]
+
+
+      # Which keys to send to columns? Can be set explicitly with sequel_writer.columns,
+      # or we'll use all non-PK columns introspected from the db schema. 
+      @column_names      = @settings["sequel_writer.columns"]
+
+      unless @column_names
+        @column_names = @sequel_db.schema( @db_table.first_source_table ).find_all do |column, info|
+          info[:primary_key] != true
+        end.collect {|pair| pair.first}
+      end
+      @column_names = @column_names.collect {|c| c.to_sym}
+      @column_names = @column_names.freeze
+
+      
+      # How many threads to use for the writer?
+      # if our thread pool settings are 0, it'll just create a null threadpool that
+      # executes in calling context. Default to 1, for waiting on DB I/O. 
+      @thread_pool_size = (@settings["sequel_writer.thread_pool"] || 1).to_i
+
+      @batch_size       = (@settings["sequel_writer.batch_size"] || 100).to_i
+
+      @batched_queue         = Queue.new
+      @thread_pool = Traject::ThreadPool.new(@thread_pool_size)
+
+      @after_send_batch_callbacks = Array(@settings["sequel_writer.after_send_batch"] || [])
+    end
+
+    # Get the logger from the settings, or default to an effectively null logger
+    def logger
+      @settings["logger"] ||= Yell.new(STDERR, :level => "gt.fatal") # null logger
+    end
+
+    def put(context)
+      @thread_pool.raise_collected_exception!
+
+      @batched_queue << context
+      if @batched_queue.size >= @batch_size
+        batch = Traject::Util.drain_queue(@batched_queue)
+        @thread_pool.maybe_in_thread_pool(batch) {|batch_arg| send_batch(batch_arg) }
+      end
+    end
+
+    def close
+      @thread_pool.raise_collected_exception!
+
+      # Finish off whatever's left. Do it in the thread pool for
+      # consistency, and to ensure expected order of operations, so
+      # it goes to the end of the queue behind any other work.
+      batch = Traject::Util.drain_queue(@batched_queue)
+      @thread_pool.maybe_in_thread_pool(batch) {|batch_arg| send_batch(batch_arg) }
+      
+
+      # Wait for shutdown, and time it.
+      logger.debug "#{self.class.name}: Shutting down thread pool, waiting if needed..."
+      elapsed = @thread_pool.shutdown_and_wait
+      if elapsed > 60
+        logger.warn "Waited #{elapsed} seconds for all threads, you may want to increase sequel_writer.thread_pool (currently #{@settings["solr_writer.thread_pool"]})"
+      end
+      logger.debug "#{self.class.name}: Thread pool shutdown complete"
+
+      # check again now that we've waited, there could still be some
+      # that didn't show up before.
+      @thread_pool.raise_collected_exception!
+
+      @sequel_db.disconnect
+    end
+
+    def send_batch(batch)
+      list_of_arrays = hashes_to_arrays(@column_names, batch.collect {|context| context.output_hash})
+
+      begin
+        db_table.import @column_names, list_of_arrays
+      rescue Sequel::DatabaseError, Sequel::PoolTimeout => batch_exception
+        # We rescue PoolTimeout too, because we're mysteriously getting those, they are maybe dropped DB connections?
+        # Try them each one by one, mostly so we can get a reasonable error message with particular record. 
+        logger.warn("SequelWriter: error (#{batch_exception}) inserting batch of #{list_of_arrays.count} starting from system_id #{batch.first.output_hash['system_id']}, retrying individually...")
+        
+        batch.each do |context|
+          send_single(context)
+        end
+      end
+
+      @after_send_batch_callbacks.each do |callback|
+        callback.call(batch, self)
+      end
+    end
+
+    def send_single(context)      
+      db_table.insert @column_names, hash_to_array(@column_names, context.output_hash)
+    rescue Sequel::DatabaseError => e
+      logger.error("SequelWriter: Could not insert row: #{context.output_hash}: #{e}")
+      raise e
+    end
+
+
+    # Turn an array of hashes into an array of arrays,
+    # with each array being a hashes values matching column_names, in that order
+    def hashes_to_arrays(column_names, list_of_hashes)
+      list_of_hashes.collect do |h| 
+        hash_to_array(column_names, h)
+      end
+    end
+
+    def hash_to_array(column_names, hash)
+      column_names.collect {|c| hash[c.to_s]}
+    end
+
+    def after_send_batch(&block)
+      @after_send_batch_callbacks << block
+    end
+
+  end
+end

data/lib/traject_sequel_writer/version.rb

@@ -0,0 +1,3 @@
+module TrajectSequelWriter
+  VERSION = "0.9.0"
+end

data/lib/traject_sequel_writer.rb

@@ -0,0 +1,5 @@
+require "traject_sequel_writer/version"
+
+module TrajectSequelWriter
+  # Your code goes here...
+end

data/test/test_helper.rb

@@ -0,0 +1,41 @@
+require 'minitest/autorun'
+require 'minitest/spec'
+
+require 'traject_sequel_writer'
+
+require 'minitest/autorun'
+
+require 'sequel'
+
+# Create a temporary sqlite db for tests, remove it when we're done
+require 'fileutils'
+FileUtils::mkdir_p 'tmp'
+
+
+TEST_SEQUEL_CONNECT_STR = if defined? JRUBY_VERSION
+  "jdbc:sqlite:tmp/testing.sqlite"
+else
+  "sqlite://tmp/testing.sqlite"
+end
+
+db = Sequel.connect(TEST_SEQUEL_CONNECT_STR)
+db.drop_table?(:test)
+db.create_table(:test) do
+  primary_key :id
+  Time :created_at
+  String :string_a
+  String :string_b
+  String :string_c
+  TrueClass :boolean_a
+  Integer   :int_a
+end
+
+db[:test].delete
+# Disconnect it now, make the code do the work
+db.disconnect
+
+
+
+MiniTest.after_run do 
+  File.unlink("tmp/testing.sqlite")
+end

data/test/test_traject_sequel_writer.rb

@@ -0,0 +1,113 @@
+require 'test_helper'
+
+require 'traject/sequel_writer'
+
+describe "Traject::SequelWriter" do
+  describe "in use" do
+    before do
+      @writer = writer
+      write_mock_docs(@writer, 63)      
+    end
+
+    after do
+      @writer.db_table.delete
+    end
+
+    it "writes" do
+      assert_equal 63, @writer.db_table.count
+
+      @writer.db_table.each do |hash|
+        assert_kind_of String, hash[:string_a], "string_a is not filled out for #{hash}"
+        assert_kind_of String, hash[:string_b], "string_b is not filled out for #{hash}"
+        assert_nil hash[:string_c]
+        assert_includes [true, false], hash[:boolean_a], "boolean_a is not true or false for #{hash}"
+        assert_kind_of Integer, hash[:int_a]
+      end
+    end
+  end
+
+
+  it "writes with sequel.database parameter instead of connection_str" do
+    sequel_db = Sequel.connect(TEST_SEQUEL_CONNECT_STR)
+
+    writer = writer("sequel_writer.connection_string" => nil, 
+      "sequel_writer.database" => sequel_db)
+
+    write_mock_docs(writer, 63)
+
+    assert_equal 63, writer.db_table.count
+
+    writer.db_table.each do |hash|
+      assert_kind_of String, hash[:string_a], "string_a is not filled out for #{hash}"
+      assert_kind_of String, hash[:string_b], "string_b is not filled out for #{hash}"
+      assert_nil hash[:string_c]
+      assert_includes [true, false], hash[:boolean_a], "boolean_a is not true or false for #{hash}"
+      assert_kind_of Integer, hash[:int_a]
+    end
+
+    writer.db_table.delete
+  end
+
+  describe "after_send_batch" do
+    before do
+      @writer = writer
+
+      @received_in_batches = Queue.new
+
+      @writer.after_send_batch do |batch, writer|
+        assert_kind_of Traject::SequelWriter, writer
+        assert_kind_of Array, batch       
+
+        batch.each {|c| @received_in_batches.push c }
+      end
+
+      @num_docs = 63
+
+      write_mock_docs(@writer, @num_docs)
+    end
+
+    after do      
+      @writer.db_table.delete
+    end
+
+    it "calls callbacks" do
+      assert_equal @num_docs, @received_in_batches.size
+    end
+  end
+
+  describe "errors" do
+    it "raises without required settings" do
+      assert_raises(ArgumentError) { Traject::SequelWriter.new }
+      assert_raises(ArgumentError) { Traject::SequelWriter.new("sequel_writer.connect_string" => "foo") }
+      assert_raises(ArgumentError) { Traject::SequelWriter.new("sequel_writer.table_name" => "foo") }
+    end
+  end
+
+
+  # Helpers
+
+  def write_mock_docs(writer, num)
+    (1..num).each do |i|
+        context = Traject::Indexer::Context.new
+        context.output_hash.merge!(
+          "id" => "ignore_me", # should ignore pk by default
+          "string_a" => "String_a #{i}",
+          "string_b" => "String_b #{i}",
+          "no_such_column" => "ignore me",
+          "boolean_a" => (i % 2 == 0) ? true : false,
+          "int_a" => i
+        )
+        writer.put context
+      end
+      writer.close
+  end
+
+  def writer(args = {})
+    args = {"sequel_writer.connection_string" => TEST_SEQUEL_CONNECT_STR,
+        "sequel_writer.table_name" => "test",
+        "sequel_writer.batch_size" => 5}.merge(args)
+    return Traject::SequelWriter.new(args)
+  end
+
+
+end

metadata

@@ -0,0 +1,126 @@
+--- !ruby/object:Gem::Specification
+name: traject_sequel_writer
+version: !ruby/object:Gem::Version
+  version: 0.9.0
+platform: ruby
+authors:
+- Jonathan Rochkind
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-07-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: traject
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: sequel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.22'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.22'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+description: 
+email:
+- jonathan@dnil.net
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/traject/sequel_writer.rb
+- lib/traject_sequel_writer.rb
+- lib/traject_sequel_writer/version.rb
+- test/test_helper.rb
+- test/test_traject_sequel_writer.rb
+homepage: https://github.com/traject/traject_sequel_writer
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: Plug-in for traject, write to an rdbms with Sequel gem
+test_files:
+- test/test_helper.rb
+- test/test_traject_sequel_writer.rb