loadmop 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: cdf767e8d88611fc7c22d76f75fdd7d84f26243a
+  data.tar.gz: 82a718413d1ad1ea267713fe341329f08bd47b2b
+SHA512:
+  metadata.gz: 225f8f6fd5d8ad11322d54bbe0c8523ede5048a4417dd51e751335fec70e38c4b812c4769cc5c1f968138da0cadc295daf3c94c0935496f85222cbb4d4041bdf
+  data.tar.gz: 5f4c71425f17b4c045a139da9cc18c325571a328e86c91b94897441070228d9e093075b1d7a002ce82aa0dac6825073020ba88bcf03d38e55211b3eb5dfd1ff8

data/.gitignore

@@ -0,0 +1,22 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+## 0.0.2 - 2014-07-11
+
+### Added
+- Nothing.
+
+### Deprecated
+- Nothing.
+
+### Removed
+- Nothing.
+
+### Fixed
+- Bug where schemas directory was not found when loadmop was installed as a gem
+
+## 0.0.1 - 2014-07-10
+
+### Added
+- This project
+
+### Deprecated
+- Nothing.
+
+### Removed
+- Nothing.
+
+### Fixed
+- Nothing.

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+# Specify your gem's dependencies in loadmop.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Outcomes Insights, Inc.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,105 @@
+# Loadmop
+
+Helps load up the OMOP [Vocabulary](http://imeds.reaganudall.org/Vocabularies)/[CDM](http://omop.org/CDM) into the database of your choice.
+
+## Requirements
+
+- Ruby 2.0+
+- [Bundler](http://bundler.io/)
+- Some sort of RDBMS to store OMOP Vocabulary files in
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'loadmop'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install loadmop
+
+## Usage
+Run `bundle exec loadmop` with no arguments to get a help screen and figure out which command you want to run
+
+### Preparation
+Create a .env file in the root directory of the clone, specifying:
+
+- SEQUELIZER_DATABASE
+    - The name of the database you want to install to
+    - Required
+- SEQUELIZER_ADAPTER
+    - The name of the adapter to use (e.g. postgres, sqlite, mysql, oracle)
+    - Required
+- SEQUELIZER_HOST
+    - The host on which the database lives
+    - Optional
+- SEQUELIZER_USERNAME
+    - The username to connect to the database with
+    - Optional
+- SEQUELIZER_PASSWORD
+    - The password to connect to the database with
+    - Optional
+- SEQUELIZER_SEARCH_PATH
+    - At least for PostgreSQL, specifies a schema to install into
+    - Optional
+
+See the [Sequelizer Gem](https://github.com/outcomesinsights/sequelizer) for [some .env examples](https://github.com/outcomesinsights/sequelizer/tree/master/examples)
+
+Then:
+
+- **Create the database you just specified in your .env file**
+    - loadmop isn't (yet) cool enough to actually create the database for you
+    - If you're using SQLite, you don't have to create the database file
+- cd into a directory where you've defined a config/database.yml or .env file that is compatible with Sequelizer
+- run `bundle install` to make sure you have all the needed dependencies installed
+- run `bundle exec sequelizer config` to ensure your connection parameters are correctly set
+- run `bundle exec sequelizer update_gemfile` to ensure your Gemfile has the right database gem
+
+### Loading Vocabulary Files
+
+- Download the [OMOP Vocabulary Files](http://vocabbuild.omop.org/vocabulary-release) and unzip them to some directory.
+    - I recommend you use the restricted files since they include CPT codes and other useful vocabularies.
+- Run `bundle exec loadmop create_vocab_database /path/to/directory/holding/unzipped/vocabulary/files`
+    - This runs all the steps for setting up the vocabulary database, namely
+        - Creating the proper tables
+        - Prepping the CSV files to load into the database
+        - Loading the CSV files into the database
+        - Adding some useful indexes to the vocabulary tables
+
+
+### Loading CDM Data
+- Run `bundle exec loadmop create_vocab_database /path/to/directory/holding/cdm/data/files`
+    - This runs all the steps for loading CDM data into a database, namely
+        - Creating the proper CDM tables
+        - Prepping the CSV files to load into the database
+        - Loading the CSV files into the database
+        - Adding some useful indexes to the CDM tables
+
+## Pleas for Help
+
+I've written methods to quickly load the data into PostgreSQL and SQLite, but I don't regularly use many other RDBMSs.  Right now they use a method that should work for all RDBMSs, but is pretty slow.
+
+Some of the faster methods are Unix-only as well.  If there are fast, platform-independent ways to load the data, I'm interested.
+
+Please submit suggestions or pull requests to speed up loading under other RDBMSs and I'll incorporate them.  Thanks!
+
+## Contributing
+
+1. Fork it ( https://github.com/outcomesinsights/loadmop/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request
+
+## Thanks
+
+- [Outcomes Insights, Inc.](http://outins.com)
+    - Many thanks for allowing me to release a portion of my work as Open Source Software!
+- [IMEDS](http://imeds.reaganudall.org/Vocabularies)/[OMOP](http://omop.org)/[OHDSI](http://ohdsi.org)
+    - For developing the vocabulary files.  What a fantastic resource!
+- [OMOP](http://omop.org)/[OHDSI](http://ohdsi.org)
+    - For developing their [CDM](http://omop.org/CDM)

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/bin/loadmop

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+require_relative '../lib/loadmop/cli'
+
+Loadmop::CLI.start(ARGV)

data/config/database.yml.example

@@ -0,0 +1,2 @@
+adapter: sqlite
+database: omop_vocabulary.sqlite

data/lib/loadmop.rb

@@ -0,0 +1,5 @@
+require 'loadmop/version'
+
+module Loadmop
+  # Your code goes here...
+end

data/lib/loadmop/cdmv4_loader.rb

@@ -0,0 +1,54 @@
+require_relative 'loader'
+
+module Loadmop
+  class CDMv4Loader < Loader
+
+    def initialize(*args)
+      super(*args)
+      set_schema_if_necessary
+    end
+
+    private
+
+    def schemas_dir
+      'schemas/cdmv4'
+    end
+
+    def files
+      ordered_file_names.map { |name| data_files_dir + name }.select(&:exist?)
+    end
+
+    def set_schema_if_necessary
+      return unless adapter == 'postgres'
+      return unless options[:schema]
+      db.create_schema(options[:schema], if_not_exists: true)
+      db.execute("SET search_path TO #{options[:schema]}")
+    end
+
+    def ordered_tables
+      %w(
+        person
+        visit_occurrence
+        condition_occurrence
+        procedure_occurrence
+        procedure_cost
+        observation
+        observation_period
+        payer_plan_period
+        drug_exposure
+        drug_cost
+        location
+        organization
+        provider
+        care_site
+        condition_era
+        death
+        cohort
+      )
+    end
+
+    def ordered_file_names
+      ordered_tables.map { |f| f + '.csv' }
+    end
+  end
+end

data/lib/loadmop/cli.rb

@@ -0,0 +1,23 @@
+require 'thor'
+require_relative 'cdmv4_loader'
+require_relative 'vocab_loader'
+
+module Loadmop
+  class CLI < Thor
+    include Sequelizer
+
+    desc 'create_vocab_database vocab_files_dir', 'Connects to database specified in the .env file and loads the OMOP Vocabulary schema and data into it'
+    def create_vocab_database(vocab_files_dir)
+      loader = VocabLoader.new(vocab_files_dir)
+      loader.create_database
+    end
+
+    desc 'create_cdmv4_data data_dir', 'Creates a schema in a database for OMOP CDMv4 then loads data into it as specified by data_files_dir'
+    option :schema, type: :string, desc: 'PostgreSQL ONLY - specifies the schema within the database that will house the CDM tables and data'
+    def create_cdmv4_database(data_files_dir)
+      cdm_loader = CDMv4Loader.new(data_files_dir, options)
+      cdm_loader.create_database
+    end
+  end
+end
+

data/lib/loadmop/loader.rb

@@ -0,0 +1,169 @@
+require 'pathname'
+require 'csv'
+require 'sequelizer'
+
+module Loadmop
+  class Loader
+    include Sequelizer
+
+    attr :options, :data_files_dir, :headers
+
+    def initialize(data_files_dir, options = {})
+      @data_files_dir = Pathname.new(data_files_dir)
+      @options = options
+      @headers = {}
+    end
+
+    def create_database
+      create_tables
+      load_files
+      create_indexes
+    end
+
+    private
+    def create_tables
+      Sequel.extension :migration
+      Sequel::Migrator.run(db, base_dir + schemas_dir, target: 1)
+    end
+
+    def create_indexes
+      Sequel.extension :migration
+      Sequel::Migrator.run(db, base_dir + schemas_dir, target: 2)
+    end
+
+    def all_files
+      @all_files ||= make_all_files
+    end
+
+    def load_files
+      fast_load || slow_load
+    end
+
+    def fast_load
+      case adapter
+      when 'postgres'
+        fast_load_postgres
+      when 'sqlite'
+        fast_load_sqlite
+      else
+        nil
+      end
+    end
+
+    def fast_load_postgres
+      all_files.each do |table_name, files|
+        db[table_name].truncate(cascade: true)
+        files.each do |file|
+          puts "Loading #{file} into #{table_name}"
+          db.copy_into(
+            table_name,
+            format:  :csv,
+            columns: headers[table_name],
+            data:    File.read(file)
+          )
+        end
+      end
+      true
+    end
+
+    def fast_load_sqlite
+      return nil if `which sqlite3` =~ /not found/i
+      all_files.each do |table_name, files|
+        run_sqlite_commands(%Q(DELETE FROM #{table_name};))
+        commands = []
+        commands << %Q(.echo on)
+        commands << %Q(.log stdout)
+        commands << %Q(.mode csv)
+        commands << "placeholder"
+        files.each do |file|
+          puts "Loading #{file} into #{table_name}"
+          commands.pop
+          commands << ".import #{file} #{table_name}"
+          run_sqlite_commands(commands)
+        end
+      end
+      true
+    end
+
+    def run_sqlite_commands(*commands)
+      db_file_path = database
+      File.open('/tmp/sqlite.load', 'w') do |command_file|
+        command_file.puts commands.flatten.join("\n")
+      end
+      command = "sqlite3 #{db_file_path} '.read /tmp/sqlite.load'"
+      puts command
+      system(command)
+    end
+
+    def slow_load
+      all_files.each do |table_name, files|
+        files.each do |file|
+          puts "Loading #{file} into #{table_name}"
+          CSV.open(file, headers: true) do |csv|
+            csv.each_slice(1000) do |rows|
+              print '.'
+              db[table_name].import(headers_for(file), rows.map(&:fields))
+            end
+          end
+          puts
+        end
+      end
+    end
+
+    def adapter
+      db.sequelizer_options.adapter
+    end
+
+    def database
+      db.sequelizer_options.database
+    end
+
+    def headers_for(file)
+      header_line = File.open(file, &:readline).downcase.gsub(/\|$/, '')
+      CSV.parse(header_line).first.map(&:to_sym)
+    end
+
+    def files_of_interest
+      Pathname.glob(data_files_dir + '*.csv')
+    end
+
+    def lines_per_split
+      100000
+    end
+
+    def additional_cleaning_steps
+      []
+    end
+
+    def make_all_files
+      split_dir = data_files_dir + 'split'
+      split_dir.mkdir unless split_dir.exist?
+      files = files_of_interest.map do |file|
+        table_name = file.basename('.*').to_s.downcase.to_sym
+        headers[table_name] = headers_for(file)
+        dir = split_dir + table_name.to_s
+        unless dir.exist?
+          dir.mkdir
+          Dir.chdir(dir) do
+            puts "Splitting #{file}"
+            steps = []
+            steps << "tail -n +2 #{file.expand_path}"
+            steps += additional_cleaning_steps
+            steps << "split -a 5 -l #{lines_per_split}"
+            system(steps.compact.join(" | "))
+          end
+        end
+        [table_name, dir.children.sort]
+      end
+
+      Hash[files]
+    end
+
+    # When installed as a gem, we need to find where the gem is installed
+    # and look for files relative to that path.  base_dir returns the
+    # path to the loadmop gem
+    def base_dir
+      Pathname.new(__FILE__).dirname + '../..'
+    end
+  end
+end

data/lib/loadmop/version.rb

@@ -0,0 +1,3 @@
+module Loadmop
+  VERSION = "0.0.2"
+end

data/lib/loadmop/vocab_loader.rb

@@ -0,0 +1,17 @@
+require_relative 'loader'
+
+module Loadmop
+  class VocabLoader < Loader
+
+    private
+
+    def schemas_dir
+      'schemas/vocabulary'
+    end
+
+    def additional_cleaning_steps
+      ["sed 's/\|$//'"]
+    end
+  end
+end
+

data/loadmop.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'loadmop/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'loadmop'
+  spec.version       = Loadmop::VERSION
+  spec.authors       = ['Ryan Duryea']
+  spec.email         = ['aguynamedryan@gmail.com']
+  spec.summary       = %q{Ruby-based script to load OMOP Vocabulary/CDMv4 data into a database}
+  spec.description   = %q{loadmop assits in loading OMOP Vocabulary files and OMOP CDMv4-compatible CSV files into your database}
+  spec.homepage      = 'https://github.com/outcomesinsights/loadmop'
+  spec.license       = 'MIT'
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ['lib']
+
+  spec.add_development_dependency 'bundler', '~> 1.6'
+  spec.add_dependency 'rake', '~> 10.3'
+  spec.add_dependency 'sequelizer', '~> 0.0'
+  spec.add_dependency 'thor', '~> 0.19'
+end

data/schemas/cdmv4/001_tables.rb

@@ -0,0 +1,256 @@
+Sequel.migration do
+  change do
+    create_table(:care_site, :ignore_index_errors=>true) do
+      Bignum :care_site_id, :null=>false
+      Bignum :location_id, :null=>false
+      Bignum :organization_id, :null=>false
+      Bignum :place_of_service_concept_id
+      String :care_site_source_value, :size=>50
+      String :place_of_service_source_value, :size=>50, :null=>false
+
+      primary_key [:care_site_id]
+    end
+
+    create_table(:cohort, :ignore_index_errors=>true) do
+      Bignum :cohort_id, :null=>false
+      Bignum :cohort_concept_id, :null=>false
+      Date :cohort_start_date, :null=>false
+      Date :cohort_end_date
+      Bignum :subject_id, :null=>false
+      String :stop_reason, :size=>20
+
+      primary_key [:cohort_id]
+    end
+
+    create_table(:location, :ignore_index_errors=>true) do
+      Bignum :location_id, :null=>false
+      String :address_1, :size=>50
+      String :address_2, :size=>50
+      String :city, :size=>50
+      String :state, :size=>2, :fixed=>true
+      String :zip, :size=>9
+      String :county, :size=>20
+      String :location_source_value, :size=>50
+
+      primary_key [:location_id]
+    end
+
+    create_table(:provider, :ignore_index_errors=>true) do
+      Bignum :provider_id, :null=>false
+      String :npi, :size=>20
+      String :dea, :size=>20
+      Bignum :specialty_concept_id
+      Bignum :care_site_id, :null=>false
+      String :provider_source_value, :size=>50, :null=>false
+      String :specialty_source_value, :size=>50
+
+      primary_key [:provider_id]
+    end
+
+    create_table(:organization, :ignore_index_errors=>true) do
+      Bignum :organization_id, :null=>false
+      Bignum :place_of_service_concept_id
+      foreign_key :location_id, :location, :type=>Bignum, :key=>[:location_id]
+      String :organization_source_value, :size=>50, :null=>false
+      String :place_of_service_source_value, :size=>50
+
+      primary_key [:organization_id]
+    end
+
+    create_table(:person, :ignore_index_errors=>true) do
+      Bignum :person_id, :null=>false
+      Bignum :gender_concept_id, :null=>false
+      Integer :year_of_birth, :null=>false
+      Integer :month_of_birth
+      Integer :day_of_birth
+      Bignum :race_concept_id
+      Bignum :ethnicity_concept_id
+      foreign_key :location_id, :location, :type=>Bignum, :key=>[:location_id]
+      foreign_key :provider_id, :provider, :type=>Bignum, :key=>[:provider_id]
+      Bignum :care_site_id
+      String :person_source_value, :size=>50
+      String :gender_source_value, :size=>50
+      String :race_source_value, :size=>50
+      String :ethnicity_source_value, :size=>50
+
+      primary_key [:person_id]
+    end
+
+    create_table(:condition_era, :ignore_index_errors=>true) do
+      Bignum :condition_era_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Bignum :condition_concept_id, :null=>false
+      Date :condition_era_start_date, :null=>false
+      Date :condition_era_end_date, :null=>false
+      Bignum :condition_type_concept_id, :null=>false
+      Integer :condition_occurrence_count
+
+      primary_key [:condition_era_id]
+    end
+
+    create_table(:condition_occurrence, :ignore_index_errors=>true) do
+      Bignum :condition_occurrence_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Bignum :condition_concept_id, :null=>false
+      Date :condition_start_date, :null=>false
+      Date :condition_end_date
+      Bignum :condition_type_concept_id, :null=>false
+      String :stop_reason, :size=>20
+      Bignum :associated_provider_id
+      Bignum :visit_occurrence_id
+      String :condition_source_value, :size=>50
+
+      primary_key [:condition_occurrence_id]
+    end
+
+    create_table(:death, :ignore_index_errors=>true) do
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Date :death_date, :null=>false
+      Bignum :death_type_concept_id, :null=>false
+      Bignum :cause_of_death_concept_id
+      String :cause_of_death_source_value, :size=>50
+
+      primary_key [:person_id]
+    end
+
+    create_table(:drug_era, :ignore_index_errors=>true) do
+      Bignum :drug_era_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Bignum :drug_concept_id, :null=>false
+      Date :drug_era_start_date, :null=>false
+      Date :drug_era_end_date, :null=>false
+      Bignum :drug_type_concept_id, :null=>false
+      Integer :drug_exposure_count
+
+      primary_key [:drug_era_id]
+    end
+
+    create_table(:drug_exposure, :ignore_index_errors=>true) do
+      Bignum :drug_exposure_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Bignum :drug_concept_id, :null=>false
+      Date :drug_exposure_start_date, :null=>false
+      Date :drug_exposure_end_date
+      Bignum :drug_type_concept_id, :null=>false
+      String :stop_reason, :size=>20
+      Integer :refills
+      Integer :quantity
+      Integer :days_supply
+      String :sig, :size=>500
+      Bignum :prescribing_provider_id
+      Bignum :visit_occurrence_id
+      Bignum :relevant_condition_concept_id
+      String :drug_source_value, :size=>50
+
+      primary_key [:drug_exposure_id]
+    end
+
+    create_table(:observation, :ignore_index_errors=>true) do
+      Bignum :observation_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Bignum :observation_concept_id, :null=>false
+      Date :observation_date, :null=>false
+      Date :observation_time
+      Float :value_as_number
+      String :value_as_string, :size=>60
+      Bignum :value_as_concept_id
+      Bignum :unit_concept_id
+      Float :range_low
+      Float :range_high
+      Bignum :observation_type_concept_id, :null=>false
+      Bignum :associated_provider_id
+      Bignum :visit_occurrence_id
+      Bignum :relevant_condition_concept_id
+      String :observation_source_value, :size=>50
+      String :units_source_value, :size=>50
+
+      primary_key [:observation_id]
+    end
+
+    create_table(:observation_period, :ignore_index_errors=>true) do
+      Bignum :observation_period_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Date :observation_period_start_date, :null=>false
+      Date :observation_period_end_date, :null=>false
+      Date :prev_ds_period_end_date
+
+      primary_key [:observation_period_id]
+    end
+
+    create_table(:payer_plan_period, :ignore_index_errors=>true) do
+      Bignum :payer_plan_period_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Date :payer_plan_period_start_date, :null=>false
+      Date :payer_plan_period_end_date, :null=>false
+      String :payer_source_value, :size=>50
+      String :plan_source_value, :size=>50
+      String :family_source_value, :size=>50
+      Date :prev_ds_period_end_date
+
+      primary_key [:payer_plan_period_id]
+    end
+
+    create_table(:procedure_occurrence, :ignore_index_errors=>true) do
+      Bignum :procedure_occurrence_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Bignum :procedure_concept_id, :null=>false
+      Date :procedure_date, :null=>false
+      Bignum :procedure_type_concept_id, :null=>false
+      Bignum :associated_provider_id
+      Bignum :visit_occurrence_id
+      Bignum :relevant_condition_concept_id
+      String :procedure_source_value, :size=>50
+
+      primary_key [:procedure_occurrence_id]
+    end
+
+    create_table(:visit_occurrence, :ignore_index_errors=>true) do
+      Bignum :visit_occurrence_id, :null=>false
+      foreign_key :person_id, :person, :type=>Bignum, :null=>false, :key=>[:person_id]
+      Date :visit_start_date, :null=>false
+      Date :visit_end_date, :null=>false
+      Bignum :place_of_service_concept_id, :null=>false
+      Bignum :care_site_id
+      String :place_of_service_source_value, :size=>50
+
+      primary_key [:visit_occurrence_id]
+    end
+
+    create_table(:drug_cost, :ignore_index_errors=>true) do
+      Bignum :drug_cost_id, :null=>false
+      foreign_key :drug_exposure_id, :drug_exposure, :type=>Bignum, :null=>false, :key=>[:drug_exposure_id]
+      Float :paid_copay
+      Float :paid_coinsurance
+      Float :paid_toward_deductible
+      Float :paid_by_payer
+      Float :paid_by_coordination_benefits
+      Float :total_out_of_pocket
+      Float :total_paid
+      Float :ingredient_cost
+      Float :dispensing_fee
+      Float :average_wholesale_price
+      Bignum :payer_plan_period_id
+
+      primary_key [:drug_cost_id]
+    end
+
+    create_table(:procedure_cost, :ignore_index_errors=>true) do
+      Bignum :procedure_cost_id, :null=>false
+      foreign_key :procedure_occurrence_id, :procedure_occurrence, :type=>Bignum, :null=>false, :key=>[:procedure_occurrence_id]
+      Float :paid_copay
+      Float :paid_coinsurance
+      Float :paid_toward_deductible
+      Float :paid_by_payer
+      Float :paid_by_coordination_benefits
+      Float :total_out_of_pocket
+      Float :total_paid
+      Bignum :disease_class_concept_id
+      Bignum :revenue_code_concept_id
+      Bignum :payer_plan_period_id
+      String :disease_class_source_value, :size=>50
+      String :revenue_code_source_value, :size=>50
+
+      primary_key [:procedure_cost_id]
+    end
+  end
+end

data/schemas/cdmv4/002_indexes.rb

@@ -0,0 +1,145 @@
+Sequel.migration do
+  change do
+    alter_table(:care_site) do
+      add_index [:location_id, :organization_id, :place_of_service_source_value], :name=>:care_site_location_id_organization_id_place_of_service_sour_key, :unique=>true, :ignore_add_index_errors=>true
+      add_index [:care_site_id], :name=>:carsit_carsitid, :ignore_add_index_errors=>true
+      add_index [:location_id], :name=>:carsit_locid, :ignore_add_index_errors=>true
+      add_index [:organization_id], :name=>:carsit_orgid, :ignore_add_index_errors=>true
+      add_index [:place_of_service_concept_id], :name=>:carsit_plaofserconid, :ignore_add_index_errors=>true
+      add_index [:location_id, :organization_id, :place_of_service_source_value, :care_site_id], :name=>:idx_care_site_org_sources, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:cohort) do
+      add_index [:cohort_concept_id], :name=>:coh_cohconid, :ignore_add_index_errors=>true
+      add_index [:cohort_id], :name=>:coh_cohid, :ignore_add_index_errors=>true
+      add_index [:subject_id], :name=>:coh_subid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:location) do
+      add_index [:location_id], :name=>:loc_locid, :ignore_add_index_errors=>true
+      add_index [:zip, :county], :name=>:location_zip_county_key, :unique=>true, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:provider) do
+      add_index [:provider_source_value, :specialty_source_value, :provider_id, :care_site_id], :name=>:idx_provider_lkp, :ignore_add_index_errors=>true
+      add_index [:care_site_id], :name=>:pro_carsitid, :ignore_add_index_errors=>true
+      add_index [:provider_id], :name=>:pro_proid, :ignore_add_index_errors=>true
+      add_index [:specialty_concept_id], :name=>:pro_speconid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:organization) do
+      add_index [:location_id], :name=>:org_locid, :ignore_add_index_errors=>true
+      add_index [:organization_id], :name=>:org_orgid, :ignore_add_index_errors=>true
+      add_index [:place_of_service_concept_id], :name=>:org_plaofserconid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:person) do
+      add_index [:care_site_id], :name=>:per_carsitid, :ignore_add_index_errors=>true
+      add_index [:ethnicity_concept_id], :name=>:per_ethconid, :ignore_add_index_errors=>true
+      add_index [:gender_concept_id], :name=>:per_genconid, :ignore_add_index_errors=>true
+      add_index [:location_id], :name=>:per_locid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:per_perid, :ignore_add_index_errors=>true
+      add_index [:provider_id], :name=>:per_proid, :ignore_add_index_errors=>true
+      add_index [:race_concept_id], :name=>:per_racconid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:condition_era) do
+      add_index [:condition_concept_id], :name=>:conera_conconid, :ignore_add_index_errors=>true
+      add_index [:condition_era_id], :name=>:conera_coneraid, :ignore_add_index_errors=>true
+      add_index [:condition_type_concept_id], :name=>:conera_contypconid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:conera_perid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:condition_occurrence) do
+      add_index [:condition_concept_id], :name=>:cci, :ignore_add_index_errors=>true
+      add_index [:associated_provider_id], :name=>:conocc_assproid, :ignore_add_index_errors=>true
+      add_index [:condition_concept_id], :name=>:conocc_conconid, :ignore_add_index_errors=>true
+      add_index [:condition_occurrence_id], :name=>:conocc_conoccid, :ignore_add_index_errors=>true
+      add_index [:condition_source_value], :name=>:conocc_consouval, :ignore_add_index_errors=>true
+      add_index [:condition_type_concept_id], :name=>:conocc_contypconid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:conocc_perid, :ignore_add_index_errors=>true
+      add_index [:visit_occurrence_id], :name=>:conocc_visoccid, :ignore_add_index_errors=>true
+      add_index [:visit_occurrence_id], :name=>:voi, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:death) do
+      add_index [:cause_of_death_concept_id], :name=>:dea_cauofdeaconid, :ignore_add_index_errors=>true
+      add_index [:death_type_concept_id], :name=>:dea_deatypconid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:dea_perid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:drug_era) do
+      add_index [:drug_concept_id], :name=>:druera_druconid, :ignore_add_index_errors=>true
+      add_index [:drug_era_id], :name=>:druera_drueraid, :ignore_add_index_errors=>true
+      add_index [:drug_type_concept_id], :name=>:druera_drutypconid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:druera_perid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:drug_exposure) do
+      add_index [:drug_concept_id], :name=>:druexp_druconid, :ignore_add_index_errors=>true
+      add_index [:drug_exposure_id], :name=>:druexp_druexpid, :ignore_add_index_errors=>true
+      add_index [:drug_type_concept_id], :name=>:druexp_drutypconid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:druexp_perid, :ignore_add_index_errors=>true
+      add_index [:prescribing_provider_id], :name=>:druexp_preproid, :ignore_add_index_errors=>true
+      add_index [:relevant_condition_concept_id], :name=>:druexp_relconconid, :ignore_add_index_errors=>true
+      add_index [:visit_occurrence_id], :name=>:druexp_visoccid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:observation) do
+      add_index [:associated_provider_id], :name=>:obs_assproid, :ignore_add_index_errors=>true
+      add_index [:observation_concept_id], :name=>:obs_obsconid, :ignore_add_index_errors=>true
+      add_index [:observation_id], :name=>:obs_obsid, :ignore_add_index_errors=>true
+      add_index [:observation_type_concept_id], :name=>:obs_obstypconid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:obs_perid, :ignore_add_index_errors=>true
+      add_index [:relevant_condition_concept_id], :name=>:obs_relconconid, :ignore_add_index_errors=>true
+      add_index [:unit_concept_id], :name=>:obs_uniconid, :ignore_add_index_errors=>true
+      add_index [:value_as_concept_id], :name=>:obs_valasconid, :ignore_add_index_errors=>true
+      add_index [:visit_occurrence_id], :name=>:obs_visoccid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:observation_period) do
+      add_index [:person_id, :observation_period_start_date, :observation_period_end_date], :name=>:idx_observation_period_lkp, :ignore_add_index_errors=>true
+      add_index [:observation_period_id], :name=>:obsper_obsperid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:obsper_perid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:payer_plan_period) do
+      add_index [:person_id, :plan_source_value, :payer_plan_period_start_date, :payer_plan_period_end_date], :name=>:idx_payer_plan_period_lkp, :ignore_add_index_errors=>true
+      add_index [:payer_plan_period_id], :name=>:payplaper_payplaperid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:payplaper_perid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:procedure_occurrence) do
+      add_index [:associated_provider_id], :name=>:proocc_assproid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:proocc_perid, :ignore_add_index_errors=>true
+      add_index [:procedure_concept_id], :name=>:proocc_proconid, :ignore_add_index_errors=>true
+      add_index [:procedure_occurrence_id], :name=>:proocc_prooccid, :ignore_add_index_errors=>true
+      add_index [:procedure_source_value], :name=>:proocc_prosouval, :ignore_add_index_errors=>true
+      add_index [:procedure_type_concept_id], :name=>:proocc_protypconid, :ignore_add_index_errors=>true
+      add_index [:relevant_condition_concept_id], :name=>:proocc_relconconid, :ignore_add_index_errors=>true
+      add_index [:visit_occurrence_id], :name=>:proocc_visoccid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:visit_occurrence) do
+      add_index [:person_id, :visit_start_date, :place_of_service_concept_id], :name=>:visit_occurrence_person_id_visit_start_date_place_of_servic_key, :ignore_add_index_errors=>true
+      add_index [:care_site_id], :name=>:visocc_carsitid, :ignore_add_index_errors=>true
+      add_index [:person_id], :name=>:visocc_perid, :ignore_add_index_errors=>true
+      add_index [:place_of_service_concept_id], :name=>:visocc_plaofserconid, :ignore_add_index_errors=>true
+      add_index [:visit_occurrence_id], :name=>:visocc_visoccid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:drug_cost) do
+      add_index [:drug_cost_id], :name=>:drucos_drucosid, :ignore_add_index_errors=>true
+      add_index [:drug_exposure_id], :name=>:drucos_druexpid, :ignore_add_index_errors=>true
+      add_index [:payer_plan_period_id], :name=>:drucos_payplaperid, :ignore_add_index_errors=>true
+    end
+
+    alter_table(:procedure_cost) do
+      add_index [:disease_class_concept_id], :name=>:procos_disclaconid, :ignore_add_index_errors=>true
+      add_index [:payer_plan_period_id], :name=>:procos_payplaperid, :ignore_add_index_errors=>true
+      add_index [:procedure_cost_id], :name=>:procos_procosid, :ignore_add_index_errors=>true
+      add_index [:procedure_occurrence_id], :name=>:procos_prooccid, :ignore_add_index_errors=>true
+      add_index [:revenue_code_concept_id], :name=>:procos_revcodconid, :ignore_add_index_errors=>true
+    end
+  end
+end

data/schemas/vocabulary/001_tables.rb

@@ -0,0 +1,96 @@
+Sequel.migration do
+  change do
+    create_table(:concept) do
+      Bignum :concept_id, :null=>false
+      String :concept_name, :null=>false
+      BigDecimal :concept_level, :null=>false
+      String :concept_class, :null=>false
+      Bignum :vocabulary_id, :null=>false
+      String :concept_code, :null=>false
+      Date :valid_start_date, :null=>false
+      Date :valid_end_date, :null=>false
+      String :invalid_reason, :size=>1, :fixed=>true
+
+      primary_key [:concept_id]
+    end
+
+    create_table(:concept_ancestor) do
+      Bignum :ancestor_concept_id, :null=>false
+      Bignum :descendant_concept_id, :null=>false
+      BigDecimal :min_levels_of_separation
+      BigDecimal :max_levels_of_separation
+
+      primary_key [:ancestor_concept_id, :descendant_concept_id]
+    end
+
+    create_table(:concept_relationship) do
+      Bignum :concept_id_1, :null=>false
+      Bignum :concept_id_2, :null=>false
+      Bignum :relationship_id, :null=>false
+      Date :valid_start_date, :null=>false
+      Date :valid_end_date, :null=>false
+      String :invalid_reason, :size=>1, :fixed=>true
+
+      primary_key [:concept_id_1, :concept_id_2, :relationship_id]
+    end
+
+    create_table(:concept_synonym) do
+      Bignum :concept_synonym_id, :null=>false
+      Bignum :concept_id, :null=>false
+      String :concept_synonym_name, :null=>false
+
+      primary_key [:concept_synonym_id]
+    end
+
+    create_table(:drug_approval) do
+      Bignum :ingredient_concept_id, :null => false
+      Date :approval_date, :null => false
+      String :approved_by, :null => false
+    end
+
+    create_table(:drug_strength) do
+      Bignum :drug_concept_id, :null => false
+      Bignum :ingredient_concept_id, :null => false
+      BigDecimal :amount_value
+      String :amount_unit
+      BigDecimal :concentration_value
+      String :concentration_enum_unit
+      String :concentration_denom_unit
+      Date :valid_start_date, :null => false
+      Date :valid_end_date, :null => false
+      String :invalid_reason
+    end
+
+    create_table(:relationship) do
+      Bignum :relationship_id, :null=>false
+      String :relationship_name, :null=>false
+      String :is_hierarchical, :size=>1, :fixed=>true
+      String :defines_ancestry, :size=>1, :fixed=>true
+      Bignum :reverse_relationship
+
+      primary_key [:relationship_id]
+    end
+
+    create_table(:source_to_concept_map) do
+      String :source_code, :null=>false
+      Bignum :source_vocabulary_id, :null=>false
+      String :source_code_description
+      Bignum :target_concept_id, :null=>false
+      Bignum :target_vocabulary_id, :null=>false
+      String :mapping_type
+      String :primary_map, :size=>1, :fixed=>true
+      Date :valid_start_date, :null=>false
+      Date :valid_end_date, :null=>false
+      String :invalid_reason, :size=>1, :fixed=>true
+
+      primary_key [:source_code, :source_vocabulary_id, :target_concept_id, :valid_end_date]
+    end
+
+    create_table(:vocabulary) do
+      Bignum :vocabulary_id, :null=>false
+      String :vocabulary_name, :null=>false
+
+      primary_key [:vocabulary_id]
+    end
+  end
+end

data/schemas/vocabulary/002_indexes.rb

@@ -0,0 +1,24 @@
+
+Sequel.migration do
+  change do
+    puts "Adding indexes"
+    def do_index(*args)
+      puts args.inspect
+      add_index(*args)
+    rescue
+      puts $!.message
+      #nothing
+    end
+    do_index :concept, [:concept_code], name: "vocabulary_concept_concept_code_index"
+    do_index :concept, [:concept_id], name: "con_conid"
+    do_index :concept, [:vocabulary_id], name: "con_vocid"
+    do_index :concept_ancestor, [:ancestor_concept_id], name: "conanc_ancconid"
+    do_index :concept_ancestor, [:descendant_concept_id], name: "conanc_desconid"
+    do_index :concept_relationship, [:relationship_id], name: "conrel_relid"
+    do_index :relationship, [:relationship_id], name: "rel_relid"
+    do_index :source_to_concept_map, [:source_code, :source_vocabulary_id], name: "soutoconmap_soucod_souvocid"
+    do_index :source_to_concept_map, [:source_vocabulary_id], name: "soutoconmap_souvocid"
+    do_index :source_to_concept_map, [:target_concept_id], name: "soutoconmap_tarconid"
+    do_index :source_to_concept_map, [:target_vocabulary_id], name: "soutoconmap_tarvocid"
+  end
+end

metadata

@@ -0,0 +1,122 @@
+--- !ruby/object:Gem::Specification
+name: loadmop
+version: !ruby/object:Gem::Version
+  version: 0.0.2
+platform: ruby
+authors:
+- Ryan Duryea
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-07-11 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.3'
+- !ruby/object:Gem::Dependency
+  name: sequelizer
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.0'
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.19'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.19'
+description: loadmop assits in loading OMOP Vocabulary files and OMOP CDMv4-compatible
+  CSV files into your database
+email:
+- aguynamedryan@gmail.com
+executables:
+- loadmop
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- CHANGELOG.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/loadmop
+- config/database.yml.example
+- lib/loadmop.rb
+- lib/loadmop/cdmv4_loader.rb
+- lib/loadmop/cli.rb
+- lib/loadmop/loader.rb
+- lib/loadmop/version.rb
+- lib/loadmop/vocab_loader.rb
+- loadmop.gemspec
+- schemas/cdmv4/001_tables.rb
+- schemas/cdmv4/002_indexes.rb
+- schemas/vocabulary/001_tables.rb
+- schemas/vocabulary/002_indexes.rb
+homepage: https://github.com/outcomesinsights/loadmop
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Ruby-based script to load OMOP Vocabulary/CDMv4 data into a database
+test_files: []
+has_rdoc: