ncs_mdes_warehouse 0.0.1

data/.gitignore

@@ -0,0 +1,7 @@
+tmp
+Gemfile.lock
+.yardoc
+docs
+reports
+*.sqlite3
+logs

data/.rvmrc

@@ -0,0 +1 @@
+rvm ruby-1.9.2-p290@ncs_mdes_warehouse

data/.yardopts

@@ -0,0 +1,7 @@
+--no-private
+--markup markdown
+--hide-void-return
+--files CHANGELOG.md
+--readme README.md
+lib/**/*.rb
+generated_models/**/*.rb

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+NCS Navigator MDES Warehouse History
+====================================
+
+0.0.1
+-----
+
+- Initial version. Includes models for MDES 2.0 Patch and most
+  peripheral functionality. Actual ETL still pending.

data/Gemfile

@@ -0,0 +1,13 @@
+source :rubygems
+
+gemspec
+
+group :development do
+  # for YARD
+  gem 'rdiscount'
+  gem 'ci_reporter', :git => 'git://github.com/nicksieger/ci_reporter.git'
+end
+
+group :test do
+  gem 'dm-sqlite-adapter', '~> 1.2.0'
+end

data/README.md

@@ -0,0 +1,154 @@
+# NCS Navigator MDES Warehouse
+
+MDES Warehouse is the central collation and reporting engine for the
+NCS Navigator suite. It provides the infrastructure to support
+transforming data from the operational and data-collection systems and
+converting it into the format required for reporting to the National
+Children's Study program office.
+
+While it has been built for the NCS Navigator suite, it exposes
+generic attachment points for transformers, potentially allowing it to
+be used with other sets of transactional systems for the National
+Children's Study.
+
+## System Requirements
+
+MDES Warehouse is installed as a ruby gem. It's compatible with Ruby
+1.8.7 (including REE) and 1.9.2.  It relies on [PostgreSQL][pgsql] 9
+(or later).
+
+[pgsql]: http://www.postgresql.org/
+
+## Set up
+
+### Install the gem
+
+    $ gem install ncs_mdes_warehouse
+
+This will install the gem, all its dependencies, and the `mdes-wh`
+executable.
+
+### Set the environment
+
+The warehouse detects the kind of environment it's running in
+(production, staging, or development) via the `NCS_NAVIGATOR_ENV`
+environment variable. Arrange for this variable to be set to an
+appropriate value for the server where you're deploying the warehouse:
+
+    # In, e.g., /etc/bashrc
+    export NCS_NAVIGATOR_ENV=staging
+
+If you're deploying on a workstation (i.e., for development) you can
+skip this step -- the default environment is development.
+
+### Configure the NCS Navigator suite
+
+If you haven't already, create `/etc/nubic/ncs/navigator.ini` by
+following the [example in the ncs_navigator_configuration
+gem][confex].
+
+[confex]: http://rubydoc.info/gems/ncs_navigator_configuration/file/sample_configuration.ini
+
+### Create the PostgreSQL databases
+
+The warehouse requires two PostgreSQL databases.  There are several
+ways to create them. `createdb` is one:
+
+    $ createuser -e -h dbserver.my.org -PRSD mdes_warehouse
+    Enter password for new role:
+    Enter it again:
+    CREATE ROLE mdes_warehouse PASSWORD 'md5...' NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;
+    $ createdb -e -O mdes_warehouse mdes_warehouse_working
+    CREATE DATABASE mdes_warehouse_working OWNER mdes_warehouse;
+    $ createdb -e -O mdes_warehouse mdes_warehouse_reporting
+    CREATE DATABASE mdes_warehouse_reporting OWNER mdes_warehouse;
+
+### Configure bcdatabase
+
+Like the rest of the NCS Navigator suite, MDES Warehouse uses
+[bcdatabase][] to locate database access parameters and
+credentials. Bcdatabase looks for credentials under
+`/etc/nubic/db`. The specific file that the warehouse looks in will
+depend on the environment in which you're running:
+
+  * development: `local_postgresql.yml`
+  * staging: `ncsdb_staging.yml`
+  * production: `ncsdb_prod.yml`
+
+Create or update the appropriate file with the warehouse configuration
+for your environment. The below is an example; yours may differ. The
+MDES Warehouse-specifc parts are the `datamapper_adapter`
+configuration key and the two `mdes_warehouse_*` entries.
+
+    defaults:
+      host: ncsdb-staging.my.org
+      adapter: postgresql
+      datamapper_adapter: postgres
+    mdes_warehouse_working:
+      username: mdes_warehouse
+      password: whatever-i-entered-before
+    mdes_warehouse_reporting:
+      username: mdes_warehouse
+      password: whatever-i-entered-before
+
+[bcdatabase]: http://rubydoc.info/gems/bcdatabase/frames
+
+### Configure warehouse
+
+The warehouse takes its base configuration from a file under the
+directory `/etc/nubic/ncs/warehouse`. The name of the file should be
+`${NCS_NAVIGATOR_ENV}.rb`; e.g., for the production environment, the
+full path would be `/etc/nubic/ncs/warehouse/production.rb`.
+
+The minimum contents of this file is a list of transformers, which are
+components that translate data from some source system or systems:
+
+    c.add_transformer VdrXml.from_most_recent_file(c, Dir['/var/lib/ncs/mdes/COOK*.xml'])
+    c.add_transformer StaffPortal.create_transformer(c)
+
+The warehouse includes some transformers. Others are included with the
+NCS Navigator suite applications themselves. See below for more
+information (TODO).
+
+The warehouse configuration file may optionally include other
+overrides to MDES warehouse defaults. See the
+{file:sample_configuration.rb} for details.
+
+### Verify your settings
+
+You can verify most of these settings by running this command:
+
+    $ mdes-wh create-schema
+
+It will connect to the working database and create a schema matching
+the current MDES version.
+
+TODO: a more complete verification utility.
+
+## Use
+
+MDES Warehouse provides an executable named `mdes-wh` which has
+several subcommands. To get a list of the commands, use `mdes-wh
+help`.
+
+## Monitoring
+
+The warehouse produces interactive monitoring output on standard error
+for most of its actions. This output can be suppressed using the
+`--quiet` command line option, or by setting `c.output_level = :quiet`
+in the environment config file.
+
+Separately, warehouse actions are more permanently logged in a log
+file. The log file name defaults to
+`/var/log/ncs/warehouse/{env_name}.log` and can be changed in the
+environment configuration file. This log file is never cleaned out
+automatically; consider using [logrotate][] or a similar tool to
+rotate it on a daily or weekly basis.
+
+[logrotate]: http://linuxcommand.org/man_pages/logrotate8.html
+
+If a log can't be written to (e.g., because the directory doesn't
+exist), the warehouse will dump warnings and errors to standard out
+instead. The log file will contain more detail than the messages
+dumped to standard out, so don't rely on this backup behavior in
+production.

data/Rakefile

@@ -0,0 +1,45 @@
+require 'bundler/gem_tasks'
+
+require 'rspec/core/rake_task'
+require 'ci/reporter/rake/rspec'
+
+# require 'active_support/core_ext/string'
+require 'ncs_navigator/mdes'
+
+$LOAD_PATH.unshift File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift File.expand_path('../generated_models', __FILE__)
+require 'ncs_navigator/warehouse'
+
+task :spec => 'spec:all'
+
+namespace :spec do
+  RSpec::Core::RakeTask.new(:fast) do |t|
+    t.pattern = "spec/**/*_spec.rb"
+    t.rspec_opts = %q(--format nested --tag ~slow)
+  end
+
+  RSpec::Core::RakeTask.new(:all) do |t|
+    t.pattern = "spec/**/*_spec.rb"
+    t.rspec_opts = %q(--format nested)
+  end
+end
+
+namespace :generate do
+  task :models do
+    NcsNavigator::Warehouse::TableModeler.for_version(
+      '2.0',
+      :path => 'generated_models'
+    ).model!
+  end
+end
+
+namespace :ci do
+  task :all => :spec
+
+  task :spec_setup do
+    ENV['CI_REPORTS'] = 'reports/spec-xml'
+    ENV['SPEC_OPTS'] = "#{ENV['SPEC_OPTS']} --format nested"
+  end
+
+  task :spec => [:spec_setup, 'ci:setup:rspecbase', 'rake:spec']
+end

data/bin/mdes-wh

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+
+# Allow this executable to be run directly from the source as well as
+# from an installed gem.
+begin
+  %w(lib generated_models).each do |p|
+    lib = File.expand_path("../../#{p}", __FILE__)
+    unless $LOAD_PATH.include?(lib)
+      $LOAD_PATH << lib
+    end
+  end
+end
+
+require 'ncs_navigator/warehouse'
+
+begin
+  NcsNavigator::Warehouse::CLI.start
+rescue Interrupt => e
+  $stderr.puts "Interrupted"
+  exit 1
+end

data/ci-exec.sh

@@ -0,0 +1,45 @@
+#!/bin/bash -xe
+
+export NCS_NAVIGATOR_ENV=ci
+
+BUNDLER_VERSION=1.1.rc
+GEMSET=ncs_mdes_warehouse
+
+if [ -z $CI_RUBY ]; then
+    echo "CI_RUBY must be set"
+    exit 1
+fi
+
+set +xe
+echo "Initializing RVM"
+source ~/.rvm/scripts/rvm
+set -xe
+
+# On the overnight build, reinstall all gems
+if [ `date +%H` -lt 5 ]; then
+    set +xe
+    echo "Purging gemset to verify that all deps can still be installed"
+    rvm --force $CI_RUBY gemset delete $GEMSET
+    set -xe
+fi
+
+RVM_CONFIG="${CI_RUBY}@${GEMSET}"
+set +xe
+echo "Switching to ${RVM_CONFIG}"
+rvm use $RVM_CONFIG
+set -xe
+
+which ruby
+ruby -v
+
+set +e
+gem list -i bundler -v $BUNDLER_VERSION
+if [ $? -ne 0 ]; then
+  set -e
+  gem install bundler -v $BUNDLER_VERSION
+fi
+set -e
+
+bundle update
+
+bundle exec rake ci:all --trace

data/generated_models/ncs_navigator/warehouse/models/two_point_zero/address.rb

@@ -0,0 +1,101 @@
+require 'data_mapper'
+require 'ncs_navigator/warehouse'
+
+module NcsNavigator::Warehouse::Models::TwoPointZero
+  class Address
+    include DataMapper::Resource
+    include NcsNavigator::Warehouse::Models::MdesModel
+
+    storage_names[:default] = 'address'
+    storage_names[:mdes_warehouse_reporting] =
+      storage_names[:mdes_warehouse_working] = storage_names[:default]
+
+    property   :psu_id,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :length => 2..8, :set => ["20000054", "20000032", "20000032", "20000032", "20000032", "20000016", "20000039", "20000200", "20000028", "20000063", "20000067", "20000201", "20000202", "20000203", "20000090", "20000083", "20000204", "20000205", "20000042", "20000037", "20000206", "20000040", "20000207", "20000208", "20000209", "20000091", "20000210", "20000069", "20000211", "20000094", "20000212", "20000213", "20000102", "20000214", "20000215", "20000044", "20000216", "20000216", "20000216", "20000030", "20000217", "20000218", "20000218", "20000218", "20000219", "20000220", "20000221", "20000092", "20000222", "20000223", "20000224", "20000225", "20000225", "20000088", "20000087", "20000226", "20000103", "20000227", "20000228", "20000229", "20000230", "20000231", "20000232", "20000025", "20000233", "20000233", "20000233", "20000048", "20000234", "20000235", "20000050", "20000236", "20000035", "20000237", "20000238", "20000239", "20000240", "20000052", "20000241", "20000243", "20000244", "20000245", "20000246", "20000247", "20000248", "20000113", "20000249", "20000250", "20000251", "20000086", "20000252", "20000253", "20000254", "20000255", "20000256", "20000018", "20000058", "20000014", "20000257", "20000258", "20000259", "20000259", "20000098", "20000060", "20000260", "20000260", "20000260", "20000260", "20000261", "20000262", "20000263", "20000097", "20000264", "20000264", "20000265", "20000062", "20000117", "20000266", "20000267", "20000268", "20000269", "20000270", "20000271", "20000272", "20000273", "-4", "20000000"] }
+    property   :address_id,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :key => true, :required => true, :length => 1..36 }
+    belongs_to :person,
+               'NcsNavigator::Warehouse::Models::TwoPointZero::Person',
+               :child_key => [ :person_id ], :required => false
+    belongs_to :institute,
+               'NcsNavigator::Warehouse::Models::TwoPointZero::Institution',
+               :child_key => [ :institute_id ], :required => false
+    belongs_to :provider,
+               'NcsNavigator::Warehouse::Models::TwoPointZero::Provider',
+               :child_key => [ :provider_id ], :required => false
+    belongs_to :du,
+               'NcsNavigator::Warehouse::Models::TwoPointZero::DwellingUnit',
+               :child_key => [ :du_id ], :required => false
+    property   :address_rank,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :length => 1..2, :set => ["1", "2", "3", "4", "-5", "-4"] }
+    property   :address_rank_oth,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..255 }
+    property   :address_info_source,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :length => 1..2, :set => ["1", "2", "3", "4", "5", "6", "7", "8", "9", "-5", "-4"] }
+    property   :address_info_source_oth,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..255 }
+    property   :address_info_mode,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :length => 1..2, :set => ["1", "2", "3", "4", "5", "6", "-5", "-4"] }
+    property   :address_info_mode_oth,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..255 }
+    property   :address_info_date,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :length => 0..10, :format => /^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])?$/ }
+    property   :address_info_update,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :length => 0..10, :format => /^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])?$/ }
+    property   :address_start_date,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :length => 0..10, :format => /^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])?$/ }
+    property   :address_end_date,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :length => 0..10, :format => /^([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])?$/ }
+    property   :address_type,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :length => 1..2, :set => ["1", "2", "3", "4", "5", "-5", "-6", "-4"] }
+    property   :address_type_oth,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..255 }
+    property   :address_description,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :length => 1..2, :set => ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "-6", "-5", "-4"] }
+    property   :address_description_oth,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..255 }
+    property   :address_1,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => true, :length => 0..100 }
+    property   :address_2,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => true, :length => 0..100 }
+    property   :unit,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => true, :length => 0..10 }
+    property   :city,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => true, :length => 0..50 }
+    property   :state,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :required => true, :pii => true, :length => 1..2, :set => ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "50", "51", "-6", "-4"] }
+    property   :zip,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..5 }
+    property   :zip4,
+               NcsNavigator::Warehouse::DataMapper::NcsString,
+               { :pii => :possible, :length => 0..4 }
+    property   :address_comment,
+               NcsNavigator::Warehouse::DataMapper::NcsText,
+               { :pii => true, :length => 0..8000 }
+
+    mdes_order :psu_id, :address_id, :person_id, :institute_id, :provider_id, :du_id, :address_rank, :address_rank_oth, :address_info_source, :address_info_source_oth, :address_info_mode, :address_info_mode_oth, :address_info_date, :address_info_update, :address_start_date, :address_end_date, :address_type, :address_type_oth, :address_description, :address_description_oth, :address_1, :address_2, :unit, :city, :state, :zip, :zip4, :address_comment
+
+  end # class
+end # module NcsNavigator::Warehouse::Models::TwoPointZero