conceptql 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bb925d74153217fcb29ee4c4e5874827005246be
-  data.tar.gz: 41ae6d964b8a6ae82b37098adcda8548917543bd
+  metadata.gz: 41a911f8bcafa0d691e093e990fe37bfdd6a9b40
+  data.tar.gz: c9d90541f263d84795b24e3ad38c0d55fd704474
 SHA512:
-  metadata.gz: f62d0381dbd1ceabed67e9fff58d95975bf4cff27a957f03aa0ba495ca7726b22af3727d26acd4f683ca048a17dac858dffd2bef7f2e0707cae7e5cfd29e6677
-  data.tar.gz: 004743713781e5ea7cf7281ea071597ac8fa0959714571639d97d69a9fdd67b03692b10b9e1c77802230f57149212e9daf5077459d02db86814759051b44cc62
+  metadata.gz: 83afc484b1486e2c21e5c120b4fcd18b09ad085564f0d856033607bd8979291aa986cf93c403fe75684181898878180eeead220418d427238deebb3be05886d9
+  data.tar.gz: aa5ef37c3a85638a4cf93e1a648dce558fb84c86145f1a1b982eaccba68d104c07a4ad2979bfe951e3d05f32eb1d363ed4f300d6e4182ce3d8caa6d50953f6e0

data/.gitignore

@@ -6,6 +6,7 @@
 .config
 .yardoc
 .idea
+.env
 Gemfile.lock
 InstalledFiles
 _yardoc
@@ -21,3 +22,4 @@ test/tmp
 test/version_tmp
 tmp
 yamls/
+doc/.?????????*

data/CHANGELOG.md

@@ -1,8 +1,29 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
-## Unreleased
+## [0.3.0] - 2016-11-08
+### Added
+- New operators
+    - PlaceOfServiceFilter
+    - Provenance
+    - ProviderFilter
+    - CoReported
+- Support for :impala_mem_limit option
+- Initial support for dynamic columns
+- Additional columns for observation domains and drug domains
+- Support for :within, :occurrences, :at_least
+
+### Changed
+- Format of this [CHANGELOG](http://keepachangelog.com/en/0.3.0/)
+- Read searches Condition, Procedure, Drug, Observation
+- Rewrote OneInTwoOut to produce faster queries
+- Except operator uses different strategy under Impala
+
+### Fixed
+- Fixed date casting and other issues for Impala compatibility.
+- Fixed ConditionType to accept concept_ids
 
+## [0.2.0] - 2015-12-11
 ### Added
 - Optimized After/Before nodes when multi-person results are in the right stream.
 - Set end_date as coalesce(end_date, start_date) to make range if end_date missing.
@@ -40,9 +61,6 @@ All notable changes to this project will be documented in this file.
 - Recall now uses any labeled operator as if that operator was fed to "Define" operator
 - Nodes now called Operators
 
-### Deprecated
-- Nothing.
-
 ### Removed
 - Let/Define operators
 
@@ -52,87 +70,44 @@ All notable changes to this project will be documented in this file.
 - Concept node now works again
 - Define ensures tables are built
 
-
-## 0.1.1 - 2014-09-18
-
-### Added
-- Nothing.
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
+## [0.1.1] - 2014-09-18
 ### Fixed
 - Calling Query#sql no longer creates a bunch of temporary tables.
 
-
-## 0.1.0 - 2014-09-04
-
+## [0.1.0] - 2014-09-04
 ### Added
 - Support for numeric, string, and concept_ids returned in results.
 - Many updates to the ConceptQL Specification document.
 - Added doc/implementation_notes.md to capture thoughts and bad ideas.
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
 - "Fake" graphs are now drawn correctly.
 - bin/conceptql doesn't bomb out drawing "fake" graphs
 
-
-## 0.0.9 - 2014-09-03
-
+## [0.0.9] - 2014-09-03
 ### Added
 - Support for MSSQL (SQL Server).
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
 - Removed debug output from Node#namify
 
-
-## 0.0.8 - 2014-08-28
-
+## [0.0.8] - 2014-08-28
 ### Added
 - Support for Oracle.
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
 - DateAdjuster/TimeWindow use Sequel's date_arithmetic extension to produce database agnostic date manipulation.
 - Breakage from Node#tree in GraphNodifier.
 - All tests are back to passing.
 - Changed from SHA1 to CRC32 hash since Oracle can't handle table names longer than 30 characters.
 
-
-## 0.0.6 - 2014-08-23
-
+## [0.0.6] - 2014-08-23
 ### Added
 - Support for Oracle
 - Tree#defined to pass type information between Define and Recall.
 - Node#sql to produce SQL for each node.
 - Graph includes row count on each edge in the diagram.
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
 - Bug in CastingNode that generate SQL returning multiple columns in a subquery.
 - Made ruby-graphviz a dependency so calling programs don't bomb out.
@@ -140,32 +115,16 @@ All notable changes to this project will be documented in this file.
 - DateAdjuster/TimeWindow use Sequel's date_arithmetic extension to produce database agnostic date manipulation.
 - All tests are back to passing.
 
-
-## 0.0.5 - 2014-08-19
-
-### Added
-- Nothing.
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
+## [0.0.5] - 2014-08-19
 ### Fixed
 - Bug in GraphNodifier not displaying types for `recall` nodes.
 
-
-## 0.0.4 - 2014-08-19
-
+## [0.0.4] - 2014-08-19
 ### Added
 - Support for 5 instead of 13 column internal representation of results.
 - `define` node, used to create "variables" in ConceptQL.
 - `recall` node, used to pull results from "variables" in ConceptQL.
 
-### Deprecated
-- Nothing.
-
 ### Removed
 - Support for 13 column results.
 - Dependency on a set of views to run SQL queries.
@@ -173,9 +132,7 @@ All notable changes to this project will be documented in this file.
 ### Fixed
 - Bug where `place_of_service_code` wasn't limited to vocabulary_id 14
 
-
-## 0.0.3 - 2014-08-12
-
+## [0.0.3] - 2014-08-12
 ### Added
 - FakeGrapher class to make it easier to generate diagrams with experimental nodes
 - fake_graph command has returned to the `conceptql` program
@@ -184,45 +141,16 @@ All notable changes to this project will be documented in this file.
     - drg as procedure_occurrence
     - vsac as misc
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
 - Tree now runs #deep_symbolize_keys on incoming statements
 
-
-## 0.0.2 - 2014-07-11
-
-### Added
-- Nothing.
-
-### Deprecated
-- Nothing.
-
+## [0.0.2] - 2014-07-11
 ### Removed
 - Several commands from `conceptql` program
     - fake_graph
     - show_db_graph
     - show_and_tell_db
 
-### Fixed
-- Nothing.
-
-
-## 0.0.1 - 2014-07-11
-
+## [0.0.1] - 2014-07-11
 ### Added
 - This project.
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
-### Fixed
-- Nothing.
-

data/Gemfile

@@ -2,3 +2,5 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in conceptql.gemspec
 gemspec
+
+gem 'pg'

data/README.md

@@ -4,13 +4,18 @@ ConceptQL (pronounced concept-Q-L) is a high-level language that allows research
 
 This gem interprets the ConceptQL language and translates it into SQL queries compatible with OMOP CDM-structured data.  The ConceptQL "language" is a set of nested hashes and arrays representing search criteria and some set operations and temporal operations to glue those criteria together.
 
+## Further Reading
+
+If you're interested in reading up on ConceptQL, a rough draft of the specifications document is [available in markdown format](https://github.com/outcomesinsights/conceptql_spec).
+
 ## Motivation for ConceptQL
+
 Outcomes Insights intends to build a vast library of research algorithms and apply those algorithms to large databases of claims data.  Early into building the library, we realized we had to overcome two major issues:
 
 1. Methods sections of research papers commonly use natural language to specify the criteria used to build cohorts from a claims database.
     - Algorithms defined in natural language are often imprecise, open to multiple interpretations, and generally difficult to reproduce.
     - Researchers could benefit from a language that removes the ambiguity of natural language while increasing the reproducibility of their research algorithms.
-2. Querying against claims databases is often difficult.
+1. Querying against claims databases is often difficult.
     - Hand-coding algorithms to extract cohorts from datasets is time-consuming, error-prone, and opaque.
     - Researchers could benefit from a language that allows algorithms to be defined at a high-level and then gets translated into the appropriate queries against a database.
 
@@ -19,6 +24,7 @@ We developed ConceptQL to address these two issues.
 We are writing a tool that can read research algorithms defined in ConceptQL.  The tool can create a diagram for the algorithm which makes it easy to visualize and understand.  The tool can also translate the algorithm into a SQL query which runs against data structured in [OMOP's Common Data Model (CDM)](http://omop.org/CDM).  The purpose of the CDM is to standardize the format and content of observational data, so standardized applications, tools and methods can be applied to them.
 
 For instance, using ConceptQL we can take a statement that looks like this:
+
 ```YAML
 :icd9: '412'
 ```
@@ -28,6 +34,7 @@ And generate a diagram that looks like this:
 ![](doc/diagram_0.png)
 
 And generate SQL that looks like this:
+
 ```SQL
 SELECT *
 FROM cdm_data.condition_occurrence AS co
@@ -37,19 +44,12 @@ AND scm.source_vocabulary_id = 2
 AND scm.source_code = co.condition_source_value
 ```
 
-As stated above, one of the goals of ConcegtQL is to make it easy to assemble fairly complex queries without having to roll up our sleeves and write raw SQL.  To accommodate this complexity, ConceptQL itself has some complexities of its own.  That said, we believe ConceptQL will help researchers define, hone, and share their research algorithms.
-
-
-## Further Reading
-
-If you're interested in reading more about ConceptQL, a rought draft of the specifications document is [available as a PDF](https://github.com/outcomesinsights/conceptql/blob/master/doc/ConceptQL%20Specification%20(alpha).pdf?raw=true) in this repository.
-
+As stated above, one of the goals of ConceptQL is to make it easy to assemble fairly complex queries without having to roll up our sleeves and write raw SQL.  To accommodate this complexity, ConceptQL itself has some complexities of its own.  That said, we believe ConceptQL will help researchers define, hone, and share their research algorithms.
 
 ## Try Before You Buy
 
 If you'd like to interact with ConceptQL a bit before deciding to dive in, head over to the [ConceptQL Sandbox](http://sandbox.cohortjigsaw.com) for an online demonstration of the language and its features.
 
-
 ## Requirements
 
 ConceptQL is in an early-alpha state.  For now it is limited to working with [OMOP CDM](http://omop.org/CDM)-structured data stored in the [PostgreSQL](http://www.postgresql.org/) database.  It has been tested under [Ubuntu Linux](http://www.ubuntu.com/) and Mac OS X 10.8+.  The interpreter is written in Ruby and theoretically should be platform independent, but your mileage may vary.
@@ -72,44 +72,66 @@ Add this line to your application's Gemfile:
 
 And then execute:
 
-    $ bundle
+    bundle
 
 Or install it yourself as:
 
-    $ gem install conceptql
+    gem install conceptql
 
 ## Usage
 
-The easiest way to try out ConceptQL locally is to use the [conceptql-dev-box](http://github.com/outcomesinsights/conceptql-dev-box)
-
 ConceptQL comes with a [Thor](http://whatisthor.com/)-based command-line program: `conceptql`
 
 `conceptql` allows you to translate a ConceptQL statement into a SQL statement or a graph depicting the ConceptQL statement.
 
 `conceptql` will print a basic help menu for each command if you run `conceptql`
 
-See [conceptql-dev-box](http://github.com/outcomesinsights/conceptql-dev-box) for more examples on how to play with ConceptQL
-
 ## Writing Your Own ConceptQL
 
 Right now, the only way to create a new ConceptQL statement is to write it out by hand in either Ruby, JSON, or YAML.
 
 If you're feeling bold, feel free to try your hand at authoring a ConceptQL statement.
 
-You might want to just take an existing statement from the statements in [test_conceptql](https://github.com/outcomesinsights/test_conceptql) and modify that.
+## Testing
+
+### Setup
+
+In order to run the tests for ConceptQL, you first have to create a database and load the OMOP vocabularies into it.  You can use [loadmop](https://github.com/outcomesinsights/loadmop/tree/develop) do to so.
+
+Follow the instructions in [loadmop](https://github.com/outcomesinsights/loadmop/#preparation) to setup the vocabulary files correctly.
+
+After loading the vocabulary data file into the database,
+create an .env file in the root of the conceptql directory,
+similar or identical to the one used for loadmop.  Then
+run `rake test_db_setup`.  This will load the ConceptQL test
+data into the database.  This only needs to be done once, not
+every time you run the tests.
+
+### Running
+
+If any of the CSV files in `test/data` has been updated since you
+last updated the database, you should run `rake test_db_teardown test_db_setup`
+to reset the data in the database.  Updates to the CSV files in
+`test/data` should be infrequent.
+
+After the test database has been setup, you can run the tests
+using `rake`, as the default task is set to run the tests.
 
 ## Contributing
 
-1. Fork it ( http://github.com/outcomesinsights/conceptql/fork )
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+1. Fork it ( [http://github.com/outcomesinsights/conceptql/fork](http://github.com/outcomesinsights/conceptql/fork) )
+1. Create your feature branch (`git checkout -b my-new-feature`)
+1. Commit your changes (`git commit -am 'Add some feature'`)
+1. Push to the branch (`git push origin my-new-feature`)
+1. Create new Pull Request
 
 Or, use the [conceptql-dev-box](https://github.com/outcomesinsights/conceptql-dev-box) to get yourself setup and hack on ConceptQL that way.
 
 ## Thanks
+
 - [Outcomes Insights, Inc.](http://outins.com)
     - Many thanks for allowing me to release a portion of my work as Open Source Software!
 - [OMOP](http://omop.org)/[OHDSI](http://www.ohdsi.org/)
     - Thank you for providing a great [Common Data Model](http://omop.org/CDM) and fantastic set of standardized [Vocabularies](http://omop.org/Vocabularies) upon which ConceptQL rests.
+- [Jeremy Evans](http://code.jeremyevans.net/)
+    - Thank you for the great contributions, ideas, and commits.

data/Rakefile

@@ -1 +1,35 @@
 require "bundler/gem_tasks"
+ENV['DATA_MODEL'] ||= 'omopv4'
+
+desc "Setup test database"
+task :test_db_setup do
+  require_relative 'test/db_setup'
+end
+
+desc "Setup test database"
+task :test_db_teardown do
+  require_relative 'test/db_teardown'
+end
+
+run_spec = lambda do |data_model|
+  sh "DATA_MODEL=#{data_model} #{FileUtils::RUBY} test/all.rb"
+end
+
+desc "Run tests with omopv4 data model"
+task :test_omopv4 do
+  run_spec.call(:omopv4)
+end
+
+desc "Run tests with omopv4_plus data model"
+task :test_omopv4_plus do
+  run_spec.call(:omopv4_plus)
+end
+
+desc "Run tests with omopv4 data model with coverage"
+task :test_cov do
+  ENV['COVERAGE'] = '1'
+  run_spec.call(:omopv4)
+end
+
+desc "Run tests with omopv4 data model"
+task :default => :test_omopv4

data/conceptql.gemspec

@@ -19,14 +19,13 @@ Gem::Specification.new do |spec|
   spec.require_paths = ['lib']
 
   spec.add_dependency 'facets', '~> 3.0'
-  spec.add_dependency 'sequelizer', '~> 0.0'
+  spec.add_dependency 'sequelizer', '~> 0.1'
   spec.add_dependency 'thor', '~> 0.19'
-  spec.add_dependency 'pg', '~> 0.17'
   spec.add_dependency 'ruby-graphviz', '~> 1.2'
   spec.add_dependency 'csv2xlsx', '~> 0'
   spec.add_development_dependency 'bundler', '~> 1.5'
   spec.add_development_dependency 'rake', '~> 10.3'
-  spec.add_development_dependency 'rspec', '~> 3.2'
-  spec.add_development_dependency 'guard-rspec', '~> 4.5'
+  spec.add_development_dependency 'minitest', '~> 5.0'
   spec.add_development_dependency 'byebug', '~> 4.0'
+  spec.add_development_dependency 'simplecov', '~> 0.11'
 end