conceptql 0.0.9 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5750de5f196ce44a4f679c8c28613bd7476824a1
-  data.tar.gz: 66c448563f934f4a64e5a030e343eceef5a691aa
+  metadata.gz: 297853bf4ddbd97f096bd42b929843cf25b88774
+  data.tar.gz: b6ec45dc7694ffdd8e205a2fa0c911501cb8ebb4
 SHA512:
-  metadata.gz: 286d1046bab811f33f2a83d0066cc608a2d205c46beb188bc3cd9227db36a4cad2b23bbc0ee968ddb49ba34394800bd2d312661787d77bb8467f6bb80a13d09e
-  data.tar.gz: b6978826328af7dd3d1cc3af40447a20ccd3afe83074c0ecbce3528f56dbc6cb48621131e784444194a050f10549dc523dccd0a9228a453892facb9faa099bb1
+  metadata.gz: c858413d3020e1168b793c43d54cf1a9e7a9574cc5ee0a1edd50237e71e4abc07ba2a6a9036fe8a82249b3908dce5ff6ea8d4c9500f05665223dd36c6b08bd7c
+  data.tar.gz: 3cd9354b86045d6d66061d131b32fb2451f1b9fdd582e905bdb7fbf0ea7d28b1657663e0b63c261b2454af30f234e81aaa509e04a49793c1150032d4de2e6a60

data/CHANGELOG.md

@@ -1,6 +1,24 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
+## 0.1.0 - 2014-09-04
+
+### Added
+- Support for numeric, string, and concept_ids returned in results.
+- Many updates to the ConceptQL Specification document.
+- Added doc/implementation_notes.md to capture thoughts and bad ideas.
+
+### Deprecated
+- Nothing.
+
+### Removed
+- Nothing.
+
+### Fixed
+- "Fake" graphs are now drawn correctly.
+- bin/conceptql doesn't bomb out drawing "fake" graphs
+
+
 ## 0.0.9 - 2014-09-03
 
 ### Added

data/doc/implementation_notes.md

@@ -0,0 +1,39 @@
+# ConceptQL Implementation Notes
+And here is where I record information about some of the decisions I made.
+
+## Define/Recall
+- Sequel's create table statement runs out-of-band with rest of ConceptQL statemnt
+    - Gets executed immediately
+    - This is going to be very slow for large datasets :-(
+- I had to retool Tree and Query and Graph to expect an array of concepts in a ConceptQL statement
+    - I'm not sure I like this
+    - A ConceptQL statement perhaps should be only a single statement at the end
+- Defines need to occur before they are used
+    - Most languages have a "forward definition" ability
+        - I have no use cases for when we might need those?
+        - Perhaps a definition that uses a definition that doesn't exist?
+        - Is that recursive?
+    - Is that something we want/need in ConceptQL?
+
+
+## Values
+- I had considered comparing a result set with a between operator, but this implies that the R stream needs a range somehow
+    - I don't think supporting a range is a good idea right now
+
+
+## Bad Ideas
+Here is where I intend to record bad ideas that appear to be blind allies
+
+### Removing person-only rows from a stream after ANDing
+- Theoretically, any result set that passes up through an AND node carries all the valid person IDs with it, so it is redundant to carry results that are just person-only.  It would make sense to eliminate them.
+- NO BAD IDEA
+    - We have to check to see if the stream is exclusively person-only.  In that case, it is unsafe to eliniate those results because we'd end up removing all results from the stream
+        - We could implement this check, but let's do that later when we want to optimize things
+             - Also, what happens if we cast to visit later and we've removed all person-only results?  This could cause odd behavior
+
+### Treating Patient Streams as "Eternal" when they encounter a temporal node
+I had created a rather sophisticated system of how to handle a patient stream entering a temporal node.
+
+The basic premise is that patient information (gender, race, etc) is "timeless" or "eternal" and so if a patient stream is the R stream in a temporal node and the L stream is, say, a stream of MIs, what is the result?  I proposed that the MI would be filtered down to only those patients that appear in the R stream.
+
+Likewise, if the MI stream is the R stream and the person stream is the L stream, what's the result?  Same thing.  Only patients common to both streams are passed through the L stream.  But patients DO have a date associated with them: their date of birth.  And, by passing a patient stream through a time_shift node of say, +50yr, we can use that patient stream in the R stream of a temporal node to filter the L stream by the patient being 50 years old.

data/doc/spec.md

@@ -478,7 +478,7 @@ For situations where we need to represent pre-defined date ranges, we can use "d
 - *Not yet implemented*
 
 
-#### What is \<date-format\>?
+#### What is <date-format\>?
 Dates follow these formats:
 
 - "YYYY-MM-DD"
@@ -848,6 +848,90 @@ And don't forget the left-hand side can have multiple types of streams:
 }
 ```
 
+
+## Sub-concepts within a Larger Concept
+If a concept is particularly complex, or has a stream of results that are used more than once, it can be helpful to break the concept into a set of sub-concepts.  This can be done using two nodes: define and recall
+
+#### define
+- Takes 2 arguments
+    - First argument is a string of arbitrary length that describe the stream to be save.  This is the "name" assigned to the stream for later recall
+    - Second argument is the stream to save under the name specified
+
+
+#### recall
+- Takes 1 argument
+    - The "name" of the stream previously saved using the `define` node
+
+
+A stream must be `define`d before `recall` can use it.
+
+```ConceptQL
+# Save away a stream of results to build the 1 inpatient, 2 outpatient pattern used in claims data algorithms
+[
+  {
+    define: [
+      'Heart Attack Visit',
+      { visit_occurrence: { icd9: '412' } }
+    ]
+  },
+
+  {
+    define: [
+      'Inpatient Heart Attack',
+      {
+        intersect: [
+          { recall: 'Heart Attack Visit'},
+          { place_of_service_code: 21 }
+        ]
+      }
+    ]
+  },
+
+  {
+    define: [
+      'Outpatient Heart Attack',
+      {
+        intersect: [
+          { recall: 'Heart Attack Visit'},
+          {
+            complement: {
+              place_of_service_code: 21
+            }
+          }
+        ]
+      }
+    ]
+  },
+
+  {
+    define: [
+      'Earlier of Two Outpatient Heart Attacks',
+      {
+        before: {
+          left: { recall: 'Outpatient Heart Attack' },
+          right: {
+            time_window: [
+              { recall: 'Outpatient Heart Attack' },
+              { start: '-30d', end: '0' }
+            ]
+          }
+        }
+      }
+    ]
+  },
+
+  {
+    first: {
+      union: [
+        { recall: 'Inpatient Heart Attack' },
+        { recall: 'Earlier of Two Outpatient Heart Attacks'}
+      ]
+    }
+  }
+]
+```
+
+
 ## Concepts within Concepts
 One of the main motivations behind keeping ConceptQL so flexible is to allow users to build ConceptQL statements from other ConceptQL statements.  This section loosely describes how this feature will work.  Its actual execution and implementation will differ from what is presented here.
 
@@ -894,6 +978,155 @@ In the actual implementation of the concept node, each ConceptQL statement will
 }
 ```
 
+
+## Values
+A result can carry forward three different types of values, modeled after the behavior of the observation table:
+
+- value_as_numeric
+    - For values like lab values, counts of occurrence of results, cost information
+- value_as_string
+    - For value_as_string from observation table, or notes captured in EHR data
+- value_as_concept_id
+    - For values that are like factors from the observation value_as_concept_id column
+
+
+By default, all value fields are set to NULL, unless a criterion node is explicitly written to populate one or more of those fields.
+
+There are many operations that can be performed on the value_as\_\* columns and as those operations are implemented, this section will grow.
+
+For now we'll cover some of the general behavior of the value_as_numeric column and it's associated nodes.
+
+#### numeric
+- Takes 2 arguments
+    - A stream
+    - And a numeric value or a symbol representing the name of a column in CDM
+
+Passing streams through a `numeric` node changes the number stored in the value column:
+
+```ConceptQL
+# All MIs, setting value_as_numeric to 2
+{
+    numeric: [
+        { icd9: '412' },
+        2
+   ]
+}
+```
+
+`numeric` can also take a column name instead of a number.  It will derive the results row's value from the value stored in the column specified.
+```ConceptQL
+# All copays for 99214s
+{
+    numeric: [
+        { procedure_cost: { cpt: '99214' } },
+        :paid_copay
+    ]
+}
+```
+
+If something nonsensical happens, like the column specified isn't present in the table pointed to by a result row, value_as_numeric in the result row will be unaffected:
+```ConceptQL
+# Still all MIs with value_as_numeric defaulted to NULL.  condition_occurrence table doesn't have a "paid_copay" column
+{
+    value: [
+        { icd9: '412' },
+        :paid_copay
+   ]
+}
+```
+
+Or if the column specified exists, but refers to a non-numerical column, we'll set the value to 0
+```ConceptQL
+# All MIs, with value set to 0 since the column specified by value node is a non-numerical column
+{
+    value: [
+        { icd9: '412' },
+        :stop_reason
+   ]
+}
+```
+
+With a `numeric` node defined, we could introduce a sum node that will sum by patient and type.  This allows us to implement the Charlson comorbidity algorithm:
+```ConceptQL
+{
+   sum: [
+     {
+        union: [
+          {
+            numeric: [
+                  { person: { icd9: '412' } },
+                  1
+             ]
+          },
+          {
+             numeric: [
+                  { person: { icd9: '278.02' } },
+                  2
+             ]
+          }
+        ]
+     }
+   ]
+}
+```
+
+### Counting
+It might be helpful to count the number of occurrences of a result row in a stream.  A simple "count" node could group identical rows and store the number of occurrences in the value_as_numeric column.
+
+I need examples of algorithms that could benefit from this node.  I'm concerned that we'll want to roll up occurrences by person most of the time and that would require us to first cast streams to person before passing the person stream to count.
+```ConceptQL
+# Count the number of times each person was irritable
+{
+    count: { person: { icd9: '799.22' } }
+}
+```
+
+We could do dumb things like count the number of times a row shows up in a union:
+```ConceptQL
+# All rows with a value of 2 would be rows that were both MI and Primary
+{
+    count: {
+        union: [
+           { icd9: '412' },
+           { primary_diagnosis: true}
+        ]
+   }
+}
+```
+
+#### Numeric Value Comparison
+Acts like any other binary node.  L and R streams, joined by person.  Any L that pass comparison go downstream.  R is thrown out.  Comparison based on result row's value column.
+
+- Less than
+- Less than or equal
+- Equal
+- Greater than or equal
+- Greater than
+- Not equal
+
+
+### numeric as criterion node
+Numeric doesn't have to take a stream.  If it doesn't have a stream as an argument, it acts like a criterion node much like date_range
+```ConceptQL
+# People with more than 1 MI
+{
+
+    greater_than: {
+        left: { count: { person: { icd9: '412' }}},
+        right: { numeric: 1 }
+    }
+}
+```
+
+#### sum
+- Takes a stream of results and does some wild things
+    - Groups all results by person and type
+        - Sums the value_as_numeric column within that grouping
+        - Sets start_date to the earliest start_date in the group
+        - Sets the end_date to the most recent end_date in the group
+        - Sets criterion_id to 0 since there is no particular single row that the result refers to anymore
+
+
 # Appendix A - Criterion Nodes
 
 | Node Name | Stream Type | Arguments | Returns |
@@ -1031,15 +1264,14 @@ ConceptQL is not yet fully specified.  These are modifications/enhancements that
 5. How do we want to look up standard vocab concepts?
      - I think Marc’s approach is a bit heavy-handed
 
-
-### Slots and Variables
 Some statements maybe very useful and it would be handy to reuse the bulk of the statement, but perhaps vary just a few things about it.  ConceptQL supports the idea of using variables to represent sub-expressions.  The variable node is used as a place holder to say "some criteria set belongs here".  That variable can be defined in another part of the criteria set and will be used in all places the variable node appears.
 
-If a variable node is used, but not defined, the concept is still valid, but will fail to run until a definition for all missing variables is provided.
 
-I don't have a good feel for:
+### Future Work for Define and Recall
+I'd like to make it so if a variable node is used, but not defined, the concept is still valid, but will fail to run until a definition for all missing variables is provided.
+
+But I don't have a good feel for:
 
-- How to represent a variable node in a diagram
 - Whether we should have users name the variables, or auto-assign a name?
     - We risk name collisions if a concept includes a sub-concept with the same variable name
     - Probably need to name space all variables
@@ -1048,173 +1280,26 @@ I don't have a good feel for:
 - We'll need to do a pass through a concept to find all variables and prompt a user, then do another pass through the concept before attempting to execute it to ensure all variables have values
     - Do we throw an exception if not?
     - Do we require calling programs to invoke a check on the concept before generating the query?
+- Perhaps slot is a different node from "define"
 
 
-##### Update 2014-08-13
-I've hacked some variable support into an experimental branch of ConceptQL.  So far, here's how it works:
-- Define node
-    - Give it two params, a name, and a stream
-    - Stream is used to populate a temporary table that is named after the name
-    - name is plain english sentence that is then turned into a hexdigest for a name
-        - Avoids collisions with other names
-        - Avoids truncation issues if name is WAY long
-- Recall node
-    - Takes a single argument: the name used in a define node
-    - Re-written to fetch results from the temp table that has the name provided
-
-Current issues:
-
-- Sequel's create table statement runs out-of-band with rest of ConceptQL statemnt
-    - Gets executed immediately
-- Type information in "define" needs to be made available to "from"
-    - Currently attempting to pass this information from define to from using an attribute tacked onto the shared db connection
-    - Recall may not have access to this information until #query is called
-    - This is bad and needs to be fixed/rethought
-
-
-Considerations for the future:
-- Probably want to rename these nodes to something better
-- It would still be nice to drop a concept into a concept that has "slots" waiting
-    - Perhaps slot is a different node from "define"
-- I had to retool Tree and Query and Graph to expect an array of concepts in a ConceptQL statement
-    - I'm not sure I like this
-    - A ConceptQL statement perhaps should be only a single statement at the end
-    - If an array of sub-concepts is fed into Query, maybe we only execute the last one after parsing the others
-        - This is consistent with how Sequel wants to live and would yield a single set of results
-        - I think I like this
-- Defines need to occur before they are used
-    - Most languages have a "forward definition" ability
-        - I have no use cases for when we might need those?
-        - Perhaps a definition that uses a definition that doesn't exist?
-        - Is that recursive?
-    - Is that something we want/need in ConceptQL?
-
-
-### Value Nodes
-So far, we can’t recreate the Charlson comorbidity index using ConceptQL.  If we added a “value” node, we could.
-
-By default each result row will carry a value column, set to 1.  Some examples:
-```ConceptQL
-# All MIs, defaulting value to 1
-{ icd9: '412' }
-```
-
-Passing streams through a value node changes the number stored in the value column:
-
-```ConceptQL
-# All MIs, changing value to 2
-{
-    value: [
-        { icd9: '412' },
-        2
-   ]
-}
-```
-
-Value can also take a column name instead of a number.  It will derive the results row's value from the value stored in the column specified.
-```ConceptQL
-# All copays for 99214s
-{
-    value: [
-        { procedure_cost: { cpt: '99214' } },
-        :paid_copay
-    ]
-}
-```
-
-If something nonsensical happens, like the column specified isn't present in the table pointed to by a result row, the value in the result row will be unaffected:
-```ConceptQL
-# Still all MIs with value defaulted to 1.  condition_occurrence table doesn't have a "paid_copay" column
-{
-    value: [
-        { icd9: '412' },
-        :paid_copay
-   ]
-}
-```
-
-Or if the column specified exists, but refers to a non-numerical column, we'll set the value to 0
-```ConceptQL
-# All MIs, with value set to 0 since the column specified by value node is a non-numerical column
-{
-    value: [
-        { icd9: '412' },
-        :stop_reason
-   ]
-}
-```
-
-With a value node defined, we could introduce a sum node that will sum by patient.  This allows us to implement the Charlson comorbidity algorithm:
-```ConceptQL
-{
-   sum: [
-     {
-        union: [
-          {
-            value: [
-                  { person: { icd9: '412' } },
-                  1
-             ]
-          },
-          {
-             value: [
-                  { person: { icd9: '278.02' } },
-                  2
-             ]
-          }
-        ]
-     }
-   ]
-}
-```
-
-### Counting
-It might be helpful to count the number of occurrences of a result row in a stream.  A simple "count" node could group identical rows and store the number of occurrences in the value column.
+### Considerations for Values
+I'm considering defaulting each value_as\_\* column to some value.
+- numeric => 1
+- concept_id => 0
+    - Or maybe the concept_id of the main concept_id value from the row?
+        - This would be confusing when pulling from the observation table
+        - What's the "main" concept_id of a person?
+        - Hm.  This feels a bit less like a good idea now
+- string
+    - source_value?
+    - Boy, this one is even harder to default
 
-I need examples of algorithms that could benefit from this node.  I'm concerned that we'll want to roll up occurrences by person most of the time and that would require us to first cast streams to person before passing the person stream to count.
 ```ConceptQL
-# Count the number of times each person was irritable
-{
-    count: { person: { icd9: '799.22' } }
-}
-```
-
-We could do dumb things like count the number of times a row shows up in a union:
-```ConceptQL
-# All rows with a value of 2 would be rows that were both MI and Primary
-{
-    count: {
-        union: [
-           { icd9: '412' },
-           { primary_diagnosis: true}
-        ]
-   }
-}
+# All MIs, defaulting value_as_numeric to 1, concept_id to concept id for 412, string to condition_source_value
+{ icd9: '412' }
 ```
 
-### Value Comparison
-Acts like any other binary node.  L and R streams, joined by person.  Any L that pass comparison go downstream.  R is thrown out.  Comparison based on result row's value column.
-
-- Less than
-- Less than or equal
-- Equal
-- Greater than or equal
-- Greater than
-- Not equal
-- Between
-
-
-### value_literal
-```ConceptQL
-# People with more than 1 MI
-{
-
-    greater_than: {
-        left: { count: { person: { icd9: '412' }}},
-        right: { value_literal: 1 }
-    }
-}
-```
 
 ### Filter Node
 Inspired by person_filter, why not just have a "filter" node that filters L by R.  Takes L, R, and an "as" option.  As option temporarily casts the L and R streams to the type specified by :as and then does person by person comparison, only keeping rows that occur on both sides.  Handy for keeping procedures that coincide with conditions without fully casting the streams:

data/lib/conceptql/cli.rb

@@ -89,7 +89,7 @@ module ConceptQL
       puts 'JSON'
       puts JSON.pretty_generate(q.statement)
       STDIN.gets
-      graph_it(statement, my_db, title)
+      graph_it(statement, title)
       STDIN.gets
       puts q.query.sql
       STDIN.gets

data/lib/conceptql/graph_nodifier.rb

@@ -59,7 +59,7 @@ module ConceptQL
       attr :values, :name
       def initialize(name, values)
         @name = name.to_s
-        super(nil, values)
+        super(values)
       end
 
       def display_name

data/lib/conceptql/nodes/casting_node.rb

@@ -61,7 +61,7 @@ module ConceptQL
           wheres << Sequel.expr(person_id: uncastable_person_ids)
         end
 
-        destination_type_id = type_id(my_type)
+        destination_type_id = make_type_id(my_type)
 
         unless to_me_types.empty?
           # For each castable type in the stream, setup a query that
@@ -72,7 +72,7 @@ module ConceptQL
               .where(criterion_type: source_type.to_s)
               .select_group(:criterion_id)
             source_table = make_table_name(source_type)
-            source_type_id = type_id(source_type)
+            source_type_id = make_type_id(source_type)
 
             db.from(source_table)
               .where(source_type_id => source_ids)
@@ -85,7 +85,7 @@ module ConceptQL
 
         unless from_me_types.empty?
           from_me_types.each do |from_me_type|
-            fk_type_id = type_id(from_me_type)
+            fk_type_id = make_type_id(from_me_type)
             wheres << Sequel.expr(fk_type_id => db.from(stream_query).where(criterion_type: from_me_type.to_s).select_group(:criterion_id))
           end
         end

data/lib/conceptql/nodes/complement.rb

@@ -11,7 +11,7 @@ module ConceptQL
             .exclude(:criterion_id => nil)
             .where(:criterion_type => type.to_s)
           query = db.from(make_table_name(type))
-            .exclude(type_id(type) => positive_query)
+            .exclude(make_type_id(type) => positive_query)
           db.from(select_it(query, type))
         end.inject do |union_query, q|
           union_query.union(q, all: true)

data/lib/conceptql/nodes/count.rb

@@ -0,0 +1,23 @@
+require_relative 'pass_thru'
+
+module ConceptQL
+  module Nodes
+    class Count < PassThru
+      def query(db)
+        db.from(unioned(db))
+          .group(*COLUMNS)
+          .select(*(COLUMNS - [:value_as_numeric]))
+          .select_append{count(1).as(:value_as_numeric)}
+          .from_self
+      end
+
+      def unioned(db)
+        children.map { |c| c.evaluate(db) }.inject do |uni, q|
+          uni.union(q)
+        end
+      end
+    end
+  end
+end
+
+

data/lib/conceptql/nodes/define.rb

@@ -42,7 +42,13 @@ module ConceptQL
       # Also, things will blow up if you try to use a variable that hasn't been
       # defined yet.
       def query(db)
-        db.create_table!(table_name, temp: true, as: stream.evaluate(db))
+        # We'll wrap the creation of the temp table in memoization
+        # That way we can call #query multiple times, but only suffer the
+        # cost of creating the temp table just once
+        @_run ||= begin
+          db.create_table!(table_name, temp: true, as: stream.evaluate(db))
+          true
+        end
         db.from(table_name)
       end
 

data/lib/conceptql/nodes/equal.rb

@@ -0,0 +1,11 @@
+require_relative 'temporal_node'
+
+module ConceptQL
+  module Nodes
+    class Equal < TemporalNode
+      def where_clause
+        { r__value_as_numeric: :l__value_as_numeric }
+      end
+    end
+  end
+end

data/lib/conceptql/nodes/node.rb

@@ -3,6 +3,16 @@ require 'active_support/core_ext/hash'
 module ConceptQL
   module Nodes
     class Node
+      COLUMNS = [
+        :person_id,
+        :criterion_id,
+        :criterion_type,
+        :start_date,
+        :end_date,
+        :value_as_numeric,
+        :value_as_string,
+        :value_as_concept_id
+      ]
       attr :values, :options
       attr_accessor :tree
       def initialize(*args)
@@ -44,13 +54,15 @@ module ConceptQL
       end
 
       def columns(query, local_type = nil)
-        criterion_type = Sequel.expr(:criterion_type)
+        criterion_type = :criterion_type
         if local_type
-          criterion_type = Sequel.cast_string(local_type.to_s)
+          criterion_type = Sequel.cast_string(local_type.to_s).as(:criterion_type)
         end
-        [:person_id___person_id,
-         Sequel.expr(type_id(local_type)).as(:criterion_id),
-         criterion_type.as(:criterion_type)] + date_columns(query, local_type)
+        columns = [:person_id,
+                    type_id(local_type),
+                    criterion_type]
+        columns += date_columns(query, local_type)
+        columns += value_columns(query)
       end
 
       private
@@ -77,6 +89,10 @@ module ConceptQL
       def type_id(type = nil)
         return :criterion_id if type.nil?
         type = :person if type == :death
+        Sequel.expr(make_type_id(type)).as(:criterion_id)
+      end
+
+      def make_type_id(type)
         (type.to_s + '_id').to_sym
       end
 
@@ -84,7 +100,31 @@ module ConceptQL
         "#{table}___tab".to_sym
       end
 
+      def value_columns(query)
+        [
+          numeric_value(query),
+          string_value(query),
+          concept_id_value(query)
+        ]
+      end
+
+      def numeric_value(query)
+        return :value_as_numeric if query.columns.include?(:value_as_numeric)
+        Sequel.cast_numeric(nil, Float).as(:value_as_numeric)
+      end
+
+      def string_value(query)
+        return :value_as_string if query.columns.include?(:value_as_string)
+        Sequel.cast_string(nil).as(:value_as_string)
+      end
+
+      def concept_id_value(query)
+        return :value_as_concept_id if query.columns.include?(:value_as_concept_id)
+        Sequel.cast_numeric(nil).as(:value_as_concept_id)
+      end
+
       def date_columns(query, type = nil)
+        return [:start_date, :end_date] if (query.columns.include?(:start_date) && query.columns.include?(:end_date))
         return [:start_date, :end_date] unless type
         sd = start_date_column(query, type)
         sd = Sequel.expr(sd).cast(:date).as(:start_date) unless sd == :start_date

data/lib/conceptql/nodes/numeric.rb

@@ -0,0 +1,40 @@
+require_relative 'pass_thru'
+
+module ConceptQL
+  module Nodes
+    # Represents a node that will either:
+    # - create a value_as_numeric value for every person in the database
+    # - change the value_as_numeric value for every every result passed in
+    #   - either to a numeric
+    #   - or a value from a column in the origin row
+    #
+    # Accepts two params:
+    # - Either a numeric value or a symbol representing a column name
+    # - An optional stream
+    class Numeric < PassThru
+      def query(db)
+        stream.nil? ? as_criterion(db) : with_kids(db)
+      end
+
+      def types
+        stream.nil? ? [:person] : super
+      end
+
+      private
+      def with_kids(db)
+        db.from(stream.evaluate(db))
+          .select(*(COLUMNS - [:value_as_numeric]))
+          .select_append(Sequel.lit('?', arguments.first).cast(Float).as(:value_as_numeric))
+          .from_self
+      end
+
+      def as_criterion(db)
+        db.from(select_it(db.from(:person), :person))
+          .select(*(COLUMNS - [:value_as_numeric]))
+          .select_append(Sequel.lit('?', arguments.first).cast(Float).as(:value_as_numeric))
+          .from_self
+      end
+    end
+  end
+end
+

data/lib/conceptql/nodes/pass_thru.rb

@@ -4,7 +4,7 @@ module ConceptQL
   module Nodes
     class PassThru < Node
       def types
-        values.map(&:types).flatten.uniq
+        children.map(&:types).flatten.uniq
       end
     end
   end

data/lib/conceptql/nodes/recall.rb

@@ -20,17 +20,26 @@ module ConceptQL
       # before we call #query.  Probably time to reevaluate how we're caching
       # the type information.
       def query(db)
+        # We're going to call evaluate on definition to ensure the definition
+        # has been created.  We were running into odd timing issues when
+        # drawing graphs where the recall node was being drawn before definition
+        # was drawn.
+        definition.evaluate(db)
         db.from(table_name)
       end
 
       def types
-        tree.defined[table_name].types
+        definition.types
       end
 
       private
       def table_name
         @table_name ||= namify(arguments.first)
       end
+
+      def definition
+        tree.defined[table_name]
+      end
     end
   end
 end

data/lib/conceptql/nodes/sum.rb

@@ -0,0 +1,24 @@
+require_relative 'pass_thru'
+
+module ConceptQL
+  module Nodes
+    class Sum < PassThru
+      def query(db)
+        db.from(unioned(db))
+          .select_group(*(COLUMNS - [:start_date, :end_date, :criterion_id, :value_as_numeric]))
+          .select_append(Sequel.lit('?', 0).as(:criterion_id))
+          .select_append{ min(start_date).as(:start_date) }
+          .select_append{ max(end_date).as(:end_date) }
+          .select_append{sum(value_as_numeric).as(:value_as_numeric)}
+          .from_self
+      end
+
+      def unioned(db)
+        children.map { |c| c.evaluate(db) }.inject do |uni, q|
+          uni.union(q)
+        end
+      end
+    end
+  end
+end
+

data/lib/conceptql/version.rb

@@ -1,3 +1,3 @@
 module ConceptQL
-  VERSION = "0.0.9"
+  VERSION = "0.1.0"
 end

data/spec/conceptql/nodes/complement_spec.rb

@@ -10,6 +10,6 @@ describe ConceptQL::Nodes::Complement do
   it 'generates complement for single criteria' do
     double1 = QueryDouble.new(1)
     double1.must_behave_like(:evaluator)
-    ConceptQL::Nodes::Complement.new(double1).query(Sequel.mock).sql.must_equal "SELECT * FROM (SELECT person_id AS person_id, visit_occurrence_id AS criterion_id, CAST('visit_occurrence' AS varchar(255)) AS criterion_type, CAST(visit_start_date AS date) AS start_date, CAST(visit_end_date AS date) AS end_date FROM visit_occurrence AS tab WHERE (visit_occurrence_id NOT IN (SELECT criterion_id FROM (SELECT * FROM table1) AS t1 WHERE ((criterion_id IS NOT NULL) AND (criterion_type = 'visit_occurrence'))))) AS t1"
+    ConceptQL::Nodes::Complement.new(double1).query(Sequel.mock).sql.must_equal "SELECT * FROM (SELECT person_id, visit_occurrence_id AS criterion_id, CAST('visit_occurrence' AS varchar(255)) AS criterion_type, CAST(visit_start_date AS date) AS start_date, CAST(visit_end_date AS date) AS end_date, CAST(NULL AS numeric) AS value_as_numeric, CAST(NULL AS varchar(255)) AS value_as_string, CAST(NULL AS integer) AS value_as_concept_id FROM visit_occurrence AS tab WHERE (visit_occurrence_id NOT IN (SELECT criterion_id FROM (SELECT * FROM table1) AS t1 WHERE ((criterion_id IS NOT NULL) AND (criterion_type = 'visit_occurrence'))))) AS t1"
   end
 end

data/spec/conceptql/nodes/time_window_spec.rb

@@ -9,7 +9,7 @@ describe ConceptQL::Nodes::TimeWindow do
     end
 
     def query(db)
-      db
+      db.from(:table)
     end
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: conceptql
 version: !ruby/object:Gem::Version
-  version: 0.0.9
+  version: 0.1.0
 platform: ruby
 authors:
 - Ryan Duryea
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-09-03 00:00:00.000000000 Z
+date: 2014-09-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -156,6 +156,7 @@ files:
 - conceptql.gemspec
 - doc/ConceptQL Specification (alpha).pdf
 - doc/diagram_0.png
+- doc/implementation_notes.md
 - doc/spec.md
 - lib/conceptql.rb
 - lib/conceptql/behaviors/dottable.rb
@@ -172,12 +173,14 @@ files:
 - lib/conceptql/nodes/complement.rb
 - lib/conceptql/nodes/concept.rb
 - lib/conceptql/nodes/condition_type.rb
+- lib/conceptql/nodes/count.rb
 - lib/conceptql/nodes/cpt.rb
 - lib/conceptql/nodes/date_range.rb
 - lib/conceptql/nodes/death.rb
 - lib/conceptql/nodes/define.rb
 - lib/conceptql/nodes/drug_type_concept.rb
 - lib/conceptql/nodes/during.rb
+- lib/conceptql/nodes/equal.rb
 - lib/conceptql/nodes/except.rb
 - lib/conceptql/nodes/first.rb
 - lib/conceptql/nodes/from.rb
@@ -190,6 +193,7 @@ files:
 - lib/conceptql/nodes/last.rb
 - lib/conceptql/nodes/loinc.rb
 - lib/conceptql/nodes/node.rb
+- lib/conceptql/nodes/numeric.rb
 - lib/conceptql/nodes/occurrence.rb
 - lib/conceptql/nodes/pass_thru.rb
 - lib/conceptql/nodes/person.rb
@@ -203,6 +207,7 @@ files:
 - lib/conceptql/nodes/source_vocabulary_node.rb
 - lib/conceptql/nodes/standard_vocabulary_node.rb
 - lib/conceptql/nodes/started_by.rb
+- lib/conceptql/nodes/sum.rb
 - lib/conceptql/nodes/temporal_node.rb
 - lib/conceptql/nodes/time_window.rb
 - lib/conceptql/nodes/union.rb