conceptql 0.5.0 → 0.6.0

data/doc/spec.md

@@ -33,7 +33,7 @@ GDM keeps the data in a small number of tables:
 - deaths
   - One row for each recorded death
 
-ConceptQL supports two variants of GDM. The data model `gdm` reads clinical codes from clinical_codes. The data model `gdm_wide` reads them from observations, a wide table that holds the same rows as clinical_codes with the columns of their contexts and collections joined on. ConceptQL looks up codes, such as ICD-9 412, in a vocabulary database to find their concept IDs.
+ConceptQL supports two variants of GDM. The data model `gdm` reads clinical codes from clinical_codes. The data model `gdm_wide` reads them from observations, a wide table that holds the same rows as clinical_codes with the columns of their contexts and collections joined on. ConceptQL looks up codes, such as ICD-9 412, in a vocabulary database to find their concept IDs. When the data database is a real database, and neither its search path nor the database at LEXICON_URL has the vocabulary tables, ConceptQL raises an error instead of running, because without a vocabulary every code would match no rows.
 
 For instance, using ConceptQL we can take a statement that looks like this:
 
@@ -256,6 +256,43 @@ Selection operators are the parts of a ConceptQL query that search for specific
 
 There are _many_ selection operators. A list of currently implemented operators is available in Appendix A.
 
+### DRG Codes
+
+CMS renumbered the Diagnosis-Related Group (DRG) codes on 2007-10-01, when the MS-DRG system replaced the CMS-DRG system. Most codes got a new meaning: DRG 205 meant a liver disorder in CMS-DRG and means other respiratory system diagnoses in MS-DRG. One vocabulary, DRG, holds the codes of both systems, and a clinical code does not record which meaning applied. So a statement names the DRG system, and ConceptQL selects by the code and by the start_date of the clinical code:
+
+- `ms_drg` selects the DRG codes whose start_date is on or after 2007-10-01.
+- `cms_drg` selects the DRG codes whose start_date is before 2007-10-01.
+
+```YAML
+---
+- ms_drg
+- '205'
+
+```
+
+![All MS-DRG 205 events: other respiratory system diagnoses with a major complication or comorbidity](spec/2f8919588f782c227a2d45b6579a2bf51059bf3a38e5a5524b3edb06e0be9900.png)
+
+| person_id | criterion_id | criterion_table | criterion_domain     | start_date | end_date   | source_value |
+| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
+| 17        | 1365         | observations    | procedure_occurrence | 2010-06-02 | 2010-06-06 | 205          |
+| 68        | 18           | observations    | procedure_occurrence | 2008-04-19 | 2008-04-23 | 205          |
+| 95        | 603          | observations    | procedure_occurrence | 2008-08-27 | 2008-09-01 | 205          |
+
+```YAML
+---
+- cms_drg
+- '205'
+
+```
+
+![All CMS-DRG 205 events (a liver disorder): none, because the test data starts in December 2007](spec/ba330009cc3750e3d781ff6f4a52b1a4e7c6e440b1f950b8e43419f10b1e56dc.png)
+
+`No Results found.`
+
+Both operators check codes as other vocabulary operators do: a DRG code has 3 digits, and a code that the DRG vocabulary does not have gets a warning. When the vocabulary records the system of each meaning (the OHDSI concept class MS-DRG or DRG), the code list of a statement names each code by its meaning in the operator's system. Otherwise it names the code by all of its meanings.
+
+The `drg` operator is deprecated. It selects the DRG codes of both systems, whatever the date. It still runs, but annotation shows a warning on it, and a later release will make it an error.
+
 ## All Other Operators i.e. Mutation Operators
 
 Virtually all other operators add, remove, filter, or otherwise alter streams of results. They are discussed in this section.
@@ -269,7 +306,9 @@ Because streams represent sets of results, its makes sense to include a operator
 - Takes any number of upstream operators and aggregates their streams
   - Unions together streams with identical types
     - Think of streams with the same type flowing together into a single stream
-    - Union does not remove duplicates: a result that arrives from two upstream operators can appear twice
+    - Union removes duplicates: a result that arrives from more than one upstream operator appears once
+      - Every stream is a set, so no operator passes along two identical results
+      - Two results are identical when they have the same value in every column, label included
   - Streams with the different types flow along together concurrently without interacting
     - It does not make sense to union, say, condition_occurrence results with procedure_occurrence results, so streams with different types won't mingle together, but will continue to flow downstream in parallel
 
@@ -353,9 +392,35 @@ Because streams represent sets of results, its makes sense to include a operator
 | 17        | 31480        | observations    | condition_occurrence | 2009-05-21 | 2009-05-21 | 486          |
 | 17        | 53417        | observations    | procedure_occurrence | 2008-04-05 | 2008-04-05 | 99231        |
 
+```YAML
+---
+- union
+- - first
+  - - icd9
+    - '412'
+- - icd9
+  - '412'
+
+```
+
+![The first MI of each patient is also one of the patient's MIs, so it appears once: the union holds the same results as icd9 412 alone](spec/4e6081d437226f76213819b0626b91ce3ba4e340da55c7366bf61ffa276a9b2c.png)
+
+| person_id | criterion_id | criterion_table | criterion_domain     | start_date | end_date   | source_value |
+| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
+| 11        | 82614        | observations    | condition_occurrence | 2008-10-29 | 2008-10-29 | 412          |
+| 17        | 27269        | observations    | condition_occurrence | 2008-08-25 | 2008-08-25 | 412          |
+| 17        | 46165        | observations    | condition_occurrence | 2009-04-30 | 2009-04-30 | 412          |
+| 38        | 40556        | observations    | condition_occurrence | 2010-02-12 | 2010-02-12 | 412          |
+| 54        | 74606        | observations    | condition_occurrence | 2008-06-05 | 2008-06-05 | 412          |
+| 60        | 986          | observations    | condition_occurrence | 2009-07-19 | 2009-07-22 | 412          |
+| 66        | 16171        | observations    | condition_occurrence | 2009-07-25 | 2009-07-25 | 412          |
+| 73        | 52843        | observations    | condition_occurrence | 2008-11-16 | 2008-11-16 | 412          |
+| 77        | 58165        | observations    | condition_occurrence | 2010-10-06 | 2010-10-06 | 412          |
+| 81        | 1405         | observations    | condition_occurrence | 2009-01-28 | 2009-01-30 | 412          |
+
 ### Filter
 
-This operator takes two sets of incoming streams, a left-hand stream and a right-hand stream. It passes along each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same person_id, criterion_id, criterion_domain, start_date, and end_date. The operator discards all results in the right-hand stream. Use filter to keep only the results that two streams have in common, as an intersection would.
+This operator takes two sets of incoming streams, a left-hand stream and a right-hand stream. It passes along each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date, and end_date. A criterion_id identifies a result only within its criterion_table, so results from two tables never match, even when their ids are equal. The operator discards all results in the right-hand stream. Use filter to keep only the results that two streams have in common, as an intersection would.
 
 ```YAML
 ---
@@ -524,6 +589,62 @@ And just to show how multiple streams behave:
 | 66        | 16171        | observations    | condition_occurrence | 2009-07-25 | 2009-07-25 | 412          |
 | 73        | 52843        | observations    | condition_occurrence | 2008-11-16 | 2008-11-16 | 412          |
 
+## Filtering by Provenance
+
+### provenance
+
+`provenance` passes along the events of its upstream stream that come from a given kind of source record. Each event has two provenances. Its file provenance tells which kind of file the record came from, such as `inpatient`, `outpatient` or `carrier_claim`. Its code provenance tells the position of the code in the record, such as `primary`, `first` or `admitting`. Each argument of `provenance` is a keyword from the file or the code provenance vocabulary, and the operator keeps an event when its provenance is that keyword's concept or one of its descendants.
+
+A plain keyword, such as `inpatient` or `primary_or_first`, keeps an event when either provenance matches it. A keyword of the form `<file keyword>:<code keyword>` keeps an event when its file provenance matches the file keyword AND its code provenance matches the code keyword. The left half must be a file provenance keyword and the right half a code provenance keyword. Any other keyword with a `:` is an error.
+
+When `provenance` has several keywords, it keeps the events that match any one of them. So `["inpatient:primary", "outpatient:first"]` keeps the inpatient events with a primary code and the outpatient events with a first code.
+
+```YAML
+---
+- provenance
+- - icd9
+  - '250.00'
+- inpatient:primary_or_first
+
+```
+
+![Diabetes diagnoses that are the primary or first code of an inpatient record](spec/14942bd4e7713897ad9aa646c71cc348d5690cccafadd50e4a7dcd65b9a81061.png)
+
+| person_id | criterion_id | criterion_table | criterion_domain     | start_date | end_date   | source_value |
+| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
+| 37        | 310          | observations    | condition_occurrence | 2009-01-19 | 2009-01-20 | 25000        |
+| 68        | 19           | observations    | condition_occurrence | 2008-04-19 | 2008-04-23 | 25000        |
+| 68        | 1596         | observations    | condition_occurrence | 2008-11-10 | 2008-11-11 | 25000        |
+| 131       | 676          | observations    | condition_occurrence | 2008-03-21 | 2008-03-24 | 25000        |
+| 207       | 106          | observations    | condition_occurrence | 2010-04-28 | 2010-05-02 | 25000        |
+| 237       | 918          | observations    | condition_occurrence | 2010-07-16 | 2010-07-19 | 25000        |
+
+The keyword `*` keeps an event that has any file or code provenance. It drops only the events whose two provenances are both empty. Several keywords are ORed, so `*` with other keywords keeps the same events as `*` alone. A `*` in a `<file keyword>:<code keyword>` keyword is an error. `provenance` needs at least one keyword: with none, it is the error `no provenance keywords; use "*" for any provenance`.
+
+```YAML
+---
+- provenance
+- - icd9
+  - '250.00'
+- "*"
+
+```
+
+![Diabetes diagnoses with any provenance](spec/8614aacee51d8264bc6c61b5e202aa9c0c9318fd66c2ff254cdb45456bdda9fc.png)
+
+| person_id | criterion_id | criterion_table | criterion_domain     | start_date | end_date   | source_value |
+| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
+| 2         | 89752        | observations    | condition_occurrence | 2009-05-11 | 2009-05-11 | 25000        |
+| 2         | 57897        | observations    | condition_occurrence | 2009-05-16 | 2009-05-16 | 25000        |
+| 2         | 76985        | observations    | condition_occurrence | 2009-07-31 | 2009-07-31 | 25000        |
+| 3         | 93400        | observations    | condition_occurrence | 2009-11-17 | 2009-11-17 | 25000        |
+| 5         | 89717        | observations    | condition_occurrence | 2009-06-28 | 2009-06-28 | 25000        |
+| 7         | 15942        | observations    | condition_occurrence | 2008-04-23 | 2008-04-23 | 25000        |
+| 7         | 69706        | observations    | condition_occurrence | 2008-05-12 | 2008-05-12 | 25000        |
+| 7         | 38732        | observations    | condition_occurrence | 2008-06-09 | 2008-06-09 | 25000        |
+| 7         | 38733        | observations    | condition_occurrence | 2008-06-09 | 2008-06-09 | 25000        |
+| 7         | 70454        | observations    | condition_occurrence | 2008-10-02 | 2008-10-02 | 25000        |
+
 ## Co-reported Events
 
 ### co_reported
@@ -856,16 +977,25 @@ There are situations when the date columns associated with a result should have
 
 - Takes 2 arguments
   - First argument is the stream on which to operate
-  - Second argument is a hash with two keys: [:start, :end] each with a value in the following format: "(-?\\d+[dmy])+"
+  - Second argument is a hash with two keys: [:start, :end] each with a value in the following format: "[res]\*([-+]?\\d\*[dwmy])+" or "[res]\*[-+]?\\d+"
+    - d is days, w is weeks (7 days), m is months and y is years
+    - A unit without a number means 1 of that unit
     - Both start and end must be defined, even if you are only adjusting one of the dates
   - Some examples
     - 30d => 30 days
     - 20 => 20 days
     - d => 1 day
+    - 2w => 2 weeks (14 days)
     - 1y => 1 year
     - -1m => -1 month
     - 10d3m => 3 months and 10 days
     - -2y10m-3d => -2 years, +10 months, -3 days
+  - An optional prefix of the letters r, e and s comes before the amounts
+    - r reverses the adjustment: each amount counts back instead of forward, so R365d means 365 days before the date and R-1m means 1 month after it
+    - e adjusts from the result's end_date, so E30d means end_date + 30 days
+    - s adjusts from the result's start_date, so S7d means start_date + 7 days
+    - Without e or s, the start value adjusts start_date and the end value adjusts end_date
+  - A bare number means days, also after an r, e or s prefix: R365, E30 and S7 mean the same as R365d, E30d and S7d
   - The start or end value can also be '', '0', or nil
     - This will leave the date unaffected
   - The start or end value can also be the string 'start' or 'end'
@@ -1036,6 +1166,27 @@ Person streams carry a patient's date of birth in their date columns. This makes
 | 152       | 82988        | observations    | condition_occurrence | 2008-08-16 | 2008-08-16 | 412          |
 | 152       | 84298        | observations    | condition_occurrence | 2010-11-22 | 2010-11-22 | 412          |
 
+### trim_date_end and trim_date_start
+
+- trim_date_end takes a left stream and a right stream. For each person, it trims the end_date of each left record to the earliest start_date of the person's right records.
+  - A left record whose start_date comes after that right start_date is discarded.
+  - A left record passes through unchanged when its person has no right records, or when that right start_date comes after its end_date.
+- trim_date_start trims the other end. For each person, it trims the start_date of each left record to the latest end_date of the person's right records.
+  - A left record whose end_date comes before that right end_date is discarded.
+  - A left record passes through unchanged when its person has no right records, or when that right end_date comes before its start_date.
+- With join_columns, the person's right records are the right records of the person with the same join_columns values as the left record.
+- With compare_all: true, trim_date_end and trim_date_start compare each left record with every right record of the person (and of the join_columns), not only with the person's first or last right record, and return each left record once.
+  - trim_date_end trims its end_date to the earliest right start_date on or after its start_date.
+  - trim_date_start trims its start_date to the latest right end_date on or before its end_date.
+  - A left record is discarded only when it has right records and none of them qualifies.
+
+### Window tables
+
+- The `window` statement runs a statement inside windows: `["window", <statement>, {"window_table": <window table>}]`.
+- The window table has the columns person_id, start_date and end_date. It is the name of a table or a ConceptQL statement.
+- Each selection operator keeps only the results whose person has a window and whose start_date falls in that window. The column window_id of a result identifies its window.
+- If the window table is a ConceptQL statement with errors, annotate reports them under window_table and counts nothing. Running the statement raises InvalidStatementError.
+
 ## Type Conversion
 
 There are situations where it is appropriate to convert the type of a stream of results into a different type. In programmer parlance, we say "typecasting" or "casting", which is the terminology we'll use here. A good analogy and mnemonic for casting is to think of taking a piece of metal, say a candle holder, melting it down, and recasting it into, say, a lamp. We'll do something similar with streams. We'll take, for example, a condition_occurrence stream and recast it into a stream of person.
@@ -1518,64 +1669,37 @@ With a `numeric` operator defined, we can use the `sum` operator to sum the valu
 
 ### Counting
 
-The `count` operator groups identical results and stores the number of results in each group in the value_as_number column. Two results are identical only when all of their columns match, including criterion_id. So `count` does not count the events of each person. Casting a stream to person first does not help, because the cast yields one result for each person.
+The `count` operator is deprecated. It still runs, but annotation shows a warning on it, and a later release will make it an error. `count` groups identical results and stores the number of results in each group in the value_as_number column. Two results are identical only when all of their columns match, including criterion_id. Every stream is a set, so `count` returns 1 for nearly every result. It does not count the events of each person.
 
-To count the events of each person, set value_as_number to 1 with `numeric`, and then use `sum`:
+To count the events of each person, set value_as_number to 1 with `numeric`, and then use `sum` with its `group_by_person` option. `sum` gives one person result for each person, and value_as_number holds the number of events. Without `group_by_person`, `sum` puts the results of one person in several groups when they have different source_values, so the count of the person is split over several results.
 
 ```YAML
 ---
 - sum
 - - numeric
   - 1
-  - - icd9
-    - '486'
-
-```
-
-![Count the number of times each person had pneumonia](spec/385ede8312e323fa1ae2337c303c8e89f252ebcb8ba229961fa3af1a3caec886.png)
-
-| person_id | criterion_id | criterion_table | criterion_domain     | start_date | end_date   | source_value | value_as_number |
-| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ | --------------- |
-| 14        | 0            | observations    | condition_occurrence | 2008-09-12 | 2009-04-05 | 486          | 2.0             |
-| 17        | 0            | observations    | condition_occurrence | 2009-05-21 | 2009-05-21 | 486          | 1.0             |
-| 19        | 0            | observations    | condition_occurrence | 2008-02-02 | 2008-02-03 | 486          | 1.0             |
-| 23        | 0            | observations    | condition_occurrence | 2009-08-18 | 2010-02-17 | 486          | 3.0             |
-| 25        | 0            | observations    | condition_occurrence | 2009-07-08 | 2009-07-08 | 486          | 1.0             |
-| 27        | 0            | observations    | condition_occurrence | 2010-09-30 | 2010-09-30 | 486          | 1.0             |
-| 28        | 0            | observations    | condition_occurrence | 2009-09-12 | 2009-09-12 | 486          | 1.0             |
-| 38        | 0            | observations    | condition_occurrence | 2010-02-24 | 2010-02-24 | 486          | 3.0             |
-| 44        | 0            | observations    | condition_occurrence | 2009-09-07 | 2010-08-30 | 486          | 6.0             |
-| 51        | 0            | observations    | condition_occurrence | 2008-03-08 | 2008-10-18 | 486          | 2.0             |
-
-We could do dumb things like count the number of times a row shows up in a union:
-
-```YAML
----
-- count
-- - union
   - - icd9
     - '412'
-  - - provenance
-    - - icd9
-      - '412'
-    - inpatient
+    - '414.01'
+    - '250.00'
+- :group_by_person: true
 
 ```
 
-![All rows with a value of 2 would be rows that were both MI and from an inpatient record](spec/ecede6b94c97cf0b036a664e4df704d119be7c31607a1defe6eb77f294fd1393.png)
-
-| person_id | criterion_id | criterion_table | criterion_domain     | start_date | end_date   | source_value | value_as_number |
-| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ | --------------- |
-| 11        | 82614        | observations    | condition_occurrence | 2008-10-29 | 2008-10-29 | 412          | 1               |
-| 17        | 27269        | observations    | condition_occurrence | 2008-08-25 | 2008-08-25 | 412          | 1               |
-| 17        | 46165        | observations    | condition_occurrence | 2009-04-30 | 2009-04-30 | 412          | 1               |
-| 38        | 40556        | observations    | condition_occurrence | 2010-02-12 | 2010-02-12 | 412          | 1               |
-| 54        | 74606        | observations    | condition_occurrence | 2008-06-05 | 2008-06-05 | 412          | 1               |
-| 60        | 986          | observations    | condition_occurrence | 2009-07-19 | 2009-07-22 | 412          | 2               |
-| 66        | 16171        | observations    | condition_occurrence | 2009-07-25 | 2009-07-25 | 412          | 1               |
-| 73        | 52843        | observations    | condition_occurrence | 2008-11-16 | 2008-11-16 | 412          | 1               |
-| 77        | 58165        | observations    | condition_occurrence | 2010-10-06 | 2010-10-06 | 412          | 1               |
-| 81        | 1405         | observations    | condition_occurrence | 2009-01-28 | 2009-01-30 | 412          | 2               |
+![Count the number of times each person had a heart attack or diabetes](spec/8324a33ed37a82d29fb36ba9ed6fe84b445b7589af1c0c016813d2d98b2c6d75.png)
+
+| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date   | source_value | value_as_number |
+| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ | --------------- |
+| 2         | 0            | patients        | person           | 2009-05-11 | 2009-07-31 |              | 3.0             |
+| 3         | 0            | patients        | person           | 2009-11-17 | 2009-11-17 |              | 1.0             |
+| 5         | 0            | patients        | person           | 2009-06-28 | 2009-06-28 |              | 1.0             |
+| 6         | 0            | patients        | person           | 2010-09-26 | 2010-09-26 |              | 1.0             |
+| 7         | 0            | patients        | person           | 2008-04-23 | 2010-07-24 |              | 30.0            |
+| 8         | 0            | patients        | person           | 2008-10-17 | 2009-07-16 |              | 3.0             |
+| 9         | 0            | patients        | person           | 2010-08-01 | 2010-09-12 |              | 2.0             |
+| 11        | 0            | patients        | person           | 2008-06-17 | 2009-11-12 |              | 4.0             |
+| 12        | 0            | patients        | person           | 2009-04-20 | 2009-05-21 |              | 3.0             |
+| 13        | 0            | patients        | person           | 2008-03-05 | 2010-09-08 |              | 9.0             |
 
 #### Numeric Value Comparison
 
@@ -1593,11 +1717,12 @@ The `numeric_filter` operator takes one stream. It passes along the results whos
       - 1
       - - icd9
         - '412'
+    - :group_by_person: true
   - :greater_than_or_equal_to: 2
 
 ```
 
-![People with more than 1 MI](spec/0d8546d33c77d25b522ae23b74c87bf9ee77f5564c6de688af8a2b270b615cfb.png)
+![People with more than 1 MI](spec/6613c788b33483210fbc0c40fdcab5942bd3fca204c8a188ae176d51435041b0.png)
 
 | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date   | source_value     | value_as_number |
 | --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- | --------------- |
@@ -1646,27 +1771,42 @@ Numeric doesn't have to take a stream. If it doesn't have a stream as an argumen
     - Sets start_date to the earliest start_date in the group
     - Sets the end_date to the most recent end_date in the group
     - Sets criterion_id to 0 since there is no particular single row that the result refers to anymore
+- Takes one option, `group_by_person`, which is true or false. When it is true, `sum` puts all of the results of each person in one group, whatever their type and source_value. It gives one person result for each person:
+  - person_id is the person
+  - criterion_domain is person and criterion_table is the person table, as a cast to `person` gives them
+  - criterion_id is 0
+  - start_date is the earliest start_date and end_date is the most recent end_date of the person's results
+  - value_as_number is the sum of the value_as_number column of the person's results
+  - The other columns, such as source_value, are NULL
+  - In a window table, `sum` gives one result for each person and window
 
 ## Appendix A - Selection Operators
 
 ConceptQL creates a vocabulary operator for each vocabulary in its vocabulary list (`config/vocabularies.csv`) and in its vocabulary database. This table lists the most common selection operators.
 
-| Operator Name       | Stream Type          | Arguments                                | Returns                                                        |
-| ------------------- | -------------------- | ---------------------------------------- | -------------------------------------------------------------- |
-| cpt                 | procedure_occurrence | 1 or more CPT codes                      | All clinical codes that match any of the CPT codes             |
-| icd9                | condition_occurrence | 1 or more ICD-9-CM diagnosis codes       | All clinical codes that match any of the ICD-9 codes           |
-| icd9_procedure      | procedure_occurrence | 1 or more ICD-9-CM procedure codes       | All clinical codes that match any of the ICD-9 procedure codes |
-| icd10cm             | condition_occurrence | 1 or more ICD-10-CM codes                | All clinical codes that match any of the ICD-10-CM codes       |
-| hcpcs               | procedure_occurrence | 1 or more HCPCS codes                    | All clinical codes that match any of the HCPCS codes           |
-| loinc               | observation          | 1 or more LOINC codes                    | All clinical codes that match any of the LOINC codes           |
-| ndc                 | drug_exposure        | 1 or more NDC codes                      | All clinical codes that match any of the NDC codes             |
-| rxnorm              | drug_exposure        | 1 or more RxNorm codes                   | All clinical codes that match any of the RxNorm codes          |
-| snomed              | condition_occurrence | 1 or more SNOMED codes                   | All clinical codes that match any of the SNOMED codes          |
-| gender              | person               | 1 or more of Male, Female, and Unknown   | All patients whose gender matches any of the arguments         |
-| race                | person               | 1 or more race names or race concept IDs | All patients whose race matches any of the arguments           |
-| death               | death                | None                                     | All death records                                              |
-| information_periods | observation_period   | None                                     | All information periods                                        |
-| date_range          | person               | A start date and an end date             | One result for each patient, with the given dates              |
+| Operator Name       | Stream Type          | Arguments                                | Returns                                                              |
+| ------------------- | -------------------- | ---------------------------------------- | -------------------------------------------------------------------- |
+| cpt                 | procedure_occurrence | 1 or more CPT codes                      | All clinical codes that match any of the CPT codes                   |
+| icd9                | condition_occurrence | 1 or more ICD-9-CM diagnosis codes       | All clinical codes that match any of the ICD-9 codes                 |
+| icd9_procedure      | procedure_occurrence | 1 or more ICD-9-CM procedure codes       | All clinical codes that match any of the ICD-9 procedure codes       |
+| icd10cm             | condition_occurrence | 1 or more ICD-10-CM codes                | All clinical codes that match any of the ICD-10-CM codes             |
+| hcpcs               | procedure_occurrence | 1 or more HCPCS codes                    | All clinical codes that match any of the HCPCS codes                 |
+| ms_drg              | procedure_occurrence | 1 or more DRG codes                      | All clinical codes from 2007-10-01 that match any of the DRG codes   |
+| cms_drg             | procedure_occurrence | 1 or more DRG codes                      | All clinical codes before 2007-10-01 that match any of the DRG codes |
+| loinc               | observation          | 1 or more LOINC codes                    | All clinical codes that match any of the LOINC codes                 |
+| ndc                 | drug_exposure        | 1 or more NDC codes                      | All clinical codes that match any of the NDC codes                   |
+| rxnorm              | drug_exposure        | 1 or more RxNorm codes                   | All clinical codes that match any of the RxNorm codes                |
+| snomed              | condition_occurrence | 1 or more SNOMED codes                   | All clinical codes that match any of the SNOMED codes                |
+| gender              | person               | 1 or more of Male, Female, and Unknown   | All patients whose gender matches any of the arguments               |
+| race                | person               | 1 or more race names or race concept IDs | All patients whose race matches any of the arguments                 |
+| death               | death                | None                                     | All death records                                                    |
+| information_periods | observation_period   | None                                     | All information periods                                              |
+| date_range          | person               | A start date and an end date             | One result for each patient, with the given dates                    |
+| hospitalization     | condition_occurrence | None                                     | All inpatient admissions                                             |
+| hospice             | condition_occurrence | None                                     | All hospice admissions                                               |
+| snf                 | condition_occurrence | None                                     | All skilled nursing facility admissions                              |
+
+hospitalization, hospice and snf select admissions. On gdm, they read collections and admission_details. On gdm_wide, they read observations, but their results still have criterion_table collections. Their results have criterion_domain condition_occurrence, because GDM has no visit or admission domain. primary_concept_id, source_value and source_vocabulary_id hold one code of the admission that has primary provenance. That code is usually a diagnosis, but it can be a procedure or a DRG. When the admission has no code with primary provenance, these three columns are NULL. Because the results have the condition_occurrence type, union combines them with the results of condition operators such as icd9 into one stream.
 
 ## Appendix B - Algorithm Showcase
 

data/lib/conceptql/behaviors/metadatable.rb

@@ -38,6 +38,15 @@ module ConceptQL
       @conceptql_spec_id = value
     end
 
+    # Marks an operator as deprecated. It still runs. Operator#validate adds
+    # the message to its annotation as a warning, and its metadata says
+    # deprecated: true. A subclass does not inherit the mark.
+    def deprecated(message = nil)
+      return @deprecated unless message
+
+      @deprecated = message
+    end
+
     def predominant_domains(*values)
       return @predominant_domains if values.empty?
 
@@ -150,11 +159,12 @@ module ConceptQL
         max_upstreams: @max_upstreams || 0,
         arguments: @arguments || [],
         options: process_options(@options),
-        predominant_domains: @domains || @predominant_domains || [],
+        # Strings, as JSON gives them: domains (for example death) are symbols
+        predominant_domains: (@domains || @predominant_domains || []).map(&:to_s),
         desc: get_desc,
         categories: @categories || [],
         basic_type: @basic_type,
-        deprecated: nil, # kept for metadata consumers; no operator is deprecated
+        deprecated: (true if deprecated), # nil for an operator that is not deprecated
         arity: arity,
         aliases: @aliases,
         spec_id: spec_id
@@ -176,8 +186,10 @@ module ConceptQL
       end
     end
 
+    # The desc that the operator set (no_desc sets ""), else the standard
+    # vocabulary text. A subclass does not inherit its parent's desc.
     def get_desc
-      @get_desc ||= standard_description
+      @desc || standard_description
     end
 
     def process_options(options)

data/lib/conceptql/behaviors/provenanceable.rb

@@ -1,26 +1,100 @@
 # frozen_string_literal: true
 
 module ConceptQL
+  # Provenance keywords select records by their file provenance
+  # (file_provenance_type) and their code provenance (code_provenance_type).
+  #
+  # A plain keyword, such as "inpatient" or "primary", names a concept in the
+  # file or the code provenance vocabulary. It keeps a record when either
+  # column is in the descendants of that concept.
+  #
+  # A keyword "<file keyword>:<code keyword>", such as
+  # "inpatient:primary_or_first", keeps a record when its file provenance is
+  # in the descendants of the file keyword AND its code provenance is in the
+  # descendants of the code keyword.
+  #
+  # Several keywords are ORed: a record is kept when any keyword keeps it.
+  #
+  # The keyword "*" keeps a record when either column is any provenance
+  # concept. Because keywords are ORed, "*" with other keywords is "*".
   module Provenanceable
+    PAIR_SEPARATOR = ":"
+    ANY = "*"
+
     def self.included(base)
       base.require_column :file_provenance_type
       base.require_column :code_provenance_type
     end
 
-    def prov_of(db, ancestors)
-      dm.descendants_of(db,
-                        dm.concept_ids(db, [ dm.file_provenance_types_vocab, dm.code_provenance_types_vocab ].flatten,
-                                       ancestors))
+    # No keyword at all is a validation error of Provenance. It still builds
+    # the condition of "*", as an invalid pair still builds a condition.
+    def build_where_from_codes(db, codes)
+      return plain_condition(db, []) if codes.empty? || codes.include?(ANY)
+
+      pairs, plain = codes.partition { |code| pair_keyword?(code) }
+      conditions = pairs.map { |pair| pair_condition(db, pair) }
+      conditions.unshift(plain_condition(db, plain)) if plain.any?
+      return conditions.first if conditions.one?
+
+      Sequel.|(*conditions)
     end
 
-    def build_where_from_codes(db, codes)
+    # The keywords as written that name no provenance concept. A pair is
+    # recognized only when it has one separator, a file keyword on the left
+    # and a code keyword on the right. Case is compared exactly, as the lookup
+    # compares it: the provenance vocabularies have no vocabulary_formats row
+    # that declares case-insensitivity (seed conceptql-oji ruling "b"). So
+    # "Inpatient" names no concept, and the concept codes are in lower case.
+    # "*" is known. It is not a half of a pair.
+    def find_bad_keywords(db, codes)
+      pairs, plain = (codes - [ ANY ]).partition { |code| pair_keyword?(code) }
+      known = known_keywords(db, all_provenance_types_vocab)
+      bad = plain.reject { |code| known.include?(code.to_s) }
+      bad += pairs.reject { |pair| known_pair?(db, pair) }
+      codes & bad
+    end
+
+    private
+
+    def pair_keyword?(code)
+      code.to_s.include?(PAIR_SEPARATOR)
+    end
+
+    def known_pair?(db, pair)
+      file, code, *rest = pair.split(PAIR_SEPARATOR, -1)
+      rest.empty? &&
+        known_keywords(db, dm.file_provenance_types_vocab).include?(file) &&
+        known_keywords(db, dm.code_provenance_types_vocab).include?(code)
+    end
+
+    # The concept codes of the vocabularies.
+    def known_keywords(db, vocabularies)
+      dm.concept_codes(db, vocabularies)
+    end
+
+    def all_provenance_types_vocab
+      [ dm.file_provenance_types_vocab, dm.code_provenance_types_vocab ].flatten
+    end
+
+    def prov_of(db, vocabularies, ancestors)
+      dm.descendants_of(db, dm.concept_ids(db, vocabularies, ancestors))
+    end
+
+    def plain_condition(db, codes)
       Sequel.|(
-        { file_provenance_type: prov_of(db, codes) }, { code_provenance_type: prov_of(db, codes) }
+        { file_provenance_type: prov_of(db, all_provenance_types_vocab, codes) },
+        { code_provenance_type: prov_of(db, all_provenance_types_vocab, codes) }
       )
     end
 
-    def find_bad_keywords(db, codes)
-      codes - dm.concept_codes(db, [ dm.file_provenance_types_vocab, dm.code_provenance_types_vocab ].flatten)
+    # An invalid pair (validation reports it) still builds a condition. A
+    # half with no concept matches nothing, so the pair keeps no record.
+    def pair_condition(db, pair)
+      file, code = pair.split(PAIR_SEPARATOR, 2)
+      Sequel.&(
+        { file_provenance_type: prov_of(db, dm.file_provenance_types_vocab, [ file ]) },
+        { code_provenance_type: prov_of(db, dm.code_provenance_types_vocab, [ code ]) }
+      )
     end
   end
 end

data/lib/conceptql/behaviors/utilizable.rb

@@ -41,13 +41,17 @@ module ConceptQL
       # With the Database's share_utilization option, the list is read from a
       # table that every statement on the Database shares. The WITH entry only
       # carries the list to Scope#with_ctes, which takes it out of the SQL and
-      # creates the table when the query first reads rows.
+      # creates the table when the query first reads rows. With
+      # CONCEPTQL_AVOID_CTES, with_ctes extracts nothing, so the scope keeps
+      # the table instead.
       def evaluate(db)
         cdb = nodifier.cdb
         return super unless cdb.opts[:share_utilization]
 
         list = select_it(query(db))
         name = cdb.shared_utilization_table(list, scope)
+        return scope.add_shared_table(db, name, list) if ConceptQL.avoid_ctes?
+
         db[name].with(name, list, Scope::SHARED_TABLE => true)
       end
 

data/lib/conceptql/data_model/base.rb

@@ -19,6 +19,7 @@ module ConceptQL
                      :concepts_by_name,
                      :concepts_ds,
                      :concepts_to_codes,
+                     :concepts_to_codes_in_class,
                      :descendants_of,
                      :known_codes,
                      :related_concept_ids
@@ -258,10 +259,7 @@ module ConceptQL
       end
 
       def table_columns(*tables)
-        tables.map { |t| table_cols(t) }.flatten.tap do |o|
-          p o
-          p tables
-        end
+        tables.flat_map { |t| table_cols(t) }
       end
 
       def table_source_value(table)

data/lib/conceptql/data_model/gdm.rb

@@ -115,8 +115,10 @@ module ConceptQL
         :context_id
       end
 
+      # The schema of a data model does not change, so the answer is kept, as
+      # for start_date_columns and the other column lookups.
       def source_vocabulary_ids
-        @source_vocabulary_ids = assign_column_to_table do |table, columns|
+        @source_vocabulary_ids ||= assign_column_to_table do |table, columns|
           next if %w[patients death].any? { |tn| table.to_s =~ /#{tn}/ }
 
           reggy = /#{table.to_s.split('_').first}_vocabulary_id/

data/lib/conceptql/database.rb

@@ -95,8 +95,10 @@ module ConceptQL
       end
 
       # The LEXICON_URL database, or nil when it is unset. Lexicon tries the
-      # data db first and, when neither has the vocabulary tables, falls back
-      # to LexiconNoDB, so no database (and no sqlite3 gem) is needed here.
+      # data db first. When neither has the vocabulary tables, a real data db
+      # raises Lexicon::NoVocabularyError, and a db-less Database (db nil or a
+      # Sequel mock) falls back to LexiconNoDB, so no database (and no sqlite3
+      # gem) is needed here.
       def make_lexicon_db
         return unless ENV["LEXICON_URL"]
 
@@ -126,6 +128,9 @@ module ConceptQL
       @source_tables.fetch(key) { db.table_exists?(table) && (@source_tables[key] = true) }
     end
 
+    # The Lexicon passed as lexicon:, or one built from the data db and the
+    # LEXICON_URL database. Database.new calls it, so a real data db without
+    # vocabulary tables raises Lexicon::NoVocabularyError there.
     def lexicon
       @lexicon ||= Lexicon.new(self.class.lexicon_db, db, cache: @opts[:cache_lookups])
     end