conceptql 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +176 -0
- data/README.md +4 -2
- data/assets/diagram/diagram.css +331 -0
- data/assets/diagram/fonts/OFL.txt +93 -0
- data/assets/diagram/fonts/Roboto-400.ttf +0 -0
- data/assets/diagram/fonts/Roboto-700.ttf +0 -0
- data/assets/diagram/fonts/Roboto-Italic-400.ttf +0 -0
- data/assets/diagram/fonts/Roboto-Italic-Latin.woff2 +0 -0
- data/assets/diagram/fonts/Roboto-Normal-Latin.woff2 +0 -0
- data/assets/diagram/operators.svg +224 -0
- data/config/vocabularies.csv +85 -85
- data/docs/spec/README.md +2624 -0
- data/docs/spec/examples/after-operator.json +562 -0
- data/docs/spec/examples/any-overlap-operator.json +106 -0
- data/docs/spec/examples/before-operator.json +437 -0
- data/docs/spec/examples/co-reported-operator.json +168 -0
- data/docs/spec/examples/contains-operator.json +97 -0
- data/docs/spec/examples/date-range-operator.json +275 -0
- data/docs/spec/examples/death-operator.json +103 -0
- data/docs/spec/examples/during-operator.json +178 -0
- data/docs/spec/examples/episode-operator.json +80 -0
- data/docs/spec/examples/equal-operator.json +79 -0
- data/docs/spec/examples/ethnicity-operator.json +86 -0
- data/docs/spec/examples/except-operator.json +536 -0
- data/docs/spec/examples/filter-operator.json +218 -0
- data/docs/spec/examples/first-operator.json +148 -0
- data/docs/spec/examples/gender-operator.json +257 -0
- data/docs/spec/examples/hospice-operator.json +40 -0
- data/docs/spec/examples/hospitalization-operator.json +131 -0
- data/docs/spec/examples/information-periods-operator.json +131 -0
- data/docs/spec/examples/last-operator.json +148 -0
- data/docs/spec/examples/match-operator.json +186 -0
- data/docs/spec/examples/nth-occurrence-operator.json +300 -0
- data/docs/spec/examples/numeric-filter-operator.json +77 -0
- data/docs/spec/examples/one-in-two-out-operator.json +158 -0
- data/docs/spec/examples/person-filter-operator.json +919 -0
- data/docs/spec/examples/person-operator.json +274 -0
- data/docs/spec/examples/place-of-service-filter-operator.json +57 -0
- data/docs/spec/examples/provenance-operator.json +148 -0
- data/docs/spec/examples/provider-filter-operator.json +77 -0
- data/docs/spec/examples/race-operator.json +131 -0
- data/docs/spec/examples/recall-operator.json +289 -0
- data/docs/spec/examples/snf-operator.json +40 -0
- data/docs/spec/examples/time-window-operator.json +736 -0
- data/docs/spec/examples/trim-date-end-operator.json +201 -0
- data/docs/spec/examples/trim-date-start-operator.json +201 -0
- data/docs/spec/examples/union-operator.json +547 -0
- data/docs/spec/examples/vocabulary-operator.json +232 -0
- data/lib/conceptql/behaviors/metadatable.rb +68 -2
- data/lib/conceptql/behaviors/provenanceable.rb +4 -1
- data/lib/conceptql/behaviors/utilizable.rb +3 -0
- data/lib/conceptql/code_case_mismatch_error.rb +50 -0
- data/lib/conceptql/data_model/gdm.rb +23 -2
- data/lib/conceptql/diagram_renderer/app.rb +59 -0
- data/lib/conceptql/diagram_renderer/configuration.rb +87 -0
- data/lib/conceptql/diagram_renderer/diagram_data.rb +80 -0
- data/lib/conceptql/diagram_renderer/drawing.rb +255 -0
- data/lib/conceptql/diagram_renderer/geometry.rb +380 -0
- data/lib/conceptql/diagram_renderer/input_error.rb +11 -0
- data/lib/conceptql/diagram_renderer/js_format.rb +112 -0
- data/lib/conceptql/diagram_renderer/resvg.rb +120 -0
- data/lib/conceptql/diagram_renderer/sprite.rb +73 -0
- data/lib/conceptql/diagram_renderer/static.rb +70 -0
- data/lib/conceptql/diagram_renderer/svg.rb +73 -0
- data/lib/conceptql/diagram_renderer/tree_layout.rb +287 -0
- data/lib/conceptql/diagram_renderer.rb +82 -0
- data/lib/conceptql/lexicon/lexicon_strategy.rb +16 -0
- data/lib/conceptql/lexicon.rb +63 -11
- data/lib/conceptql/nodifier.rb +5 -3
- data/lib/conceptql/operators/after.rb +15 -1
- data/lib/conceptql/operators/any_overlap.rb +6 -1
- data/lib/conceptql/operators/before.rb +11 -1
- data/lib/conceptql/operators/binary_operator_operator.rb +12 -2
- data/lib/conceptql/operators/co_reported.rb +9 -4
- data/lib/conceptql/operators/contains.rb +6 -1
- data/lib/conceptql/operators/count.rb +2 -0
- data/lib/conceptql/operators/date_range.rb +15 -3
- data/lib/conceptql/operators/death.rb +5 -1
- data/lib/conceptql/operators/dedup.rb +2 -0
- data/lib/conceptql/operators/during.rb +6 -1
- data/lib/conceptql/operators/episode.rb +13 -2
- data/lib/conceptql/operators/equal.rb +11 -1
- data/lib/conceptql/operators/ethnicity.rb +19 -7
- data/lib/conceptql/operators/except.rb +10 -1
- data/lib/conceptql/operators/filter.rb +6 -1
- data/lib/conceptql/operators/first.rb +19 -9
- data/lib/conceptql/operators/from.rb +3 -0
- data/lib/conceptql/operators/gender.rb +10 -2
- data/lib/conceptql/operators/hospice.rb +7 -1
- data/lib/conceptql/operators/hospitalization.rb +6 -1
- data/lib/conceptql/operators/information_periods.rb +26 -7
- data/lib/conceptql/operators/invalid.rb +4 -0
- data/lib/conceptql/operators/last.rb +19 -9
- data/lib/conceptql/operators/match.rb +6 -1
- data/lib/conceptql/operators/multiple_vocabularies.rb +17 -3
- data/lib/conceptql/operators/numeric.rb +4 -0
- data/lib/conceptql/operators/numeric_filter.rb +13 -3
- data/lib/conceptql/operators/occurrence.rb +33 -12
- data/lib/conceptql/operators/one_in_two_out.rb +22 -5
- data/lib/conceptql/operators/operator.rb +8 -4
- data/lib/conceptql/operators/person.rb +5 -1
- data/lib/conceptql/operators/person_filter.rb +14 -1
- data/lib/conceptql/operators/place_of_service_filter.rb +14 -3
- data/lib/conceptql/operators/provenance.rb +17 -8
- data/lib/conceptql/operators/provider_filter.rb +18 -6
- data/lib/conceptql/operators/race.rb +14 -9
- data/lib/conceptql/operators/read.rb +11 -1
- data/lib/conceptql/operators/recall.rb +34 -2
- data/lib/conceptql/operators/snf.rb +7 -1
- data/lib/conceptql/operators/sum.rb +4 -0
- data/lib/conceptql/operators/temporal_operator.rb +4 -2
- data/lib/conceptql/operators/time_window.rb +27 -4
- data/lib/conceptql/operators/trim_date.rb +6 -14
- data/lib/conceptql/operators/trim_date_end.rb +9 -3
- data/lib/conceptql/operators/trim_date_start.rb +9 -3
- data/lib/conceptql/operators/union.rb +13 -1
- data/lib/conceptql/operators/vocabulary.rb +80 -6
- data/lib/conceptql/paths.rb +9 -1
- data/lib/conceptql/version.rb +1 -1
- data/lib/conceptql/vocabularies/drg_systems.rb +3 -2
- data/lib/conceptql/vocabularies/entry.rb +14 -11
- data/lib/conceptql/vocabularies/vocabulary_class_factory.rb +2 -1
- data/lib/conceptql.rb +3 -0
- metadata +75 -2
- data/doc/spec.md +0 -2091
data/docs/spec/README.md
ADDED
|
@@ -0,0 +1,2624 @@
|
|
|
1
|
+
<!-- just spec writes this file from README.cql.md. Edit README.cql.md, not this file. -->
|
|
2
|
+
|
|
3
|
+
# ConceptQL Specification
|
|
4
|
+
|
|
5
|
+
[ConceptQL](https://github.com/outcomesinsights/conceptql) (pronounced concept-Q-L) is a high-level language that lets researchers define their research algorithms without ambiguity. It has run in production since 2014 as the core of our study-building software, [Jigsaw](https://jigsaw.io).
|
|
6
|
+
|
|
7
|
+
Its implementation has changed considerably over that time, but it has kept its original goals.
|
|
8
|
+
|
|
9
|
+
## Motivation for ConceptQL
|
|
10
|
+
|
|
11
|
+
Outcomes Insights has built a large library of research algorithms and has applied them to large databases of claims data. Early in that work, we had to solve two problems:
|
|
12
|
+
|
|
13
|
+
1. Methods sections of research papers commonly use natural language to specify the criteria used to build cohorts from a claims database.
|
|
14
|
+
- Algorithms defined in natural language are often imprecise, open to multiple interpretations, and difficult to reproduce.
|
|
15
|
+
- Researchers need a language that removes the ambiguity of natural language and makes their algorithms reproducible.
|
|
16
|
+
2. Querying claims databases is difficult.
|
|
17
|
+
- Hand-coding the queries that extract cohorts from a dataset is slow, error-prone, and opaque.
|
|
18
|
+
- Researchers need a language in which they define an algorithm at a high level, and which a tool translates into the correct database queries.
|
|
19
|
+
|
|
20
|
+
We developed ConceptQL to solve these two problems.
|
|
21
|
+
|
|
22
|
+
The reference implementation of ConceptQL reads an algorithm written in ConceptQL. It draws a diagram of the algorithm, which makes the algorithm easy to see and understand. It also translates the algorithm into SQL that runs against data in the [Generalized Data Model (GDM)](https://github.com/outcomesinsights/generalized_data_model). The GDM standardizes the format and content of observational data, so that standard applications, tools and methods can use them.
|
|
23
|
+
|
|
24
|
+
For example, ConceptQL takes a statement like this:
|
|
25
|
+
|
|
26
|
+
```JSON
|
|
27
|
+
[ "icd9", "412" ]
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
It draws this diagram:
|
|
31
|
+
|
|
32
|
+
______________________________________________________________________
|
|
33
|
+
|
|
34
|
+
**Example 1**
|
|
35
|
+
|
|
36
|
+
```JSON
|
|
37
|
+
["icd9","412"]
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+

|
|
41
|
+
|
|
42
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
43
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
44
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
45
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
46
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
47
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
48
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
49
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
50
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
51
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
52
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
53
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
54
|
+
|
|
55
|
+
______________________________________________________________________
|
|
56
|
+
|
|
57
|
+
And it generates SQL like this (simplified):
|
|
58
|
+
|
|
59
|
+
```SQL
|
|
60
|
+
SELECT patient_id AS person_id,
|
|
61
|
+
id AS criterion_id,
|
|
62
|
+
'clinical_codes' AS criterion_table,
|
|
63
|
+
'condition_occurrence' AS criterion_domain,
|
|
64
|
+
start_date,
|
|
65
|
+
end_date,
|
|
66
|
+
clinical_code_source_value AS source_value
|
|
67
|
+
FROM clinical_codes
|
|
68
|
+
WHERE clinical_code_vocabulary_id = 'ICD9CM'
|
|
69
|
+
AND clinical_code_concept_id IN (44820864)
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
ConceptQL looks up the concept ID of each code in the vocabulary tables when it builds the SQL. Here, 44820864 is the concept for ICD-9-CM 412.
|
|
73
|
+
|
|
74
|
+
One goal of ConceptQL is to let researchers assemble complex queries without writing SQL. To support that complexity, ConceptQL has some complexity of its own. We believe that ConceptQL helps researchers define, refine, and share their research algorithms.
|
|
75
|
+
|
|
76
|
+
## ConceptQL Overview
|
|
77
|
+
|
|
78
|
+
### What ConceptQL Looks Like
|
|
79
|
+
|
|
80
|
+
A ConceptQL statement is a set of nested arrays and hashes. This example is in YAML, but JSON or any other format that can represent nested arrays and hashes works equally well. If you squint, it looks a bit like [Lisp](<https://en.wikipedia.org/wiki/Lisp_(programming_language)>): the nested arrays are [S-expressions](https://en.wikipedia.org/wiki/S-expression).
|
|
81
|
+
|
|
82
|
+
ConceptQL is a domain-specific, declarative language. It represents search criteria, and it combines them with set operations and temporal operations.
|
|
83
|
+
|
|
84
|
+
```YAML
|
|
85
|
+
# Example 1: A simple example in YAML
|
|
86
|
+
# The first element is the operator, icd9cm. The second element is a code, 412.
|
|
87
|
+
# This statement selects every record with the ICD-9-CM diagnosis code 412.
|
|
88
|
+
---
|
|
89
|
+
- icd9cm
|
|
90
|
+
- '412'
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
The first element of each array is the name of an operator. The other elements are the operator's arguments, its upstream operators, and a hash of options. Codes are strings, so quote them in YAML. Operator names are not case-sensitive.
|
|
94
|
+
|
|
95
|
+
### ConceptQL Diagrams
|
|
96
|
+
|
|
97
|
+
ConceptQL in YAML or JSON can be hard to read. A directed graph is often easier to explore. This is the diagram for the YAML example above:
|
|
98
|
+
|
|
99
|
+
______________________________________________________________________
|
|
100
|
+
|
|
101
|
+
**Example 2 - All Conditions Matching MI**
|
|
102
|
+
|
|
103
|
+
```JSON
|
|
104
|
+
["icd9","412"]
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+

|
|
108
|
+
|
|
109
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
110
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
111
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
112
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
113
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
114
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
115
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
116
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
117
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
118
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
119
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
120
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
121
|
+
|
|
122
|
+
______________________________________________________________________
|
|
123
|
+
|
|
124
|
+
Each node in a diagram is an operator. A line between two operators shows that the records of the upper operator flow down into the lower operator. Each line is labeled with the number of records (`rows=`) and people (`n=`) that flow along it in our test data. Here is a slightly larger example:
|
|
125
|
+
|
|
126
|
+
______________________________________________________________________
|
|
127
|
+
|
|
128
|
+
**Example 3 - First Office Visit Per Patient**
|
|
129
|
+
|
|
130
|
+
```JSON
|
|
131
|
+
["first",["cpt","99214"]]
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+

|
|
135
|
+
|
|
136
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
137
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
138
|
+
| 2 | 58127 | observations | procedure_occurrence | 2009-07-03 | 2009-07-03 | 99214 |
|
|
139
|
+
| 5 | 38313 | observations | procedure_occurrence | 2008-12-14 | 2008-12-14 | 99214 |
|
|
140
|
+
| 6 | 48992 | observations | procedure_occurrence | 2009-09-12 | 2009-09-12 | 99214 |
|
|
141
|
+
| 7 | 63797 | observations | procedure_occurrence | 2008-08-13 | 2008-08-13 | 99214 |
|
|
142
|
+
| 8 | 55871 | observations | procedure_occurrence | 2008-02-27 | 2008-02-27 | 99214 |
|
|
143
|
+
| 9 | 45498 | observations | procedure_occurrence | 2009-09-11 | 2009-09-11 | 99214 |
|
|
144
|
+
| 10 | 48752 | observations | procedure_occurrence | 2009-05-09 | 2009-05-09 | 99214 |
|
|
145
|
+
| 11 | 73107 | observations | procedure_occurrence | 2008-06-17 | 2008-06-17 | 99214 |
|
|
146
|
+
| 12 | 71635 | observations | procedure_occurrence | 2008-02-11 | 2008-02-11 | 99214 |
|
|
147
|
+
| 13 | 74458 | observations | procedure_occurrence | 2008-01-24 | 2008-01-24 | 99214 |
|
|
148
|
+
|
|
149
|
+
______________________________________________________________________
|
|
150
|
+
|
|
151
|
+
The diagram above reads "get all procedures that match CPT 99214 (office visit), and then keep only the first one (by date) for each person". More briefly: "get the first office visit for each patient". To read a diagram accurately, you need to know how each operator works. This document describes each operator.
|
|
152
|
+
|
|
153
|
+
Every diagram ends with a line that leads to nothing. The next section explains why.
|
|
154
|
+
|
|
155
|
+
### Think of Records as a Stream
|
|
156
|
+
|
|
157
|
+
A ConceptQL diagram has the "leaf" operators at the top and the "trunk" operator at the bottom. Think of the records of a ConceptQL statement as a stream of data. The leaf operators select records from the database, and they act as the sources of the stream. The records flow down through the diagram. Some operators combine streams, and others remove records, until one stream comes out at the bottom of the diagram.
|
|
158
|
+
|
|
159
|
+
The trailing line in each diagram is a reminder that every ConceptQL statement yields a stream of records.
|
|
160
|
+
|
|
161
|
+
### Streams Have Types
|
|
162
|
+
|
|
163
|
+
Each record has a type. The type is the `criterion_domain` column of the record. A vocabulary operator takes its type from the vocabulary's domain. For example, ICD-9-CM records are `condition_occurrence` records, and CPT records are `procedure_occurrence` records. Other operators produce a fixed type, such as `person` or `death`.
|
|
164
|
+
|
|
165
|
+
The diagrams use color to show types. The node of an operator and the lines that leave it have the color of the types that it outputs:
|
|
166
|
+
|
|
167
|
+
| Type | Color | Comes from, for example |
|
|
168
|
+
| -------------------- | ------ | ------------------------------------------------------------------------ |
|
|
169
|
+
| condition_occurrence | red | diagnosis codes (icd9cm, icd10cm, snomed), hospitalization, snf, hospice |
|
|
170
|
+
| death | brown | death |
|
|
171
|
+
| drug_exposure | purple | drug codes (ndc, rxnorm) |
|
|
172
|
+
| person | blue | person, gender, race, ethnicity, date_range |
|
|
173
|
+
| procedure_occurrence | green | procedure codes (cpt4, hcpcs, icd9proc, icd10pcs) |
|
|
174
|
+
| observation_period | teal | information_periods |
|
|
175
|
+
| observation | gray | lab codes (loinc) |
|
|
176
|
+
| episode | gray | episode |
|
|
177
|
+
|
|
178
|
+
An operator that outputs more than one type has a gray node, and the line that leaves it is drawn as one colored line for each type, side by side.
|
|
179
|
+
|
|
180
|
+
The type stays with each record as it moves through the operators. Some operators use the type when they compare records. For example, `filter` and `except` match a record only with a record from the same table. Other operators, such as `first` and `person_filter`, ignore the type. Each operator's description says which applies.
|
|
181
|
+
|
|
182
|
+
### Why Types?
|
|
183
|
+
|
|
184
|
+
ConceptQL was first written to query [OMOP's CDM v4.0](https://github.com/OHDSI/CommonDataModel/releases/tag/v4). In that model, each table held one type of data (condition_occurrence, procedure_occurrence, visit_occurrence, and so on), and ConceptQL treated each type as distinct.
|
|
185
|
+
|
|
186
|
+
ConceptQL now runs against the [GDM](https://github.com/outcomesinsights/generalized_data_model), where most codes of all types are in one table, `clinical_codes`. Types matter less now. ConceptQL keeps them to help users see which kinds of data a statement selects.
|
|
187
|
+
|
|
188
|
+
### What *are* Streams Really?
|
|
189
|
+
|
|
190
|
+
Thinking in streams is helpful, but sometimes we need to know what happens in the database.
|
|
191
|
+
|
|
192
|
+
A stream is a set of rows. Each row points back to one row of a GDM table, and it always has these columns:
|
|
193
|
+
|
|
194
|
+
- `person_id`: the patient.
|
|
195
|
+
- `criterion_id`: the ID of the row in its table of origin.
|
|
196
|
+
- `criterion_table`: the table of origin, such as `clinical_codes`, `patients` or `deaths`.
|
|
197
|
+
- `criterion_domain`: the type of the record (see above).
|
|
198
|
+
- `start_date` and `end_date`: the dates of the record.
|
|
199
|
+
- `source_value` and `source_vocabulary_id`: the code of the record and its vocabulary, when the record has a code.
|
|
200
|
+
- `label`: the label of the nearest upstream operator that has one (see [`label`](#label-option-and-its-features)).
|
|
201
|
+
|
|
202
|
+
Some operators add columns. For example, lab records (loinc) add `value_as_number` and unit columns, and the utilization operators add admission and discharge columns.
|
|
203
|
+
|
|
204
|
+
So this statement yields one row for each male patient, pointing at the `patients` table:
|
|
205
|
+
|
|
206
|
+
______________________________________________________________________
|
|
207
|
+
|
|
208
|
+
**Example 4 - All Male Patients**
|
|
209
|
+
|
|
210
|
+
```JSON
|
|
211
|
+
["gender","Male"]
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+

|
|
215
|
+
|
|
216
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
217
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
218
|
+
| 1 | 1 | patients | person | 1923-05-01 | 1923-05-01 | 00013D2EFD8E45D1 |
|
|
219
|
+
| 2 | 2 | patients | person | 1943-01-01 | 1943-01-01 | 00016F745862898F |
|
|
220
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1941-06-01 | 00021CA6FF03E670 |
|
|
221
|
+
| 5 | 5 | patients | person | 1936-08-01 | 1936-08-01 | 00024B3D2352D2D0 |
|
|
222
|
+
| 6 | 6 | patients | person | 1943-10-01 | 1943-10-01 | 0002DAE1C81CC70D |
|
|
223
|
+
| 7 | 7 | patients | person | 1922-07-01 | 1922-07-01 | 0002F28CE057345B |
|
|
224
|
+
| 8 | 8 | patients | person | 1935-09-01 | 1935-09-01 | 000308435E3E5B76 |
|
|
225
|
+
| 12 | 12 | patients | person | 1929-06-01 | 1929-06-01 | 00048EF1F4791C68 |
|
|
226
|
+
| 14 | 14 | patients | person | 1934-05-01 | 1934-05-01 | 00052705243EA128 |
|
|
227
|
+
| 16 | 16 | patients | person | 1934-01-01 | 1934-01-01 | 0007E57CC13CE880 |
|
|
228
|
+
|
|
229
|
+
______________________________________________________________________
|
|
230
|
+
|
|
231
|
+
This statement yields one row for each `clinical_codes` row that has the ICD-9-CM code 250.01:
|
|
232
|
+
|
|
233
|
+
______________________________________________________________________
|
|
234
|
+
|
|
235
|
+
**Example 5 - All Condition Occurrences that match ICD-9CM 250.01**
|
|
236
|
+
|
|
237
|
+
```JSON
|
|
238
|
+
["icd9","250.01"]
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+

|
|
242
|
+
|
|
243
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
244
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
245
|
+
| 2 | 55131 | observations | condition_occurrence | 2010-02-19 | 2010-02-19 | 25001 |
|
|
246
|
+
| 13 | 68892 | observations | condition_occurrence | 2010-09-14 | 2010-09-14 | 25001 |
|
|
247
|
+
| 14 | 28741 | observations | condition_occurrence | 2009-06-27 | 2009-06-27 | 25001 |
|
|
248
|
+
| 17 | 59056 | observations | condition_occurrence | 2009-11-25 | 2009-11-25 | 25001 |
|
|
249
|
+
| 17 | 30214 | observations | condition_occurrence | 2010-03-21 | 2010-03-21 | 25001 |
|
|
250
|
+
| 23 | 77713 | observations | condition_occurrence | 2010-01-30 | 2010-01-30 | 25001 |
|
|
251
|
+
| 25 | 84696 | observations | condition_occurrence | 2009-12-31 | 2009-12-31 | 25001 |
|
|
252
|
+
| 46 | 82760 | observations | condition_occurrence | 2010-12-26 | 2010-12-26 | 25001 |
|
|
253
|
+
| 56 | 37909 | observations | condition_occurrence | 2009-09-13 | 2009-09-13 | 25001 |
|
|
254
|
+
| 58 | 55348 | observations | condition_occurrence | 2009-01-25 | 2009-01-25 | 25001 |
|
|
255
|
+
|
|
256
|
+
______________________________________________________________________
|
|
257
|
+
|
|
258
|
+
It usually helps to think of these queries as a "stream of people" or a "stream of conditions", and not to think about tables and IDs.
|
|
259
|
+
|
|
260
|
+
A ConceptQL statement yields the rows that pass all of its criteria. What to do with those rows is up to the system that runs the statement. For example, a system that has all 250.01 records could:
|
|
261
|
+
|
|
262
|
+
- Find the first and last date of occurrence for each person
|
|
263
|
+
- Count the occurrences for each person
|
|
264
|
+
- Count the people with the condition
|
|
265
|
+
- Count the occurrences in the whole population
|
|
266
|
+
|
|
267
|
+
That kind of aggregation and analysis is outside the scope of ConceptQL. ConceptQL finds the rows that you are interested in. Other parts of the system decide what to do with them.
|
|
268
|
+
|
|
269
|
+
The rest of this document describes each operator. [Appendix E](#appendix-e---operator-quick-reference) lists every operator in one table.
|
|
270
|
+
|
|
271
|
+
### Statements That ConceptQL Refuses
|
|
272
|
+
|
|
273
|
+
ConceptQL checks each statement before it builds SQL for it. A statement with an error is invalid. Examples of errors are an unknown operator, a required option that is missing, and a `recall` of a label that the statement does not have. ConceptQL refuses to build SQL for an invalid statement, and so it does not run it. It reports each error with the name of the operator that has the error.
|
|
274
|
+
|
|
275
|
+
A warning does not make a statement invalid. Examples of warnings are an unknown code, an improperly formatted code and a deprecated operator. A statement with warnings runs. When ConceptQL draws the diagram of a statement, it shows the errors and the warnings of each operator, and it does not refuse an invalid statement.
|
|
276
|
+
|
|
277
|
+
ConceptQL finds the concept of each code in the vocabulary tables. It uses either the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`) or the GDM vocabulary tables (`concepts`, `ancestors`, `mappings` and `vocabularies`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no records, and a statement would return no records with no warning. Without a database, for example when ConceptQL only draws a diagram, it uses no vocabulary tables.
|
|
278
|
+
|
|
279
|
+
## Selection Operators
|
|
280
|
+
|
|
281
|
+
Selection operators select records from the database. For example, a search for diagnoses of old myocardial infarction (ICD-9-CM 412) is a selection. Most selection operators are leaf operators: no other operator flows into them. `person` and `death` can also take one upstream operator (see [`person`](#person-operator)).
|
|
282
|
+
|
|
283
|
+
The vocabulary operators select records by their codes. The next section describes them, and [Other Selection Operators](#other-selection-operators) describes the rest.
|
|
284
|
+
|
|
285
|
+
## `Vocabulary` Operator
|
|
286
|
+
|
|
287
|
+
Selects the records whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once.
|
|
288
|
+
|
|
289
|
+
- **Upstream:** none.
|
|
290
|
+
- **Arguments:**
|
|
291
|
+
- `codes` (codelist): One or more codes of the vocabulary. Write each code as the vocabulary writes it, including any dots: 250.01, not 25001. A code matches exactly, including case, and it does not select its child codes. A code alone of * selects every record of the vocabulary, and there is no other wildcard. A code that does not have the vocabulary's format matches nothing and gives a warning.
|
|
292
|
+
- **Returns:** Every record whose code is one of the given codes in the operator's vocabulary. The type of the records depends on the vocabulary, for example condition_occurrence for icd9cm and procedure_occurrence for cpt4. An operator does not follow mappings to other vocabularies.
|
|
293
|
+
|
|
294
|
+
______________________________________________________________________
|
|
295
|
+
|
|
296
|
+
**Example 6 - All records with the ICD-9-CM diagnosis code 412 (old myocardial infarction)**
|
|
297
|
+
|
|
298
|
+
```JSON
|
|
299
|
+
["icd9cm","412"]
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+

|
|
303
|
+
|
|
304
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
305
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
306
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
307
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
308
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
309
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
310
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
311
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
312
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
313
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
314
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
315
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
316
|
+
|
|
317
|
+
______________________________________________________________________
|
|
318
|
+
|
|
319
|
+
______________________________________________________________________
|
|
320
|
+
|
|
321
|
+
**Example 7 - All MS-DRG 205 records: other respiratory system diagnoses with a major complication or comorbidity**
|
|
322
|
+
|
|
323
|
+
```JSON
|
|
324
|
+
["ms_drg","205"]
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+

|
|
328
|
+
|
|
329
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
330
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
331
|
+
| 17 | 1365 | observations | procedure_occurrence | 2010-06-02 | 2010-06-06 | 205 |
|
|
332
|
+
| 68 | 18 | observations | procedure_occurrence | 2008-04-19 | 2008-04-23 | 205 |
|
|
333
|
+
| 95 | 603 | observations | procedure_occurrence | 2008-08-27 | 2008-09-01 | 205 |
|
|
334
|
+
|
|
335
|
+
______________________________________________________________________
|
|
336
|
+
|
|
337
|
+
______________________________________________________________________
|
|
338
|
+
|
|
339
|
+
**Example 8 - All CMS-DRG 205 records (a liver disorder)**
|
|
340
|
+
|
|
341
|
+
```JSON
|
|
342
|
+
["cms_drg","205"]
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+

|
|
346
|
+
|
|
347
|
+
No results.
|
|
348
|
+
|
|
349
|
+
______________________________________________________________________
|
|
350
|
+
|
|
351
|
+
ConceptQL creates one vocabulary operator for each vocabulary that it knows, so the list of operators depends on the vocabulary tables. The operator name is the vocabulary ID in lower case, with spaces and punctuation replaced by `_`. Some operators also have a shorter alias. [All Vocabulary Operators](#all-vocabulary-operators) lists every vocabulary operator.
|
|
352
|
+
|
|
353
|
+
These are the most common vocabulary operators:
|
|
354
|
+
|
|
355
|
+
| Operator | Alias | Vocabulary | Stream type |
|
|
356
|
+
| ------------ | -------------- | -------------------------------------------- | -------------------- |
|
|
357
|
+
| cpt4 | cpt | CPT-4 procedure codes | procedure_occurrence |
|
|
358
|
+
| hcpcs | | HCPCS procedure codes | procedure_occurrence |
|
|
359
|
+
| cpt_or_hcpcs | | CPT-4 and HCPCS together | procedure_occurrence |
|
|
360
|
+
| icd9cm | icd9 | ICD-9-CM diagnosis codes | condition_occurrence |
|
|
361
|
+
| icd9proc | icd9_procedure | ICD-9-CM procedure codes (volume 3) | procedure_occurrence |
|
|
362
|
+
| icd10cm | | ICD-10-CM diagnosis codes | condition_occurrence |
|
|
363
|
+
| icd10pcs | icd10_pcs | ICD-10-PCS procedure codes | procedure_occurrence |
|
|
364
|
+
| ms_drg | | MS-DRG codes, for events from 2007-10-01 | procedure_occurrence |
|
|
365
|
+
| cms_drg | | CMS-DRG codes, for events before 2007-10-01 | procedure_occurrence |
|
|
366
|
+
| revenue_code | | UB-04 revenue codes | procedure_occurrence |
|
|
367
|
+
| loinc | | LOINC lab codes | observation |
|
|
368
|
+
| ndc | | National Drug Codes (11-digit package codes) | drug_exposure |
|
|
369
|
+
| rxnorm | | RxNorm codes | drug_exposure |
|
|
370
|
+
| snomed | | SNOMED CT codes | condition_occurrence |
|
|
371
|
+
|
|
372
|
+
Examples of how codes match:
|
|
373
|
+
|
|
374
|
+
- `V58.61` matches, but `v58.61` does not.
|
|
375
|
+
- ConceptQL does not expand a code to its child codes, so `584` does not match `584.9`.
|
|
376
|
+
- `410*` matches nothing, because `*` is a wildcard only as a code alone.
|
|
377
|
+
- A code that does not have the vocabulary's format gives the warning "improperly formatted code".
|
|
378
|
+
- A code that the vocabulary does not contain gives the warning "unknown code(s)".
|
|
379
|
+
- A code that has more than one meaning in the vocabulary gives the warning "code(s) with more than one meaning".
|
|
380
|
+
- Some data stores CPT-4 and HCPCS codes only in the merged vocabulary CPT4_HCPCS. On that data, `cpt` and `hcpcs` select nothing, because they read only CPT4 and HCPCS. They give the warning "the data has no records of this vocabulary", which names `cpt_or_hcpcs`. Use `cpt_or_hcpcs`, which reads all three vocabularies.
|
|
381
|
+
- `rxnorm` does not find records coded with NDC, because an operator does not follow mappings between vocabularies.
|
|
382
|
+
|
|
383
|
+
### DRG Codes
|
|
384
|
+
|
|
385
|
+
CMS renumbered the Diagnosis-Related Group (DRG) codes on 2007-10-01, when the MS-DRG system replaced the CMS-DRG system. Most codes got a new meaning. For example, DRG 205 meant a liver disorder in CMS-DRG, and it means other respiratory system diagnoses in MS-DRG. One vocabulary, DRG, holds the codes of both systems, and a record does not say which meaning applied. So a statement names the DRG system, and ConceptQL selects each record by its code and by its start_date:
|
|
386
|
+
|
|
387
|
+
- `ms_drg` selects the DRG records whose start_date is on or after 2007-10-01.
|
|
388
|
+
- `cms_drg` selects the DRG records whose start_date is before 2007-10-01.
|
|
389
|
+
|
|
390
|
+
The examples above show both operators. Our test data has no DRG record before 2007-10-01, so the `cms_drg` example returns no records.
|
|
391
|
+
|
|
392
|
+
Both operators check codes as the other vocabulary operators do. A DRG code has 3 digits, with leading zeros. A code that the DRG vocabulary does not have gets the warning "unknown code(s)". These operators never give the warning "code(s) with more than one meaning", because the operator chooses the meaning.
|
|
393
|
+
|
|
394
|
+
The `drg` operator is deprecated. It selects the DRG records of both systems, whatever their dates. A statement that uses `drg` still runs, but `drg` gets the warning "deprecated operator", and a later release will make it an error. Use `ms_drg` or `cms_drg`.
|
|
395
|
+
|
|
396
|
+
### All Vocabulary Operators
|
|
397
|
+
|
|
398
|
+
The build writes this table from the metadata of the operators. It lists the vocabulary operators of a ConceptQL database with no vocabulary tables, ordered by name. Each row gives the full name of the operator's vocabulary and the domain, which is the type of the records that the operator returns. The vocabulary tables of a database can add more operators. A deprecated operator still runs, with a warning.
|
|
399
|
+
|
|
400
|
+
| Operator | Vocabulary | Domain |
|
|
401
|
+
| ------------------------ | ------------------------------------------------------------------------------------------------------ | -------------------- |
|
|
402
|
+
| `admsrce` | ADMSRCE | condition_occurrence |
|
|
403
|
+
| `admtype` | ADMTYPE | condition_occurrence |
|
|
404
|
+
| `atc` | WHO Anatomic Therapeutic Chemical Classification | drug_exposure |
|
|
405
|
+
| `betos` | BETOS | condition_occurrence |
|
|
406
|
+
| `clinical_code_type` | Clinical Code Type | condition_occurrence |
|
|
407
|
+
| `cms_drg` | CMS Diagnosis-Related Group, events before 2007-10-01 | procedure_occurrence |
|
|
408
|
+
| `concept_class` | OMOP Concept Class | condition_occurrence |
|
|
409
|
+
| `condition_type` | OMOP Condition Occurrence Type | condition_occurrence |
|
|
410
|
+
| `cpt4` | Current Procedural Terminology version 4 (AMA) | procedure_occurrence |
|
|
411
|
+
| `cpt_or_hcpcs` | CPT4, HCPCS and CPT4_HCPCS | procedure_occurrence |
|
|
412
|
+
| `cs_schema` | CS_Schema | condition_occurrence |
|
|
413
|
+
| `currency` | International Currency Symbol (ISO 4217) | condition_occurrence |
|
|
414
|
+
| `death_type` | OMOP Death Type | death |
|
|
415
|
+
| `device_type` | OMOP Device Type | observation |
|
|
416
|
+
| `domain` | OMOP Domain | condition_occurrence |
|
|
417
|
+
| `drg` (deprecated) | Diagnosis-related group (CMS) | procedure_occurrence |
|
|
418
|
+
| `drug_type` | OMOP Drug Exposure Type | drug_exposure |
|
|
419
|
+
| `drugtextstring` | Drug Text String | drug_exposure |
|
|
420
|
+
| `duration_unit` | DURATION_UNIT | condition_occurrence |
|
|
421
|
+
| `gdm_cen_tract_meas` | Census Tract Measures from SEER | observation |
|
|
422
|
+
| `gemscript` | Gemscript | drug_exposure |
|
|
423
|
+
| `hcfaspec` | HCFASPEC | condition_occurrence |
|
|
424
|
+
| `hcfatype` | HCFATYPE | condition_occurrence |
|
|
425
|
+
| `hcpcs` | Healthcare Common Procedure Coding System (CMS) | procedure_occurrence |
|
|
426
|
+
| `icd03_morphology` | ICD03_Morphology | condition_occurrence |
|
|
427
|
+
| `icd03_topography` | ICD03_Topography | condition_occurrence |
|
|
428
|
+
| `icd10cm` | International Classification of Diseases, Tenth Revision, Clinical Modification (NCHS) | condition_occurrence |
|
|
429
|
+
| `icd10pcs` | ICD-10 Procedure Coding System (CMS) | procedure_occurrence |
|
|
430
|
+
| `icd10who` | International Classification of Diseases, Tenth Revision (WHO) | condition_occurrence |
|
|
431
|
+
| `icd9cm` | International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 1 and 2 (NCHS) | condition_occurrence |
|
|
432
|
+
| `icd9proc` | International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 3 (NCHS) | procedure_occurrence |
|
|
433
|
+
| `loinc` | Logical Observation Identifiers Names and Codes (Regenstrief Institute) | observation |
|
|
434
|
+
| `marcla_obs_type` | Market Clarity Observation Type | observation |
|
|
435
|
+
| `meas_type` | OMOP Measurement Type | observation |
|
|
436
|
+
| `ms_drg` | Medicare Severity Diagnosis-Related Group (CMS), events from 2007-10-01 | procedure_occurrence |
|
|
437
|
+
| `ndc` | National Drug Code (FDA and manufacturers) | drug_exposure |
|
|
438
|
+
| `note_type` | OMOP Note Type | condition_occurrence |
|
|
439
|
+
| `nucc` | National Uniform Claim Committee Health Care Provider Taxonomy Code Set (NUCC) | condition_occurrence |
|
|
440
|
+
| `obs_period_type` | OMOP Observation Period Type | observation_period |
|
|
441
|
+
| `observation_type` | OMOP Observation Type | observation |
|
|
442
|
+
| `oi_file_type` | OI_FILE_TYPE | condition_occurrence |
|
|
443
|
+
| `oi_information_type` | OI_INFORMATION_TYPE | condition_occurrence |
|
|
444
|
+
| `oi_prov` | OI_PROV | condition_occurrence |
|
|
445
|
+
| `plcsrvc` | PLCSRVC | condition_occurrence |
|
|
446
|
+
| `procedure_type` | OMOP Procedure Occurrence Type | procedure_occurrence |
|
|
447
|
+
| `provider_facility_type` | PROVIDER_FACILITY_TYPE | condition_occurrence |
|
|
448
|
+
| `read` | NHS UK Read Codes Version 2 (HSCIC) | condition_occurrence |
|
|
449
|
+
| `relationship` | OMOP Relationship | condition_occurrence |
|
|
450
|
+
| `revenue_code` | UB04/CMS1450 Revenue Codes (CMS) | procedure_occurrence |
|
|
451
|
+
| `rxnorm` | RxNorm (NLM) | drug_exposure |
|
|
452
|
+
| `seer_adjajc6m` | SEER_ADJAJC6M | condition_occurrence |
|
|
453
|
+
| `seer_adjajc6n` | SEER_ADJAJC6N | condition_occurrence |
|
|
454
|
+
| `seer_adjajc6t` | SEER_ADJAJC6T | condition_occurrence |
|
|
455
|
+
| `seer_beh` | SEER_BEH | condition_occurrence |
|
|
456
|
+
| `seer_census_pov_ind` | SEER_CENSUS_POV_IND | condition_occurrence |
|
|
457
|
+
| `seer_dajcc7` | SEER_DAJCC7 | condition_occurrence |
|
|
458
|
+
| `seer_dajccstg` | SEER_DAJCCSTG | condition_occurrence |
|
|
459
|
+
| `seer_dxconf` | SEER_DXCONF | condition_occurrence |
|
|
460
|
+
| `seer_grade` | SEER_GRADE | condition_occurrence |
|
|
461
|
+
| `seer_lat` | SEER_LAT | condition_occurrence |
|
|
462
|
+
| `seer_marst` | SEER_MARST | condition_occurrence |
|
|
463
|
+
| `seer_msex` | SEER_MSEX | condition_occurrence |
|
|
464
|
+
| `seer_nhiade` | SEER_NHIADE | condition_occurrence |
|
|
465
|
+
| `seer_nosrg` | SEER_NOSRG | condition_occurrence |
|
|
466
|
+
| `seer_rad` | SEER_RAD | condition_occurrence |
|
|
467
|
+
| `seer_radsurg` | SEER_RADSURG | condition_occurrence |
|
|
468
|
+
| `seer_seq` | SEER_SEQ | condition_occurrence |
|
|
469
|
+
| `seer_siterwho` | SEER_SITERWHO | condition_occurrence |
|
|
470
|
+
| `seer_srace` | SEER_SRACE | condition_occurrence |
|
|
471
|
+
| `seer_src` | SEER_SRC | condition_occurrence |
|
|
472
|
+
| `seer_sxprif` | SEER_SXPRIF | condition_occurrence |
|
|
473
|
+
| `seer_sxsitf` | SEER_SXSITF | condition_occurrence |
|
|
474
|
+
| `seer_typefu` | SEER_TYPEFU | condition_occurrence |
|
|
475
|
+
| `seer_urbrur` | SEER_URBRUR | condition_occurrence |
|
|
476
|
+
| `seer_yost_acs_06_10` | SEER's Yost Index calculated from ACS 2006-2010 | observation |
|
|
477
|
+
| `seer_yost_acs_10_14` | SEER's Yost Index calculated from ACS 2010-2014 | observation |
|
|
478
|
+
| `seer_yost_acs_13_17` | SEER's Yost Index calculated from ACS 2013-2017 | observation |
|
|
479
|
+
| `snomed` | SNOMED-CT | condition_occurrence |
|
|
480
|
+
| `stus_cd` | STUS_CD | condition_occurrence |
|
|
481
|
+
| `typesrvc` | TYPESRVC | condition_occurrence |
|
|
482
|
+
| `ucum` | Unified Code for Units of Measure (Regenstrief Institute) | condition_occurrence |
|
|
483
|
+
| `visit` | OMOP Visit | visit_occurrence |
|
|
484
|
+
| `visit_type` | OMOP Visit Type | visit_occurrence |
|
|
485
|
+
| `vocabulary` | OMOP Vocabulary | condition_occurrence |
|
|
486
|
+
|
|
487
|
+
## Other Selection Operators
|
|
488
|
+
|
|
489
|
+
This section describes the selection operators that are not vocabulary operators.
|
|
490
|
+
|
|
491
|
+
### Person-related Operators
|
|
492
|
+
|
|
493
|
+
Person operators select records from the `patients` table, which holds the patient demographics. The type of these records is `person`. The start_date and end_date of a person record are both the patient's birth date. [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) explains why that is useful.
|
|
494
|
+
|
|
495
|
+
#### `Gender` Operator
|
|
496
|
+
|
|
497
|
+
Selects the patients with the given genders.
|
|
498
|
+
|
|
499
|
+
- **Upstream:** none.
|
|
500
|
+
- **Arguments:**
|
|
501
|
+
- `gender` (string, one of `Male`, `Female`, `Unknown`): One or more genders. A value that starts with M selects the male patients, and a value that starts with F selects the female patients. Any other value, such as Unknown, selects the patients whose gender is neither male nor female, including the patients with no gender. Case does not matter.
|
|
502
|
+
- **Returns:** One person record for each patient whose gender is one of the given genders. The start_date and the end_date of each record are the patient's birth date.
|
|
503
|
+
|
|
504
|
+
______________________________________________________________________
|
|
505
|
+
|
|
506
|
+
**Example 9 - All female patients**
|
|
507
|
+
|
|
508
|
+
```JSON
|
|
509
|
+
["gender","Female"]
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+

|
|
513
|
+
|
|
514
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
515
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
516
|
+
| 3 | 3 | patients | person | 1936-09-01 | 1936-09-01 | 0001FDD721E223DC |
|
|
517
|
+
| 9 | 9 | patients | person | 1976-09-01 | 1976-09-01 | 000345A39D4157C9 |
|
|
518
|
+
| 10 | 10 | patients | person | 1938-10-01 | 1938-10-01 | 00036A21B65B0206 |
|
|
519
|
+
| 11 | 11 | patients | person | 1934-02-01 | 1934-02-01 | 000489E7EAAD463F |
|
|
520
|
+
| 13 | 13 | patients | person | 1936-07-01 | 1936-07-01 | 0004F0ABD505251D |
|
|
521
|
+
| 15 | 15 | patients | person | 1936-03-01 | 1936-03-01 | 00070B63745BE497 |
|
|
522
|
+
| 17 | 17 | patients | person | 1919-09-01 | 1919-09-01 | 0007F12A492FD25D |
|
|
523
|
+
| 18 | 18 | patients | person | 1919-10-01 | 1919-10-01 | 000A005BA0BED3EA |
|
|
524
|
+
| 19 | 19 | patients | person | 1942-07-01 | 1942-07-01 | 000B4662348C35B4 |
|
|
525
|
+
| 26 | 26 | patients | person | 1939-12-01 | 1939-12-01 | 0010D6F80D245D62 |
|
|
526
|
+
|
|
527
|
+
______________________________________________________________________
|
|
528
|
+
|
|
529
|
+
______________________________________________________________________
|
|
530
|
+
|
|
531
|
+
**Example 10 - All male patients**
|
|
532
|
+
|
|
533
|
+
```JSON
|
|
534
|
+
["gender","Male"]
|
|
535
|
+
```
|
|
536
|
+
|
|
537
|
+

|
|
538
|
+
|
|
539
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
540
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
541
|
+
| 1 | 1 | patients | person | 1923-05-01 | 1923-05-01 | 00013D2EFD8E45D1 |
|
|
542
|
+
| 2 | 2 | patients | person | 1943-01-01 | 1943-01-01 | 00016F745862898F |
|
|
543
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1941-06-01 | 00021CA6FF03E670 |
|
|
544
|
+
| 5 | 5 | patients | person | 1936-08-01 | 1936-08-01 | 00024B3D2352D2D0 |
|
|
545
|
+
| 6 | 6 | patients | person | 1943-10-01 | 1943-10-01 | 0002DAE1C81CC70D |
|
|
546
|
+
| 7 | 7 | patients | person | 1922-07-01 | 1922-07-01 | 0002F28CE057345B |
|
|
547
|
+
| 8 | 8 | patients | person | 1935-09-01 | 1935-09-01 | 000308435E3E5B76 |
|
|
548
|
+
| 12 | 12 | patients | person | 1929-06-01 | 1929-06-01 | 00048EF1F4791C68 |
|
|
549
|
+
| 14 | 14 | patients | person | 1934-05-01 | 1934-05-01 | 00052705243EA128 |
|
|
550
|
+
| 16 | 16 | patients | person | 1934-01-01 | 1934-01-01 | 0007E57CC13CE880 |
|
|
551
|
+
|
|
552
|
+
______________________________________________________________________
|
|
553
|
+
|
|
554
|
+
#### `Race` Operator
|
|
555
|
+
|
|
556
|
+
Selects the patients with the given races.
|
|
557
|
+
|
|
558
|
+
- **Upstream:** none.
|
|
559
|
+
- **Arguments:**
|
|
560
|
+
- `races` (codelist): One or more race names, such as White, or race concept IDs. A name matches the concepts with that name in any vocabulary, and case does not matter. The operator also selects the more detailed races under each given race, and each concept that is a kind of one of the race concepts through an is-a relationship. A value that contains Unknown also selects the patients who have no race or race concept ID 0.
|
|
561
|
+
- **Returns:** One person record for each patient whose race is one of the given races. The start_date and the end_date of each record are the patient's birth date.
|
|
562
|
+
|
|
563
|
+
______________________________________________________________________
|
|
564
|
+
|
|
565
|
+
**Example 11 - All patients recorded as white**
|
|
566
|
+
|
|
567
|
+
```JSON
|
|
568
|
+
["race","White"]
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+

|
|
572
|
+
|
|
573
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
574
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
575
|
+
| 1 | 1 | patients | person | 1923-05-01 | 1923-05-01 | 00013D2EFD8E45D1 |
|
|
576
|
+
| 2 | 2 | patients | person | 1943-01-01 | 1943-01-01 | 00016F745862898F |
|
|
577
|
+
| 3 | 3 | patients | person | 1936-09-01 | 1936-09-01 | 0001FDD721E223DC |
|
|
578
|
+
| 5 | 5 | patients | person | 1936-08-01 | 1936-08-01 | 00024B3D2352D2D0 |
|
|
579
|
+
| 7 | 7 | patients | person | 1922-07-01 | 1922-07-01 | 0002F28CE057345B |
|
|
580
|
+
| 8 | 8 | patients | person | 1935-09-01 | 1935-09-01 | 000308435E3E5B76 |
|
|
581
|
+
| 9 | 9 | patients | person | 1976-09-01 | 1976-09-01 | 000345A39D4157C9 |
|
|
582
|
+
| 11 | 11 | patients | person | 1934-02-01 | 1934-02-01 | 000489E7EAAD463F |
|
|
583
|
+
| 12 | 12 | patients | person | 1929-06-01 | 1929-06-01 | 00048EF1F4791C68 |
|
|
584
|
+
| 13 | 13 | patients | person | 1936-07-01 | 1936-07-01 | 0004F0ABD505251D |
|
|
585
|
+
|
|
586
|
+
______________________________________________________________________
|
|
587
|
+
|
|
588
|
+
#### `Ethnicity` Operator
|
|
589
|
+
|
|
590
|
+
Selects the patients with the given ethnicities.
|
|
591
|
+
|
|
592
|
+
- **Upstream:** none.
|
|
593
|
+
- **Arguments:**
|
|
594
|
+
- `ethnicity` (string, one of `Hispanic or Latino`, `Not Hispanic or Latino`): One or more ethnicities. A value that starts with Hispanic selects the Hispanic or Latino patients, and a value that starts with Not Hispanic selects the patients who are not Hispanic or Latino. A value that contains Unknown selects the patients who have no ethnicity or ethnicity concept ID 0. Case does not matter, and any other value selects no patients.
|
|
595
|
+
- **Returns:** One person record for each patient whose ethnicity is one of the given ethnicities. The start_date and the end_date of each record are the patient's birth date.
|
|
596
|
+
|
|
597
|
+
______________________________________________________________________
|
|
598
|
+
|
|
599
|
+
**Example 12 - All patients recorded as Hispanic**
|
|
600
|
+
|
|
601
|
+
```JSON
|
|
602
|
+
["ethnicity","Hispanic"]
|
|
603
|
+
```
|
|
604
|
+
|
|
605
|
+

|
|
606
|
+
|
|
607
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
608
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
609
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1941-06-01 | 00021CA6FF03E670 |
|
|
610
|
+
| 22 | 22 | patients | person | 1942-08-01 | 1942-08-01 | 000D6D88463D8A76 |
|
|
611
|
+
| 72 | 72 | patients | person | 1933-04-01 | 1933-04-01 | 002354398A00234E |
|
|
612
|
+
| 76 | 76 | patients | person | 1938-03-01 | 1938-03-01 | 00242FF5E7A3F2D9 |
|
|
613
|
+
| 181 | 181 | patients | person | 1939-11-01 | 1939-11-01 | 005E0AB5172E715F |
|
|
614
|
+
|
|
615
|
+
______________________________________________________________________
|
|
616
|
+
|
|
617
|
+
#### `Person` Operator
|
|
618
|
+
|
|
619
|
+
Selects every patient. With one upstream operator, it selects the patients who have a record in the upstream stream.
|
|
620
|
+
|
|
621
|
+
- **Upstream:** none, or one operator.
|
|
622
|
+
- **Returns:** One record for each selected patient, from the patients table. The type of the records is person, and their start_date and end_date are both the patient's birth date.
|
|
623
|
+
|
|
624
|
+
______________________________________________________________________
|
|
625
|
+
|
|
626
|
+
**Example 13 - All patients**
|
|
627
|
+
|
|
628
|
+
```JSON
|
|
629
|
+
["person"]
|
|
630
|
+
```
|
|
631
|
+
|
|
632
|
+

|
|
633
|
+
|
|
634
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
635
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
636
|
+
| 1 | 1 | patients | person | 1923-05-01 | 1923-05-01 | 00013D2EFD8E45D1 |
|
|
637
|
+
| 2 | 2 | patients | person | 1943-01-01 | 1943-01-01 | 00016F745862898F |
|
|
638
|
+
| 3 | 3 | patients | person | 1936-09-01 | 1936-09-01 | 0001FDD721E223DC |
|
|
639
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1941-06-01 | 00021CA6FF03E670 |
|
|
640
|
+
| 5 | 5 | patients | person | 1936-08-01 | 1936-08-01 | 00024B3D2352D2D0 |
|
|
641
|
+
| 6 | 6 | patients | person | 1943-10-01 | 1943-10-01 | 0002DAE1C81CC70D |
|
|
642
|
+
| 7 | 7 | patients | person | 1922-07-01 | 1922-07-01 | 0002F28CE057345B |
|
|
643
|
+
| 8 | 8 | patients | person | 1935-09-01 | 1935-09-01 | 000308435E3E5B76 |
|
|
644
|
+
| 9 | 9 | patients | person | 1976-09-01 | 1976-09-01 | 000345A39D4157C9 |
|
|
645
|
+
| 10 | 10 | patients | person | 1938-10-01 | 1938-10-01 | 00036A21B65B0206 |
|
|
646
|
+
|
|
647
|
+
______________________________________________________________________
|
|
648
|
+
|
|
649
|
+
______________________________________________________________________
|
|
650
|
+
|
|
651
|
+
**Example 14 - The patients with an old myocardial infarction (ICD-9-CM 412)**
|
|
652
|
+
|
|
653
|
+
```JSON
|
|
654
|
+
["person",["icd9cm","412"]]
|
|
655
|
+
```
|
|
656
|
+
|
|
657
|
+

|
|
658
|
+
|
|
659
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
660
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
661
|
+
| 11 | 11 | patients | person | 1934-02-01 | 1934-02-01 | 000489E7EAAD463F |
|
|
662
|
+
| 17 | 17 | patients | person | 1919-09-01 | 1919-09-01 | 0007F12A492FD25D |
|
|
663
|
+
| 38 | 38 | patients | person | 1922-12-01 | 1922-12-01 | 001731EB127233DA |
|
|
664
|
+
| 54 | 54 | patients | person | 1931-02-01 | 1931-02-01 | 001CAFF084B21E14 |
|
|
665
|
+
| 60 | 60 | patients | person | 1925-07-01 | 1925-07-01 | 001EA2F4DB30F105 |
|
|
666
|
+
| 66 | 66 | patients | person | 1920-04-01 | 1920-04-01 | 0021B3C854C968C8 |
|
|
667
|
+
| 73 | 73 | patients | person | 1925-09-01 | 1925-09-01 | 00237322613CFC3C |
|
|
668
|
+
| 77 | 77 | patients | person | 1929-03-01 | 1929-03-01 | 00244B6D9AB50F9B |
|
|
669
|
+
| 81 | 81 | patients | person | 1940-06-01 | 1940-06-01 | 0024E5A7B7272E75 |
|
|
670
|
+
| 88 | 88 | patients | person | 1922-05-01 | 1922-05-01 | 00291F39917544B1 |
|
|
671
|
+
|
|
672
|
+
______________________________________________________________________
|
|
673
|
+
|
|
674
|
+
### Additional Selection Operators
|
|
675
|
+
|
|
676
|
+
#### `Death` Operator
|
|
677
|
+
|
|
678
|
+
Selects every death record. With one upstream operator, it selects the deaths of the patients who have a record in the upstream stream.
|
|
679
|
+
|
|
680
|
+
- **Upstream:** none, or one operator.
|
|
681
|
+
- **Returns:** One record for each selected death, from the deaths table. The type of the records is death, and their start_date and end_date are both the date of death.
|
|
682
|
+
|
|
683
|
+
______________________________________________________________________
|
|
684
|
+
|
|
685
|
+
**Example 15 - All death records**
|
|
686
|
+
|
|
687
|
+
```JSON
|
|
688
|
+
["death"]
|
|
689
|
+
```
|
|
690
|
+
|
|
691
|
+

|
|
692
|
+
|
|
693
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
694
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
|
|
695
|
+
| 16 | 1 | deaths | death | 2010-12-01 | 2010-12-01 | |
|
|
696
|
+
| 26 | 2 | deaths | death | 2009-07-01 | 2009-07-01 | |
|
|
697
|
+
| 65 | 3 | deaths | death | 2009-01-01 | 2009-01-01 | |
|
|
698
|
+
| 119 | 4 | deaths | death | 2008-09-01 | 2008-09-01 | |
|
|
699
|
+
| 166 | 5 | deaths | death | 2008-07-01 | 2008-07-01 | |
|
|
700
|
+
| 181 | 6 | deaths | death | 2010-07-01 | 2010-07-01 | |
|
|
701
|
+
| 190 | 7 | deaths | death | 2009-09-01 | 2009-09-01 | |
|
|
702
|
+
| 191 | 8 | deaths | death | 2009-09-01 | 2009-09-01 | |
|
|
703
|
+
| 201 | 9 | deaths | death | 2008-05-01 | 2008-05-01 | |
|
|
704
|
+
| 215 | 10 | deaths | death | 2008-01-01 | 2008-01-01 | |
|
|
705
|
+
|
|
706
|
+
______________________________________________________________________
|
|
707
|
+
|
|
708
|
+
______________________________________________________________________
|
|
709
|
+
|
|
710
|
+
**Example 16 - The deaths of the patients with an old myocardial infarction (ICD-9-CM 412)**
|
|
711
|
+
|
|
712
|
+
```JSON
|
|
713
|
+
["death",["icd9cm","412"]]
|
|
714
|
+
```
|
|
715
|
+
|
|
716
|
+

|
|
717
|
+
|
|
718
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
719
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
|
|
720
|
+
| 190 | 7 | deaths | death | 2009-09-01 | 2009-09-01 | |
|
|
721
|
+
|
|
722
|
+
______________________________________________________________________
|
|
723
|
+
|
|
724
|
+
#### `Information Periods` Operator
|
|
725
|
+
|
|
726
|
+
Selects information periods: the spans of time in which the data has information about a patient, such as periods of insurance enrollment.
|
|
727
|
+
|
|
728
|
+
- **Upstream:** none.
|
|
729
|
+
- **Arguments:**
|
|
730
|
+
- `information_types` (codelist): Zero or more information types, each as a concept ID (an integer) or a concept code of the GDM_INFORMATION_TYPE vocabulary, such as PARTD (Medicare Part D). A statement may mix both. With no information types, the operator selects every information period. With information types, it selects only the information periods whose type is one of them. An unknown code gets a warning and selects no period.
|
|
731
|
+
- **Returns:** One record for each selected information period, from the information_periods table. The type of the records is observation_period.
|
|
732
|
+
|
|
733
|
+
______________________________________________________________________
|
|
734
|
+
|
|
735
|
+
**Example 17 - All information periods**
|
|
736
|
+
|
|
737
|
+
```JSON
|
|
738
|
+
["information_periods"]
|
|
739
|
+
```
|
|
740
|
+
|
|
741
|
+

|
|
742
|
+
|
|
743
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
744
|
+
| --------- | ------------ | ------------------- | ------------------ | ---------- | ---------- | ------------ |
|
|
745
|
+
| 1 | 1 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
746
|
+
| 1 | 2 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
747
|
+
| 1 | 3 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
748
|
+
| 1 | 4 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
749
|
+
| 2 | 5 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
750
|
+
| 2 | 6 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
751
|
+
| 2 | 793 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
752
|
+
| 2 | 7 | information_periods | observation_period | 2009-01-01 | 2010-12-31 | |
|
|
753
|
+
| 3 | 8 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
754
|
+
| 3 | 9 | information_periods | observation_period | 2008-01-01 | 2010-12-31 | |
|
|
755
|
+
|
|
756
|
+
______________________________________________________________________
|
|
757
|
+
|
|
758
|
+
### Utilization Operators
|
|
759
|
+
|
|
760
|
+
Utilization operators select admissions. Each operator returns one record for each admission whose claim came from the matching kind of file. For example, `hospitalization` uses the claims whose file provenance is inpatient.
|
|
761
|
+
|
|
762
|
+
The type of these records is `condition_occurrence`, because the GDM has no visit or admission type. So `union` combines them with diagnosis records into one stream. `source_value` usually holds a diagnosis.
|
|
763
|
+
|
|
764
|
+
Our test data has no SNF claims and no hospice claims, so the examples of `snf` and `hospice` return no records.
|
|
765
|
+
|
|
766
|
+
#### `SNF` Operator
|
|
767
|
+
|
|
768
|
+
Selects admissions to skilled nursing facilities (SNF): one record for each admission whose claim came from an SNF file.
|
|
769
|
+
|
|
770
|
+
- **Upstream:** none.
|
|
771
|
+
- **Returns:** One record for each SNF admission. The type of the records is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each record also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
|
|
772
|
+
|
|
773
|
+
______________________________________________________________________
|
|
774
|
+
|
|
775
|
+
**Example 18 - All SNF admissions**
|
|
776
|
+
|
|
777
|
+
```JSON
|
|
778
|
+
["snf"]
|
|
779
|
+
```
|
|
780
|
+
|
|
781
|
+

|
|
782
|
+
|
|
783
|
+
No results.
|
|
784
|
+
|
|
785
|
+
______________________________________________________________________
|
|
786
|
+
|
|
787
|
+
#### `Hospice` Operator
|
|
788
|
+
|
|
789
|
+
Selects hospice admissions: one record for each admission whose claim came from a hospice file.
|
|
790
|
+
|
|
791
|
+
- **Upstream:** none.
|
|
792
|
+
- **Returns:** One record for each hospice admission. The type of the records is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each record also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
|
|
793
|
+
|
|
794
|
+
______________________________________________________________________
|
|
795
|
+
|
|
796
|
+
**Example 19 - All hospice admissions**
|
|
797
|
+
|
|
798
|
+
```JSON
|
|
799
|
+
["hospice"]
|
|
800
|
+
```
|
|
801
|
+
|
|
802
|
+

|
|
803
|
+
|
|
804
|
+
No results.
|
|
805
|
+
|
|
806
|
+
______________________________________________________________________
|
|
807
|
+
|
|
808
|
+
#### `Hospitalization` Operator
|
|
809
|
+
|
|
810
|
+
Selects inpatient hospital admissions: one record for each admission whose claim came from an inpatient file.
|
|
811
|
+
|
|
812
|
+
- **Upstream:** none.
|
|
813
|
+
- **Returns:** One record for each inpatient admission. The type of the records is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each record also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
|
|
814
|
+
|
|
815
|
+
______________________________________________________________________
|
|
816
|
+
|
|
817
|
+
**Example 20 - All inpatient hospital admissions**
|
|
818
|
+
|
|
819
|
+
```JSON
|
|
820
|
+
["hospitalization"]
|
|
821
|
+
```
|
|
822
|
+
|
|
823
|
+

|
|
824
|
+
|
|
825
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
826
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
827
|
+
| 1 | 95 | collections | condition_occurrence | 2010-03-12 | 2010-03-13 | V4501 |
|
|
828
|
+
| 2 | 37 | collections | condition_occurrence | 2009-04-12 | 2009-04-18 | 2724 |
|
|
829
|
+
| 2 | 96 | collections | condition_occurrence | 2009-08-31 | 2009-09-02 | 7092 |
|
|
830
|
+
| 2 | 12 | collections | condition_occurrence | 2009-09-17 | 2009-09-20 | 32723 |
|
|
831
|
+
| 2 | 43 | collections | condition_occurrence | 2010-06-26 | 2010-07-01 | 7856 |
|
|
832
|
+
| 14 | 151 | collections | condition_occurrence | 2008-09-12 | 2008-09-12 | 2724 |
|
|
833
|
+
| 17 | 93 | collections | condition_occurrence | 2008-09-19 | 2008-09-22 | 33811 |
|
|
834
|
+
| 17 | 83 | collections | condition_occurrence | 2010-05-22 | 2010-06-12 | 5845 |
|
|
835
|
+
| 17 | 122 | collections | condition_occurrence | 2010-06-02 | 2010-06-06 | 4280 |
|
|
836
|
+
| 17 | 124 | collections | condition_occurrence | 2010-06-16 | 2010-06-19 | V4365 |
|
|
837
|
+
|
|
838
|
+
______________________________________________________________________
|
|
839
|
+
|
|
840
|
+
## Set Operators
|
|
841
|
+
|
|
842
|
+
Streams are sets of records, so ConceptQL has operators that work on sets.
|
|
843
|
+
|
|
844
|
+
### `Union` Operator
|
|
845
|
+
|
|
846
|
+
Combines the records of all its upstream operators into one stream. Each record keeps its own type, so one stream can hold several types.
|
|
847
|
+
|
|
848
|
+
- **Upstream:** one or more operators.
|
|
849
|
+
- **Returns:** Each record of each upstream. A record that comes from more than one upstream comes out once. Two records are the same only when every column is equal, including label.
|
|
850
|
+
|
|
851
|
+
______________________________________________________________________
|
|
852
|
+
|
|
853
|
+
**Example 21 - Two streams of the same type (condition_occurrence) combined into one stream**
|
|
854
|
+
|
|
855
|
+
```JSON
|
|
856
|
+
["union",["icd9","412"],["icd9","250.01"]]
|
|
857
|
+
```
|
|
858
|
+
|
|
859
|
+

|
|
860
|
+
|
|
861
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
862
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
863
|
+
| 2 | 55131 | observations | condition_occurrence | 2010-02-19 | 2010-02-19 | 25001 |
|
|
864
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
865
|
+
| 13 | 68892 | observations | condition_occurrence | 2010-09-14 | 2010-09-14 | 25001 |
|
|
866
|
+
| 14 | 28741 | observations | condition_occurrence | 2009-06-27 | 2009-06-27 | 25001 |
|
|
867
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
868
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
869
|
+
| 17 | 59056 | observations | condition_occurrence | 2009-11-25 | 2009-11-25 | 25001 |
|
|
870
|
+
| 17 | 30214 | observations | condition_occurrence | 2010-03-21 | 2010-03-21 | 25001 |
|
|
871
|
+
| 23 | 77713 | observations | condition_occurrence | 2010-01-30 | 2010-01-30 | 25001 |
|
|
872
|
+
| 25 | 84696 | observations | condition_occurrence | 2009-12-31 | 2009-12-31 | 25001 |
|
|
873
|
+
|
|
874
|
+
______________________________________________________________________
|
|
875
|
+
|
|
876
|
+
______________________________________________________________________
|
|
877
|
+
|
|
878
|
+
**Example 22 - Two condition_occurrence streams combined, then combined with a procedure_occurrence stream**
|
|
879
|
+
|
|
880
|
+
```JSON
|
|
881
|
+
["union",["union",["icd9","412"],["icd9","250.01"]],["cpt","99214"]]
|
|
882
|
+
```
|
|
883
|
+
|
|
884
|
+

|
|
885
|
+
|
|
886
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
887
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
888
|
+
| 2 | 55131 | observations | condition_occurrence | 2010-02-19 | 2010-02-19 | 25001 |
|
|
889
|
+
| 2 | 58127 | observations | procedure_occurrence | 2009-07-03 | 2009-07-03 | 99214 |
|
|
890
|
+
| 2 | 83339 | observations | procedure_occurrence | 2009-12-02 | 2009-12-02 | 99214 |
|
|
891
|
+
| 5 | 38313 | observations | procedure_occurrence | 2008-12-14 | 2008-12-14 | 99214 |
|
|
892
|
+
| 5 | 76788 | observations | procedure_occurrence | 2009-07-04 | 2009-07-04 | 99214 |
|
|
893
|
+
| 5 | 63847 | observations | procedure_occurrence | 2009-12-21 | 2009-12-21 | 99214 |
|
|
894
|
+
| 6 | 48992 | observations | procedure_occurrence | 2009-09-12 | 2009-09-12 | 99214 |
|
|
895
|
+
| 6 | 76651 | observations | procedure_occurrence | 2010-02-23 | 2010-02-23 | 99214 |
|
|
896
|
+
| 7 | 63797 | observations | procedure_occurrence | 2008-08-13 | 2008-08-13 | 99214 |
|
|
897
|
+
| 7 | 59555 | observations | procedure_occurrence | 2008-09-21 | 2008-09-21 | 99214 |
|
|
898
|
+
|
|
899
|
+
______________________________________________________________________
|
|
900
|
+
|
|
901
|
+
______________________________________________________________________
|
|
902
|
+
|
|
903
|
+
**Example 23 - Two condition_occurrence streams and a procedure_occurrence stream combined by a single union**
|
|
904
|
+
|
|
905
|
+
```JSON
|
|
906
|
+
["union",["icd9","412"],["icd9","250.01"],["cpt","99214"]]
|
|
907
|
+
```
|
|
908
|
+
|
|
909
|
+

|
|
910
|
+
|
|
911
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
912
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
913
|
+
| 2 | 55131 | observations | condition_occurrence | 2010-02-19 | 2010-02-19 | 25001 |
|
|
914
|
+
| 2 | 58127 | observations | procedure_occurrence | 2009-07-03 | 2009-07-03 | 99214 |
|
|
915
|
+
| 2 | 83339 | observations | procedure_occurrence | 2009-12-02 | 2009-12-02 | 99214 |
|
|
916
|
+
| 5 | 38313 | observations | procedure_occurrence | 2008-12-14 | 2008-12-14 | 99214 |
|
|
917
|
+
| 5 | 76788 | observations | procedure_occurrence | 2009-07-04 | 2009-07-04 | 99214 |
|
|
918
|
+
| 5 | 63847 | observations | procedure_occurrence | 2009-12-21 | 2009-12-21 | 99214 |
|
|
919
|
+
| 6 | 48992 | observations | procedure_occurrence | 2009-09-12 | 2009-09-12 | 99214 |
|
|
920
|
+
| 6 | 76651 | observations | procedure_occurrence | 2010-02-23 | 2010-02-23 | 99214 |
|
|
921
|
+
| 7 | 63797 | observations | procedure_occurrence | 2008-08-13 | 2008-08-13 | 99214 |
|
|
922
|
+
| 7 | 59555 | observations | procedure_occurrence | 2008-09-21 | 2008-09-21 | 99214 |
|
|
923
|
+
|
|
924
|
+
______________________________________________________________________
|
|
925
|
+
|
|
926
|
+
Union does not change records of different types. Each later operator decides whether it treats the types separately or together.
|
|
927
|
+
|
|
928
|
+
### `Except` Operator
|
|
929
|
+
|
|
930
|
+
Removes each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same criterion_table, criterion_id, start_date and end_date.
|
|
931
|
+
|
|
932
|
+
- **Operands:**
|
|
933
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
934
|
+
- `right` (required): The right-hand stream. The operator removes each left-hand record that is also in this stream, and does not pass these records on.
|
|
935
|
+
- **Returns:** The left-hand records that have no right-hand record with the same criterion_table, criterion_id, start_date and end_date. A left-hand record whose table has no right-hand record passes through unchanged. The right-hand records are not returned.
|
|
936
|
+
|
|
937
|
+
______________________________________________________________________
|
|
938
|
+
|
|
939
|
+
**Example 24 - All males who are not white**
|
|
940
|
+
|
|
941
|
+
```JSON
|
|
942
|
+
["except",{"left":["gender","male"],"right":["race","white"]}]
|
|
943
|
+
```
|
|
944
|
+
|
|
945
|
+

|
|
946
|
+
|
|
947
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
948
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
949
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1941-06-01 | 00021CA6FF03E670 |
|
|
950
|
+
| 6 | 6 | patients | person | 1943-10-01 | 1943-10-01 | 0002DAE1C81CC70D |
|
|
951
|
+
| 21 | 21 | patients | person | 1932-08-01 | 1932-08-01 | 000C7486B11E7030 |
|
|
952
|
+
| 22 | 22 | patients | person | 1942-08-01 | 1942-08-01 | 000D6D88463D8A76 |
|
|
953
|
+
| 50 | 50 | patients | person | 1941-08-01 | 1941-08-01 | 001AEDD510C92C87 |
|
|
954
|
+
| 68 | 68 | patients | person | 1954-11-01 | 1954-11-01 | 0021D4CDAFC0609F |
|
|
955
|
+
| 69 | 69 | patients | person | 1925-05-01 | 1925-05-01 | 00225409819CF5F6 |
|
|
956
|
+
| 71 | 71 | patients | person | 1931-01-01 | 1931-01-01 | 0022CC1943D038E2 |
|
|
957
|
+
| 72 | 72 | patients | person | 1933-04-01 | 1933-04-01 | 002354398A00234E |
|
|
958
|
+
| 75 | 75 | patients | person | 1932-08-01 | 1932-08-01 | 00241223F034F97B |
|
|
959
|
+
|
|
960
|
+
______________________________________________________________________
|
|
961
|
+
|
|
962
|
+
______________________________________________________________________
|
|
963
|
+
|
|
964
|
+
**Example 25 - All Conditions that are MI**
|
|
965
|
+
|
|
966
|
+
```JSON
|
|
967
|
+
["except",{"left":["icd9","412"],"right":["cpt","99214"]}]
|
|
968
|
+
```
|
|
969
|
+
|
|
970
|
+

|
|
971
|
+
|
|
972
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
973
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
974
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
975
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
976
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
977
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
978
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
979
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
980
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
981
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
982
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
983
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
984
|
+
|
|
985
|
+
______________________________________________________________________
|
|
986
|
+
|
|
987
|
+
______________________________________________________________________
|
|
988
|
+
|
|
989
|
+
**Example 26 - Passes two types downstream: Conditions that are MI (unaffected by the right-hand stream) and People that are Male but not White**
|
|
990
|
+
|
|
991
|
+
```JSON
|
|
992
|
+
["except",{"left":["union",["icd9","412"],["gender","Male"]],"right":["race","White"]}]
|
|
993
|
+
```
|
|
994
|
+
|
|
995
|
+

|
|
996
|
+
|
|
997
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
998
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ---------------- |
|
|
999
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1941-06-01 | 00021CA6FF03E670 |
|
|
1000
|
+
| 6 | 6 | patients | person | 1943-10-01 | 1943-10-01 | 0002DAE1C81CC70D |
|
|
1001
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
1002
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1003
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1004
|
+
| 21 | 21 | patients | person | 1932-08-01 | 1932-08-01 | 000C7486B11E7030 |
|
|
1005
|
+
| 22 | 22 | patients | person | 1942-08-01 | 1942-08-01 | 000D6D88463D8A76 |
|
|
1006
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1007
|
+
| 50 | 50 | patients | person | 1941-08-01 | 1941-08-01 | 001AEDD510C92C87 |
|
|
1008
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1009
|
+
|
|
1010
|
+
______________________________________________________________________
|
|
1011
|
+
|
|
1012
|
+
A left-hand record matches only a right-hand record from the same table, and only when the dates are the same. For example, a record whose dates `time_window` changed no longer matches the original record. The second example above shows that a left-hand record passes through unchanged when no right-hand record comes from its table. The third example has several types in the left-hand stream.
|
|
1013
|
+
|
|
1014
|
+
### Discussion About Set Operators
|
|
1015
|
+
|
|
1016
|
+
Why can one stream hold records of several types? Because operators such as `first` then act across all the types. For example, the statement below finds the first MI or death for each patient. Add a few more criteria, and you can find the first censoring event for each patient.
|
|
1017
|
+
|
|
1018
|
+
______________________________________________________________________
|
|
1019
|
+
|
|
1020
|
+
**Example 27 - First occurrence of either MI or Death for each patient**
|
|
1021
|
+
|
|
1022
|
+
```JSON
|
|
1023
|
+
["first",["union",["icd9","412"],["death"]]]
|
|
1024
|
+
```
|
|
1025
|
+
|
|
1026
|
+

|
|
1027
|
+
|
|
1028
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1029
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1030
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
1031
|
+
| 16 | 1 | deaths | death | 2010-12-01 | 2010-12-01 | |
|
|
1032
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1033
|
+
| 26 | 2 | deaths | death | 2009-07-01 | 2009-07-01 | |
|
|
1034
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1035
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1036
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
1037
|
+
| 65 | 3 | deaths | death | 2009-01-01 | 2009-01-01 | |
|
|
1038
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1039
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
1040
|
+
|
|
1041
|
+
______________________________________________________________________
|
|
1042
|
+
|
|
1043
|
+
## Time-oriented Operators
|
|
1044
|
+
|
|
1045
|
+
Every record has a start_date and an end_date. All temporal comparisons use these two columns. Each record takes its dates from its row in its table of origin.
|
|
1046
|
+
|
|
1047
|
+
If the table of origin has only one date, the record uses that date as both its start_date and its end_date. For example, a death record uses the `date` column of its row in `deaths`.
|
|
1048
|
+
|
|
1049
|
+
Person records are a special case. A person record uses the patient's birth date as its start_date and end_date. This may sound strange, but [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) shows why it is useful.
|
|
1050
|
+
|
|
1051
|
+
## Relative Temporal Operators
|
|
1052
|
+
|
|
1053
|
+
For each patient, we may want only the chronologically first or last record, or the second, or the second-to-last. Relative temporal operators provide this filtering. They order each patient's records by start_date. When two records have the same start_date, the record with the lower `criterion_id` comes first.
|
|
1054
|
+
|
|
1055
|
+
### `Nth Occurrence` Operator
|
|
1056
|
+
|
|
1057
|
+
For each person, orders the records by start_date and passes along the record at the given position. A positive position counts from the earliest record and a negative one from the most recent record.
|
|
1058
|
+
|
|
1059
|
+
- **Upstream:** one operator.
|
|
1060
|
+
- **Arguments:**
|
|
1061
|
+
- `occurrence` (integer): The position of the record to pass along, as an integer. A positive number counts from the earliest record, so 1 is the first record and 2 is the second. A negative number counts from the most recent record, so -1 is the last record and -3 is the third from the last. 0 selects nothing.
|
|
1062
|
+
- **Options:**
|
|
1063
|
+
- `at_least` (string): A date adjustment, such as 30d. For each record, the operator counts that record and the later records that start at least this long after it ends, and passes along the record at the given position. So a person can have more than one record in the result.
|
|
1064
|
+
- `within` (string): A date adjustment, such as 365d. For each record, the operator counts that record and the later records that start at most this long after it ends, and passes along the record at the given position. So a person can have more than one record in the result.
|
|
1065
|
+
- `group_by_date` (boolean): Treats the records of a person on one start_date as one occurrence, so the position counts dates. From the date at that position, the operator passes along the record with the lowest criterion_id, or the highest criterion_id for a negative position.
|
|
1066
|
+
- **Returns:** One record per person: the record at the given position. A person who has fewer records than the position asks for is not passed along. With at_least or within, a person can have more than one record.
|
|
1067
|
+
|
|
1068
|
+
______________________________________________________________________
|
|
1069
|
+
|
|
1070
|
+
**Example 28 - For each patient, select the Condition that represents the second occurrence of an MI**
|
|
1071
|
+
|
|
1072
|
+
```JSON
|
|
1073
|
+
["occurrence",2,["icd9","412"]]
|
|
1074
|
+
```
|
|
1075
|
+
|
|
1076
|
+

|
|
1077
|
+
|
|
1078
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1079
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1080
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1081
|
+
| 88 | 73803 | observations | condition_occurrence | 2009-05-30 | 2009-05-30 | 412 |
|
|
1082
|
+
| 152 | 84298 | observations | condition_occurrence | 2010-11-22 | 2010-11-22 | 412 |
|
|
1083
|
+
| 177 | 507 | observations | condition_occurrence | 2009-06-13 | 2009-06-16 | 412 |
|
|
1084
|
+
| 187 | 87583 | observations | condition_occurrence | 2010-12-26 | 2010-12-26 | 412 |
|
|
1085
|
+
| 212 | 34052 | observations | condition_occurrence | 2009-10-07 | 2009-10-07 | 412 |
|
|
1086
|
+
| 213 | 15005 | observations | condition_occurrence | 2010-02-07 | 2010-02-07 | 412 |
|
|
1087
|
+
| 214 | 83257 | observations | condition_occurrence | 2009-11-12 | 2009-11-12 | 412 |
|
|
1088
|
+
| 220 | 20660 | observations | condition_occurrence | 2009-10-31 | 2009-10-31 | 412 |
|
|
1089
|
+
| 223 | 39893 | observations | condition_occurrence | 2008-10-31 | 2008-10-31 | 412 |
|
|
1090
|
+
|
|
1091
|
+
______________________________________________________________________
|
|
1092
|
+
|
|
1093
|
+
______________________________________________________________________
|
|
1094
|
+
|
|
1095
|
+
**Example 29 - For each patient, the first record on the second day with any of these diagnoses**
|
|
1096
|
+
|
|
1097
|
+
```JSON
|
|
1098
|
+
["occurrence",2,["icd9","410.00","410.01","250.00","250.01"],{"group_by_date":true}]
|
|
1099
|
+
```
|
|
1100
|
+
|
|
1101
|
+

|
|
1102
|
+
|
|
1103
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1104
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1105
|
+
| 2 | 57897 | observations | condition_occurrence | 2009-05-16 | 2009-05-16 | 25000 |
|
|
1106
|
+
| 7 | 69706 | observations | condition_occurrence | 2008-05-12 | 2008-05-12 | 25000 |
|
|
1107
|
+
| 8 | 90541 | observations | condition_occurrence | 2009-03-01 | 2009-03-03 | 25000 |
|
|
1108
|
+
| 9 | 84726 | observations | condition_occurrence | 2010-09-12 | 2010-09-12 | 25000 |
|
|
1109
|
+
| 11 | 35227 | observations | condition_occurrence | 2008-12-08 | 2008-12-08 | 25000 |
|
|
1110
|
+
| 12 | 43800 | observations | condition_occurrence | 2009-05-21 | 2009-05-21 | 25000 |
|
|
1111
|
+
| 13 | 57738 | observations | condition_occurrence | 2008-11-22 | 2008-11-22 | 25000 |
|
|
1112
|
+
| 14 | 28741 | observations | condition_occurrence | 2009-06-27 | 2009-06-27 | 25001 |
|
|
1113
|
+
| 15 | 68873 | observations | condition_occurrence | 2010-11-18 | 2010-11-18 | 25000 |
|
|
1114
|
+
| 17 | 73012 | observations | condition_occurrence | 2008-04-26 | 2008-04-26 | 25000 |
|
|
1115
|
+
|
|
1116
|
+
______________________________________________________________________
|
|
1117
|
+
|
|
1118
|
+
`at_least` and `within` take date adjustments, as [`time_window`](#time-window-operator) does. With `group_by_date`, several codes on the same day count as one occurrence. The second example above finds, for each patient, the second day on which any of its codes appears.
|
|
1119
|
+
|
|
1120
|
+
### `First` Operator
|
|
1121
|
+
|
|
1122
|
+
For each person, passes along the record with the earliest start_date.
|
|
1123
|
+
|
|
1124
|
+
- **Upstream:** one operator.
|
|
1125
|
+
- **Returns:** One record per person: the upstream record with the earliest start_date. When several records have that start_date, the one with the lowest criterion_id.
|
|
1126
|
+
|
|
1127
|
+
______________________________________________________________________
|
|
1128
|
+
|
|
1129
|
+
**Example 30 - For each patient, select the Condition that represents the first occurrence of an MI**
|
|
1130
|
+
|
|
1131
|
+
```JSON
|
|
1132
|
+
["first",["icd9","412"]]
|
|
1133
|
+
```
|
|
1134
|
+
|
|
1135
|
+

|
|
1136
|
+
|
|
1137
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1138
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1139
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
1140
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1141
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1142
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1143
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
1144
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1145
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
1146
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
1147
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
1148
|
+
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
1149
|
+
|
|
1150
|
+
______________________________________________________________________
|
|
1151
|
+
|
|
1152
|
+
`first` is the same as `["occurrence", 1, ...]`. It does not accept `at_least`, `within` or `group_by_date`.
|
|
1153
|
+
|
|
1154
|
+
### `Last` Operator
|
|
1155
|
+
|
|
1156
|
+
For each person, passes along the record with the most recent start_date.
|
|
1157
|
+
|
|
1158
|
+
- **Upstream:** one operator.
|
|
1159
|
+
- **Returns:** One record per person: the upstream record with the latest start_date. When several records have that start_date, the one with the highest criterion_id.
|
|
1160
|
+
|
|
1161
|
+
______________________________________________________________________
|
|
1162
|
+
|
|
1163
|
+
**Example 31 - For each patient, select the Condition that represents the last occurrence of an MI**
|
|
1164
|
+
|
|
1165
|
+
```JSON
|
|
1166
|
+
["last",["icd9","412"]]
|
|
1167
|
+
```
|
|
1168
|
+
|
|
1169
|
+

|
|
1170
|
+
|
|
1171
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1172
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1173
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
1174
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1175
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1176
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1177
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
1178
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1179
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
1180
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
1181
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
1182
|
+
| 88 | 73803 | observations | condition_occurrence | 2009-05-30 | 2009-05-30 | 412 |
|
|
1183
|
+
|
|
1184
|
+
______________________________________________________________________
|
|
1185
|
+
|
|
1186
|
+
`last` is the same as `["occurrence", -1, ...]`. It does not accept `at_least`, `within` or `group_by_date`.
|
|
1187
|
+
|
|
1188
|
+
## Date Literals
|
|
1189
|
+
|
|
1190
|
+
To represent a fixed range of dates, use the `date_range` operator.
|
|
1191
|
+
|
|
1192
|
+
### `Date Range` Operator
|
|
1193
|
+
|
|
1194
|
+
Makes one person record for each patient, with the given start_date and end_date. Use it to compare other records with a fixed range of dates.
|
|
1195
|
+
|
|
1196
|
+
- **Upstream:** none.
|
|
1197
|
+
- **Options:**
|
|
1198
|
+
- `start` (string, required): The start_date of each record. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive.
|
|
1199
|
+
- `end` (string, required): The end_date of each record. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive. For a single day, give the same date as the start option.
|
|
1200
|
+
- **Returns:** One record for each patient. The type of the records is person, and each record has the start_date and the end_date that the options give.
|
|
1201
|
+
|
|
1202
|
+
______________________________________________________________________
|
|
1203
|
+
|
|
1204
|
+
**Example 32 - Each patient, with the dates of the year 2010**
|
|
1205
|
+
|
|
1206
|
+
```JSON
|
|
1207
|
+
["date_range",{"start":"2010-01-01","end":"2010-12-31"}]
|
|
1208
|
+
```
|
|
1209
|
+
|
|
1210
|
+

|
|
1211
|
+
|
|
1212
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1213
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
1214
|
+
| 1 | 1 | patients | person | 2010-01-01 | 2010-12-31 | 00013D2EFD8E45D1 |
|
|
1215
|
+
| 2 | 2 | patients | person | 2010-01-01 | 2010-12-31 | 00016F745862898F |
|
|
1216
|
+
| 3 | 3 | patients | person | 2010-01-01 | 2010-12-31 | 0001FDD721E223DC |
|
|
1217
|
+
| 4 | 4 | patients | person | 2010-01-01 | 2010-12-31 | 00021CA6FF03E670 |
|
|
1218
|
+
| 5 | 5 | patients | person | 2010-01-01 | 2010-12-31 | 00024B3D2352D2D0 |
|
|
1219
|
+
| 6 | 6 | patients | person | 2010-01-01 | 2010-12-31 | 0002DAE1C81CC70D |
|
|
1220
|
+
| 7 | 7 | patients | person | 2010-01-01 | 2010-12-31 | 0002F28CE057345B |
|
|
1221
|
+
| 8 | 8 | patients | person | 2010-01-01 | 2010-12-31 | 000308435E3E5B76 |
|
|
1222
|
+
| 9 | 9 | patients | person | 2010-01-01 | 2010-12-31 | 000345A39D4157C9 |
|
|
1223
|
+
| 10 | 10 | patients | person | 2010-01-01 | 2010-12-31 | 00036A21B65B0206 |
|
|
1224
|
+
|
|
1225
|
+
______________________________________________________________________
|
|
1226
|
+
|
|
1227
|
+
______________________________________________________________________
|
|
1228
|
+
|
|
1229
|
+
**Example 33 - Each patient, with the dates from the start to the end of the data**
|
|
1230
|
+
|
|
1231
|
+
```JSON
|
|
1232
|
+
["date_range",{"start":"START","end":"END"}]
|
|
1233
|
+
```
|
|
1234
|
+
|
|
1235
|
+

|
|
1236
|
+
|
|
1237
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1238
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
1239
|
+
| 1 | 1 | patients | person | 2008-01-01 | 2010-12-31 | 00013D2EFD8E45D1 |
|
|
1240
|
+
| 2 | 2 | patients | person | 2008-01-01 | 2010-12-31 | 00016F745862898F |
|
|
1241
|
+
| 3 | 3 | patients | person | 2008-01-01 | 2010-12-31 | 0001FDD721E223DC |
|
|
1242
|
+
| 4 | 4 | patients | person | 2008-01-01 | 2010-12-31 | 00021CA6FF03E670 |
|
|
1243
|
+
| 5 | 5 | patients | person | 2008-01-01 | 2010-12-31 | 00024B3D2352D2D0 |
|
|
1244
|
+
| 6 | 6 | patients | person | 2008-01-01 | 2010-12-31 | 0002DAE1C81CC70D |
|
|
1245
|
+
| 7 | 7 | patients | person | 2008-01-01 | 2010-12-31 | 0002F28CE057345B |
|
|
1246
|
+
| 8 | 8 | patients | person | 2008-01-01 | 2010-12-31 | 000308435E3E5B76 |
|
|
1247
|
+
| 9 | 9 | patients | person | 2008-01-01 | 2010-12-31 | 000345A39D4157C9 |
|
|
1248
|
+
| 10 | 10 | patients | person | 2008-01-01 | 2010-12-31 | 00036A21B65B0206 |
|
|
1249
|
+
|
|
1250
|
+
______________________________________________________________________
|
|
1251
|
+
|
|
1252
|
+
The range includes both dates. A `YYYY-MM-DD` date has a four-digit year, a two-digit month, and a two-digit day, with leading zeros. There is no operator for a single day. Use `date_range` with the same start and end.
|
|
1253
|
+
|
|
1254
|
+
## Temporal Comparison Operators
|
|
1255
|
+
|
|
1256
|
+
As described above, each record has a start date and an end date, which define its own date range. Temporal operators filter streams by comparing these date ranges.
|
|
1257
|
+
|
|
1258
|
+
A temporal operator takes two streams: the left-hand records (LHR) and the right-hand records (RHR). It compares each patient's LHR with the same patient's RHR, and it passes along the LHR that pass the comparison. It discards the RHR after the comparison.
|
|
1259
|
+
|
|
1260
|
+
- Each LHR is passed along at most once, however many RHR it matches.
|
|
1261
|
+
- A patient who has LHR but no RHR loses all of their LHR.
|
|
1262
|
+
- A record that is in both streams can match itself. For example, a record is during itself.
|
|
1263
|
+
|
|
1264
|
+
Edge behaviors: `during`, `contains` and `any_overlap` compare each LHR with every RHR of the patient. `before` and `after` do not. Without the `within` option, `before` compares each LHR only with the latest start_date of the patient's RHR, and `after` compares each LHR only with the earliest end_date of the patient's RHR. [`before`](#before-operator) and [`after`](#after-operator) explain why, and [Working Around `Before`/`After` Behavior](#working-around-beforeafter-behavior) shows how to choose another RHR.
|
|
1265
|
+
|
|
1266
|
+
Each operator compares the dates in a different way. All of them take [some options](#temporal-comparison-operator-options), which are described later.
|
|
1267
|
+
|
|
1268
|
+
### `During` Operator
|
|
1269
|
+
|
|
1270
|
+
For each patient, keeps the left-hand records whose date range is inside the date range of a right-hand record. The left-hand record starts on or after the start_date of the right-hand record and ends on or before its end_date.
|
|
1271
|
+
|
|
1272
|
+
- **Operands:**
|
|
1273
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
1274
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
|
|
1275
|
+
- **Options:**
|
|
1276
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand record is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
|
|
1277
|
+
- **Returns:** The left-hand records that are inside the date range of at least one right-hand record of the same patient, each record once. The right-hand records are not returned.
|
|
1278
|
+
|
|
1279
|
+
______________________________________________________________________
|
|
1280
|
+
|
|
1281
|
+
**Example 34 - All MIs for the year 2010**
|
|
1282
|
+
|
|
1283
|
+
```JSON
|
|
1284
|
+
["during",{"left":["icd9","412"],"right":["date_range",{"start":"2010-01-01","end":"2010-12-31"}]}]
|
|
1285
|
+
```
|
|
1286
|
+
|
|
1287
|
+

|
|
1288
|
+
|
|
1289
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1290
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1291
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1292
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
1293
|
+
| 90 | 33940 | observations | condition_occurrence | 2010-05-26 | 2010-05-26 | 412 |
|
|
1294
|
+
| 109 | 89334 | observations | condition_occurrence | 2010-07-06 | 2010-07-06 | 412 |
|
|
1295
|
+
| 110 | 62592 | observations | condition_occurrence | 2010-06-27 | 2010-06-27 | 412 |
|
|
1296
|
+
| 152 | 84298 | observations | condition_occurrence | 2010-11-22 | 2010-11-22 | 412 |
|
|
1297
|
+
| 187 | 87583 | observations | condition_occurrence | 2010-12-26 | 2010-12-26 | 412 |
|
|
1298
|
+
| 213 | 15005 | observations | condition_occurrence | 2010-02-07 | 2010-02-07 | 412 |
|
|
1299
|
+
| 217 | 56467 | observations | condition_occurrence | 2010-06-22 | 2010-06-23 | 412 |
|
|
1300
|
+
| 239 | 49718 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1301
|
+
|
|
1302
|
+
______________________________________________________________________
|
|
1303
|
+
|
|
1304
|
+
### `Contains` Operator
|
|
1305
|
+
|
|
1306
|
+
For each patient, keeps the left-hand records whose date range encloses the date range of a right-hand record. The left-hand record starts on or before the start_date of the right-hand record and ends on or after its end_date.
|
|
1307
|
+
|
|
1308
|
+
- **Operands:**
|
|
1309
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
1310
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
|
|
1311
|
+
- **Options:**
|
|
1312
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each left-hand record is widened by this amount in each direction: its start_date moves back and its end_date moves forward. The returned records keep their own dates. Use a positive amount.
|
|
1313
|
+
- **Returns:** The left-hand records that enclose the date range of at least one right-hand record of the same patient, each record once. The right-hand records are not returned.
|
|
1314
|
+
|
|
1315
|
+
______________________________________________________________________
|
|
1316
|
+
|
|
1317
|
+
**Example 35 - All MIs occurring on 2010-06-22**
|
|
1318
|
+
|
|
1319
|
+
```JSON
|
|
1320
|
+
["contains",{"left":["icd9","412"],"right":["date_range",{"start":"2010-06-22","end":"2010-06-22"}]}]
|
|
1321
|
+
```
|
|
1322
|
+
|
|
1323
|
+

|
|
1324
|
+
|
|
1325
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1326
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1327
|
+
| 217 | 56467 | observations | condition_occurrence | 2010-06-22 | 2010-06-23 | 412 |
|
|
1328
|
+
|
|
1329
|
+
______________________________________________________________________
|
|
1330
|
+
|
|
1331
|
+
### `Any Overlap` Operator
|
|
1332
|
+
|
|
1333
|
+
For each patient, keeps the left-hand records whose date range shares at least one day with the date range of a right-hand record.
|
|
1334
|
+
|
|
1335
|
+
- **Operands:**
|
|
1336
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
1337
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
|
|
1338
|
+
- **Options:**
|
|
1339
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand record is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
|
|
1340
|
+
- **Returns:** The left-hand records that overlap the date range of at least one right-hand record of the same patient, each record once. The right-hand records are not returned.
|
|
1341
|
+
|
|
1342
|
+
______________________________________________________________________
|
|
1343
|
+
|
|
1344
|
+
**Example 36 - All MIs that overlap with the month of 2010-06**
|
|
1345
|
+
|
|
1346
|
+
```JSON
|
|
1347
|
+
["any_overlap",{"left":["icd9","412"],"right":["date_range",{"start":"2010-06-01","end":"2010-06-30"}]}]
|
|
1348
|
+
```
|
|
1349
|
+
|
|
1350
|
+

|
|
1351
|
+
|
|
1352
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1353
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1354
|
+
| 110 | 62592 | observations | condition_occurrence | 2010-06-27 | 2010-06-27 | 412 |
|
|
1355
|
+
| 217 | 56467 | observations | condition_occurrence | 2010-06-22 | 2010-06-23 | 412 |
|
|
1356
|
+
|
|
1357
|
+
______________________________________________________________________
|
|
1358
|
+
|
|
1359
|
+
This diagram shows each kind of L record that has "any overlap" with an R record:
|
|
1360
|
+
|
|
1361
|
+

|
|
1362
|
+
|
|
1363
|
+
### `Before` Operator
|
|
1364
|
+
|
|
1365
|
+
For each patient, keeps the left-hand records whose end_date is before the latest start_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead.
|
|
1366
|
+
|
|
1367
|
+
- **Operands:**
|
|
1368
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
1369
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
|
|
1370
|
+
- **Options:**
|
|
1371
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand record must also end no more than this amount before the start_date of the right-hand record. With this option, each left-hand record is compared with every right-hand record of the patient, not only with the one that starts last. Use a positive amount.
|
|
1372
|
+
- `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand record must end more than this amount before the start_date of the right-hand record. Use a positive amount.
|
|
1373
|
+
- **Returns:** The left-hand records that end before a right-hand record of the same patient starts, each record once. The right-hand records are not returned.
|
|
1374
|
+
|
|
1375
|
+
______________________________________________________________________
|
|
1376
|
+
|
|
1377
|
+
**Example 37 - All MIs that occurred before a patient's last case of diabetes (250.01)**
|
|
1378
|
+
|
|
1379
|
+
```JSON
|
|
1380
|
+
["before",{"left":["icd9","412"],"right":["icd9","250.01"]}]
|
|
1381
|
+
```
|
|
1382
|
+
|
|
1383
|
+

|
|
1384
|
+
|
|
1385
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1386
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1387
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1388
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1389
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1390
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
1391
|
+
| 109 | 89334 | observations | condition_occurrence | 2010-07-06 | 2010-07-06 | 412 |
|
|
1392
|
+
| 131 | 172 | observations | condition_occurrence | 2008-03-22 | 2008-03-23 | 412 |
|
|
1393
|
+
| 152 | 82988 | observations | condition_occurrence | 2008-08-16 | 2008-08-16 | 412 |
|
|
1394
|
+
| 184 | 24281 | observations | condition_occurrence | 2008-01-07 | 2008-01-07 | 412 |
|
|
1395
|
+
| 187 | 90638 | observations | condition_occurrence | 2009-01-31 | 2009-01-31 | 412 |
|
|
1396
|
+
| 214 | 89463 | observations | condition_occurrence | 2008-02-17 | 2008-02-17 | 412 |
|
|
1397
|
+
|
|
1398
|
+
______________________________________________________________________
|
|
1399
|
+
|
|
1400
|
+
______________________________________________________________________
|
|
1401
|
+
|
|
1402
|
+
**Example 38 - All MIs that occurred before a patient's first case of diabetes (250.01)**
|
|
1403
|
+
|
|
1404
|
+
```JSON
|
|
1405
|
+
["before",{"left":["icd9","412"],"right":["first",["icd9","250.01"]]}]
|
|
1406
|
+
```
|
|
1407
|
+
|
|
1408
|
+

|
|
1409
|
+
|
|
1410
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1411
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1412
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1413
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1414
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1415
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
1416
|
+
| 109 | 89334 | observations | condition_occurrence | 2010-07-06 | 2010-07-06 | 412 |
|
|
1417
|
+
| 131 | 172 | observations | condition_occurrence | 2008-03-22 | 2008-03-23 | 412 |
|
|
1418
|
+
| 152 | 82988 | observations | condition_occurrence | 2008-08-16 | 2008-08-16 | 412 |
|
|
1419
|
+
| 184 | 24281 | observations | condition_occurrence | 2008-01-07 | 2008-01-07 | 412 |
|
|
1420
|
+
| 187 | 90638 | observations | condition_occurrence | 2009-01-31 | 2009-01-31 | 412 |
|
|
1421
|
+
| 214 | 89463 | observations | condition_occurrence | 2008-02-17 | 2008-02-17 | 412 |
|
|
1422
|
+
|
|
1423
|
+
______________________________________________________________________
|
|
1424
|
+
|
|
1425
|
+
______________________________________________________________________
|
|
1426
|
+
|
|
1427
|
+
**Example 39 - Heart attacks that end no more than 1 year before death**
|
|
1428
|
+
|
|
1429
|
+
```JSON
|
|
1430
|
+
["before",{"left":["icd9","410.00","410.01","410.10","410.11"],"right":["death"],"within":"1y"}]
|
|
1431
|
+
```
|
|
1432
|
+
|
|
1433
|
+

|
|
1434
|
+
|
|
1435
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1436
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1437
|
+
| 191 | 42977 | observations | condition_occurrence | 2009-05-21 | 2009-05-21 | 41010 |
|
|
1438
|
+
|
|
1439
|
+
______________________________________________________________________
|
|
1440
|
+
|
|
1441
|
+
The comparison is strict, so an LHR that ends on the day the RHR starts does not pass.
|
|
1442
|
+
|
|
1443
|
+
This rule needs some explanation. Imagine the events A-A-B-A-B-A. How many A's come before a B? One could argue that only the first two A's qualify, because they come before the first B. But three A's come before some B. The `before` operator uses the second answer, so it compares every LHR with the start_date of the patient's last RHR.
|
|
1444
|
+
|
|
1445
|
+
### `After` Operator
|
|
1446
|
+
|
|
1447
|
+
For each patient, keeps the left-hand records whose start_date is after the earliest end_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead.
|
|
1448
|
+
|
|
1449
|
+
- **Operands:**
|
|
1450
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
1451
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
|
|
1452
|
+
- **Options:**
|
|
1453
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand record must also start no more than this amount after the end_date of the right-hand record. With this option, each left-hand record is compared with every right-hand record of the patient, not only with the one that ends first. Use a positive amount.
|
|
1454
|
+
- `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand record must start more than this amount after the end_date of the right-hand record. Use a positive amount.
|
|
1455
|
+
- **Returns:** The left-hand records that start after a right-hand record of the same patient ends, each record once. The right-hand records are not returned.
|
|
1456
|
+
|
|
1457
|
+
______________________________________________________________________
|
|
1458
|
+
|
|
1459
|
+
**Example 40 - All MIs that occurred after a patient's first case of diabetes (250.01)**
|
|
1460
|
+
|
|
1461
|
+
```JSON
|
|
1462
|
+
["after",{"left":["icd9","412"],"right":["icd9","250.01"]}]
|
|
1463
|
+
```
|
|
1464
|
+
|
|
1465
|
+

|
|
1466
|
+
|
|
1467
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1468
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1469
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
1470
|
+
| 96 | 20748 | observations | condition_occurrence | 2008-09-21 | 2008-09-24 | 412 |
|
|
1471
|
+
| 110 | 62592 | observations | condition_occurrence | 2010-06-27 | 2010-06-27 | 412 |
|
|
1472
|
+
| 152 | 84298 | observations | condition_occurrence | 2010-11-22 | 2010-11-22 | 412 |
|
|
1473
|
+
| 187 | 87583 | observations | condition_occurrence | 2010-12-26 | 2010-12-26 | 412 |
|
|
1474
|
+
| 190 | 61423 | observations | condition_occurrence | 2009-04-09 | 2009-04-09 | 412 |
|
|
1475
|
+
| 214 | 83257 | observations | condition_occurrence | 2009-11-12 | 2009-11-12 | 412 |
|
|
1476
|
+
| 223 | 38682 | observations | condition_occurrence | 2008-07-28 | 2008-07-28 | 412 |
|
|
1477
|
+
| 223 | 39893 | observations | condition_occurrence | 2008-10-31 | 2008-10-31 | 412 |
|
|
1478
|
+
|
|
1479
|
+
______________________________________________________________________
|
|
1480
|
+
|
|
1481
|
+
______________________________________________________________________
|
|
1482
|
+
|
|
1483
|
+
**Example 41 - Prescriptions of interest within three days after an office visit**
|
|
1484
|
+
|
|
1485
|
+
```JSON
|
|
1486
|
+
["after",{"left":["ndc","61392070054","65084025214","65726040125",{"label":"Prescriptions of Interest"}],"right":["cpt","99214",{"label":"Office Visit"}],"within":"3d"}]
|
|
1487
|
+
```
|
|
1488
|
+
|
|
1489
|
+

|
|
1490
|
+
|
|
1491
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1492
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
|
|
1493
|
+
| 207 | 11312 | observations | drug_exposure | 2010-08-13 | 2010-08-13 | 61392070054 |
|
|
1494
|
+
|
|
1495
|
+
______________________________________________________________________
|
|
1496
|
+
|
|
1497
|
+
______________________________________________________________________
|
|
1498
|
+
|
|
1499
|
+
**Example 42 - Heart attack diagnoses more than 1 week after a diagnosis of diabetes**
|
|
1500
|
+
|
|
1501
|
+
```JSON
|
|
1502
|
+
["after",{"left":["icd9","410.00","410.01","410.10","410.11",{"label":"Heart Attack Dx"}],"right":["icd9","250.01",{"label":"Diabetes Dx"}],"at_least":"1w"}]
|
|
1503
|
+
```
|
|
1504
|
+
|
|
1505
|
+

|
|
1506
|
+
|
|
1507
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1508
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1509
|
+
| 58 | 79480 | observations | condition_occurrence | 2009-02-24 | 2009-02-24 | 41011 |
|
|
1510
|
+
|
|
1511
|
+
______________________________________________________________________
|
|
1512
|
+
|
|
1513
|
+
______________________________________________________________________
|
|
1514
|
+
|
|
1515
|
+
**Example 43 - All MIs that occurred after a male patient's 50th birthday**
|
|
1516
|
+
|
|
1517
|
+
```JSON
|
|
1518
|
+
["after",{"left":["icd9","412"],"right":["time_window",["gender","Male"],{"start":"50y","end":"50y"}]}]
|
|
1519
|
+
```
|
|
1520
|
+
|
|
1521
|
+

|
|
1522
|
+
|
|
1523
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1524
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1525
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1526
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
1527
|
+
| 93 | 82295 | observations | condition_occurrence | 2009-11-08 | 2009-11-08 | 412 |
|
|
1528
|
+
| 96 | 20748 | observations | condition_occurrence | 2008-09-21 | 2008-09-24 | 412 |
|
|
1529
|
+
| 109 | 89334 | observations | condition_occurrence | 2010-07-06 | 2010-07-06 | 412 |
|
|
1530
|
+
| 110 | 62592 | observations | condition_occurrence | 2010-06-27 | 2010-06-27 | 412 |
|
|
1531
|
+
| 131 | 172 | observations | condition_occurrence | 2008-03-22 | 2008-03-23 | 412 |
|
|
1532
|
+
| 149 | 28518 | observations | condition_occurrence | 2008-04-07 | 2008-04-07 | 412 |
|
|
1533
|
+
| 152 | 82988 | observations | condition_occurrence | 2008-08-16 | 2008-08-16 | 412 |
|
|
1534
|
+
| 152 | 84298 | observations | condition_occurrence | 2010-11-22 | 2010-11-22 | 412 |
|
|
1535
|
+
|
|
1536
|
+
______________________________________________________________________
|
|
1537
|
+
|
|
1538
|
+
The comparison is strict, so an LHR that starts on the day the RHR ends does not pass.
|
|
1539
|
+
|
|
1540
|
+
Imagine the events A-A-B-A-B-A again. How many A's come after a B? One could argue that only the last A qualifies, because it comes after the last B. But two A's come after some B. The `after` operator uses the second answer, so it compares every LHR with the earliest end_date of the patient's RHR.
|
|
1541
|
+
|
|
1542
|
+
### Working Around `Before`/`After` Behavior
|
|
1543
|
+
|
|
1544
|
+
If you want a different RHR for the comparison, use a relative temporal operator to select it. The second example of [`before`](#before-operator) uses `first` to compare each MI with the patient's first case of diabetes.
|
|
1545
|
+
|
|
1546
|
+
## Time Manipulation Operators
|
|
1547
|
+
|
|
1548
|
+
### `Time Window` Operator
|
|
1549
|
+
|
|
1550
|
+
Moves the start_date and the end_date of each record. Use it to shift or widen a record's dates before you compare them with other records.
|
|
1551
|
+
|
|
1552
|
+
- **Upstream:** one operator.
|
|
1553
|
+
- **Options:**
|
|
1554
|
+
- `start` (string): How to change the start_date. If you omit it, the start_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the record's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
|
|
1555
|
+
- `end` (string): How to change the end_date. If you omit it, the end_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the record's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
|
|
1556
|
+
- **Returns:** One record for each upstream record, with the start_date and the end_date that the options give. The other columns do not change.
|
|
1557
|
+
|
|
1558
|
+
______________________________________________________________________
|
|
1559
|
+
|
|
1560
|
+
**Example 44 - All Diagnoses of Diabetes (ICD-9 250.01) within 30 days of an MI**
|
|
1561
|
+
|
|
1562
|
+
```JSON
|
|
1563
|
+
["during",{"left":["icd9","250.01"],"right":["time_window",["icd9","412"],{"start":"-30d","end":"30d"}]}]
|
|
1564
|
+
```
|
|
1565
|
+
|
|
1566
|
+

|
|
1567
|
+
|
|
1568
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1569
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1570
|
+
| 96 | 46366 | observations | condition_occurrence | 2008-09-13 | 2008-09-13 | 25001 |
|
|
1571
|
+
| 223 | 81480 | observations | condition_occurrence | 2008-07-03 | 2008-07-03 | 25001 |
|
|
1572
|
+
|
|
1573
|
+
______________________________________________________________________
|
|
1574
|
+
|
|
1575
|
+
______________________________________________________________________
|
|
1576
|
+
|
|
1577
|
+
**Example 45 - Shift the dates of all MIs back by 200 years**
|
|
1578
|
+
|
|
1579
|
+
```JSON
|
|
1580
|
+
["time_window",["icd9","412"],{"start":"-200y","end":"-200y"}]
|
|
1581
|
+
```
|
|
1582
|
+
|
|
1583
|
+

|
|
1584
|
+
|
|
1585
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1586
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1587
|
+
| 11 | 82614 | observations | condition_occurrence | 1808-10-29 | 1808-10-29 | 412 |
|
|
1588
|
+
| 17 | 27269 | observations | condition_occurrence | 1808-08-25 | 1808-08-25 | 412 |
|
|
1589
|
+
| 17 | 46165 | observations | condition_occurrence | 1809-04-30 | 1809-04-30 | 412 |
|
|
1590
|
+
| 38 | 40556 | observations | condition_occurrence | 1810-02-12 | 1810-02-12 | 412 |
|
|
1591
|
+
| 54 | 74606 | observations | condition_occurrence | 1808-06-05 | 1808-06-05 | 412 |
|
|
1592
|
+
| 60 | 986 | observations | condition_occurrence | 1809-07-19 | 1809-07-22 | 412 |
|
|
1593
|
+
| 66 | 16171 | observations | condition_occurrence | 1809-07-25 | 1809-07-25 | 412 |
|
|
1594
|
+
| 73 | 52843 | observations | condition_occurrence | 1808-11-16 | 1808-11-16 | 412 |
|
|
1595
|
+
| 77 | 58165 | observations | condition_occurrence | 1810-10-06 | 1810-10-06 | 412 |
|
|
1596
|
+
| 81 | 1405 | observations | condition_occurrence | 1809-01-28 | 1809-01-30 | 412 |
|
|
1597
|
+
|
|
1598
|
+
______________________________________________________________________
|
|
1599
|
+
|
|
1600
|
+
______________________________________________________________________
|
|
1601
|
+
|
|
1602
|
+
**Example 46 - Widen the dates of all MIs to run from 2 months and 2 days before the MI to 1 year and 3 days after it**
|
|
1603
|
+
|
|
1604
|
+
```JSON
|
|
1605
|
+
["time_window",["icd9","412"],{"start":"-2m-2d","end":"3d1y"}]
|
|
1606
|
+
```
|
|
1607
|
+
|
|
1608
|
+

|
|
1609
|
+
|
|
1610
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1611
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1612
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-08-27 | 2009-11-01 | 412 |
|
|
1613
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-06-23 | 2009-08-28 | 412 |
|
|
1614
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-02-26 | 2010-05-03 | 412 |
|
|
1615
|
+
| 38 | 40556 | observations | condition_occurrence | 2009-12-10 | 2011-02-15 | 412 |
|
|
1616
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-04-03 | 2009-06-08 | 412 |
|
|
1617
|
+
| 60 | 986 | observations | condition_occurrence | 2009-05-17 | 2010-07-25 | 412 |
|
|
1618
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-05-23 | 2010-07-28 | 412 |
|
|
1619
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-09-14 | 2009-11-19 | 412 |
|
|
1620
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-08-04 | 2011-10-09 | 412 |
|
|
1621
|
+
| 81 | 1405 | observations | condition_occurrence | 2008-11-26 | 2010-02-02 | 412 |
|
|
1622
|
+
|
|
1623
|
+
______________________________________________________________________
|
|
1624
|
+
|
|
1625
|
+
______________________________________________________________________
|
|
1626
|
+
|
|
1627
|
+
**Example 47 - Collapse the date range of all MIs to the start date: leave start_date unchanged and set end_date to start_date**
|
|
1628
|
+
|
|
1629
|
+
```JSON
|
|
1630
|
+
["time_window",["icd9","412"],{"start":"","end":"start"}]
|
|
1631
|
+
```
|
|
1632
|
+
|
|
1633
|
+

|
|
1634
|
+
|
|
1635
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1636
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1637
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
1638
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1639
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1640
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1641
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1642
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-19 | 412 |
|
|
1643
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1644
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
1645
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
1646
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-28 | 412 |
|
|
1647
|
+
|
|
1648
|
+
______________________________________________________________________
|
|
1649
|
+
|
|
1650
|
+
______________________________________________________________________
|
|
1651
|
+
|
|
1652
|
+
**Example 48 - Nonsensical, but allowed: swap the start_date and end_date**
|
|
1653
|
+
|
|
1654
|
+
```JSON
|
|
1655
|
+
["time_window",["icd9","412"],{"start":"end","end":"start"}]
|
|
1656
|
+
```
|
|
1657
|
+
|
|
1658
|
+

|
|
1659
|
+
|
|
1660
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1661
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1662
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
1663
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
1664
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
1665
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
1666
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
1667
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-22 | 2009-07-19 | 412 |
|
|
1668
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
1669
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
1670
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
1671
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-30 | 2009-01-28 | 412 |
|
|
1672
|
+
|
|
1673
|
+
______________________________________________________________________
|
|
1674
|
+
|
|
1675
|
+
More examples of date adjustments:
|
|
1676
|
+
|
|
1677
|
+
- `30d` is 30 days.
|
|
1678
|
+
- `20` is 20 days.
|
|
1679
|
+
- `d` is 1 day.
|
|
1680
|
+
- `1y` is 1 year.
|
|
1681
|
+
- `-2y10m-3d` is back 2 years, forward 10 months, and back 3 days.
|
|
1682
|
+
|
|
1683
|
+
### `Trim Date Start` Operator
|
|
1684
|
+
|
|
1685
|
+
For each patient, moves the start_date of each left-hand record forward to the end_date of the right-hand record with the latest start_date. A left-hand record that ends before that end_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged.
|
|
1686
|
+
|
|
1687
|
+
- **Operands:**
|
|
1688
|
+
- `left` (required): The left-hand stream. The operator moves the start_date of these records and passes them on.
|
|
1689
|
+
- `right` (required): The right-hand stream. For each patient, the end_date of the record with the latest start_date sets the earliest start_date of the left-hand records. These records are not passed on.
|
|
1690
|
+
- **Options:**
|
|
1691
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is added to the end_date of the right-hand record before the comparison. The adjusted date is the one that moves the start_date of the left-hand record.
|
|
1692
|
+
- **Returns:** The left-hand records, each with its start_date set to the later of its own start_date and the end_date of the patient's right-hand record. Left-hand records that end before that end_date are not returned. The right-hand records are not returned.
|
|
1693
|
+
|
|
1694
|
+
______________________________________________________________________
|
|
1695
|
+
|
|
1696
|
+
**Example 49 - Each patient's span from birth to age 50, trimmed to start no earlier than 1980-01-01 (patients who were 50 before 1980 are dropped)**
|
|
1697
|
+
|
|
1698
|
+
```JSON
|
|
1699
|
+
["trim_date_start",{"left":["time_window",["person"],{"end":"+50y"}],"right":["date_range",{"start":"1980-01-01","end":"1980-01-01"}]}]
|
|
1700
|
+
```
|
|
1701
|
+
|
|
1702
|
+

|
|
1703
|
+
|
|
1704
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1705
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
1706
|
+
| 2 | 2 | patients | person | 1980-01-01 | 1993-01-01 | 00016F745862898F |
|
|
1707
|
+
| 3 | 3 | patients | person | 1980-01-01 | 1986-09-01 | 0001FDD721E223DC |
|
|
1708
|
+
| 4 | 4 | patients | person | 1980-01-01 | 1991-06-01 | 00021CA6FF03E670 |
|
|
1709
|
+
| 5 | 5 | patients | person | 1980-01-01 | 1986-08-01 | 00024B3D2352D2D0 |
|
|
1710
|
+
| 6 | 6 | patients | person | 1980-01-01 | 1993-10-01 | 0002DAE1C81CC70D |
|
|
1711
|
+
| 8 | 8 | patients | person | 1980-01-01 | 1985-09-01 | 000308435E3E5B76 |
|
|
1712
|
+
| 9 | 9 | patients | person | 1980-01-01 | 2026-09-01 | 000345A39D4157C9 |
|
|
1713
|
+
| 10 | 10 | patients | person | 1980-01-01 | 1988-10-01 | 00036A21B65B0206 |
|
|
1714
|
+
| 11 | 11 | patients | person | 1980-01-01 | 1984-02-01 | 000489E7EAAD463F |
|
|
1715
|
+
| 13 | 13 | patients | person | 1980-01-01 | 1986-07-01 | 0004F0ABD505251D |
|
|
1716
|
+
|
|
1717
|
+
______________________________________________________________________
|
|
1718
|
+
|
|
1719
|
+
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's records. There are four cases:
|
|
1720
|
+
|
|
1721
|
+
- The patient has no RHR.
|
|
1722
|
+
- The LHR passes through unchanged.
|
|
1723
|
+
- The RHR end_date is before the LHR start_date.
|
|
1724
|
+
- The LHR passes through unchanged.
|
|
1725
|
+
- The RHR end_date is after the LHR end_date.
|
|
1726
|
+
- The LHR is discarded.
|
|
1727
|
+
- The RHR end_date is between the LHR start_date and end_date.
|
|
1728
|
+
- The LHR's start_date becomes the RHR's end_date.
|
|
1729
|
+
|
|
1730
|
+
### `Trim Date End` Operator
|
|
1731
|
+
|
|
1732
|
+
For each patient, moves the end_date of each left-hand record back to the start_date of the right-hand record with the earliest start_date. A left-hand record that starts after that start_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged.
|
|
1733
|
+
|
|
1734
|
+
- **Operands:**
|
|
1735
|
+
- `left` (required): The left-hand stream. The operator moves the end_date of these records and passes them on.
|
|
1736
|
+
- `right` (required): The right-hand stream. For each patient, the earliest start_date of these records sets the latest end_date of the left-hand records. These records are not passed on.
|
|
1737
|
+
- **Options:**
|
|
1738
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is subtracted from the start_date of the right-hand record before the comparison. The adjusted date is the one that moves the end_date of the left-hand record.
|
|
1739
|
+
- **Returns:** The left-hand records, each with its end_date set to the earlier of its own end_date and the start_date of the patient's right-hand record. Left-hand records that start after that start_date are not returned. The right-hand records are not returned.
|
|
1740
|
+
|
|
1741
|
+
______________________________________________________________________
|
|
1742
|
+
|
|
1743
|
+
**Example 50 - Each patient's span from birth to age 50 or to 1970-01-01, whichever is earlier (patients born after 1970 are dropped)**
|
|
1744
|
+
|
|
1745
|
+
```JSON
|
|
1746
|
+
["trim_date_end",{"left":["time_window",["person"],{"end":"+50y"}],"right":["date_range",{"start":"1970-01-01","end":"1970-01-01"}]}]
|
|
1747
|
+
```
|
|
1748
|
+
|
|
1749
|
+

|
|
1750
|
+
|
|
1751
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1752
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
1753
|
+
| 1 | 1 | patients | person | 1923-05-01 | 1970-01-01 | 00013D2EFD8E45D1 |
|
|
1754
|
+
| 2 | 2 | patients | person | 1943-01-01 | 1970-01-01 | 00016F745862898F |
|
|
1755
|
+
| 3 | 3 | patients | person | 1936-09-01 | 1970-01-01 | 0001FDD721E223DC |
|
|
1756
|
+
| 4 | 4 | patients | person | 1941-06-01 | 1970-01-01 | 00021CA6FF03E670 |
|
|
1757
|
+
| 5 | 5 | patients | person | 1936-08-01 | 1970-01-01 | 00024B3D2352D2D0 |
|
|
1758
|
+
| 6 | 6 | patients | person | 1943-10-01 | 1970-01-01 | 0002DAE1C81CC70D |
|
|
1759
|
+
| 7 | 7 | patients | person | 1922-07-01 | 1970-01-01 | 0002F28CE057345B |
|
|
1760
|
+
| 8 | 8 | patients | person | 1935-09-01 | 1970-01-01 | 000308435E3E5B76 |
|
|
1761
|
+
| 10 | 10 | patients | person | 1938-10-01 | 1970-01-01 | 00036A21B65B0206 |
|
|
1762
|
+
| 11 | 11 | patients | person | 1934-02-01 | 1970-01-01 | 000489E7EAAD463F |
|
|
1763
|
+
|
|
1764
|
+
______________________________________________________________________
|
|
1765
|
+
|
|
1766
|
+
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's records. There are four cases:
|
|
1767
|
+
|
|
1768
|
+
- The patient has no RHR.
|
|
1769
|
+
- The LHR passes through unchanged.
|
|
1770
|
+
- The RHR start_date is after the LHR end_date.
|
|
1771
|
+
- The LHR passes through unchanged.
|
|
1772
|
+
- The RHR start_date is before the LHR start_date.
|
|
1773
|
+
- The LHR is discarded.
|
|
1774
|
+
- The RHR start_date is between the LHR start_date and end_date.
|
|
1775
|
+
- The LHR's end_date becomes the RHR's start_date.
|
|
1776
|
+
|
|
1777
|
+
### The `join_columns`, `compare_all` and `uuid` Options
|
|
1778
|
+
|
|
1779
|
+
Systems that build statements, such as Jigsaw's export engine, give `trim_date_start` and `trim_date_end` two more options. Jigsaw's diagram editor does not show them.
|
|
1780
|
+
|
|
1781
|
+
- `join_columns`: a list of column names. Each LHR is compared only with the RHR of the same patient that have the same values in these columns. Without `compare_all`, the operator uses the first RHR (`trim_date_end`) or the last RHR (`trim_date_start`) of each patient and each set of `join_columns` values. The RHR are ordered by start_date and then by `criterion_id`.
|
|
1782
|
+
- `compare_all` (boolean): compare each LHR with every RHR of the patient (and of the `join_columns` values), not only with the first or the last RHR. Each LHR comes out at most once.
|
|
1783
|
+
- `trim_date_end` moves the end_date of the LHR back to the earliest RHR start_date that is on or after the LHR start_date.
|
|
1784
|
+
- `trim_date_start` moves the start_date of the LHR forward to the latest RHR end_date that is on or before the LHR end_date.
|
|
1785
|
+
- The `within` option moves each RHR date before the comparison, as above.
|
|
1786
|
+
- An LHR is discarded only when it has RHR (of its patient and its `join_columns` values) and none of them qualifies. An LHR without RHR passes through unchanged.
|
|
1787
|
+
|
|
1788
|
+
These systems often join on the column `uuid`. A selection operator, such as a vocabulary operator, takes the option `uuid` (`{"uuid": true}`), which adds the column `uuid` to its records. The uuid is the `person_id`, `criterion_id`, `criterion_table`, start_date and end_date of the record, joined with `/`, such as `17/27269/observations/2008-08-25/2008-08-25`. An empty part is an empty string, so a record without an end_date has a uuid such as `1/2/observations/2008-03-10/`. The uuid keeps its value when a later operator, such as `time_window`, moves the dates. So two streams made from the same records can be joined record by record.
|
|
1789
|
+
|
|
1790
|
+
This example makes a one-year window after each MI. Then it trims each window to end 30 days after its own MI. The `join_columns` option makes sure that no other MI of the same patient trims the window:
|
|
1791
|
+
|
|
1792
|
+
______________________________________________________________________
|
|
1793
|
+
|
|
1794
|
+
**Example 51 - Each MI's following year, trimmed to end 30 days after that same MI**
|
|
1795
|
+
|
|
1796
|
+
```JSON
|
|
1797
|
+
["trim_date_end",{"left":["time_window",["icd9","412",{"uuid":true}],{"end":"+1y"}],"right":["time_window",["icd9","412",{"uuid":true}],{"start":"+30d","end":"+30d"}],"join_columns":["uuid"],"compare_all":true}]
|
|
1798
|
+
```
|
|
1799
|
+
|
|
1800
|
+

|
|
1801
|
+
|
|
1802
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1803
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1804
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-11-28 | 412 |
|
|
1805
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-09-24 | 412 |
|
|
1806
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-05-30 | 412 |
|
|
1807
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-03-14 | 412 |
|
|
1808
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-07-05 | 412 |
|
|
1809
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-08-18 | 412 |
|
|
1810
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-08-24 | 412 |
|
|
1811
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-12-16 | 412 |
|
|
1812
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-11-05 | 412 |
|
|
1813
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-02-27 | 412 |
|
|
1814
|
+
|
|
1815
|
+
______________________________________________________________________
|
|
1816
|
+
|
|
1817
|
+
## Temporal Comparison Operator Options
|
|
1818
|
+
|
|
1819
|
+
It can be hard to reason with [`time_window`](#time-window-operator) when you use temporal comparison operators. The options `within` and `at_least` describe some common temporal relationships more directly. Every temporal comparison operator except `equal` takes `within`, and `before` and `after` also take `at_least`. The reference section of each operator says what the options do for it.
|
|
1820
|
+
|
|
1821
|
+
When `before` or `after` has the `within` option, it compares each LHR with every RHR of the patient, not only with the patient's last or earliest RHR.
|
|
1822
|
+
|
|
1823
|
+
Walk-through of the second example of [`after`](#after-operator), which finds the prescriptions of interest within three days after an office visit:
|
|
1824
|
+
|
|
1825
|
+
- Select some prescriptions of interest as the LHR.
|
|
1826
|
+
- Select office visits as the RHR.
|
|
1827
|
+
- Compare each LHR with each RHR of the same patient. Keep the LHR whose start_date is after the RHR end_date, and no more than 3 days after it.
|
|
1828
|
+
|
|
1829
|
+
Walk-through of the third example of [`before`](#before-operator), which asks whether anyone dies within a year of a heart attack:
|
|
1830
|
+
|
|
1831
|
+
- Select heart attack diagnoses as the LHR.
|
|
1832
|
+
- Select death records as the RHR.
|
|
1833
|
+
- Keep the LHR whose end_date is before the date of death, and no more than 1 year before it.
|
|
1834
|
+
|
|
1835
|
+
The third example of [`after`](#after-operator) shows how `at_least` works.
|
|
1836
|
+
|
|
1837
|
+
### Temporal Operators and Person Streams
|
|
1838
|
+
|
|
1839
|
+
The start and end dates of a person record are the patient's birth date. So a person stream is rarely useful as the left-hand stream of a temporal operator. But it is useful as the right-hand stream. If we move the birth date with `time_window`, we can filter by the patient's age. The last example of [`after`](#after-operator) selects the MIs that occurred after a male patient's 50th birthday.
|
|
1840
|
+
|
|
1841
|
+
## `Episode` Operator
|
|
1842
|
+
|
|
1843
|
+
Merges each patient's records into episodes: spans of time in which the records are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug.
|
|
1844
|
+
|
|
1845
|
+
- **Upstream:** one operator.
|
|
1846
|
+
- **Options:**
|
|
1847
|
+
- `gap_of` (integer): The largest gap in days that stays in the same episode. The operator orders each patient's records by start_date and then by end_date. A record stays in the current episode when its start_date is no more than this many days after the latest end_date of the earlier records, not only of the record just before. Otherwise it starts a new episode. A record without an end_date counts as ending on its start_date. The default is 0.
|
|
1848
|
+
- **Returns:** One record for each episode. Its start_date is the earliest start_date in the episode, and its end_date is the latest end_date. Its criterion_table and criterion_domain are episode, and its criterion_id is 0, because an episode merges several records.
|
|
1849
|
+
|
|
1850
|
+
______________________________________________________________________
|
|
1851
|
+
|
|
1852
|
+
**Example 52 - Episodes of diabetes diagnoses**
|
|
1853
|
+
|
|
1854
|
+
```JSON
|
|
1855
|
+
["episode",["icd9","250.00"],{"gap_of":"90"}]
|
|
1856
|
+
```
|
|
1857
|
+
|
|
1858
|
+

|
|
1859
|
+
|
|
1860
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1861
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
|
|
1862
|
+
| 2 | 0 | episode | episode | 2009-05-11 | 2009-07-31 | |
|
|
1863
|
+
| 3 | 0 | episode | episode | 2009-11-17 | 2009-11-17 | |
|
|
1864
|
+
| 5 | 0 | episode | episode | 2009-06-28 | 2009-06-28 | |
|
|
1865
|
+
| 7 | 0 | episode | episode | 2008-04-23 | 2008-06-09 | |
|
|
1866
|
+
| 7 | 0 | episode | episode | 2008-10-02 | 2008-11-30 | |
|
|
1867
|
+
| 7 | 0 | episode | episode | 2009-07-12 | 2009-09-13 | |
|
|
1868
|
+
| 7 | 0 | episode | episode | 2010-04-22 | 2010-07-18 | |
|
|
1869
|
+
| 8 | 0 | episode | episode | 2008-10-17 | 2008-10-17 | |
|
|
1870
|
+
| 8 | 0 | episode | episode | 2009-03-01 | 2009-03-03 | |
|
|
1871
|
+
| 8 | 0 | episode | episode | 2009-07-16 | 2009-07-16 | |
|
|
1872
|
+
|
|
1873
|
+
______________________________________________________________________
|
|
1874
|
+
|
|
1875
|
+
A long record keeps the episode open: the records 2020-01-01 to 2020-03-01 and 2020-01-05 to 2020-01-10 and 2020-02-15 to 2020-02-20 make one episode with `gap_of` 0, because the first record ends after the third record starts. An episode stream is often most useful as the right-hand stream of a [temporal operator](#temporal-comparison-operators).
|
|
1876
|
+
|
|
1877
|
+
## Inline-Filter Operators
|
|
1878
|
+
|
|
1879
|
+
These operators filter one incoming stream and pass along the records that remain.
|
|
1880
|
+
|
|
1881
|
+
### `Place of Service Filter` Operator
|
|
1882
|
+
|
|
1883
|
+
Passes along the records whose place of service is one of the given CMS Place of Service codes. A record that has no place of service is not passed along.
|
|
1884
|
+
|
|
1885
|
+
- **Upstream:** one operator.
|
|
1886
|
+
- **Arguments:**
|
|
1887
|
+
- `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a record whose place of service is any code of that vocabulary. An unknown code makes the statement invalid.
|
|
1888
|
+
- **Returns:** The upstream records whose place of service is one of the given codes.
|
|
1889
|
+
|
|
1890
|
+
______________________________________________________________________
|
|
1891
|
+
|
|
1892
|
+
**Example 53 - Diabetes diagnoses (ICD-9 250.00) with the place of service outpatient hospital (22)**
|
|
1893
|
+
|
|
1894
|
+
```JSON
|
|
1895
|
+
["place_of_service_filter",22,["icd9","250.00"]]
|
|
1896
|
+
```
|
|
1897
|
+
|
|
1898
|
+

|
|
1899
|
+
|
|
1900
|
+
No results.
|
|
1901
|
+
|
|
1902
|
+
______________________________________________________________________
|
|
1903
|
+
|
|
1904
|
+
Claims often report the place of service of an event with a CMS place of service code. Our test data has no place of service information, so the example returns no records.
|
|
1905
|
+
|
|
1906
|
+
### `Provenance` Operator
|
|
1907
|
+
|
|
1908
|
+
Passes along the records whose provenance matches at least one of the given keywords. The provenance of a record says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary.
|
|
1909
|
+
|
|
1910
|
+
- **Upstream:** one operator.
|
|
1911
|
+
- **Arguments:**
|
|
1912
|
+
- `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a record came from. A code keyword, such as primary or first, says where the code was on the claim. A plain keyword matches a record when its file provenance or its code provenance matches it. A keyword also matches the provenances under it, so inpatient includes snf. A keyword of the form file:code, such as inpatient:primary, needs both the file and the code provenance to match. The keyword * matches a record that has any provenance. A statement with no keyword, or with an unknown keyword, is invalid.
|
|
1913
|
+
- **Returns:** The upstream records whose provenance matches at least one of the keywords.
|
|
1914
|
+
|
|
1915
|
+
______________________________________________________________________
|
|
1916
|
+
|
|
1917
|
+
**Example 54 - Diabetes diagnoses (ICD-9 250.00) from inpatient files**
|
|
1918
|
+
|
|
1919
|
+
```JSON
|
|
1920
|
+
["provenance","inpatient",["icd9","250.00"]]
|
|
1921
|
+
```
|
|
1922
|
+
|
|
1923
|
+

|
|
1924
|
+
|
|
1925
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1926
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1927
|
+
| 17 | 929 | observations | condition_occurrence | 2010-05-22 | 2010-06-12 | 25000 |
|
|
1928
|
+
| 21 | 1024 | observations | condition_occurrence | 2008-10-15 | 2008-10-21 | 25000 |
|
|
1929
|
+
| 28 | 1446 | observations | condition_occurrence | 2008-04-21 | 2008-04-26 | 25000 |
|
|
1930
|
+
| 32 | 257 | observations | condition_occurrence | 2010-08-30 | 2010-09-07 | 25000 |
|
|
1931
|
+
| 37 | 310 | observations | condition_occurrence | 2009-01-19 | 2009-01-20 | 25000 |
|
|
1932
|
+
| 60 | 983 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 25000 |
|
|
1933
|
+
| 66 | 1146 | observations | condition_occurrence | 2008-12-08 | 2008-12-13 | 25000 |
|
|
1934
|
+
| 66 | 1507 | observations | condition_occurrence | 2009-12-29 | 2009-12-31 | 25000 |
|
|
1935
|
+
| 68 | 19 | observations | condition_occurrence | 2008-04-19 | 2008-04-23 | 25000 |
|
|
1936
|
+
| 68 | 1596 | observations | condition_occurrence | 2008-11-10 | 2008-11-11 | 25000 |
|
|
1937
|
+
|
|
1938
|
+
______________________________________________________________________
|
|
1939
|
+
|
|
1940
|
+
Provenance is information about the origin of claims data. It is sometimes an important factor in research algorithms. For example, authors often describe "inpatient records", by which they mean claims from the inpatient file of a claims dataset. The [GDM](https://github.com/outcomesinsights/generalized_data_model#contexts) records the provenance of the data during ETL.
|
|
1941
|
+
|
|
1942
|
+
These are the keywords of each kind:
|
|
1943
|
+
|
|
1944
|
+
- File keywords: `inpatient`, `outpatient`, `carrier_claim`, `claim`, `dme`, `hha`, `hospice`, `snf`, `prescription_drug` and `demographics`.
|
|
1945
|
+
- Code keywords: `primary`, `primary_or_first`, `first`, `second`, `admitting`, `discharge`, `problem_list`, `chief_complaint` and `unspecified`.
|
|
1946
|
+
|
|
1947
|
+
Besides `inpatient`, which includes `snf`, `outpatient` includes `hha` and `hospice`, and `claim` includes every kind of claim file.
|
|
1948
|
+
|
|
1949
|
+
### `Provider Filter` Operator
|
|
1950
|
+
|
|
1951
|
+
Passes along the records whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value.
|
|
1952
|
+
|
|
1953
|
+
- **Upstream:** one operator.
|
|
1954
|
+
- **Options:**
|
|
1955
|
+
- `specialties` (string, required): A comma-separated list of specialty concept IDs, such as 38003868 (Endocrinology, Diabetes & Metabolism Physician), or * for any specialty. A provider matches when the specialty recorded for the provider on the claim, or the provider's own specialty, is in the list.
|
|
1956
|
+
- `roles` (string, required): A comma-separated list of role concept IDs of the GDM_ROLE vocabulary, such as 100000067 (Attending physician) or 100176860 (Billing provider), or * for any role.
|
|
1957
|
+
- **Returns:** The upstream records that have a provider who matches both the specialties and the roles. A record can have more than one provider, so a record comes out once for each provider who matches. Each record gets the columns provider_id, context_specialty_concept_id, practitioner_specialty_concept_id and role_type_concept_id.
|
|
1958
|
+
|
|
1959
|
+
______________________________________________________________________
|
|
1960
|
+
|
|
1961
|
+
**Example 55 - Diabetes diagnoses with an endocrinologist as the attending physician**
|
|
1962
|
+
|
|
1963
|
+
```JSON
|
|
1964
|
+
["provider_filter",["icd9","250.00"],{"specialties":"38003868","roles":"100000067"}]
|
|
1965
|
+
```
|
|
1966
|
+
|
|
1967
|
+

|
|
1968
|
+
|
|
1969
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1970
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
1971
|
+
| 17 | 18184 | observations | condition_occurrence | 2010-08-22 | 2010-08-22 | 25000 |
|
|
1972
|
+
|
|
1973
|
+
______________________________________________________________________
|
|
1974
|
+
|
|
1975
|
+
These are the roles of the GDM_ROLE vocabulary: `100000066` (Performing physician), `100000067` (Attending physician), `100000068` (Operating physician), `100000069` (Other physician), `100176860` (Billing provider), `100176861` (Rendering provider) and `100176862` (Prescribing provider). The operator takes concept IDs, not names of specialties and roles.
|
|
1976
|
+
|
|
1977
|
+
Use `*` to skip one half of the filter. This example keeps the diabetes diagnoses that had an attending physician of any specialty. The results show every column, so you can see the provider, both specialty columns, and the role that matched:
|
|
1978
|
+
|
|
1979
|
+
______________________________________________________________________
|
|
1980
|
+
|
|
1981
|
+
**Example 56 - Diabetes diagnoses with an attending physician of any specialty**
|
|
1982
|
+
|
|
1983
|
+
```JSON
|
|
1984
|
+
["provider_filter",["icd9","250.00"],{"specialties":"*","roles":"100000067"}]
|
|
1985
|
+
```
|
|
1986
|
+
|
|
1987
|
+

|
|
1988
|
+
|
|
1989
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value | source_vocabulary_id | label | column_family | provider_id | context_specialty_concept_id | practitioner_specialty_concept_id | role_type_concept_id |
|
|
1990
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ | -------------------- | ----- | ------------- | ----------- | ---------------------------- | --------------------------------- | -------------------- |
|
|
1991
|
+
| 7 | 15942 | observations | condition_occurrence | 2008-04-23 | 2008-04-23 | 25000 | ICD9CM | | default | 8406 | | 38003694 | 100000067 |
|
|
1992
|
+
| 9 | 24929 | observations | condition_occurrence | 2010-08-01 | 2010-08-01 | 25000 | ICD9CM | | default | 2221 | 38004018 | 38003931 | 100000067 |
|
|
1993
|
+
| 12 | 15443 | observations | condition_occurrence | 2009-04-20 | 2009-04-20 | 25000 | ICD9CM | | default | 3246 | | 38004431 | 100000067 |
|
|
1994
|
+
| 13 | 19968 | observations | condition_occurrence | 2008-03-05 | 2008-03-05 | 25000 | ICD9CM | | default | 9453 | 38003793 | | 100000067 |
|
|
1995
|
+
| 13 | 20654 | observations | condition_occurrence | 2009-01-16 | 2009-01-16 | 25000 | ICD9CM | | default | 5160 | 38004263 | | 100000067 |
|
|
1996
|
+
| 14 | 24380 | observations | condition_occurrence | 2009-09-13 | 2009-09-13 | 25000 | ICD9CM | | default | 7875 | | | 100000067 |
|
|
1997
|
+
| 17 | 27268 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 25000 | ICD9CM | | default | 7256 | | | 100000067 |
|
|
1998
|
+
| 17 | 23309 | observations | condition_occurrence | 2009-07-19 | 2009-07-19 | 25000 | ICD9CM | | default | 1413 | | | 100000067 |
|
|
1999
|
+
| 17 | 28431 | observations | condition_occurrence | 2009-11-28 | 2009-11-28 | 25000 | ICD9CM | | default | 5747 | 38004113 | | 100000067 |
|
|
2000
|
+
| 17 | 929 | observations | condition_occurrence | 2010-05-22 | 2010-06-12 | 25000 | ICD9CM | | default | 1976 | | 38003861 | 100000067 |
|
|
2001
|
+
|
|
2002
|
+
______________________________________________________________________
|
|
2003
|
+
|
|
2004
|
+
### `One In Two Out` Operator
|
|
2005
|
+
|
|
2006
|
+
For each person, passes along the earliest confirmed condition record. One inpatient record confirms a condition, and so do two outpatient records whose start dates are a given gap apart.
|
|
2007
|
+
|
|
2008
|
+
- **Upstream:** one operator.
|
|
2009
|
+
- **Options:**
|
|
2010
|
+
- `inpatient_length_of_stay` (integer): An inpatient stay must be longer than this many days: an inpatient record counts only when its discharge date is more than this many days after its admission date. 0 or no value means no length-of-stay check.
|
|
2011
|
+
- `outpatient_minimum_gap` (string): A date adjustment, such as 30d. The confirming outpatient record must start at least this long after the initial outpatient record. Without this option, the gap is 30 days.
|
|
2012
|
+
- `outpatient_maximum_gap` (string): A date adjustment, such as 365d. The confirming outpatient record must start no more than this long after the initial outpatient record. Without this option, there is no maximum gap.
|
|
2013
|
+
- `outpatient_event_to_return` (string, one of `Initial Event`, `Confirming Event`): Which outpatient record the operator passes on when two outpatient records confirm the condition: the initial record or the confirming record. Without this option, or with any value other than Initial Event, it passes on the confirming record.
|
|
2014
|
+
- **Returns:** One record per person: the earliest confirmed condition record. A returned inpatient record has the admission date as its start_date and the discharge date as its end_date, when the claim has them. Only condition records with inpatient, outpatient or carrier_claim provenance can be confirmed.
|
|
2015
|
+
|
|
2016
|
+
______________________________________________________________________
|
|
2017
|
+
|
|
2018
|
+
**Example 57 - First diabetes diagnosis, either found once in an inpatient record, or twice in outpatient records at least 30 days apart**
|
|
2019
|
+
|
|
2020
|
+
```JSON
|
|
2021
|
+
["one_in_two_out",["icd9","250.00"],{"outpatient_minimum_gap":"30d","outpatient_event_to_return":"Initial Event"}]
|
|
2022
|
+
```
|
|
2023
|
+
|
|
2024
|
+

|
|
2025
|
+
|
|
2026
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2027
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2028
|
+
| 2 | 89752 | observations | condition_occurrence | 2009-05-11 | 2009-05-11 | 25000 |
|
|
2029
|
+
| 7 | 15942 | observations | condition_occurrence | 2008-04-23 | 2008-04-23 | 25000 |
|
|
2030
|
+
| 8 | 95135 | observations | condition_occurrence | 2008-10-17 | 2008-10-17 | 25000 |
|
|
2031
|
+
| 9 | 24929 | observations | condition_occurrence | 2010-08-01 | 2010-08-01 | 25000 |
|
|
2032
|
+
| 11 | 73103 | observations | condition_occurrence | 2008-06-17 | 2008-06-17 | 25000 |
|
|
2033
|
+
| 12 | 15443 | observations | condition_occurrence | 2009-04-20 | 2009-04-20 | 25000 |
|
|
2034
|
+
| 13 | 19968 | observations | condition_occurrence | 2008-03-05 | 2008-03-05 | 25000 |
|
|
2035
|
+
| 14 | 77991 | observations | condition_occurrence | 2009-06-05 | 2009-06-05 | 25000 |
|
|
2036
|
+
| 15 | 47045 | observations | condition_occurrence | 2008-09-24 | 2008-09-24 | 25000 |
|
|
2037
|
+
| 17 | 83187 | observations | condition_occurrence | 2008-04-14 | 2008-04-14 | 25000 |
|
|
2038
|
+
|
|
2039
|
+
______________________________________________________________________
|
|
2040
|
+
|
|
2041
|
+
The operator decides between inpatient and outpatient by the provenance of each record (see [`provenance`](#provenance-operator)):
|
|
2042
|
+
|
|
2043
|
+
- A record with `inpatient` provenance (which includes SNF) is an inpatient record.
|
|
2044
|
+
- A record with `outpatient` or `carrier_claim` provenance is an outpatient record.
|
|
2045
|
+
- Other records, and records whose type is not `condition_occurrence`, are removed.
|
|
2046
|
+
|
|
2047
|
+
The place of service of a record does not matter. The two outpatient records need not have the same code. Any two records from the upstream stream can confirm each other. For outpatient records, the operator compares start_dates.
|
|
2048
|
+
|
|
2049
|
+
More about the options:
|
|
2050
|
+
|
|
2051
|
+
- `outpatient_maximum_gap` helps with acute diseases. For example, to confirm the flu, set a maximum gap so that the two diagnoses come from the same illness. Jigsaw fills in 365 days.
|
|
2052
|
+
- `outpatient_event_to_return`: use `Initial Event` in most cases, for example when the algorithm records an exposure of interest or an outcome. Use `Confirming Event` when the algorithm finds an index event. The initial event would introduce immortal time bias. Jigsaw fills in `Initial Event`, but a statement that omits the option gets `Confirming Event`. So always set this option.
|
|
2053
|
+
|
|
2054
|
+
### `Person Filter` Operator
|
|
2055
|
+
|
|
2056
|
+
Keeps each left-hand record whose patient has at least one record in the right-hand stream. The type and the dates of the right-hand records do not matter.
|
|
2057
|
+
|
|
2058
|
+
- **Operands:**
|
|
2059
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
2060
|
+
- `right` (required): The right-hand stream. The operator keeps the left-hand records of each patient who has a record in this stream, and does not pass these records on.
|
|
2061
|
+
- **Returns:** The left-hand records of every type whose patient has at least one right-hand record. The right-hand records are not returned.
|
|
2062
|
+
|
|
2063
|
+
______________________________________________________________________
|
|
2064
|
+
|
|
2065
|
+
**Example 58 - All MI Conditions for people who are male**
|
|
2066
|
+
|
|
2067
|
+
```JSON
|
|
2068
|
+
["person_filter",{"left":["icd9","412"],"right":["gender","Male"]}]
|
|
2069
|
+
```
|
|
2070
|
+
|
|
2071
|
+

|
|
2072
|
+
|
|
2073
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2074
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2075
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2076
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2077
|
+
| 93 | 82295 | observations | condition_occurrence | 2009-11-08 | 2009-11-08 | 412 |
|
|
2078
|
+
| 96 | 20748 | observations | condition_occurrence | 2008-09-21 | 2008-09-24 | 412 |
|
|
2079
|
+
| 109 | 89334 | observations | condition_occurrence | 2010-07-06 | 2010-07-06 | 412 |
|
|
2080
|
+
| 110 | 62592 | observations | condition_occurrence | 2010-06-27 | 2010-06-27 | 412 |
|
|
2081
|
+
| 131 | 172 | observations | condition_occurrence | 2008-03-22 | 2008-03-23 | 412 |
|
|
2082
|
+
| 149 | 28518 | observations | condition_occurrence | 2008-04-07 | 2008-04-07 | 412 |
|
|
2083
|
+
| 152 | 82988 | observations | condition_occurrence | 2008-08-16 | 2008-08-16 | 412 |
|
|
2084
|
+
| 152 | 84298 | observations | condition_occurrence | 2010-11-22 | 2010-11-22 | 412 |
|
|
2085
|
+
|
|
2086
|
+
______________________________________________________________________
|
|
2087
|
+
|
|
2088
|
+
______________________________________________________________________
|
|
2089
|
+
|
|
2090
|
+
**Example 59 - All MI Conditions for people who had an office visit at some point in the data**
|
|
2091
|
+
|
|
2092
|
+
```JSON
|
|
2093
|
+
["person_filter",{"left":["icd9","412"],"right":["cpt","99214"]}]
|
|
2094
|
+
```
|
|
2095
|
+
|
|
2096
|
+

|
|
2097
|
+
|
|
2098
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2099
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2100
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2101
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2102
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
2103
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2104
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2105
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2106
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2107
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2108
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
2109
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2110
|
+
|
|
2111
|
+
______________________________________________________________________
|
|
2112
|
+
|
|
2113
|
+
______________________________________________________________________
|
|
2114
|
+
|
|
2115
|
+
**Example 60 - The same result as the example above, with the office visits converted to patients first**
|
|
2116
|
+
|
|
2117
|
+
```JSON
|
|
2118
|
+
["person_filter",{"left":["icd9","412"],"right":["person",["cpt","99214"]]}]
|
|
2119
|
+
```
|
|
2120
|
+
|
|
2121
|
+

|
|
2122
|
+
|
|
2123
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2124
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2125
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2126
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2127
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
2128
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2129
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2130
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2131
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2132
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2133
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
2134
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2135
|
+
|
|
2136
|
+
______________________________________________________________________
|
|
2137
|
+
|
|
2138
|
+
______________________________________________________________________
|
|
2139
|
+
|
|
2140
|
+
**Example 61 - All MI Conditions for people who are Male OR had an office visit at some point in the data**
|
|
2141
|
+
|
|
2142
|
+
```JSON
|
|
2143
|
+
["person_filter",{"left":["icd9","412"],"right":["union",["cpt","99214"],["gender","Male"]]}]
|
|
2144
|
+
```
|
|
2145
|
+
|
|
2146
|
+

|
|
2147
|
+
|
|
2148
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2149
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2150
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2151
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2152
|
+
| 17 | 46165 | observations | condition_occurrence | 2009-04-30 | 2009-04-30 | 412 |
|
|
2153
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2154
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2155
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2156
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2157
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2158
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
2159
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2160
|
+
|
|
2161
|
+
______________________________________________________________________
|
|
2162
|
+
|
|
2163
|
+
______________________________________________________________________
|
|
2164
|
+
|
|
2165
|
+
**Example 62 - Yields two types: all MI Conditions for people who are Male, and all office visit Procedures for people who are Male**
|
|
2166
|
+
|
|
2167
|
+
```JSON
|
|
2168
|
+
["person_filter",{"left":["union",["icd9","412"],["cpt","99214"]],"right":["gender","Male"]}]
|
|
2169
|
+
```
|
|
2170
|
+
|
|
2171
|
+

|
|
2172
|
+
|
|
2173
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2174
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2175
|
+
| 2 | 58127 | observations | procedure_occurrence | 2009-07-03 | 2009-07-03 | 99214 |
|
|
2176
|
+
| 2 | 83339 | observations | procedure_occurrence | 2009-12-02 | 2009-12-02 | 99214 |
|
|
2177
|
+
| 5 | 38313 | observations | procedure_occurrence | 2008-12-14 | 2008-12-14 | 99214 |
|
|
2178
|
+
| 5 | 76788 | observations | procedure_occurrence | 2009-07-04 | 2009-07-04 | 99214 |
|
|
2179
|
+
| 5 | 63847 | observations | procedure_occurrence | 2009-12-21 | 2009-12-21 | 99214 |
|
|
2180
|
+
| 6 | 48992 | observations | procedure_occurrence | 2009-09-12 | 2009-09-12 | 99214 |
|
|
2181
|
+
| 6 | 76651 | observations | procedure_occurrence | 2010-02-23 | 2010-02-23 | 99214 |
|
|
2182
|
+
| 7 | 63797 | observations | procedure_occurrence | 2008-08-13 | 2008-08-13 | 99214 |
|
|
2183
|
+
| 7 | 59555 | observations | procedure_occurrence | 2008-09-21 | 2008-09-21 | 99214 |
|
|
2184
|
+
| 7 | 71597 | observations | procedure_occurrence | 2008-09-22 | 2008-09-22 | 99214 |
|
|
2185
|
+
|
|
2186
|
+
______________________________________________________________________
|
|
2187
|
+
|
|
2188
|
+
For example, to find all MIs of males, use `person_filter`. The right-hand stream need not be a person stream, because any record of a patient counts. The left-hand stream can also have several types, as the last example shows.
|
|
2189
|
+
|
|
2190
|
+
## Comparison Operators
|
|
2191
|
+
|
|
2192
|
+
These operators compare records of incoming streams to produce a set of records.
|
|
2193
|
+
|
|
2194
|
+
### `Filter` Operator
|
|
2195
|
+
|
|
2196
|
+
Keeps each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date.
|
|
2197
|
+
|
|
2198
|
+
- **Operands:**
|
|
2199
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
2200
|
+
- `right` (required): The right-hand stream. The operator keeps each left-hand record that is also in this stream, and does not pass these records on.
|
|
2201
|
+
- **Returns:** The left-hand records that have a right-hand record with the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. The right-hand records are not returned.
|
|
2202
|
+
|
|
2203
|
+
______________________________________________________________________
|
|
2204
|
+
|
|
2205
|
+
**Example 63 - MIs that are a patient's first diagnosis of MI or diabetes**
|
|
2206
|
+
|
|
2207
|
+
```JSON
|
|
2208
|
+
["filter",{"left":["icd9","412"],"right":["first",["union",["icd9","412"],["icd9","250.01"]]]}]
|
|
2209
|
+
```
|
|
2210
|
+
|
|
2211
|
+

|
|
2212
|
+
|
|
2213
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2214
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2215
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2216
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2217
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2218
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2219
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2220
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2221
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2222
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2223
|
+
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
2224
|
+
| 90 | 33940 | observations | condition_occurrence | 2010-05-26 | 2010-05-26 | 412 |
|
|
2225
|
+
|
|
2226
|
+
______________________________________________________________________
|
|
2227
|
+
|
|
2228
|
+
It is an intersection of the two streams. Filter compares records, not date ranges. A record whose dates `time_window` changed no longer matches the original record.
|
|
2229
|
+
|
|
2230
|
+
### `Match` Operator
|
|
2231
|
+
|
|
2232
|
+
Keeps each left-hand record that is equal to a right-hand record in every column, including label. Two empty (NULL) values count as equal.
|
|
2233
|
+
|
|
2234
|
+
- **Operands:**
|
|
2235
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
2236
|
+
- `right` (required): The right-hand stream. The operator keeps each left-hand record that is equal to a record of this stream, and does not pass these records on.
|
|
2237
|
+
- **Returns:** The left-hand records that have a right-hand record with the same value in every column. The right-hand records are not returned.
|
|
2238
|
+
|
|
2239
|
+
______________________________________________________________________
|
|
2240
|
+
|
|
2241
|
+
**Example 64 - MIs that are a patient's first MI**
|
|
2242
|
+
|
|
2243
|
+
```JSON
|
|
2244
|
+
["match",{"left":["icd9","412"],"right":["first",["icd9","412"]]}]
|
|
2245
|
+
```
|
|
2246
|
+
|
|
2247
|
+

|
|
2248
|
+
|
|
2249
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2250
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2251
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2252
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2253
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2254
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2255
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2256
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2257
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2258
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
2259
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2260
|
+
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
2261
|
+
|
|
2262
|
+
______________________________________________________________________
|
|
2263
|
+
|
|
2264
|
+
It is stricter than [`filter`](#filter-operator), so most users want `filter`. Systems that build statements, such as Jigsaw's export engine, use match with options that Jigsaw does not show.
|
|
2265
|
+
|
|
2266
|
+
### `Co-Reported` Operator
|
|
2267
|
+
|
|
2268
|
+
Passes along the records of its upstreams that were reported in the same context as a record of every other upstream. In the GDM, a context is the claim or other source record that reported the record.
|
|
2269
|
+
|
|
2270
|
+
- **Upstream:** one or more operators.
|
|
2271
|
+
- **Returns:** The records of each upstream whose context also has a record of every other upstream. The operator does not compare dates. If one upstream has no record in a context, the records of all upstreams in that context are not passed along. Person and death records have no context, so they never pass.
|
|
2272
|
+
|
|
2273
|
+
______________________________________________________________________
|
|
2274
|
+
|
|
2275
|
+
**Example 65 - Diabetes diagnoses reported as a reason for an office visit, and those office visits**
|
|
2276
|
+
|
|
2277
|
+
```JSON
|
|
2278
|
+
["co_reported",["icd9","250.00"],["cpt","99214"]]
|
|
2279
|
+
```
|
|
2280
|
+
|
|
2281
|
+

|
|
2282
|
+
|
|
2283
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2284
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2285
|
+
| 25 | 93217 | observations | condition_occurrence | 2008-09-18 | 2008-09-18 | 25000 |
|
|
2286
|
+
| 25 | 93220 | observations | procedure_occurrence | 2008-09-18 | 2008-09-18 | 99214 |
|
|
2287
|
+
| 31 | 87695 | observations | condition_occurrence | 2008-06-17 | 2008-06-17 | 25000 |
|
|
2288
|
+
| 31 | 87696 | observations | procedure_occurrence | 2008-06-17 | 2008-06-17 | 99214 |
|
|
2289
|
+
| 40 | 81973 | observations | condition_occurrence | 2009-05-01 | 2009-05-01 | 25000 |
|
|
2290
|
+
| 40 | 81993 | observations | procedure_occurrence | 2009-05-01 | 2009-05-01 | 99214 |
|
|
2291
|
+
| 44 | 44702 | observations | condition_occurrence | 2008-01-10 | 2008-01-10 | 25000 |
|
|
2292
|
+
| 44 | 44715 | observations | procedure_occurrence | 2008-01-10 | 2008-01-10 | 99214 |
|
|
2293
|
+
| 58 | 35093 | observations | condition_occurrence | 2009-06-15 | 2009-06-15 | 25000 |
|
|
2294
|
+
| 58 | 35095 | observations | procedure_occurrence | 2009-06-15 | 2009-06-15 | 99214 |
|
|
2295
|
+
|
|
2296
|
+
______________________________________________________________________
|
|
2297
|
+
|
|
2298
|
+
Claims data often reports diagnoses on the same claim as a procedure. This is done for billing: the provider says "I performed this procedure because the patient has these conditions". The `co_reported` operator finds this relationship. In the [GDM](https://github.com/outcomesinsights/generalized_data_model), "the same claim" means the same [context](https://github.com/outcomesinsights/generalized_data_model#contexts). In the example, both the diagnoses and the procedures pass through the operator.
|
|
2299
|
+
|
|
2300
|
+
## `label` Option and Its Features
|
|
2301
|
+
|
|
2302
|
+
Every ConceptQL operator takes a `label` option, so the reference sections of the operators do not list it:
|
|
2303
|
+
|
|
2304
|
+
A short description of the operator's records. The operator writes it into the label column of each of its records, and a recall operator selects those records by it. Each label in a statement must be different.
|
|
2305
|
+
|
|
2306
|
+
An operator without a label passes the upstream label through.
|
|
2307
|
+
|
|
2308
|
+
For example, here are some diabetes diagnoses with a label:
|
|
2309
|
+
|
|
2310
|
+
______________________________________________________________________
|
|
2311
|
+
|
|
2312
|
+
**Example 66**
|
|
2313
|
+
|
|
2314
|
+
```JSON
|
|
2315
|
+
["icd9","250.00",{"label":"diabetes"}]
|
|
2316
|
+
```
|
|
2317
|
+
|
|
2318
|
+

|
|
2319
|
+
|
|
2320
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value | source_vocabulary_id | label | column_family |
|
|
2321
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ | -------------------- | -------- | ------------- |
|
|
2322
|
+
| 2 | 89752 | observations | condition_occurrence | 2009-05-11 | 2009-05-11 | 25000 | ICD9CM | diabetes | default |
|
|
2323
|
+
| 2 | 57897 | observations | condition_occurrence | 2009-05-16 | 2009-05-16 | 25000 | ICD9CM | diabetes | default |
|
|
2324
|
+
| 2 | 76985 | observations | condition_occurrence | 2009-07-31 | 2009-07-31 | 25000 | ICD9CM | diabetes | default |
|
|
2325
|
+
| 3 | 93400 | observations | condition_occurrence | 2009-11-17 | 2009-11-17 | 25000 | ICD9CM | diabetes | default |
|
|
2326
|
+
| 5 | 89717 | observations | condition_occurrence | 2009-06-28 | 2009-06-28 | 25000 | ICD9CM | diabetes | default |
|
|
2327
|
+
| 7 | 15942 | observations | condition_occurrence | 2008-04-23 | 2008-04-23 | 25000 | ICD9CM | diabetes | default |
|
|
2328
|
+
| 7 | 69706 | observations | condition_occurrence | 2008-05-12 | 2008-05-12 | 25000 | ICD9CM | diabetes | default |
|
|
2329
|
+
| 7 | 38732 | observations | condition_occurrence | 2008-06-09 | 2008-06-09 | 25000 | ICD9CM | diabetes | default |
|
|
2330
|
+
| 7 | 38733 | observations | condition_occurrence | 2008-06-09 | 2008-06-09 | 25000 | ICD9CM | diabetes | default |
|
|
2331
|
+
| 7 | 70454 | observations | condition_occurrence | 2008-10-02 | 2008-10-02 | 25000 | ICD9CM | diabetes | default |
|
|
2332
|
+
|
|
2333
|
+
______________________________________________________________________
|
|
2334
|
+
|
|
2335
|
+
The label of the nearest labeled operator is the one in the output:
|
|
2336
|
+
|
|
2337
|
+
______________________________________________________________________
|
|
2338
|
+
|
|
2339
|
+
**Example 67**
|
|
2340
|
+
|
|
2341
|
+
```JSON
|
|
2342
|
+
["first",["icd9","250.00",{"label":"won't show"}],{"label":"1st diabetes"}]
|
|
2343
|
+
```
|
|
2344
|
+
|
|
2345
|
+

|
|
2346
|
+
|
|
2347
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value | source_vocabulary_id | label | column_family |
|
|
2348
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ | -------------------- | ------------ | ------------- |
|
|
2349
|
+
| 2 | 89752 | observations | condition_occurrence | 2009-05-11 | 2009-05-11 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2350
|
+
| 3 | 93400 | observations | condition_occurrence | 2009-11-17 | 2009-11-17 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2351
|
+
| 5 | 89717 | observations | condition_occurrence | 2009-06-28 | 2009-06-28 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2352
|
+
| 7 | 15942 | observations | condition_occurrence | 2008-04-23 | 2008-04-23 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2353
|
+
| 8 | 95135 | observations | condition_occurrence | 2008-10-17 | 2008-10-17 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2354
|
+
| 9 | 24929 | observations | condition_occurrence | 2010-08-01 | 2010-08-01 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2355
|
+
| 11 | 73103 | observations | condition_occurrence | 2008-06-17 | 2008-06-17 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2356
|
+
| 12 | 15443 | observations | condition_occurrence | 2009-04-20 | 2009-04-20 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2357
|
+
| 13 | 19968 | observations | condition_occurrence | 2008-03-05 | 2008-03-05 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2358
|
+
| 14 | 77991 | observations | condition_occurrence | 2009-06-05 | 2009-06-05 | 25000 | ICD9CM | 1st diabetes | default |
|
|
2359
|
+
|
|
2360
|
+
______________________________________________________________________
|
|
2361
|
+
|
|
2362
|
+
Labels give hints about the output:
|
|
2363
|
+
|
|
2364
|
+
______________________________________________________________________
|
|
2365
|
+
|
|
2366
|
+
**Example 68**
|
|
2367
|
+
|
|
2368
|
+
```JSON
|
|
2369
|
+
["first",["union",["icd9","250.00",{"label":"diabetes"}],["icd9","401.9",{"label":"hypertension"}]]]
|
|
2370
|
+
```
|
|
2371
|
+
|
|
2372
|
+

|
|
2373
|
+
|
|
2374
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value | source_vocabulary_id | label | column_family |
|
|
2375
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ | -------------------- | ------------ | ------------- |
|
|
2376
|
+
| 1 | 1065 | observations | condition_occurrence | 2010-03-12 | 2010-03-13 | 4019 | ICD9CM | hypertension | default |
|
|
2377
|
+
| 2 | 57517 | observations | condition_occurrence | 2009-03-30 | 2009-03-30 | 4019 | ICD9CM | hypertension | default |
|
|
2378
|
+
| 3 | 93400 | observations | condition_occurrence | 2009-11-17 | 2009-11-17 | 25000 | ICD9CM | diabetes | default |
|
|
2379
|
+
| 5 | 38310 | observations | condition_occurrence | 2008-12-14 | 2008-12-14 | 4019 | ICD9CM | hypertension | default |
|
|
2380
|
+
| 6 | 76645 | observations | condition_occurrence | 2010-02-23 | 2010-02-23 | 4019 | ICD9CM | hypertension | default |
|
|
2381
|
+
| 7 | 15942 | observations | condition_occurrence | 2008-04-23 | 2008-04-23 | 25000 | ICD9CM | diabetes | default |
|
|
2382
|
+
| 8 | 55867 | observations | condition_occurrence | 2008-02-27 | 2008-02-27 | 4019 | ICD9CM | hypertension | default |
|
|
2383
|
+
| 9 | 27621 | observations | condition_occurrence | 2009-10-12 | 2009-10-19 | 4019 | ICD9CM | hypertension | default |
|
|
2384
|
+
| 11 | 73103 | observations | condition_occurrence | 2008-06-17 | 2008-06-17 | 25000 | ICD9CM | diabetes | default |
|
|
2385
|
+
| 12 | 34937 | observations | condition_occurrence | 2008-09-17 | 2008-09-17 | 4019 | ICD9CM | hypertension | default |
|
|
2386
|
+
|
|
2387
|
+
______________________________________________________________________
|
|
2388
|
+
|
|
2389
|
+
A label must be a string.
|
|
2390
|
+
|
|
2391
|
+
### `Recall` Operator
|
|
2392
|
+
|
|
2393
|
+
Passes on the records of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of records more than once.
|
|
2394
|
+
|
|
2395
|
+
- **Upstream:** none.
|
|
2396
|
+
- **Arguments:**
|
|
2397
|
+
- `name` (string): The label of the operator whose records to pass on. It must match the label exactly, including case. The labeled operator can come before or after the recall in the statement, but labeled streams that recall each other in a loop are invalid.
|
|
2398
|
+
- **Returns:** The records of the operator that has the given label.
|
|
2399
|
+
|
|
2400
|
+
______________________________________________________________________
|
|
2401
|
+
|
|
2402
|
+
**Example 69 - Label neurological diagnoses as "neuro dxs". Recall them to remove those that were co-reported with a nervous-system MS-DRG. Then apply the one inpatient, two outpatient rule. (The code lists are shortened.)**
|
|
2403
|
+
|
|
2404
|
+
```JSON
|
|
2405
|
+
["one_in_two_out",["except",{"left":["union",["icd9","331.0","332.0","340","345.90"],["icd10cm","G20","G30.9","G35","G40.909"],{"label":"neuro dxs"}],"right":["co_reported",["recall","neuro dxs"],["ms_drg","056","057","058","059","060","100","101"]]}],{"outpatient_minimum_gap":"30d","outpatient_maximum_gap":"365d","outpatient_event_to_return":"Confirming Event"}]
|
|
2406
|
+
```
|
|
2407
|
+
|
|
2408
|
+

|
|
2409
|
+
|
|
2410
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2411
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2412
|
+
| 2 | 122 | observations | condition_occurrence | 2009-09-17 | 2009-09-20 | 34590 |
|
|
2413
|
+
| 7 | 27644 | observations | condition_occurrence | 2008-09-21 | 2008-09-30 | 3320 |
|
|
2414
|
+
| 28 | 131 | observations | condition_occurrence | 2008-04-26 | 2008-04-30 | 3310 |
|
|
2415
|
+
| 35 | 34819 | observations | condition_occurrence | 2009-05-02 | 2009-05-02 | 340 |
|
|
2416
|
+
| 42 | 72000 | observations | condition_occurrence | 2008-08-28 | 2008-08-28 | 3310 |
|
|
2417
|
+
| 44 | 39145 | observations | condition_occurrence | 2008-08-31 | 2008-08-31 | 3310 |
|
|
2418
|
+
| 68 | 28936 | observations | condition_occurrence | 2009-12-01 | 2009-12-01 | 340 |
|
|
2419
|
+
| 72 | 91104 | observations | condition_occurrence | 2009-09-07 | 2009-09-07 | 3320 |
|
|
2420
|
+
| 83 | 58845 | observations | condition_occurrence | 2010-09-05 | 2010-09-05 | 3320 |
|
|
2421
|
+
| 88 | 89097 | observations | condition_occurrence | 2009-02-03 | 2009-02-03 | 3320 |
|
|
2422
|
+
|
|
2423
|
+
______________________________________________________________________
|
|
2424
|
+
|
|
2425
|
+
## Window Tables
|
|
2426
|
+
|
|
2427
|
+
A system can run a statement inside windows. A window is a date range of one patient. Jigsaw's export engine uses windows. The system gives the windows in a window table, and it wraps the statement like this:
|
|
2428
|
+
|
|
2429
|
+
```JSON
|
|
2430
|
+
[ "window", <statement>, { "window_table": <window table> } ]
|
|
2431
|
+
```
|
|
2432
|
+
|
|
2433
|
+
- The window table has the columns `person_id`, `start_date` and `end_date`. It is the name of a database table or a ConceptQL statement.
|
|
2434
|
+
- Each selection operator keeps only the records whose patient has a window and whose start_date is in that window. Both dates of the window are in the window. The person operators, such as `person` and `gender`, keep each patient who has a window, whatever the dates.
|
|
2435
|
+
- Each record gets the column `window_id`, which identifies its window. A record that is in two windows of its patient comes out once for each window.
|
|
2436
|
+
- Operators that work on each patient, such as `first`, `episode` and the temporal operators, work on each patient and window. For example, `first` keeps the first record of each window.
|
|
2437
|
+
- If the window table is a ConceptQL statement with errors, the statement that it wraps is invalid. A diagram reports the errors under `window_table`, and counts nothing.
|
|
2438
|
+
|
|
2439
|
+
## Appendix A - Experimental Operators
|
|
2440
|
+
|
|
2441
|
+
These operators work, but they may change or be removed. They compare the `value_as_number` column, which lab records (such as `loinc`) carry.
|
|
2442
|
+
|
|
2443
|
+
### `Numeric Filter` Operator
|
|
2444
|
+
|
|
2445
|
+
Passes along the records whose value_as_number is in the given range. Lab records, such as those of loinc, carry this value.
|
|
2446
|
+
|
|
2447
|
+
- **Upstream:** one operator.
|
|
2448
|
+
- **Options:**
|
|
2449
|
+
- `greater_than_or_equal_to` (float): The lowest value_as_number that passes. Without this option, the range has no lower limit.
|
|
2450
|
+
- `less_than_or_equal_to` (float): The highest value_as_number that passes. Without this option, the range has no upper limit.
|
|
2451
|
+
- **Returns:** The upstream records whose value_as_number is in the range. A record that has no value_as_number is not passed along.
|
|
2452
|
+
|
|
2453
|
+
______________________________________________________________________
|
|
2454
|
+
|
|
2455
|
+
**Example 70 - Hemoglobin A1c results (LOINC 4548-4) with a value from 5 to 7**
|
|
2456
|
+
|
|
2457
|
+
```JSON
|
|
2458
|
+
["numeric_filter",["loinc","4548-4"],{"greater_than_or_equal_to":5,"less_than_or_equal_to":7}]
|
|
2459
|
+
```
|
|
2460
|
+
|
|
2461
|
+

|
|
2462
|
+
|
|
2463
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2464
|
+
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
|
|
2465
|
+
| 200 | 100001 | observations | observation | 2009-03-03 | 2009-03-03 | 4548-4 |
|
|
2466
|
+
|
|
2467
|
+
______________________________________________________________________
|
|
2468
|
+
|
|
2469
|
+
It is an [inline filter operator](#inline-filter-operators).
|
|
2470
|
+
|
|
2471
|
+
### `Equal` Operator
|
|
2472
|
+
|
|
2473
|
+
For each patient, keeps the left-hand records whose value_as_number equals the value_as_number of a right-hand record. It does not compare dates.
|
|
2474
|
+
|
|
2475
|
+
- **Operands:**
|
|
2476
|
+
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
2477
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
|
|
2478
|
+
- **Returns:** The left-hand records that have a right-hand record of the same patient with the same value_as_number, each record once. Left-hand records without a value_as_number (NULL) are not returned. The right-hand records are not returned.
|
|
2479
|
+
|
|
2480
|
+
______________________________________________________________________
|
|
2481
|
+
|
|
2482
|
+
**Example 71 - HbA1c results (LOINC 4548-4) with the same value as an HbA1c result by HPLC (LOINC 17856-6) of the same patient**
|
|
2483
|
+
|
|
2484
|
+
```JSON
|
|
2485
|
+
["equal",{"left":["loinc","4548-4"],"right":["loinc","17856-6"]}]
|
|
2486
|
+
```
|
|
2487
|
+
|
|
2488
|
+

|
|
2489
|
+
|
|
2490
|
+
No results.
|
|
2491
|
+
|
|
2492
|
+
______________________________________________________________________
|
|
2493
|
+
|
|
2494
|
+
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams.
|
|
2495
|
+
|
|
2496
|
+
## Appendix B - Algorithm Showcase
|
|
2497
|
+
|
|
2498
|
+
These algorithms are based on [OMOP's Health Outcomes of Interest](http://omop.org/HOI). Some code lists are shortened to keep the diagrams small.
|
|
2499
|
+
|
|
2500
|
+
### Acute Kidney Injury - Narrow Definition and diagnostic procedure
|
|
2501
|
+
|
|
2502
|
+
- The first diagnosis of acute kidney failure (ICD-9-CM 584.5 to 584.9)
|
|
2503
|
+
- AND
|
|
2504
|
+
- an ICD-9 procedure code 39.95 or 54.98 within 60 days after the diagnosis
|
|
2505
|
+
- AND NOT
|
|
2506
|
+
- a diagnosis of chronic dialysis at any time before the diagnosis
|
|
2507
|
+
- V45.1, V56.0, V56.31, V56.32, V56.8
|
|
2508
|
+
|
|
2509
|
+
______________________________________________________________________
|
|
2510
|
+
|
|
2511
|
+
**Example 72**
|
|
2512
|
+
|
|
2513
|
+
```JSON
|
|
2514
|
+
["during",{"left":["except",{"left":["first",["icd9","584.5","584.6","584.7","584.8","584.9"],{"label":"First AKI diagnosis"}],"right":["after",{"left":["recall","First AKI diagnosis"],"right":["icd9","V45.1","V56.0","V56.31","V56.32","V56.8"]}]}],"right":["time_window",["icd9_procedure","39.95","54.98"],{"start":"-60d","end":"0"}]}]
|
|
2515
|
+
```
|
|
2516
|
+
|
|
2517
|
+

|
|
2518
|
+
|
|
2519
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2520
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2521
|
+
| 153 | 102 | observations | condition_occurrence | 2010-01-29 | 2010-02-08 | 5849 |
|
|
2522
|
+
|
|
2523
|
+
______________________________________________________________________
|
|
2524
|
+
|
|
2525
|
+
### Mortality after Myocardial Infarction #3
|
|
2526
|
+
|
|
2527
|
+
- The person died
|
|
2528
|
+
- AND an acute MI diagnosis (ICD-9-CM 410.x) before death
|
|
2529
|
+
- AND either
|
|
2530
|
+
- a diagnostic test or procedure for MI within 30 days before the MI diagnosis
|
|
2531
|
+
- an MI therapy procedure within 60 days after the MI diagnosis
|
|
2532
|
+
|
|
2533
|
+
______________________________________________________________________
|
|
2534
|
+
|
|
2535
|
+
**Example 73**
|
|
2536
|
+
|
|
2537
|
+
```JSON
|
|
2538
|
+
["during",{"left":["before",{"left":["icd9","410.00","410.01","410.02","410.10","410.11","410.12","410.20","410.21","410.22","410.30","410.31","410.32","410.40","410.41","410.42","410.50","410.51","410.52","410.60","410.61","410.62","410.70","410.71","410.72","410.80","410.81","410.82","410.90","410.91","410.92"],"right":["death"]}],"right":["union",["time_window",["union",["cpt","0146T","75898","82554","92980","93010","93233","93508","93540","93545"],["icd9_procedure","00.24","36.02","89.53","89.57","89.69"],["loinc","10839-9","13969-1","18843-3","2154-3","33204-9","48425-3","49259-5","6597-9","8634-8"]],{"start":"0","end":"30d"}],["time_window",["union",["cpt","0146T","75898","82554","92980","93010","93233"],["icd9_procedure","00.24","36.02","89.53","89.57","89.69"]],{"start":"-60d","end":"0"}]]}]
|
|
2539
|
+
```
|
|
2540
|
+
|
|
2541
|
+

|
|
2542
|
+
|
|
2543
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2544
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2545
|
+
| 191 | 64751 | observations | condition_occurrence | 2008-03-11 | 2008-03-11 | 41022 |
|
|
2546
|
+
| 191 | 64752 | observations | condition_occurrence | 2008-03-11 | 2008-03-11 | 41071 |
|
|
2547
|
+
| 191 | 51654 | observations | condition_occurrence | 2008-11-08 | 2008-11-11 | 41090 |
|
|
2548
|
+
|
|
2549
|
+
______________________________________________________________________
|
|
2550
|
+
|
|
2551
|
+
## Appendix C - History of ConceptQL and Its Evolution
|
|
2552
|
+
|
|
2553
|
+
ConceptQL was first developed to query data in the [OMOP Common Data Model (CDM) v4.0](https://github.com/OHDSI/CommonDataModel/releases/tag/v4). ConceptQL took simple statements like `[ "icd9", "412" ]` and found the correct table to query (in this case `condition_occurrence`), along with the correct source code and vocabulary ID to search for in that table.
|
|
2554
|
+
|
|
2555
|
+
As the OMOP CDM continued to change and its ETL requirements continued to grow, [Outcomes Insights, Inc](https://outins.com) developed the [GDM](https://github.com/outcomesinsights/generalized_data_model), and ConceptQL was adapted to query both data models. ConceptQL 0.5.0 removed support for the OMOP CDM, because no production application of ConceptQL used it. This experience shows that ConceptQL can be adapted to other data models when the need arises.
|
|
2556
|
+
|
|
2557
|
+
ConceptQL was first written for PostgreSQL, and it was later adapted to MSSQL, Oracle, and Impala. It now supports PostgreSQL, DuckDB, and Apache Spark. With a modest amount of work, ConceptQL can support most databases that have the features of [SQL:2003](https://en.wikipedia.org/wiki/SQL:2003), such as window functions. Most SQL that ConceptQL generates is standard SQL. It does not depend on features or query hints of a particular database.
|
|
2558
|
+
|
|
2559
|
+
## Appendix D - Regenerating This Document
|
|
2560
|
+
|
|
2561
|
+
This document is part of the [ConceptQL](https://github.com/outcomesinsights/conceptql) repository, in `docs/spec/`. `README.md` is generated from `README.cql.md`. Edit `README.cql.md`, and then run `just spec` in the repository. This document describes the version of ConceptQL in the same commit.
|
|
2562
|
+
|
|
2563
|
+
The reference section of each operator, the text of the `label` option, the table of all vocabulary operators and Appendix E come from the metadata of the operators (`ConceptQL.metadata`), so the texts in the operator classes are their only copy. A directive in `README.cql.md`, an HTML comment on a line of its own, marks the place of each generated block:
|
|
2564
|
+
|
|
2565
|
+
- `<!-- conceptql-spec: operator <spec_id> -->` writes the reference section of the operators with that `spec_id`. It comes directly after the heading of the section, whose anchor is the `spec_id`. The section has the operator's description, its upstream operators or its operands (`left` and `right`), its arguments, its options, the records it returns, and its examples. Text about more than one operator, and longer explanations of one operator, follow the directive as markdown.
|
|
2566
|
+
- `<!-- conceptql-spec: label-option -->` writes the text of the `label` option, which every operator takes.
|
|
2567
|
+
- `<!-- conceptql-spec: quick-reference -->` writes the table of Appendix E, with one row for each `spec_id`.
|
|
2568
|
+
- `<!-- conceptql-spec: vocabulary-operators -->` writes the table of [All Vocabulary Operators](#all-vocabulary-operators), with one row for each vocabulary operator: its name, the full name of its vocabulary and its domain. It marks a deprecated operator.
|
|
2569
|
+
|
|
2570
|
+
The metadata comes from a ConceptQL database with the data model `gdm`, no data database and no custom vocabularies, so it decides which operators get a section. Each `spec_id` of that metadata needs exactly one directive, and the build stops when one is missing, repeated or not after its heading. The other directives appear exactly once each, and the build also stops on an unknown directive.
|
|
2571
|
+
|
|
2572
|
+
`just spec` runs `docs/spec/build/run.rb` with the working tree's ConceptQL. For each `ConceptQL` block of `README.cql.md` and each example of a reference section, the build writes the example: the statement as JSON, its diagram, and the first 10 records of its result. It orders the records by `person_id`, `criterion_domain`, start_date, `criterion_table` and `criterion_id`, so each run shows the same records. An example that is not a valid statement stops the build. Then `just spec` formats `README.md` with treefmt. If `README.md` or an image changed, `just spec` lists the changed files and fails, so that you review and commit them. A second run then changes nothing and passes.
|
|
2573
|
+
|
|
2574
|
+
The results and the counts come from ConceptQL's DuckDB test data, `synpuf_test_data.duckdb`, with the data model `gdm_wide` and the OHDSI vocabulary tables. `just spec` runs the build through `just test-duckdb`, so it needs what the DuckDB tests need:
|
|
2575
|
+
|
|
2576
|
+
- Linux on x86-64. `just test-duckdb` downloads the pinned version of the DuckDB library on the first run.
|
|
2577
|
+
- The DuckDB test data. `just test-duckdb` copies it from the path in `DUCKDB_TEST_DATA_SOURCE`, and it fails unless the copy matches the checksum in `.github/synpuf_test_data.sha256`.
|
|
2578
|
+
|
|
2579
|
+
The diagrams are drawn with ConceptQL's diagram renderer, `ConceptQL::DiagramRenderer`, which runs the resvg program. The build needs resvg 0.33.0 exactly, which `mise.toml` pins and `just setup` installs. It fails when resvg is missing or is another version. The renderer labels each line with the records (`rows=`) and the people (`n=`) that flow along it, and it lists the options of each operator. The build draws every diagram on each run, and writes each one as `images/<name>.svg`, where the name is the SHA-256 of the statement as canonical JSON. It removes the SVG images that no example uses, and it writes `images/manifest.tsv`, which gives the SHA-256 of the diagram data (with its counts) for each image.
|
|
2580
|
+
|
|
2581
|
+
Between releases, the results and the images on the main branch can be out of date. Before a release, `just release-check` runs `just spec-check`. It rebuilds the document and the diagram data, and it draws every image into a temporary directory. It fails when `README.md` does not match the code, or when an image differs from its new drawing, is missing, or is not used. It names each such image.
|
|
2582
|
+
|
|
2583
|
+
## Appendix E - Operator Quick Reference
|
|
2584
|
+
|
|
2585
|
+
The build writes this table from the metadata of the operators. It has one row for each reference section. The Jigsaw category of each operator is the group in which Jigsaw's operator palette shows it. Every operator also takes the [`label`](#label-option-and-its-features) option.
|
|
2586
|
+
|
|
2587
|
+
| Operator | Jigsaw category | What it does |
|
|
2588
|
+
| -------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
2589
|
+
| [`Vocabulary` Operator](#vocabulary-operator) | Select by Clinical Codes | Selects the records whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once. |
|
|
2590
|
+
| [`gender`](#gender-operator) | Select by Property | Selects the patients with the given genders. |
|
|
2591
|
+
| [`race`](#race-operator) | Select by Property | Selects the patients with the given races. |
|
|
2592
|
+
| [`ethnicity`](#ethnicity-operator) | Select by Property | Selects the patients with the given ethnicities. |
|
|
2593
|
+
| [`snf`](#snf-operator) | Select by Property | Selects admissions to skilled nursing facilities (SNF): one record for each admission whose claim came from an SNF file. |
|
|
2594
|
+
| [`hospice`](#hospice-operator) | Select by Property | Selects hospice admissions: one record for each admission whose claim came from a hospice file. |
|
|
2595
|
+
| [`hospitalization`](#hospitalization-operator) | Select by Property | Selects inpatient hospital admissions: one record for each admission whose claim came from an inpatient file. |
|
|
2596
|
+
| [`date_range`](#date-range-operator) | Select by Property | Makes one person record for each patient, with the given start_date and end_date. Use it to compare other records with a fixed range of dates. |
|
|
2597
|
+
| [`person`](#person-operator) | Get Related Data | Selects every patient. With one upstream operator, it selects the patients who have a record in the upstream stream. |
|
|
2598
|
+
| [`death`](#death-operator) | Get Related Data | Selects every death record. With one upstream operator, it selects the deaths of the patients who have a record in the upstream stream. |
|
|
2599
|
+
| [`information_periods`](#information-periods-operator) | Get Related Data | Selects information periods: the spans of time in which the data has information about a patient, such as periods of insurance enrollment. |
|
|
2600
|
+
| [`recall`](#recall-operator) | Get Related Data | Passes on the records of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of records more than once. |
|
|
2601
|
+
| [`time_window`](#time-window-operator) | Modify Data | Moves the start_date and the end_date of each record. Use it to shift or widen a record's dates before you compare them with other records. |
|
|
2602
|
+
| [`episode`](#episode-operator) | Modify Data | Merges each patient's records into episodes: spans of time in which the records are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug. |
|
|
2603
|
+
| [`union`](#union-operator) | Combine Streams | Combines the records of all its upstream operators into one stream. Each record keeps its own type, so one stream can hold several types. |
|
|
2604
|
+
| [`co_reported`](#co-reported-operator) | Combine Streams | Passes along the records of its upstreams that were reported in the same context as a record of every other upstream. In the GDM, a context is the claim or other source record that reported the record. |
|
|
2605
|
+
| [`except`](#except-operator) | Filter by Comparing | Removes each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same criterion_table, criterion_id, start_date and end_date. |
|
|
2606
|
+
| [`during`](#during-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose date range is inside the date range of a right-hand record. The left-hand record starts on or after the start_date of the right-hand record and ends on or before its end_date. |
|
|
2607
|
+
| [`contains`](#contains-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose date range encloses the date range of a right-hand record. The left-hand record starts on or before the start_date of the right-hand record and ends on or after its end_date. |
|
|
2608
|
+
| [`any_overlap`](#any-overlap-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose date range shares at least one day with the date range of a right-hand record. |
|
|
2609
|
+
| [`before`](#before-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose end_date is before the latest start_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead. |
|
|
2610
|
+
| [`after`](#after-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose start_date is after the earliest end_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead. |
|
|
2611
|
+
| [`trim_date_start`](#trim-date-start-operator) | Filter by Comparing | For each patient, moves the start_date of each left-hand record forward to the end_date of the right-hand record with the latest start_date. A left-hand record that ends before that end_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged. |
|
|
2612
|
+
| [`trim_date_end`](#trim-date-end-operator) | Filter by Comparing | For each patient, moves the end_date of each left-hand record back to the start_date of the right-hand record with the earliest start_date. A left-hand record that starts after that start_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged. |
|
|
2613
|
+
| [`person_filter`](#person-filter-operator) | Filter by Comparing | Keeps each left-hand record whose patient has at least one record in the right-hand stream. The type and the dates of the right-hand records do not matter. |
|
|
2614
|
+
| [`filter`](#filter-operator) | Filter by Comparing | Keeps each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. |
|
|
2615
|
+
| [`match`](#match-operator) | Filter by Comparing | Keeps each left-hand record that is equal to a right-hand record in every column, including label. Two empty (NULL) values count as equal. |
|
|
2616
|
+
| [`equal`](#equal-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose value_as_number equals the value_as_number of a right-hand record. It does not compare dates. |
|
|
2617
|
+
| [`occurrence`](#nth-occurrence-operator) | Filter Single Stream | For each person, orders the records by start_date and passes along the record at the given position. A positive position counts from the earliest record and a negative one from the most recent record. |
|
|
2618
|
+
| [`first`](#first-operator) | Filter Single Stream | For each person, passes along the record with the earliest start_date. |
|
|
2619
|
+
| [`last`](#last-operator) | Filter Single Stream | For each person, passes along the record with the most recent start_date. |
|
|
2620
|
+
| [`place_of_service_filter`](#place-of-service-filter-operator) | Filter Single Stream | Passes along the records whose place of service is one of the given CMS Place of Service codes. A record that has no place of service is not passed along. |
|
|
2621
|
+
| [`provenance`](#provenance-operator) | Filter Single Stream | Passes along the records whose provenance matches at least one of the given keywords. The provenance of a record says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary. |
|
|
2622
|
+
| [`provider_filter`](#provider-filter-operator) | Filter Single Stream | Passes along the records whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value. |
|
|
2623
|
+
| [`one_in_two_out`](#one-in-two-out-operator) | Filter Single Stream | For each person, passes along the earliest confirmed condition record. One inpatient record confirms a condition, and so do two outpatient records whose start dates are a given gap apart. |
|
|
2624
|
+
| [`numeric_filter`](#numeric-filter-operator) | Filter Single Stream | Passes along the records whose value_as_number is in the given range. Lab records, such as those of loinc, carry this value. |
|