conceptql 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +141 -0
- data/docs/spec/README.md +402 -337
- data/docs/spec/examples/date-range-operator.json +1 -1
- data/docs/spec/examples/death-operator.json +1 -1
- data/docs/spec/examples/nth-occurrence-operator.json +1 -1
- data/docs/spec/examples/one-in-two-out-operator.json +1 -1
- data/docs/spec/examples/person-filter-operator.json +2 -2
- data/docs/spec/examples/vocabulary-operator.json +3 -3
- data/lib/conceptql/behaviors/metadatable.rb +76 -62
- data/lib/conceptql/behaviors/utilizable.rb +1 -1
- data/lib/conceptql/database.rb +16 -2
- data/lib/conceptql/diagram.rb +2 -2
- data/lib/conceptql/lexicon/lexicon_no_db.rb +4 -4
- data/lib/conceptql/lexicon.rb +7 -4
- data/lib/conceptql/operators/after.rb +5 -4
- data/lib/conceptql/operators/any_overlap.rb +4 -3
- data/lib/conceptql/operators/before.rb +5 -4
- data/lib/conceptql/operators/binary_operator_operator.rb +11 -13
- data/lib/conceptql/operators/casting_operator.rb +1 -1
- data/lib/conceptql/operators/co_reported.rb +9 -7
- data/lib/conceptql/operators/contains.rb +4 -3
- data/lib/conceptql/operators/count.rb +3 -3
- data/lib/conceptql/operators/date_range.rb +9 -8
- data/lib/conceptql/operators/death.rb +4 -4
- data/lib/conceptql/operators/dedup.rb +2 -2
- data/lib/conceptql/operators/during.rb +4 -3
- data/lib/conceptql/operators/episode.rb +9 -9
- data/lib/conceptql/operators/equal.rb +3 -2
- data/lib/conceptql/operators/ethnicity.rb +7 -3
- data/lib/conceptql/operators/except.rb +4 -3
- data/lib/conceptql/operators/filter.rb +4 -3
- data/lib/conceptql/operators/first.rb +4 -3
- data/lib/conceptql/operators/from.rb +2 -1
- data/lib/conceptql/operators/gender.rb +4 -3
- data/lib/conceptql/operators/hospice.rb +4 -3
- data/lib/conceptql/operators/hospitalization.rb +4 -3
- data/lib/conceptql/operators/information_periods.rb +5 -4
- data/lib/conceptql/operators/last.rb +4 -3
- data/lib/conceptql/operators/match.rb +4 -3
- data/lib/conceptql/operators/multiple_vocabularies.rb +4 -3
- data/lib/conceptql/operators/numeric.rb +4 -4
- data/lib/conceptql/operators/numeric_filter.rb +6 -6
- data/lib/conceptql/operators/occurrence.rb +25 -18
- data/lib/conceptql/operators/one_in_two_out.rb +16 -16
- data/lib/conceptql/operators/operator.rb +16 -3
- data/lib/conceptql/operators/pass_thru.rb +1 -1
- data/lib/conceptql/operators/person.rb +3 -3
- data/lib/conceptql/operators/person_filter.rb +6 -5
- data/lib/conceptql/operators/place_of_service_filter.rb +7 -7
- data/lib/conceptql/operators/provenance.rb +9 -9
- data/lib/conceptql/operators/provider_filter.rb +6 -6
- data/lib/conceptql/operators/race.rb +4 -3
- data/lib/conceptql/operators/recall.rb +6 -5
- data/lib/conceptql/operators/snf.rb +4 -3
- data/lib/conceptql/operators/sum.rb +6 -5
- data/lib/conceptql/operators/time_window.rb +6 -6
- data/lib/conceptql/operators/trim_date_end.rb +6 -5
- data/lib/conceptql/operators/trim_date_start.rb +6 -5
- data/lib/conceptql/operators/union.rb +8 -6
- data/lib/conceptql/operators/vocabulary.rb +12 -8
- data/lib/conceptql/query.rb +41 -12
- data/lib/conceptql/scope.rb +10 -2
- data/lib/conceptql/version.rb +1 -1
- data/lib/conceptql/visitors/metadata_extractor.rb +3 -1
- data/lib/conceptql/vocabularies/drg_systems.rb +16 -14
- data/lib/conceptql/vocabularies/read_codes.rb +1 -1
- data/lib/conceptql/window.rb +23 -2
- data/schemas/gdm.yml +0 -66
- data/schemas/gdm_wide.yml +0 -66
- metadata +1 -1
data/docs/spec/README.md
CHANGED
|
@@ -84,13 +84,21 @@ ConceptQL is a domain-specific, declarative language. It represents search crite
|
|
|
84
84
|
```YAML
|
|
85
85
|
# Example 1: A simple example in YAML
|
|
86
86
|
# The first element is the operator, icd9cm. The second element is a code, 412.
|
|
87
|
-
# This statement selects every
|
|
87
|
+
# This statement selects every result with the ICD-9-CM diagnosis code 412.
|
|
88
88
|
---
|
|
89
89
|
- icd9cm
|
|
90
90
|
- '412'
|
|
91
91
|
```
|
|
92
92
|
|
|
93
|
-
|
|
93
|
+
A statement is an algorithm written in ConceptQL. It is commonly stored as JSON. Each array of a statement has these parts:
|
|
94
|
+
|
|
95
|
+
- The first element is an operator. An operator is a function that is applied to the dataset or to results. Only a selection operator reads the dataset. Every other operator works on its upstream results.
|
|
96
|
+
- The arguments follow the operator. An argument is an unnamed, positional input. It is a code, such as `412`, or an upstream statement. Many operators take a variable number of arguments.
|
|
97
|
+
- An optional hash at the end holds the options. An option is a named input. Every operator takes the option `label` (see [`label`](#label-option-and-its-features)).
|
|
98
|
+
|
|
99
|
+
The reference section of each operator lists its upstream statements under "Upstream" and its other arguments under "Arguments". Codes are strings, so quote them in YAML. Operator names are not case-sensitive.
|
|
100
|
+
|
|
101
|
+
The words upstream and downstream describe the flow of results through a statement. The upstream of an operator is the results that come into it. An upstream statement is nested inside the array of the operator that takes its results. Downstream is everything in a statement that comes after a given operator. In the second diagram below, the results of `cpt` are the upstream of `first`, and `first` is downstream of `cpt`.
|
|
94
102
|
|
|
95
103
|
### ConceptQL Diagrams
|
|
96
104
|
|
|
@@ -121,7 +129,7 @@ ______________________________________________________________________
|
|
|
121
129
|
|
|
122
130
|
______________________________________________________________________
|
|
123
131
|
|
|
124
|
-
Each node in a diagram is an operator. A line between two operators shows that the
|
|
132
|
+
Each node in a diagram is an operator. A line between two operators shows that the results of the upper operator flow down into the lower operator. Each line is labeled with the number of results (`rows=`) and people (`n=`) that flow along it in our test dataset. Here is a slightly larger example:
|
|
125
133
|
|
|
126
134
|
______________________________________________________________________
|
|
127
135
|
|
|
@@ -152,15 +160,15 @@ The diagram above reads "get all procedures that match CPT 99214 (office visit),
|
|
|
152
160
|
|
|
153
161
|
Every diagram ends with a line that leads to nothing. The next section explains why.
|
|
154
162
|
|
|
155
|
-
### Think of
|
|
163
|
+
### Think of Results as a Stream
|
|
156
164
|
|
|
157
|
-
A ConceptQL diagram has the "leaf" operators at the top and the "trunk" operator at the bottom. Think of the
|
|
165
|
+
A ConceptQL diagram has the "leaf" operators at the top and the "trunk" operator at the bottom. Think of the results of a ConceptQL statement as a stream. The leaf operators select rows from the dataset and make results from them, so they act as the sources of the stream. The results flow down through the diagram. Some operators combine streams, and others remove results, until one stream comes out at the bottom of the diagram.
|
|
158
166
|
|
|
159
|
-
The trailing line in each diagram is a reminder that every ConceptQL statement yields a stream of
|
|
167
|
+
The trailing line in each diagram is a reminder that every ConceptQL statement yields a stream of results.
|
|
160
168
|
|
|
161
169
|
### Streams Have Types
|
|
162
170
|
|
|
163
|
-
Each
|
|
171
|
+
Each result has a type. The type is the `criterion_domain` column of the result. A vocabulary operator takes its type from the vocabulary's domain. For example, ICD-9-CM results are `condition_occurrence` results, and CPT results are `procedure_occurrence` results. Other operators produce a fixed type, such as `person` or `death`.
|
|
164
172
|
|
|
165
173
|
The diagrams use color to show types. The node of an operator and the lines that leave it have the color of the types that it outputs:
|
|
166
174
|
|
|
@@ -177,7 +185,7 @@ The diagrams use color to show types. The node of an operator and the lines that
|
|
|
177
185
|
|
|
178
186
|
An operator that outputs more than one type has a gray node, and the line that leaves it is drawn as one colored line for each type, side by side.
|
|
179
187
|
|
|
180
|
-
The type stays with each
|
|
188
|
+
The type stays with each result as it moves through the operators. Some operators use the type when they compare results. For example, `filter` and `except` match a result only with a result from the same table. Other operators, such as `first` and `person_filter`, ignore the type. Each operator's description says which applies.
|
|
181
189
|
|
|
182
190
|
### Why Types?
|
|
183
191
|
|
|
@@ -189,19 +197,19 @@ ConceptQL now runs against the [GDM](https://github.com/outcomesinsights/general
|
|
|
189
197
|
|
|
190
198
|
Thinking in streams is helpful, but sometimes we need to know what happens in the database.
|
|
191
199
|
|
|
192
|
-
A stream is a set of
|
|
200
|
+
A stream is the set of results that passes from one operator to the next. ConceptQL is not a stream-processing system. Here, "stream" means only that set of results. Each result points back to one row of a GDM table, and it always has these columns:
|
|
193
201
|
|
|
194
202
|
- `person_id`: the patient.
|
|
195
203
|
- `criterion_id`: the ID of the row in its table of origin.
|
|
196
204
|
- `criterion_table`: the table of origin, such as `clinical_codes`, `patients` or `deaths`.
|
|
197
|
-
- `criterion_domain`: the type of the
|
|
198
|
-
- `start_date` and `end_date`: the dates of the
|
|
199
|
-
- `source_value` and `source_vocabulary_id`: the code of the
|
|
205
|
+
- `criterion_domain`: the type of the result (see above).
|
|
206
|
+
- `start_date` and `end_date`: the dates of the result.
|
|
207
|
+
- `source_value` and `source_vocabulary_id`: the code of the result and its vocabulary, when the result has a code.
|
|
200
208
|
- `label`: the label of the nearest upstream operator that has one (see [`label`](#label-option-and-its-features)).
|
|
201
209
|
|
|
202
|
-
Some operators add columns. For example, lab
|
|
210
|
+
Some operators add columns. For example, the results of lab codes (loinc) add `value_as_number` and unit columns, and the utilization operators add admission and discharge columns.
|
|
203
211
|
|
|
204
|
-
So this statement yields one
|
|
212
|
+
So this statement yields one result for each male patient, pointing at the `patients` table:
|
|
205
213
|
|
|
206
214
|
______________________________________________________________________
|
|
207
215
|
|
|
@@ -228,7 +236,7 @@ ______________________________________________________________________
|
|
|
228
236
|
|
|
229
237
|
______________________________________________________________________
|
|
230
238
|
|
|
231
|
-
This statement yields one
|
|
239
|
+
This statement yields one result for each `clinical_codes` row that has the ICD-9-CM code 250.01:
|
|
232
240
|
|
|
233
241
|
______________________________________________________________________
|
|
234
242
|
|
|
@@ -257,49 +265,49 @@ ______________________________________________________________________
|
|
|
257
265
|
|
|
258
266
|
It usually helps to think of these queries as a "stream of people" or a "stream of conditions", and not to think about tables and IDs.
|
|
259
267
|
|
|
260
|
-
A ConceptQL statement yields the
|
|
268
|
+
A ConceptQL statement yields the results that pass all of its criteria. What to do with those results is up to the system that runs the statement. For example, a system that has all 250.01 results could:
|
|
261
269
|
|
|
262
270
|
- Find the first and last date of occurrence for each person
|
|
263
271
|
- Count the occurrences for each person
|
|
264
272
|
- Count the people with the condition
|
|
265
273
|
- Count the occurrences in the whole population
|
|
266
274
|
|
|
267
|
-
That kind of aggregation and analysis is outside the scope of ConceptQL. ConceptQL finds the
|
|
275
|
+
That kind of aggregation and analysis is outside the scope of ConceptQL. ConceptQL finds the results that you are interested in. Other parts of the system decide what to do with them.
|
|
268
276
|
|
|
269
277
|
The rest of this document describes each operator. [Appendix E](#appendix-e---operator-quick-reference) lists every operator in one table.
|
|
270
278
|
|
|
271
279
|
### Statements That ConceptQL Refuses
|
|
272
280
|
|
|
273
|
-
ConceptQL checks each statement before it builds SQL for it. A statement with an error is invalid. Examples of errors are an unknown operator, a required option that is missing,
|
|
281
|
+
ConceptQL checks each statement before it builds SQL for it. A statement with an error is invalid. Examples of errors are an unknown operator, a required option that is missing, a `recall` of a label that the statement does not have, and a code that is a number, such as 412 instead of "412". ConceptQL refuses to build SQL for an invalid statement, and so it does not run it. It reports each error with the name of the operator that has the error.
|
|
274
282
|
|
|
275
283
|
A warning does not make a statement invalid. Examples of warnings are an unknown code, an improperly formatted code and a deprecated operator. A statement with warnings runs. When ConceptQL draws the diagram of a statement, it shows the errors and the warnings of each operator, and it does not refuse an invalid statement.
|
|
276
284
|
|
|
277
|
-
ConceptQL finds the concept of each code in the vocabulary tables. It uses the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no
|
|
285
|
+
ConceptQL finds the concept of each code in the vocabulary tables. It uses the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no rows, and a statement would return no results with no warning. Without a database, for example when ConceptQL only draws a diagram, it uses no vocabulary tables. The vocabulary tables of a dataset can be trimmed to the codes that the dataset contains. Then ConceptQL can also read a full set of vocabulary tables, the reference lexicon, but only to tell a code that the dataset does not contain from a code that does not exist. When `LEXICON_URL` is set and the program names no other reference lexicon, the `LEXICON_URL` database is the reference lexicon.
|
|
278
286
|
|
|
279
287
|
## Selection Operators
|
|
280
288
|
|
|
281
|
-
Selection operators select
|
|
289
|
+
Selection operators select rows from the dataset and make results from them. For example, a search for diagnoses of old myocardial infarction (ICD-9-CM 412) is a selection. Most selection operators are leaf operators: no other operator flows into them. `person` and `death` can also take one upstream operator (see [`person`](#person-operator)).
|
|
282
290
|
|
|
283
|
-
The vocabulary operators select
|
|
291
|
+
The vocabulary operators select rows by their codes. The next section describes them, and [Other Selection Operators](#other-selection-operators) describes the rest.
|
|
284
292
|
|
|
285
293
|
## `Vocabulary` Operator
|
|
286
294
|
|
|
287
|
-
Selects the
|
|
295
|
+
Selects the results whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once.
|
|
288
296
|
|
|
289
297
|
- **Upstream:** none.
|
|
290
298
|
- **Arguments:**
|
|
291
|
-
- `codes` (codelist): One or more codes of the vocabulary. Write each code as the vocabulary writes it, including any dots: 250.01, not 25001. A code matches exactly, including case, and it does not select its child codes. A code alone of * selects every
|
|
292
|
-
- **Returns:** Every
|
|
299
|
+
- `codes` (codelist): One or more codes of the vocabulary. Write each code as the vocabulary writes it, including any dots: 250.01, not 25001. A code matches exactly, including case, and it does not select its child codes. A code alone of * selects every result of the vocabulary, and there is no other wildcard. A code that does not have the vocabulary's format matches nothing and gives a warning.
|
|
300
|
+
- **Returns:** Every result whose code is one of the given codes in the operator's vocabulary. The type of the results depends on the vocabulary, for example condition_occurrence for icd9cm and procedure_occurrence for cpt4. An operator does not follow mappings to other vocabularies.
|
|
293
301
|
|
|
294
302
|
______________________________________________________________________
|
|
295
303
|
|
|
296
|
-
**Example 6 - All
|
|
304
|
+
**Example 6 - All results with the ICD-9-CM diagnosis code 412 (old myocardial infarction)**
|
|
297
305
|
|
|
298
306
|
```JSON
|
|
299
307
|
["icd9cm","412"]
|
|
300
308
|
```
|
|
301
309
|
|
|
302
|
-

|
|
303
311
|
|
|
304
312
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
305
313
|
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
@@ -318,13 +326,13 @@ ______________________________________________________________________
|
|
|
318
326
|
|
|
319
327
|
______________________________________________________________________
|
|
320
328
|
|
|
321
|
-
**Example 7 - All MS-DRG 205
|
|
329
|
+
**Example 7 - All MS-DRG 205 results: other respiratory system diagnoses with a major complication or comorbidity**
|
|
322
330
|
|
|
323
331
|
```JSON
|
|
324
332
|
["ms_drg","205"]
|
|
325
333
|
```
|
|
326
334
|
|
|
327
|
-

|
|
328
336
|
|
|
329
337
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
330
338
|
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
@@ -336,13 +344,13 @@ ______________________________________________________________________
|
|
|
336
344
|
|
|
337
345
|
______________________________________________________________________
|
|
338
346
|
|
|
339
|
-
**Example 8 - All CMS-DRG 205
|
|
347
|
+
**Example 8 - All CMS-DRG 205 results (a liver disorder)**
|
|
340
348
|
|
|
341
349
|
```JSON
|
|
342
350
|
["cms_drg","205"]
|
|
343
351
|
```
|
|
344
352
|
|
|
345
|
-

|
|
346
354
|
|
|
347
355
|
No results.
|
|
348
356
|
|
|
@@ -352,22 +360,22 @@ ConceptQL creates one vocabulary operator for each vocabulary that it knows, so
|
|
|
352
360
|
|
|
353
361
|
These are the most common vocabulary operators:
|
|
354
362
|
|
|
355
|
-
| Operator | Alias | Vocabulary
|
|
356
|
-
| ------------ | -------------- |
|
|
357
|
-
| cpt4 | cpt | CPT-4 procedure codes
|
|
358
|
-
| hcpcs | | HCPCS procedure codes
|
|
359
|
-
| cpt_or_hcpcs | | CPT-4 and HCPCS together
|
|
360
|
-
| icd9cm | icd9 | ICD-9-CM diagnosis codes
|
|
361
|
-
| icd9proc | icd9_procedure | ICD-9-CM procedure codes (volume 3)
|
|
362
|
-
| icd10cm | | ICD-10-CM diagnosis codes
|
|
363
|
-
| icd10pcs | icd10_pcs | ICD-10-PCS procedure codes
|
|
364
|
-
| ms_drg | | MS-DRG codes, for
|
|
365
|
-
| cms_drg | | CMS-DRG codes, for
|
|
366
|
-
| revenue_code | | UB-04 revenue codes
|
|
367
|
-
| loinc | | LOINC lab codes
|
|
368
|
-
| ndc | | National Drug Codes (11-digit package codes)
|
|
369
|
-
| rxnorm | | RxNorm codes
|
|
370
|
-
| snomed | | SNOMED CT codes
|
|
363
|
+
| Operator | Alias | Vocabulary | Stream type |
|
|
364
|
+
| ------------ | -------------- | ----------------------------------------------- | -------------------- |
|
|
365
|
+
| cpt4 | cpt | CPT-4 procedure codes | procedure_occurrence |
|
|
366
|
+
| hcpcs | | HCPCS procedure codes | procedure_occurrence |
|
|
367
|
+
| cpt_or_hcpcs | | CPT-4 and HCPCS together | procedure_occurrence |
|
|
368
|
+
| icd9cm | icd9 | ICD-9-CM diagnosis codes | condition_occurrence |
|
|
369
|
+
| icd9proc | icd9_procedure | ICD-9-CM procedure codes (volume 3) | procedure_occurrence |
|
|
370
|
+
| icd10cm | | ICD-10-CM diagnosis codes | condition_occurrence |
|
|
371
|
+
| icd10pcs | icd10_pcs | ICD-10-PCS procedure codes | procedure_occurrence |
|
|
372
|
+
| ms_drg | | MS-DRG codes, for discharges from 2007-10-01 | procedure_occurrence |
|
|
373
|
+
| cms_drg | | CMS-DRG codes, for discharges before 2007-10-01 | procedure_occurrence |
|
|
374
|
+
| revenue_code | | UB-04 revenue codes | procedure_occurrence |
|
|
375
|
+
| loinc | | LOINC lab codes | observation |
|
|
376
|
+
| ndc | | National Drug Codes (11-digit package codes) | drug_exposure |
|
|
377
|
+
| rxnorm | | RxNorm codes | drug_exposure |
|
|
378
|
+
| snomed | | SNOMED CT codes | condition_occurrence |
|
|
371
379
|
|
|
372
380
|
Examples of how codes match:
|
|
373
381
|
|
|
@@ -378,26 +386,26 @@ Examples of how codes match:
|
|
|
378
386
|
- A code that the vocabulary does not contain gives the warning "unknown code(s)".
|
|
379
387
|
- When ConceptQL has a full lexicon (the reference lexicon) beside the vocabulary tables of the dataset, a code that only the full lexicon contains gives the warning "code(s) not present in this dataset", not "unknown code(s)". It is a warning, so the statement stays valid on a dataset that lacks some of its codes.
|
|
380
388
|
- A code that has more than one meaning in the vocabulary gives the warning "code(s) with more than one meaning".
|
|
381
|
-
- Some
|
|
382
|
-
- When the
|
|
383
|
-
- `rxnorm` does not find
|
|
389
|
+
- Some datasets store CPT-4 and HCPCS codes only in the merged vocabulary CPT4_HCPCS. On such a dataset, `cpt` and `hcpcs` select nothing, because they read only CPT4 and HCPCS. They give the warning "the data has no records of this vocabulary", which names `cpt_or_hcpcs`. Use `cpt_or_hcpcs`, which reads all three vocabularies.
|
|
390
|
+
- When the dataset lists the vocabularies of its rows (the GDM table `etl_vocabulary_ids`) and the operator's vocabulary is not in the list, the operator gives the warning "the data has no records of this vocabulary". Its counts are null, not 0, because the dataset cannot show whether a patient has the codes. This warning covers all of its codes, so they get no other warning, except the warning for an improperly formatted code.
|
|
391
|
+
- `rxnorm` does not find rows coded with NDC, because an operator does not follow mappings between vocabularies.
|
|
384
392
|
|
|
385
393
|
### DRG Codes
|
|
386
394
|
|
|
387
|
-
CMS renumbered the Diagnosis-Related Group (DRG) codes on 2007-10-01, when the MS-DRG system replaced the CMS-DRG system. Most codes got a new meaning. For example, DRG 205 meant a liver disorder in CMS-DRG, and it means other respiratory system diagnoses in MS-DRG. One vocabulary, DRG, holds the codes of both systems, and a
|
|
395
|
+
CMS renumbered the Diagnosis-Related Group (DRG) codes on 2007-10-01, when the MS-DRG system replaced the CMS-DRG system. Most codes got a new meaning. For example, DRG 205 meant a liver disorder in CMS-DRG, and it means other respiratory system diagnoses in MS-DRG. One vocabulary, DRG, holds the codes of both systems, and a row does not say which meaning applied. So a statement names the DRG system, and ConceptQL selects each row by its code and by its start_date:
|
|
388
396
|
|
|
389
|
-
- `ms_drg` selects the DRG
|
|
390
|
-
- `cms_drg` selects the DRG
|
|
397
|
+
- `ms_drg` selects the DRG rows whose start_date is on or after 2007-10-01.
|
|
398
|
+
- `cms_drg` selects the DRG rows whose start_date is before 2007-10-01.
|
|
391
399
|
|
|
392
|
-
The examples above show both operators. Our test
|
|
400
|
+
The examples above show both operators. Our test dataset has no DRG row before 2007-10-01, so the `cms_drg` example returns no results.
|
|
393
401
|
|
|
394
402
|
Both operators check codes as the other vocabulary operators do. A DRG code has 3 digits, with leading zeros. A code that the DRG vocabulary does not have gets the warning "unknown code(s)". These operators never give the warning "code(s) with more than one meaning", because the operator chooses the meaning.
|
|
395
403
|
|
|
396
|
-
The `drg` operator is deprecated. It selects the DRG
|
|
404
|
+
The `drg` operator is deprecated. It selects the DRG rows of both systems, whatever their dates. A statement that uses `drg` still runs, but `drg` gets the warning "deprecated operator", and a later release will make it an error. Use `ms_drg` or `cms_drg`.
|
|
397
405
|
|
|
398
406
|
### All Vocabulary Operators
|
|
399
407
|
|
|
400
|
-
The build writes this table from the metadata of the operators. It lists the vocabulary operators of a ConceptQL database with no vocabulary tables, ordered by name. Each row gives the full name of the operator's vocabulary and the domain, which is the type of the
|
|
408
|
+
The build writes this table from the metadata of the operators. It lists the vocabulary operators of a ConceptQL database with no vocabulary tables, ordered by name. Each row gives the full name of the operator's vocabulary and the domain, which is the type of the results that the operator returns. The vocabulary tables of a database can add more operators. A deprecated operator still runs, with a warning.
|
|
401
409
|
|
|
402
410
|
| Operator | Vocabulary | Domain |
|
|
403
411
|
| ------------------------ | ------------------------------------------------------------------------------------------------------ | -------------------- |
|
|
@@ -406,7 +414,7 @@ The build writes this table from the metadata of the operators. It lists the voc
|
|
|
406
414
|
| `atc` | WHO Anatomic Therapeutic Chemical Classification | drug_exposure |
|
|
407
415
|
| `betos` | BETOS | condition_occurrence |
|
|
408
416
|
| `clinical_code_type` | Clinical Code Type | condition_occurrence |
|
|
409
|
-
| `cms_drg` | CMS Diagnosis-Related Group,
|
|
417
|
+
| `cms_drg` | CMS Diagnosis-Related Group, discharges before 2007-10-01 | procedure_occurrence |
|
|
410
418
|
| `concept_class` | OMOP Concept Class | condition_occurrence |
|
|
411
419
|
| `condition_type` | OMOP Condition Occurrence Type | condition_occurrence |
|
|
412
420
|
| `cpt4` | Current Procedural Terminology version 4 (AMA) | procedure_occurrence |
|
|
@@ -435,7 +443,7 @@ The build writes this table from the metadata of the operators. It lists the voc
|
|
|
435
443
|
| `loinc` | Logical Observation Identifiers Names and Codes (Regenstrief Institute) | observation |
|
|
436
444
|
| `marcla_obs_type` | Market Clarity Observation Type | observation |
|
|
437
445
|
| `meas_type` | OMOP Measurement Type | observation |
|
|
438
|
-
| `ms_drg` | Medicare Severity Diagnosis-Related Group (CMS),
|
|
446
|
+
| `ms_drg` | Medicare Severity Diagnosis-Related Group (CMS), discharges from 2007-10-01 | procedure_occurrence |
|
|
439
447
|
| `ndc` | National Drug Code (FDA and manufacturers) | drug_exposure |
|
|
440
448
|
| `note_type` | OMOP Note Type | condition_occurrence |
|
|
441
449
|
| `nucc` | National Uniform Claim Committee Health Care Provider Taxonomy Code Set (NUCC) | condition_occurrence |
|
|
@@ -492,7 +500,7 @@ This section describes the selection operators that are not vocabulary operators
|
|
|
492
500
|
|
|
493
501
|
### Person-related Operators
|
|
494
502
|
|
|
495
|
-
Person operators select
|
|
503
|
+
Person operators select rows from the `patients` table, which holds the patient demographics. The type of their results is `person`. The start_date and end_date of a person result are both the patient's birth date. [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) explains why that is useful.
|
|
496
504
|
|
|
497
505
|
#### `Gender` Operator
|
|
498
506
|
|
|
@@ -501,7 +509,7 @@ Selects the patients with the given genders.
|
|
|
501
509
|
- **Upstream:** none.
|
|
502
510
|
- **Arguments:**
|
|
503
511
|
- `gender` (string, one of `Male`, `Female`, `Unknown`): One or more genders. A value that starts with M selects the male patients, and a value that starts with F selects the female patients. Any other value, such as Unknown, selects the patients whose gender is neither male nor female, including the patients with no gender. Case does not matter.
|
|
504
|
-
- **Returns:** One person
|
|
512
|
+
- **Returns:** One person result for each patient whose gender is one of the given genders. The start_date and the end_date of each result are the patient's birth date.
|
|
505
513
|
|
|
506
514
|
______________________________________________________________________
|
|
507
515
|
|
|
@@ -560,7 +568,7 @@ Selects the patients with the given races.
|
|
|
560
568
|
- **Upstream:** none.
|
|
561
569
|
- **Arguments:**
|
|
562
570
|
- `races` (codelist): One or more race names, such as White, or race concept IDs. A name matches the concepts with that name in any vocabulary, and case does not matter. The operator also selects the more detailed races under each given race, and each concept that is a kind of one of the race concepts through an is-a relationship. A value that contains Unknown also selects the patients who have no race or race concept ID 0.
|
|
563
|
-
- **Returns:** One person
|
|
571
|
+
- **Returns:** One person result for each patient whose race is one of the given races. The start_date and the end_date of each result are the patient's birth date.
|
|
564
572
|
|
|
565
573
|
______________________________________________________________________
|
|
566
574
|
|
|
@@ -594,7 +602,7 @@ Selects the patients with the given ethnicities.
|
|
|
594
602
|
- **Upstream:** none.
|
|
595
603
|
- **Arguments:**
|
|
596
604
|
- `ethnicity` (string, one of `Hispanic or Latino`, `Not Hispanic or Latino`): One or more ethnicities. A value that starts with Hispanic selects the Hispanic or Latino patients, and a value that starts with Not Hispanic selects the patients who are not Hispanic or Latino. A value that contains Unknown selects the patients who have no ethnicity or ethnicity concept ID 0. Case does not matter, and any other value selects no patients.
|
|
597
|
-
- **Returns:** One person
|
|
605
|
+
- **Returns:** One person result for each patient whose ethnicity is one of the given ethnicities. The start_date and the end_date of each result are the patient's birth date.
|
|
598
606
|
|
|
599
607
|
______________________________________________________________________
|
|
600
608
|
|
|
@@ -618,10 +626,10 @@ ______________________________________________________________________
|
|
|
618
626
|
|
|
619
627
|
#### `Person` Operator
|
|
620
628
|
|
|
621
|
-
Selects every patient. With one upstream operator, it selects the patients who have a
|
|
629
|
+
Selects every patient. With one upstream operator, it selects the patients who have a result in the upstream stream.
|
|
622
630
|
|
|
623
631
|
- **Upstream:** none, or one operator.
|
|
624
|
-
- **Returns:** One
|
|
632
|
+
- **Returns:** One result for each selected patient, from the patients table. The type of the results is person, and their start_date and end_date are both the patient's birth date.
|
|
625
633
|
|
|
626
634
|
______________________________________________________________________
|
|
627
635
|
|
|
@@ -677,20 +685,20 @@ ______________________________________________________________________
|
|
|
677
685
|
|
|
678
686
|
#### `Death` Operator
|
|
679
687
|
|
|
680
|
-
Selects every death
|
|
688
|
+
Selects every death. With one upstream operator, it selects the deaths of the patients who have a result in the upstream stream.
|
|
681
689
|
|
|
682
690
|
- **Upstream:** none, or one operator.
|
|
683
|
-
- **Returns:** One
|
|
691
|
+
- **Returns:** One result for each selected death, from the deaths table. The type of the results is death, and their start_date and end_date are both the date of death.
|
|
684
692
|
|
|
685
693
|
______________________________________________________________________
|
|
686
694
|
|
|
687
|
-
**Example 15 - All
|
|
695
|
+
**Example 15 - All deaths**
|
|
688
696
|
|
|
689
697
|
```JSON
|
|
690
698
|
["death"]
|
|
691
699
|
```
|
|
692
700
|
|
|
693
|
-

|
|
694
702
|
|
|
695
703
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
696
704
|
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
|
|
@@ -725,12 +733,12 @@ ______________________________________________________________________
|
|
|
725
733
|
|
|
726
734
|
#### `Information Periods` Operator
|
|
727
735
|
|
|
728
|
-
Selects information periods: the spans of time in which the
|
|
736
|
+
Selects information periods: the spans of time in which the dataset has information about a patient, such as periods of insurance enrollment.
|
|
729
737
|
|
|
730
738
|
- **Upstream:** none.
|
|
731
739
|
- **Arguments:**
|
|
732
740
|
- `information_types` (codelist): Zero or more information types, each as a concept ID (an integer) or a concept code of the GDM_INFORMATION_TYPE vocabulary, such as PARTD (Medicare Part D). A statement may mix both. With no information types, the operator selects every information period. With information types, it selects only the information periods whose type is one of them. An unknown code gets a warning and selects no period.
|
|
733
|
-
- **Returns:** One
|
|
741
|
+
- **Returns:** One result for each selected information period, from the information_periods table. The type of the results is observation_period.
|
|
734
742
|
|
|
735
743
|
______________________________________________________________________
|
|
736
744
|
|
|
@@ -757,20 +765,30 @@ ______________________________________________________________________
|
|
|
757
765
|
|
|
758
766
|
______________________________________________________________________
|
|
759
767
|
|
|
768
|
+
An information period is a span of time in which the dataset has a given kind of information about a person. The main kinds are periods of eligibility and periods of enrollment.
|
|
769
|
+
|
|
770
|
+
A person can have several information periods of different types, and the periods can overlap. For example, a person can be enrolled in Medicare Part B for one span and in Part D for a different span. Without Part D enrollment, the dataset gives little insight into the prescription drugs of the person. So the type of a period tells which kinds of information exist for that span. In the example above, each person has several periods that overlap.
|
|
771
|
+
|
|
772
|
+
The CDM of OMOP has a similar table, [`observation_period`](https://ohdsi.github.io/CommonDataModel/cdm54.html#observation_period). The CDM describes it so: "This table contains records which define spans of time during which two conditions are expected to hold: (i) Clinical Events that happened to the Person are recorded in the Event tables, and (ii) absence of records indicate such Events did not occur during this span of time."
|
|
773
|
+
|
|
774
|
+
ConceptQL uses the name "information period", because each span says which information the dataset holds. In study design, an observation period is a choice of the analysis. It is often shorter than the span of information that exists.
|
|
775
|
+
|
|
776
|
+
In general, a period is a span of time in the dataset, and a window is a span of time in a study (see [Windows](#windows)).
|
|
777
|
+
|
|
760
778
|
### Utilization Operators
|
|
761
779
|
|
|
762
|
-
Utilization operators select admissions. Each operator returns one
|
|
780
|
+
Utilization operators select admissions. Each operator returns one result for each admission whose claim came from the matching kind of file. For example, `hospitalization` uses the claims whose file provenance is inpatient.
|
|
763
781
|
|
|
764
|
-
The type of these
|
|
782
|
+
The type of these results is `condition_occurrence`, because the GDM has no visit or admission type. So `union` combines them with diagnosis results into one stream. `source_value` usually holds a diagnosis.
|
|
765
783
|
|
|
766
|
-
Our test
|
|
784
|
+
Our test dataset has no SNF claims and no hospice claims, so the examples of `snf` and `hospice` return no results.
|
|
767
785
|
|
|
768
786
|
#### `SNF` Operator
|
|
769
787
|
|
|
770
|
-
Selects admissions to skilled nursing facilities (SNF): one
|
|
788
|
+
Selects admissions to skilled nursing facilities (SNF): one result for each admission whose claim came from an SNF file.
|
|
771
789
|
|
|
772
790
|
- **Upstream:** none.
|
|
773
|
-
- **Returns:** One
|
|
791
|
+
- **Returns:** One result for each SNF admission. The type of the results is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each result also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
|
|
774
792
|
|
|
775
793
|
______________________________________________________________________
|
|
776
794
|
|
|
@@ -788,10 +806,10 @@ ______________________________________________________________________
|
|
|
788
806
|
|
|
789
807
|
#### `Hospice` Operator
|
|
790
808
|
|
|
791
|
-
Selects hospice admissions: one
|
|
809
|
+
Selects hospice admissions: one result for each admission whose claim came from a hospice file.
|
|
792
810
|
|
|
793
811
|
- **Upstream:** none.
|
|
794
|
-
- **Returns:** One
|
|
812
|
+
- **Returns:** One result for each hospice admission. The type of the results is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each result also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
|
|
795
813
|
|
|
796
814
|
______________________________________________________________________
|
|
797
815
|
|
|
@@ -809,10 +827,10 @@ ______________________________________________________________________
|
|
|
809
827
|
|
|
810
828
|
#### `Hospitalization` Operator
|
|
811
829
|
|
|
812
|
-
Selects inpatient hospital admissions: one
|
|
830
|
+
Selects inpatient hospital admissions: one result for each admission whose claim came from an inpatient file.
|
|
813
831
|
|
|
814
832
|
- **Upstream:** none.
|
|
815
|
-
- **Returns:** One
|
|
833
|
+
- **Returns:** One result for each inpatient admission. The type of the results is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each result also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
|
|
816
834
|
|
|
817
835
|
______________________________________________________________________
|
|
818
836
|
|
|
@@ -841,14 +859,14 @@ ______________________________________________________________________
|
|
|
841
859
|
|
|
842
860
|
## Set Operators
|
|
843
861
|
|
|
844
|
-
Streams are sets of
|
|
862
|
+
Streams are sets of results, so ConceptQL has operators that work on sets.
|
|
845
863
|
|
|
846
864
|
### `Union` Operator
|
|
847
865
|
|
|
848
|
-
Combines the
|
|
866
|
+
Combines the results of all its upstream operators into one stream. Each result keeps its own type, so one stream can hold several types.
|
|
849
867
|
|
|
850
|
-
- **Upstream:**
|
|
851
|
-
- **Returns:** Each
|
|
868
|
+
- **Upstream:** two or more operators.
|
|
869
|
+
- **Returns:** Each result of each upstream. A result that comes from more than one upstream comes out once. Two results are the same only when every column is equal, including label.
|
|
852
870
|
|
|
853
871
|
______________________________________________________________________
|
|
854
872
|
|
|
@@ -925,16 +943,16 @@ ______________________________________________________________________
|
|
|
925
943
|
|
|
926
944
|
______________________________________________________________________
|
|
927
945
|
|
|
928
|
-
Union does not change
|
|
946
|
+
Union does not change results of different types. Each later operator decides whether it treats the types separately or together.
|
|
929
947
|
|
|
930
948
|
### `Except` Operator
|
|
931
949
|
|
|
932
|
-
Removes each left-hand
|
|
950
|
+
Removes each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same criterion_table, criterion_id, start_date and end_date.
|
|
933
951
|
|
|
934
952
|
- **Operands:**
|
|
935
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
936
|
-
- `right` (required): The right-hand stream. The operator removes each left-hand
|
|
937
|
-
- **Returns:** The left-hand
|
|
953
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
954
|
+
- `right` (required): The right-hand stream. The operator removes each left-hand result that is also in this stream, and does not pass these results on.
|
|
955
|
+
- **Returns:** The left-hand results that have no right-hand result with the same criterion_table, criterion_id, start_date and end_date. A left-hand result whose table has no right-hand result passes through unchanged. The right-hand results are not returned.
|
|
938
956
|
|
|
939
957
|
______________________________________________________________________
|
|
940
958
|
|
|
@@ -1011,11 +1029,11 @@ ______________________________________________________________________
|
|
|
1011
1029
|
|
|
1012
1030
|
______________________________________________________________________
|
|
1013
1031
|
|
|
1014
|
-
A left-hand
|
|
1032
|
+
A left-hand result matches only a right-hand result from the same table, and only when the dates are the same. For example, a result whose dates `time_window` changed no longer matches the original result. The second example above shows that a left-hand result passes through unchanged when no right-hand result comes from its table. The third example has several types in the left-hand stream.
|
|
1015
1033
|
|
|
1016
1034
|
### Discussion About Set Operators
|
|
1017
1035
|
|
|
1018
|
-
Why can one stream hold
|
|
1036
|
+
Why can one stream hold results of several types? Because operators such as `first` then act across all the types. For example, the statement below finds the first MI or death for each patient. Add a few more criteria, and you can find the first censoring event for each patient.
|
|
1019
1037
|
|
|
1020
1038
|
______________________________________________________________________
|
|
1021
1039
|
|
|
@@ -1044,28 +1062,28 @@ ______________________________________________________________________
|
|
|
1044
1062
|
|
|
1045
1063
|
## Time-oriented Operators
|
|
1046
1064
|
|
|
1047
|
-
Every
|
|
1065
|
+
Every result has a start_date and an end_date. All temporal comparisons use these two columns. Each result takes its dates from its row in its table of origin.
|
|
1048
1066
|
|
|
1049
|
-
If the table of origin has only one date, the
|
|
1067
|
+
If the table of origin has only one date, the result uses that date as both its start_date and its end_date. For example, a death result uses the `date` column of its row in `deaths`.
|
|
1050
1068
|
|
|
1051
|
-
Person
|
|
1069
|
+
Person results are a special case. A person result uses the patient's birth date as its start_date and end_date. This may sound strange, but [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) shows why it is useful.
|
|
1052
1070
|
|
|
1053
1071
|
## Relative Temporal Operators
|
|
1054
1072
|
|
|
1055
|
-
For each patient, we may want only the chronologically first or last
|
|
1073
|
+
For each patient, we may want only the chronologically first or last result, or the second, or the second-to-last. Relative temporal operators provide this filtering. They order each patient's results by start_date. When two results have the same start_date, the result with the lower `criterion_id` comes first.
|
|
1056
1074
|
|
|
1057
1075
|
### `Nth Occurrence` Operator
|
|
1058
1076
|
|
|
1059
|
-
For each person, orders the
|
|
1077
|
+
For each person, orders the results by start_date and passes along the result at the given position. A positive position counts from the earliest result and a negative one from the most recent result.
|
|
1060
1078
|
|
|
1061
1079
|
- **Upstream:** one operator.
|
|
1062
1080
|
- **Arguments:**
|
|
1063
|
-
- `occurrence` (integer): The position of the
|
|
1081
|
+
- `occurrence` (integer): The position of the result to pass along, as an integer. A positive number counts from the earliest result, so 1 is the first result and 2 is the second. A negative number counts from the most recent result, so -1 is the last result and -3 is the third from the last. 0 selects nothing.
|
|
1064
1082
|
- **Options:**
|
|
1065
|
-
- `at_least` (string): A date adjustment, such as 30d. For each
|
|
1066
|
-
- `within` (string): A date adjustment, such as 365d. For each
|
|
1067
|
-
- `group_by_date` (boolean): Treats the
|
|
1068
|
-
- **Returns:** One
|
|
1083
|
+
- `at_least` (string): A date adjustment, such as 30d. For each result, the operator counts that result and the later results that start at least this long after it ends, and passes along the result at the given position. So a person can have more than one result.
|
|
1084
|
+
- `within` (string): A date adjustment, such as 365d. For each result, the operator counts that result and the later results that start at most this long after it ends, and passes along the result at the given position. So a person can have more than one result.
|
|
1085
|
+
- `group_by_date` (boolean): Treats the results of a person on one start_date as one occurrence, so the position counts dates. From the date at that position, the operator passes along the result with the lowest criterion_id, or the highest criterion_id for a negative position.
|
|
1086
|
+
- **Returns:** One result per person: the result at the given position. A person who has fewer results than the position asks for is not passed along. With at_least or within, a person can have more than one result.
|
|
1069
1087
|
|
|
1070
1088
|
______________________________________________________________________
|
|
1071
1089
|
|
|
@@ -1094,13 +1112,13 @@ ______________________________________________________________________
|
|
|
1094
1112
|
|
|
1095
1113
|
______________________________________________________________________
|
|
1096
1114
|
|
|
1097
|
-
**Example 29 - For each patient, the first
|
|
1115
|
+
**Example 29 - For each patient, the first result on the second day with any of these diagnoses**
|
|
1098
1116
|
|
|
1099
1117
|
```JSON
|
|
1100
1118
|
["occurrence",2,["icd9","410.00","410.01","250.00","250.01"],{"group_by_date":true}]
|
|
1101
1119
|
```
|
|
1102
1120
|
|
|
1103
|
-

|
|
1104
1122
|
|
|
1105
1123
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1106
1124
|
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
@@ -1121,10 +1139,10 @@ ______________________________________________________________________
|
|
|
1121
1139
|
|
|
1122
1140
|
### `First` Operator
|
|
1123
1141
|
|
|
1124
|
-
For each person, passes along the
|
|
1142
|
+
For each person, passes along the result with the earliest start_date.
|
|
1125
1143
|
|
|
1126
1144
|
- **Upstream:** one operator.
|
|
1127
|
-
- **Returns:** One
|
|
1145
|
+
- **Returns:** One result per person: the upstream result with the earliest start_date. When several results have that start_date, the one with the lowest criterion_id.
|
|
1128
1146
|
|
|
1129
1147
|
______________________________________________________________________
|
|
1130
1148
|
|
|
@@ -1155,10 +1173,10 @@ ______________________________________________________________________
|
|
|
1155
1173
|
|
|
1156
1174
|
### `Last` Operator
|
|
1157
1175
|
|
|
1158
|
-
For each person, passes along the
|
|
1176
|
+
For each person, passes along the result with the most recent start_date.
|
|
1159
1177
|
|
|
1160
1178
|
- **Upstream:** one operator.
|
|
1161
|
-
- **Returns:** One
|
|
1179
|
+
- **Returns:** One result per person: the upstream result with the latest start_date. When several results have that start_date, the one with the highest criterion_id.
|
|
1162
1180
|
|
|
1163
1181
|
______________________________________________________________________
|
|
1164
1182
|
|
|
@@ -1193,13 +1211,13 @@ To represent a fixed range of dates, use the `date_range` operator.
|
|
|
1193
1211
|
|
|
1194
1212
|
### `Date Range` Operator
|
|
1195
1213
|
|
|
1196
|
-
Makes one person
|
|
1214
|
+
Makes one person result for each patient, with the given start_date and end_date. Use it to compare other results with a fixed range of dates.
|
|
1197
1215
|
|
|
1198
1216
|
- **Upstream:** none.
|
|
1199
1217
|
- **Options:**
|
|
1200
|
-
- `start` (string, required): The start_date of each
|
|
1201
|
-
- `end` (string, required): The end_date of each
|
|
1202
|
-
- **Returns:** One
|
|
1218
|
+
- `start` (string, required): The start_date of each result. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive.
|
|
1219
|
+
- `end` (string, required): The end_date of each result. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive. For a single day, give the same date as the start option.
|
|
1220
|
+
- **Returns:** One result for each patient. The type of the results is person, and each result has the start_date and the end_date that the options give.
|
|
1203
1221
|
|
|
1204
1222
|
______________________________________________________________________
|
|
1205
1223
|
|
|
@@ -1228,13 +1246,13 @@ ______________________________________________________________________
|
|
|
1228
1246
|
|
|
1229
1247
|
______________________________________________________________________
|
|
1230
1248
|
|
|
1231
|
-
**Example 33 - Each patient, with the dates from the start to the end of the
|
|
1249
|
+
**Example 33 - Each patient, with the dates from the start to the end of the dataset**
|
|
1232
1250
|
|
|
1233
1251
|
```JSON
|
|
1234
1252
|
["date_range",{"start":"START","end":"END"}]
|
|
1235
1253
|
```
|
|
1236
1254
|
|
|
1237
|
-

|
|
1238
1256
|
|
|
1239
1257
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
1240
1258
|
| --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
|
|
@@ -1255,13 +1273,13 @@ The range includes both dates. A `YYYY-MM-DD` date has a four-digit year, a two-
|
|
|
1255
1273
|
|
|
1256
1274
|
## Temporal Comparison Operators
|
|
1257
1275
|
|
|
1258
|
-
As described above, each
|
|
1276
|
+
As described above, each result has a start date and an end date, which define its own date range. Temporal operators filter streams by comparing these date ranges.
|
|
1259
1277
|
|
|
1260
|
-
A temporal operator takes two streams: the left-hand
|
|
1278
|
+
A temporal operator takes two streams: the left-hand results (LHR) and the right-hand results (RHR). It compares each patient's LHR with the same patient's RHR, and it passes along the LHR that pass the comparison. It discards the RHR after the comparison.
|
|
1261
1279
|
|
|
1262
1280
|
- Each LHR is passed along at most once, however many RHR it matches.
|
|
1263
1281
|
- A patient who has LHR but no RHR loses all of their LHR.
|
|
1264
|
-
- A
|
|
1282
|
+
- A result that is in both streams can match itself. For example, a result is during itself.
|
|
1265
1283
|
|
|
1266
1284
|
Edge behaviors: `during`, `contains` and `any_overlap` compare each LHR with every RHR of the patient. `before` and `after` do not. Without the `within` option, `before` compares each LHR only with the latest start_date of the patient's RHR, and `after` compares each LHR only with the earliest end_date of the patient's RHR. [`before`](#before-operator) and [`after`](#after-operator) explain why, and [Working Around `Before`/`After` Behavior](#working-around-beforeafter-behavior) shows how to choose another RHR.
|
|
1267
1285
|
|
|
@@ -1269,14 +1287,14 @@ Each operator compares the dates in a different way. All of them take [some opti
|
|
|
1269
1287
|
|
|
1270
1288
|
### `During` Operator
|
|
1271
1289
|
|
|
1272
|
-
For each patient, keeps the left-hand
|
|
1290
|
+
For each patient, keeps the left-hand results whose date range is inside the date range of a right-hand result. The left-hand result starts on or after the start_date of the right-hand result and ends on or before its end_date.
|
|
1273
1291
|
|
|
1274
1292
|
- **Operands:**
|
|
1275
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
1276
|
-
- `right` (required): The right-hand stream. The operator compares the left-hand
|
|
1293
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
1294
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
|
|
1277
1295
|
- **Options:**
|
|
1278
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand
|
|
1279
|
-
- **Returns:** The left-hand
|
|
1296
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand result is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
|
|
1297
|
+
- **Returns:** The left-hand results that are inside the date range of at least one right-hand result of the same patient, each result once. The right-hand results are not returned.
|
|
1280
1298
|
|
|
1281
1299
|
______________________________________________________________________
|
|
1282
1300
|
|
|
@@ -1305,14 +1323,14 @@ ______________________________________________________________________
|
|
|
1305
1323
|
|
|
1306
1324
|
### `Contains` Operator
|
|
1307
1325
|
|
|
1308
|
-
For each patient, keeps the left-hand
|
|
1326
|
+
For each patient, keeps the left-hand results whose date range encloses the date range of a right-hand result. The left-hand result starts on or before the start_date of the right-hand result and ends on or after its end_date.
|
|
1309
1327
|
|
|
1310
1328
|
- **Operands:**
|
|
1311
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
1312
|
-
- `right` (required): The right-hand stream. The operator compares the left-hand
|
|
1329
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
1330
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
|
|
1313
1331
|
- **Options:**
|
|
1314
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each left-hand
|
|
1315
|
-
- **Returns:** The left-hand
|
|
1332
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each left-hand result is widened by this amount in each direction: its start_date moves back and its end_date moves forward. The returned results keep their own dates. Use a positive amount.
|
|
1333
|
+
- **Returns:** The left-hand results that enclose the date range of at least one right-hand result of the same patient, each result once. The right-hand results are not returned.
|
|
1316
1334
|
|
|
1317
1335
|
______________________________________________________________________
|
|
1318
1336
|
|
|
@@ -1332,14 +1350,14 @@ ______________________________________________________________________
|
|
|
1332
1350
|
|
|
1333
1351
|
### `Any Overlap` Operator
|
|
1334
1352
|
|
|
1335
|
-
For each patient, keeps the left-hand
|
|
1353
|
+
For each patient, keeps the left-hand results whose date range shares at least one day with the date range of a right-hand result.
|
|
1336
1354
|
|
|
1337
1355
|
- **Operands:**
|
|
1338
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
1339
|
-
- `right` (required): The right-hand stream. The operator compares the left-hand
|
|
1356
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
1357
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
|
|
1340
1358
|
- **Options:**
|
|
1341
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand
|
|
1342
|
-
- **Returns:** The left-hand
|
|
1359
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand result is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
|
|
1360
|
+
- **Returns:** The left-hand results that overlap the date range of at least one right-hand result of the same patient, each result once. The right-hand results are not returned.
|
|
1343
1361
|
|
|
1344
1362
|
______________________________________________________________________
|
|
1345
1363
|
|
|
@@ -1358,21 +1376,21 @@ ______________________________________________________________________
|
|
|
1358
1376
|
|
|
1359
1377
|
______________________________________________________________________
|
|
1360
1378
|
|
|
1361
|
-
This diagram shows each kind of L
|
|
1379
|
+
This diagram shows each kind of L result that has "any overlap" with an R result:
|
|
1362
1380
|
|
|
1363
1381
|

|
|
1364
1382
|
|
|
1365
1383
|
### `Before` Operator
|
|
1366
1384
|
|
|
1367
|
-
For each patient, keeps the left-hand
|
|
1385
|
+
For each patient, keeps the left-hand results whose end_date is before the latest start_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead.
|
|
1368
1386
|
|
|
1369
1387
|
- **Operands:**
|
|
1370
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
1371
|
-
- `right` (required): The right-hand stream. The operator compares the left-hand
|
|
1388
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
1389
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
|
|
1372
1390
|
- **Options:**
|
|
1373
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand
|
|
1374
|
-
- `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand
|
|
1375
|
-
- **Returns:** The left-hand
|
|
1391
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand result must also end no more than this amount before the start_date of the right-hand result. With this option, each left-hand result is compared with every right-hand result of the patient, not only with the one that starts last. Use a positive amount.
|
|
1392
|
+
- `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand result must end more than this amount before the start_date of the right-hand result. Use a positive amount.
|
|
1393
|
+
- **Returns:** The left-hand results that end before a right-hand result of the same patient starts, each result once. The right-hand results are not returned.
|
|
1376
1394
|
|
|
1377
1395
|
______________________________________________________________________
|
|
1378
1396
|
|
|
@@ -1442,19 +1460,19 @@ ______________________________________________________________________
|
|
|
1442
1460
|
|
|
1443
1461
|
The comparison is strict, so an LHR that ends on the day the RHR starts does not pass.
|
|
1444
1462
|
|
|
1445
|
-
This rule needs some explanation. Imagine the
|
|
1463
|
+
This rule needs some explanation. Imagine the results A-A-B-A-B-A. How many A's come before a B? One could argue that only the first two A's qualify, because they come before the first B. But three A's come before some B. The `before` operator uses the second answer, so it compares every LHR with the start_date of the patient's last RHR.
|
|
1446
1464
|
|
|
1447
1465
|
### `After` Operator
|
|
1448
1466
|
|
|
1449
|
-
For each patient, keeps the left-hand
|
|
1467
|
+
For each patient, keeps the left-hand results whose start_date is after the earliest end_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead.
|
|
1450
1468
|
|
|
1451
1469
|
- **Operands:**
|
|
1452
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
1453
|
-
- `right` (required): The right-hand stream. The operator compares the left-hand
|
|
1470
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
1471
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
|
|
1454
1472
|
- **Options:**
|
|
1455
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand
|
|
1456
|
-
- `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand
|
|
1457
|
-
- **Returns:** The left-hand
|
|
1473
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand result must also start no more than this amount after the end_date of the right-hand result. With this option, each left-hand result is compared with every right-hand result of the patient, not only with the one that ends first. Use a positive amount.
|
|
1474
|
+
- `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand result must start more than this amount after the end_date of the right-hand result. Use a positive amount.
|
|
1475
|
+
- **Returns:** The left-hand results that start after a right-hand result of the same patient ends, each result once. The right-hand results are not returned.
|
|
1458
1476
|
|
|
1459
1477
|
______________________________________________________________________
|
|
1460
1478
|
|
|
@@ -1539,7 +1557,7 @@ ______________________________________________________________________
|
|
|
1539
1557
|
|
|
1540
1558
|
The comparison is strict, so an LHR that starts on the day the RHR ends does not pass.
|
|
1541
1559
|
|
|
1542
|
-
Imagine the
|
|
1560
|
+
Imagine the results A-A-B-A-B-A again. How many A's come after a B? One could argue that only the last A qualifies, because it comes after the last B. But two A's come after some B. The `after` operator uses the second answer, so it compares every LHR with the earliest end_date of the patient's RHR.
|
|
1543
1561
|
|
|
1544
1562
|
### Working Around `Before`/`After` Behavior
|
|
1545
1563
|
|
|
@@ -1549,13 +1567,13 @@ If you want a different RHR for the comparison, use a relative temporal operator
|
|
|
1549
1567
|
|
|
1550
1568
|
### `Time Window` Operator
|
|
1551
1569
|
|
|
1552
|
-
Moves the start_date and the end_date of each
|
|
1570
|
+
Moves the start_date and the end_date of each result. Use it to shift or widen a result's dates before you compare them with other results.
|
|
1553
1571
|
|
|
1554
1572
|
- **Upstream:** one operator.
|
|
1555
1573
|
- **Options:**
|
|
1556
|
-
- `start` (string): How to change the start_date. If you omit it, the start_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the
|
|
1557
|
-
- `end` (string): How to change the end_date. If you omit it, the end_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the
|
|
1558
|
-
- **Returns:** One
|
|
1574
|
+
- `start` (string): How to change the start_date. If you omit it, the start_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the result's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
|
|
1575
|
+
- `end` (string): How to change the end_date. If you omit it, the end_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the result's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
|
|
1576
|
+
- **Returns:** One result for each upstream result, with the start_date and the end_date that the options give. The other columns do not change.
|
|
1559
1577
|
|
|
1560
1578
|
______________________________________________________________________
|
|
1561
1579
|
|
|
@@ -1684,14 +1702,14 @@ More examples of date adjustments:
|
|
|
1684
1702
|
|
|
1685
1703
|
### `Trim Date Start` Operator
|
|
1686
1704
|
|
|
1687
|
-
For each patient, moves the start_date of each left-hand
|
|
1705
|
+
For each patient, moves the start_date of each left-hand result forward to the end_date of the right-hand result with the latest start_date. A left-hand result that ends before that end_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged.
|
|
1688
1706
|
|
|
1689
1707
|
- **Operands:**
|
|
1690
|
-
- `left` (required): The left-hand stream. The operator moves the start_date of these
|
|
1691
|
-
- `right` (required): The right-hand stream. For each patient, the end_date of the
|
|
1708
|
+
- `left` (required): The left-hand stream. The operator moves the start_date of these results and passes them on.
|
|
1709
|
+
- `right` (required): The right-hand stream. For each patient, the end_date of the result with the latest start_date sets the earliest start_date of the left-hand results. These results are not passed on.
|
|
1692
1710
|
- **Options:**
|
|
1693
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is added to the end_date of the right-hand
|
|
1694
|
-
- **Returns:** The left-hand
|
|
1711
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is added to the end_date of the right-hand result before the comparison. The adjusted date is the one that moves the start_date of the left-hand result.
|
|
1712
|
+
- **Returns:** The left-hand results, each with its start_date set to the later of its own start_date and the end_date of the patient's right-hand result. Left-hand results that end before that end_date are not returned. The right-hand results are not returned.
|
|
1695
1713
|
|
|
1696
1714
|
______________________________________________________________________
|
|
1697
1715
|
|
|
@@ -1718,7 +1736,7 @@ ______________________________________________________________________
|
|
|
1718
1736
|
|
|
1719
1737
|
______________________________________________________________________
|
|
1720
1738
|
|
|
1721
|
-
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's
|
|
1739
|
+
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's results. There are four cases:
|
|
1722
1740
|
|
|
1723
1741
|
- The patient has no RHR.
|
|
1724
1742
|
- The LHR passes through unchanged.
|
|
@@ -1731,14 +1749,14 @@ Like a [temporal comparison operator](#temporal-comparison-operators), it takes
|
|
|
1731
1749
|
|
|
1732
1750
|
### `Trim Date End` Operator
|
|
1733
1751
|
|
|
1734
|
-
For each patient, moves the end_date of each left-hand
|
|
1752
|
+
For each patient, moves the end_date of each left-hand result back to the start_date of the right-hand result with the earliest start_date. A left-hand result that starts after that start_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged.
|
|
1735
1753
|
|
|
1736
1754
|
- **Operands:**
|
|
1737
|
-
- `left` (required): The left-hand stream. The operator moves the end_date of these
|
|
1738
|
-
- `right` (required): The right-hand stream. For each patient, the earliest start_date of these
|
|
1755
|
+
- `left` (required): The left-hand stream. The operator moves the end_date of these results and passes them on.
|
|
1756
|
+
- `right` (required): The right-hand stream. For each patient, the earliest start_date of these results sets the latest end_date of the left-hand results. These results are not passed on.
|
|
1739
1757
|
- **Options:**
|
|
1740
|
-
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is subtracted from the start_date of the right-hand
|
|
1741
|
-
- **Returns:** The left-hand
|
|
1758
|
+
- `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is subtracted from the start_date of the right-hand result before the comparison. The adjusted date is the one that moves the end_date of the left-hand result.
|
|
1759
|
+
- **Returns:** The left-hand results, each with its end_date set to the earlier of its own end_date and the start_date of the patient's right-hand result. Left-hand results that start after that start_date are not returned. The right-hand results are not returned.
|
|
1742
1760
|
|
|
1743
1761
|
______________________________________________________________________
|
|
1744
1762
|
|
|
@@ -1765,7 +1783,7 @@ ______________________________________________________________________
|
|
|
1765
1783
|
|
|
1766
1784
|
______________________________________________________________________
|
|
1767
1785
|
|
|
1768
|
-
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's
|
|
1786
|
+
Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's results. There are four cases:
|
|
1769
1787
|
|
|
1770
1788
|
- The patient has no RHR.
|
|
1771
1789
|
- The LHR passes through unchanged.
|
|
@@ -1787,7 +1805,7 @@ Systems that build statements, such as Jigsaw's export engine, give `trim_date_s
|
|
|
1787
1805
|
- The `within` option moves each RHR date before the comparison, as above.
|
|
1788
1806
|
- An LHR is discarded only when it has RHR (of its patient and its `join_columns` values) and none of them qualifies. An LHR without RHR passes through unchanged.
|
|
1789
1807
|
|
|
1790
|
-
These systems often join on the column `uuid`. A selection operator, such as a vocabulary operator, takes the option `uuid` (`{"uuid": true}`), which adds the column `uuid` to its
|
|
1808
|
+
These systems often join on the column `uuid`. A selection operator, such as a vocabulary operator, takes the option `uuid` (`{"uuid": true}`), which adds the column `uuid` to its results. The uuid is the `person_id`, `criterion_id`, `criterion_table`, start_date and end_date of the result, joined with `/`, such as `17/27269/observations/2008-08-25/2008-08-25`. An empty part is an empty string, so a result without an end_date has a uuid such as `1/2/observations/2008-03-10/`. The uuid keeps its value when a later operator, such as `time_window`, moves the dates. So two streams made from the same results can be joined result by result.
|
|
1791
1809
|
|
|
1792
1810
|
This example makes a one-year window after each MI. Then it trims each window to end 30 days after its own MI. The `join_columns` option makes sure that no other MI of the same patient trims the window:
|
|
1793
1811
|
|
|
@@ -1831,23 +1849,23 @@ Walk-through of the second example of [`after`](#after-operator), which finds th
|
|
|
1831
1849
|
Walk-through of the third example of [`before`](#before-operator), which asks whether anyone dies within a year of a heart attack:
|
|
1832
1850
|
|
|
1833
1851
|
- Select heart attack diagnoses as the LHR.
|
|
1834
|
-
- Select
|
|
1852
|
+
- Select deaths as the RHR.
|
|
1835
1853
|
- Keep the LHR whose end_date is before the date of death, and no more than 1 year before it.
|
|
1836
1854
|
|
|
1837
1855
|
The third example of [`after`](#after-operator) shows how `at_least` works.
|
|
1838
1856
|
|
|
1839
1857
|
### Temporal Operators and Person Streams
|
|
1840
1858
|
|
|
1841
|
-
The start and end dates of a person
|
|
1859
|
+
The start and end dates of a person result are the patient's birth date. So a person stream is rarely useful as the left-hand stream of a temporal operator. But it is useful as the right-hand stream. If we move the birth date with `time_window`, we can filter by the patient's age. The last example of [`after`](#after-operator) selects the MIs that occurred after a male patient's 50th birthday.
|
|
1842
1860
|
|
|
1843
1861
|
## `Episode` Operator
|
|
1844
1862
|
|
|
1845
|
-
Merges each patient's
|
|
1863
|
+
Merges each patient's results into episodes: spans of time in which the results are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug.
|
|
1846
1864
|
|
|
1847
1865
|
- **Upstream:** one operator.
|
|
1848
1866
|
- **Options:**
|
|
1849
|
-
- `gap_of` (integer): The largest gap in days that stays in the same episode. The operator orders each patient's
|
|
1850
|
-
- **Returns:** One
|
|
1867
|
+
- `gap_of` (integer): The largest gap in days that stays in the same episode. The operator orders each patient's results by start_date and then by end_date. A result stays in the current episode when its start_date is no more than this many days after the latest end_date of the earlier results, not only of the result just before. Otherwise it starts a new episode. A result without an end_date counts as ending on its start_date. The default is 0.
|
|
1868
|
+
- **Returns:** One result for each episode. Its start_date is the earliest start_date in the episode, and its end_date is the latest end_date. Its criterion_table and criterion_domain are episode, and its criterion_id is 0, because an episode merges several results.
|
|
1851
1869
|
|
|
1852
1870
|
______________________________________________________________________
|
|
1853
1871
|
|
|
@@ -1874,20 +1892,20 @@ ______________________________________________________________________
|
|
|
1874
1892
|
|
|
1875
1893
|
______________________________________________________________________
|
|
1876
1894
|
|
|
1877
|
-
A long
|
|
1895
|
+
A long result keeps the episode open: the results 2020-01-01 to 2020-03-01 and 2020-01-05 to 2020-01-10 and 2020-02-15 to 2020-02-20 make one episode with `gap_of` 0, because the first result ends after the third result starts. An episode stream is often most useful as the right-hand stream of a [temporal operator](#temporal-comparison-operators).
|
|
1878
1896
|
|
|
1879
1897
|
## Inline-Filter Operators
|
|
1880
1898
|
|
|
1881
|
-
These operators filter one incoming stream and pass along the
|
|
1899
|
+
These operators filter one incoming stream and pass along the results that remain.
|
|
1882
1900
|
|
|
1883
1901
|
### `Place of Service Filter` Operator
|
|
1884
1902
|
|
|
1885
|
-
Passes along the
|
|
1903
|
+
Passes along the results whose place of service is one of the given CMS Place of Service codes. A result that has no place of service is not passed along.
|
|
1886
1904
|
|
|
1887
1905
|
- **Upstream:** one operator.
|
|
1888
1906
|
- **Arguments:**
|
|
1889
|
-
- `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a
|
|
1890
|
-
- **Returns:** The upstream
|
|
1907
|
+
- `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a result whose place of service is any code of that vocabulary. An unknown code gets a warning and matches no result.
|
|
1908
|
+
- **Returns:** The upstream results whose place of service is one of the given codes.
|
|
1891
1909
|
|
|
1892
1910
|
______________________________________________________________________
|
|
1893
1911
|
|
|
@@ -1903,16 +1921,16 @@ No results.
|
|
|
1903
1921
|
|
|
1904
1922
|
______________________________________________________________________
|
|
1905
1923
|
|
|
1906
|
-
Claims often report the place of service
|
|
1924
|
+
Claims often report the place of service with a CMS place of service code. Our test dataset has no place of service information, so the example returns no results.
|
|
1907
1925
|
|
|
1908
1926
|
### `Provenance` Operator
|
|
1909
1927
|
|
|
1910
|
-
Passes along the
|
|
1928
|
+
Passes along the results whose provenance matches at least one of the given keywords. The provenance of a result says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary.
|
|
1911
1929
|
|
|
1912
1930
|
- **Upstream:** one operator.
|
|
1913
1931
|
- **Arguments:**
|
|
1914
|
-
- `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a
|
|
1915
|
-
- **Returns:** The upstream
|
|
1932
|
+
- `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a result came from. A code keyword, such as primary or first, says where the code was on the claim. A plain keyword matches a result when its file provenance or its code provenance matches it. A keyword also matches the provenances under it, so inpatient includes snf. A keyword of the form file:code, such as inpatient:primary, needs both the file and the code provenance to match. The keyword * matches a result that has any provenance. A statement with no keyword is invalid. An unknown keyword gets a warning and matches no result.
|
|
1933
|
+
- **Returns:** The upstream results whose provenance matches at least one of the keywords.
|
|
1916
1934
|
|
|
1917
1935
|
______________________________________________________________________
|
|
1918
1936
|
|
|
@@ -1939,24 +1957,47 @@ ______________________________________________________________________
|
|
|
1939
1957
|
|
|
1940
1958
|
______________________________________________________________________
|
|
1941
1959
|
|
|
1942
|
-
Provenance is information about the origin of claims data. It is sometimes an important factor in research algorithms.
|
|
1960
|
+
Provenance is information about the origin of a row of claims data. It is sometimes an important factor in research algorithms. There are two kinds of provenance:
|
|
1961
|
+
|
|
1962
|
+
- File provenance is the file of the dataset that a row came from. Examples are the MedPAR file and the carrier claims file of Medicare data.
|
|
1963
|
+
- Code provenance is the role of a code on its claim. Examples are a primary diagnosis and an admitting diagnosis.
|
|
1964
|
+
|
|
1965
|
+
The [GDM](https://github.com/outcomesinsights/generalized_data_model#contexts) records both kinds during ETL. The file provenance of a row is the `source_type_concept_id` of its context. The code provenance of a row is its `provenance_concept_id`. ConceptQL names these two columns `file_provenance_type` and `code_provenance_type`.
|
|
1966
|
+
|
|
1967
|
+
Each keyword names a concept in a provenance vocabulary. A file keyword names a concept of the file provenance vocabulary, and a code keyword names a concept of the code provenance vocabulary. A keyword matches its concept and each descendant of that concept in the `concept_ancestor` table of the dataset. For example, in our test dataset, the concept of the MedPAR file of inpatient stays is a descendant of `inpatient`.
|
|
1943
1968
|
|
|
1944
1969
|
These are the keywords of each kind:
|
|
1945
1970
|
|
|
1946
1971
|
- File keywords: `inpatient`, `outpatient`, `carrier_claim`, `claim`, `dme`, `hha`, `hospice`, `snf`, `prescription_drug` and `demographics`.
|
|
1947
1972
|
- Code keywords: `primary`, `primary_or_first`, `first`, `second`, `admitting`, `discharge`, `problem_list`, `chief_complaint` and `unspecified`.
|
|
1948
1973
|
|
|
1949
|
-
|
|
1974
|
+
The hierarchy of these concepts is part of the vocabulary tables of the dataset. It is not a rule of ConceptQL. In the vocabulary tables of our test dataset, `inpatient` includes `snf`, and `outpatient` includes `hha` and `hospice`. There, `carrier_claim` includes `dme`, and `claim` includes each claim file of the test dataset.
|
|
1975
|
+
|
|
1976
|
+
#### Inpatient and Outpatient Settings
|
|
1977
|
+
|
|
1978
|
+
Authors often describe "inpatient records" or "outpatient visits". In ConceptQL terms, the authors mean the results whose rows came from an inpatient setting. An inpatient setting is meant to be a stay after admission to a hospital. In practice, the line between the inpatient and the outpatient setting is poorly defined. These are some of the hard cases:
|
|
1979
|
+
|
|
1980
|
+
- In some data, the file decides. For example, the MedPAR file of Medicare data holds the inpatient stays.
|
|
1981
|
+
- A carrier claim can be for a service given in a hospital.
|
|
1982
|
+
- An emergency room visit can become an admission. Then the visit has an outpatient part and an inpatient part.
|
|
1983
|
+
- The place of service code of a claim may or may not decide the setting.
|
|
1984
|
+
- A stay in a skilled nursing facility may or may not count as inpatient.
|
|
1985
|
+
|
|
1986
|
+
Outpatient visits have the same problems.
|
|
1987
|
+
|
|
1988
|
+
ConceptQL does not decide these cases. The dataset declares what counts as inpatient and what counts as outpatient, through a mapping. The GDM stores the concrete facts of the dataset, such as the file that a row came from and the place of service. The mapping, not the ETL, decides which of these rows are inpatient. So the setting is a decision of the analysis. A different analysis can use a different mapping, with no new ETL.
|
|
1989
|
+
|
|
1990
|
+
The mapping is the hierarchy of the file and code provenance concepts in the vocabulary tables of the dataset. For example, our test dataset maps its inpatient claims file to `inpatient` and its outpatient claims file to `outpatient`. [`one_in_two_out`](#one-in-two-out-operator) uses the keywords `inpatient`, `outpatient` and `carrier_claim`. As plain keywords, they match either kind of provenance. In our test dataset, only the file provenance matches them, because no code provenance of the dataset is a descendant of these keywords. The [utilization operators](#utilization-operators), such as `hospitalization`, use only the file provenance.
|
|
1950
1991
|
|
|
1951
1992
|
### `Provider Filter` Operator
|
|
1952
1993
|
|
|
1953
|
-
Passes along the
|
|
1994
|
+
Passes along the results whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value.
|
|
1954
1995
|
|
|
1955
1996
|
- **Upstream:** one operator.
|
|
1956
1997
|
- **Options:**
|
|
1957
1998
|
- `specialties` (string, required): A comma-separated list of specialty concept IDs, such as 38003868 (Endocrinology, Diabetes & Metabolism Physician), or * for any specialty. A provider matches when the specialty recorded for the provider on the claim, or the provider's own specialty, is in the list.
|
|
1958
1999
|
- `roles` (string, required): A comma-separated list of role concept IDs of the GDM_ROLE vocabulary, such as 100000067 (Attending physician) or 100176860 (Billing provider), or * for any role.
|
|
1959
|
-
- **Returns:** The upstream
|
|
2000
|
+
- **Returns:** The upstream results that have a provider who matches both the specialties and the roles. A result can have more than one provider, so a result comes out once for each provider who matches. Each result gets the columns provider_id, context_specialty_concept_id, practitioner_specialty_concept_id and role_type_concept_id.
|
|
1960
2001
|
|
|
1961
2002
|
______________________________________________________________________
|
|
1962
2003
|
|
|
@@ -2005,25 +2046,25 @@ ______________________________________________________________________
|
|
|
2005
2046
|
|
|
2006
2047
|
### `One In Two Out` Operator
|
|
2007
2048
|
|
|
2008
|
-
For each person, passes along the earliest confirmed condition
|
|
2049
|
+
For each person, passes along the earliest confirmed condition result. One inpatient result confirms a condition, and so do two outpatient results whose start dates are a given gap apart.
|
|
2009
2050
|
|
|
2010
2051
|
- **Upstream:** one operator.
|
|
2011
2052
|
- **Options:**
|
|
2012
|
-
- `inpatient_length_of_stay` (integer): An inpatient stay must be longer than this many days: an inpatient
|
|
2013
|
-
- `outpatient_minimum_gap` (string): A date adjustment, such as 30d. The confirming outpatient
|
|
2014
|
-
- `outpatient_maximum_gap` (string): A date adjustment, such as 365d. The confirming outpatient
|
|
2015
|
-
- `outpatient_event_to_return` (string, one of `Initial Event`, `Confirming Event`): Which outpatient
|
|
2016
|
-
- **Returns:** One
|
|
2053
|
+
- `inpatient_length_of_stay` (integer): An inpatient stay must be longer than this many days: an inpatient result counts only when its discharge date is more than this many days after its admission date. 0 or no value means no length-of-stay check.
|
|
2054
|
+
- `outpatient_minimum_gap` (string): A date adjustment, such as 30d. The confirming outpatient result must start at least this long after the initial outpatient result. Without this option, the gap is 30 days.
|
|
2055
|
+
- `outpatient_maximum_gap` (string): A date adjustment, such as 365d. The confirming outpatient result must start no more than this long after the initial outpatient result. Without this option, there is no maximum gap.
|
|
2056
|
+
- `outpatient_event_to_return` (string, one of `Initial Event`, `Confirming Event`): Which outpatient result the operator passes on when two outpatient results confirm the condition: the initial result or the confirming result. Without this option, or with any value other than Initial Event, it passes on the confirming result.
|
|
2057
|
+
- **Returns:** One result per person: the earliest confirmed condition result. A returned inpatient result has the admission date as its start_date and the discharge date as its end_date, when the claim has them. Only condition results with inpatient, outpatient or carrier_claim provenance can be confirmed.
|
|
2017
2058
|
|
|
2018
2059
|
______________________________________________________________________
|
|
2019
2060
|
|
|
2020
|
-
**Example 57 - First diabetes diagnosis, either
|
|
2061
|
+
**Example 57 - First diabetes diagnosis, either one inpatient result, or two outpatient results at least 30 days apart**
|
|
2021
2062
|
|
|
2022
2063
|
```JSON
|
|
2023
2064
|
["one_in_two_out",["icd9","250.00"],{"outpatient_minimum_gap":"30d","outpatient_event_to_return":"Initial Event"}]
|
|
2024
2065
|
```
|
|
2025
2066
|
|
|
2026
|
-

|
|
2027
2068
|
|
|
2028
2069
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2029
2070
|
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
@@ -2040,31 +2081,107 @@ ______________________________________________________________________
|
|
|
2040
2081
|
|
|
2041
2082
|
______________________________________________________________________
|
|
2042
2083
|
|
|
2043
|
-
The operator decides between inpatient and outpatient by the provenance of each
|
|
2084
|
+
The operator decides between inpatient and outpatient by the provenance of each result (see [`provenance`](#provenance-operator)):
|
|
2044
2085
|
|
|
2045
|
-
- A
|
|
2046
|
-
- A
|
|
2047
|
-
- Other
|
|
2086
|
+
- A result with `inpatient` provenance (which includes SNF in our test dataset) is an inpatient result.
|
|
2087
|
+
- A result with `outpatient` or `carrier_claim` provenance is an outpatient result.
|
|
2088
|
+
- Other results, and results whose type is not `condition_occurrence`, are removed.
|
|
2048
2089
|
|
|
2049
|
-
The place of service of a
|
|
2090
|
+
The place of service of a result does not matter. The two outpatient results need not have the same code. Any two results from the upstream stream can confirm each other. For outpatient results, the operator compares start_dates.
|
|
2050
2091
|
|
|
2051
2092
|
More about the options:
|
|
2052
2093
|
|
|
2053
2094
|
- `outpatient_maximum_gap` helps with acute diseases. For example, to confirm the flu, set a maximum gap so that the two diagnoses come from the same illness. Jigsaw fills in 365 days.
|
|
2054
2095
|
- `outpatient_event_to_return`: use `Initial Event` in most cases, for example when the algorithm records an exposure of interest or an outcome. Use `Confirming Event` when the algorithm finds an index event. The initial event would introduce immortal time bias. Jigsaw fills in `Initial Event`, but a statement that omits the option gets `Confirming Event`. So always set this option.
|
|
2055
2096
|
|
|
2097
|
+
## Comparison Operators
|
|
2098
|
+
|
|
2099
|
+
These operators compare the results of incoming streams to produce a set of results.
|
|
2100
|
+
|
|
2101
|
+
### `Filter` Operator
|
|
2102
|
+
|
|
2103
|
+
Keeps each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date.
|
|
2104
|
+
|
|
2105
|
+
- **Operands:**
|
|
2106
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
2107
|
+
- `right` (required): The right-hand stream. The operator keeps each left-hand result that is also in this stream, and does not pass these results on.
|
|
2108
|
+
- **Returns:** The left-hand results that have a right-hand result with the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. The right-hand results are not returned.
|
|
2109
|
+
|
|
2110
|
+
______________________________________________________________________
|
|
2111
|
+
|
|
2112
|
+
**Example 58 - MIs that are a patient's first diagnosis of MI or diabetes**
|
|
2113
|
+
|
|
2114
|
+
```JSON
|
|
2115
|
+
["filter",{"left":["icd9","412"],"right":["first",["union",["icd9","412"],["icd9","250.01"]]]}]
|
|
2116
|
+
```
|
|
2117
|
+
|
|
2118
|
+

|
|
2119
|
+
|
|
2120
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2121
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2122
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2123
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2124
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2125
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2126
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2127
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2128
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2129
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2130
|
+
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
2131
|
+
| 90 | 33940 | observations | condition_occurrence | 2010-05-26 | 2010-05-26 | 412 |
|
|
2132
|
+
|
|
2133
|
+
______________________________________________________________________
|
|
2134
|
+
|
|
2135
|
+
It is an intersection of the two streams. Filter compares results, not date ranges. A result whose dates `time_window` changed no longer matches the original result.
|
|
2136
|
+
|
|
2137
|
+
### `Match` Operator
|
|
2138
|
+
|
|
2139
|
+
Keeps each left-hand result that is equal to a right-hand result in every column, including label. Two empty (NULL) values count as equal.
|
|
2140
|
+
|
|
2141
|
+
- **Operands:**
|
|
2142
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
2143
|
+
- `right` (required): The right-hand stream. The operator keeps each left-hand result that is equal to a result of this stream, and does not pass these results on.
|
|
2144
|
+
- **Returns:** The left-hand results that have a right-hand result with the same value in every column. The right-hand results are not returned.
|
|
2145
|
+
|
|
2146
|
+
______________________________________________________________________
|
|
2147
|
+
|
|
2148
|
+
**Example 59 - MIs that are a patient's first MI**
|
|
2149
|
+
|
|
2150
|
+
```JSON
|
|
2151
|
+
["match",{"left":["icd9","412"],"right":["first",["icd9","412"]]}]
|
|
2152
|
+
```
|
|
2153
|
+
|
|
2154
|
+

|
|
2155
|
+
|
|
2156
|
+
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2157
|
+
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2158
|
+
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2159
|
+
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2160
|
+
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2161
|
+
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2162
|
+
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2163
|
+
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2164
|
+
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2165
|
+
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
2166
|
+
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2167
|
+
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
2168
|
+
|
|
2169
|
+
______________________________________________________________________
|
|
2170
|
+
|
|
2171
|
+
It is stricter than [`filter`](#filter-operator), so most users want `filter`. Systems that build statements, such as Jigsaw's export engine, use match with options that Jigsaw does not show.
|
|
2172
|
+
|
|
2056
2173
|
### `Person Filter` Operator
|
|
2057
2174
|
|
|
2058
|
-
Keeps each left-hand
|
|
2175
|
+
Keeps each left-hand result whose patient has at least one result in the right-hand stream. The type and the dates of the right-hand results do not matter.
|
|
2059
2176
|
|
|
2060
2177
|
- **Operands:**
|
|
2061
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
2062
|
-
- `right` (required): The right-hand stream. The operator keeps the left-hand
|
|
2063
|
-
- **Returns:** The left-hand
|
|
2178
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
2179
|
+
- `right` (required): The right-hand stream. The operator keeps the left-hand results of each patient who has a result in this stream, and does not pass these results on.
|
|
2180
|
+
- **Returns:** The left-hand results of every type whose patient has at least one right-hand result. The right-hand results are not returned.
|
|
2064
2181
|
|
|
2065
2182
|
______________________________________________________________________
|
|
2066
2183
|
|
|
2067
|
-
**Example
|
|
2184
|
+
**Example 60 - All MI Conditions for people who are male**
|
|
2068
2185
|
|
|
2069
2186
|
```JSON
|
|
2070
2187
|
["person_filter",{"left":["icd9","412"],"right":["gender","Male"]}]
|
|
@@ -2089,13 +2206,13 @@ ______________________________________________________________________
|
|
|
2089
2206
|
|
|
2090
2207
|
______________________________________________________________________
|
|
2091
2208
|
|
|
2092
|
-
**Example
|
|
2209
|
+
**Example 61 - All MI Conditions for people who had an office visit at some point in the dataset**
|
|
2093
2210
|
|
|
2094
2211
|
```JSON
|
|
2095
2212
|
["person_filter",{"left":["icd9","412"],"right":["cpt","99214"]}]
|
|
2096
2213
|
```
|
|
2097
2214
|
|
|
2098
|
-

|
|
2099
2216
|
|
|
2100
2217
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2101
2218
|
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
@@ -2114,7 +2231,7 @@ ______________________________________________________________________
|
|
|
2114
2231
|
|
|
2115
2232
|
______________________________________________________________________
|
|
2116
2233
|
|
|
2117
|
-
**Example
|
|
2234
|
+
**Example 62 - The same result as the example above, with the office visits converted to patients first**
|
|
2118
2235
|
|
|
2119
2236
|
```JSON
|
|
2120
2237
|
["person_filter",{"left":["icd9","412"],"right":["person",["cpt","99214"]]}]
|
|
@@ -2139,13 +2256,13 @@ ______________________________________________________________________
|
|
|
2139
2256
|
|
|
2140
2257
|
______________________________________________________________________
|
|
2141
2258
|
|
|
2142
|
-
**Example
|
|
2259
|
+
**Example 63 - All MI Conditions for people who are Male OR had an office visit at some point in the dataset**
|
|
2143
2260
|
|
|
2144
2261
|
```JSON
|
|
2145
2262
|
["person_filter",{"left":["icd9","412"],"right":["union",["cpt","99214"],["gender","Male"]]}]
|
|
2146
2263
|
```
|
|
2147
2264
|
|
|
2148
|
-

|
|
2149
2266
|
|
|
2150
2267
|
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2151
2268
|
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
@@ -2164,7 +2281,7 @@ ______________________________________________________________________
|
|
|
2164
2281
|
|
|
2165
2282
|
______________________________________________________________________
|
|
2166
2283
|
|
|
2167
|
-
**Example
|
|
2284
|
+
**Example 64 - Yields two types: all MI Conditions for people who are Male, and all office visit Procedures for people who are Male**
|
|
2168
2285
|
|
|
2169
2286
|
```JSON
|
|
2170
2287
|
["person_filter",{"left":["union",["icd9","412"],["cpt","99214"]],"right":["gender","Male"]}]
|
|
@@ -2187,90 +2304,14 @@ ______________________________________________________________________
|
|
|
2187
2304
|
|
|
2188
2305
|
______________________________________________________________________
|
|
2189
2306
|
|
|
2190
|
-
For example, to find all MIs of males, use `person_filter`. The right-hand stream need not be a person stream, because any
|
|
2191
|
-
|
|
2192
|
-
## Comparison Operators
|
|
2193
|
-
|
|
2194
|
-
These operators compare records of incoming streams to produce a set of records.
|
|
2195
|
-
|
|
2196
|
-
### `Filter` Operator
|
|
2197
|
-
|
|
2198
|
-
Keeps each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date.
|
|
2199
|
-
|
|
2200
|
-
- **Operands:**
|
|
2201
|
-
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
2202
|
-
- `right` (required): The right-hand stream. The operator keeps each left-hand record that is also in this stream, and does not pass these records on.
|
|
2203
|
-
- **Returns:** The left-hand records that have a right-hand record with the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. The right-hand records are not returned.
|
|
2204
|
-
|
|
2205
|
-
______________________________________________________________________
|
|
2206
|
-
|
|
2207
|
-
**Example 63 - MIs that are a patient's first diagnosis of MI or diabetes**
|
|
2208
|
-
|
|
2209
|
-
```JSON
|
|
2210
|
-
["filter",{"left":["icd9","412"],"right":["first",["union",["icd9","412"],["icd9","250.01"]]]}]
|
|
2211
|
-
```
|
|
2212
|
-
|
|
2213
|
-

|
|
2214
|
-
|
|
2215
|
-
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2216
|
-
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2217
|
-
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2218
|
-
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2219
|
-
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2220
|
-
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2221
|
-
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2222
|
-
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2223
|
-
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2224
|
-
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2225
|
-
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
2226
|
-
| 90 | 33940 | observations | condition_occurrence | 2010-05-26 | 2010-05-26 | 412 |
|
|
2227
|
-
|
|
2228
|
-
______________________________________________________________________
|
|
2229
|
-
|
|
2230
|
-
It is an intersection of the two streams. Filter compares records, not date ranges. A record whose dates `time_window` changed no longer matches the original record.
|
|
2231
|
-
|
|
2232
|
-
### `Match` Operator
|
|
2233
|
-
|
|
2234
|
-
Keeps each left-hand record that is equal to a right-hand record in every column, including label. Two empty (NULL) values count as equal.
|
|
2235
|
-
|
|
2236
|
-
- **Operands:**
|
|
2237
|
-
- `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
|
|
2238
|
-
- `right` (required): The right-hand stream. The operator keeps each left-hand record that is equal to a record of this stream, and does not pass these records on.
|
|
2239
|
-
- **Returns:** The left-hand records that have a right-hand record with the same value in every column. The right-hand records are not returned.
|
|
2240
|
-
|
|
2241
|
-
______________________________________________________________________
|
|
2242
|
-
|
|
2243
|
-
**Example 64 - MIs that are a patient's first MI**
|
|
2244
|
-
|
|
2245
|
-
```JSON
|
|
2246
|
-
["match",{"left":["icd9","412"],"right":["first",["icd9","412"]]}]
|
|
2247
|
-
```
|
|
2248
|
-
|
|
2249
|
-

|
|
2250
|
-
|
|
2251
|
-
| person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
|
|
2252
|
-
| --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
|
|
2253
|
-
| 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
|
|
2254
|
-
| 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
|
|
2255
|
-
| 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
|
|
2256
|
-
| 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
|
|
2257
|
-
| 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
|
|
2258
|
-
| 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
|
|
2259
|
-
| 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
|
|
2260
|
-
| 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
|
|
2261
|
-
| 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
|
|
2262
|
-
| 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
|
|
2263
|
-
|
|
2264
|
-
______________________________________________________________________
|
|
2265
|
-
|
|
2266
|
-
It is stricter than [`filter`](#filter-operator), so most users want `filter`. Systems that build statements, such as Jigsaw's export engine, use match with options that Jigsaw does not show.
|
|
2307
|
+
For example, to find all MIs of males, use `person_filter`. The right-hand stream need not be a person stream, because any result of a patient counts. The left-hand stream can also have several types, as the last example shows.
|
|
2267
2308
|
|
|
2268
2309
|
### `Co-Reported` Operator
|
|
2269
2310
|
|
|
2270
|
-
Passes along the
|
|
2311
|
+
Passes along the results of its upstreams that were reported in the same context as a result of every other upstream. In the GDM, a context is the claim or other source row of the dataset that reported the row behind a result.
|
|
2271
2312
|
|
|
2272
|
-
- **Upstream:**
|
|
2273
|
-
- **Returns:** The
|
|
2313
|
+
- **Upstream:** two or more operators.
|
|
2314
|
+
- **Returns:** The results of each upstream whose context also has a result of every other upstream. The operator does not compare dates. If one upstream has no result in a context, the results of all upstreams in that context are not passed along. Person and death results have no context, so they never pass.
|
|
2274
2315
|
|
|
2275
2316
|
______________________________________________________________________
|
|
2276
2317
|
|
|
@@ -2303,7 +2344,7 @@ Claims data often reports diagnoses on the same claim as a procedure. This is do
|
|
|
2303
2344
|
|
|
2304
2345
|
Every ConceptQL operator takes a `label` option, so the reference sections of the operators do not list it:
|
|
2305
2346
|
|
|
2306
|
-
A short description of the operator's
|
|
2347
|
+
A short description of the operator's results. The operator writes it into the label column of each of its results, and a recall operator selects those results by it. Each label in a statement must be different.
|
|
2307
2348
|
|
|
2308
2349
|
An operator without a label passes the upstream label through.
|
|
2309
2350
|
|
|
@@ -2392,12 +2433,12 @@ A label must be a string.
|
|
|
2392
2433
|
|
|
2393
2434
|
### `Recall` Operator
|
|
2394
2435
|
|
|
2395
|
-
Passes on the
|
|
2436
|
+
Passes on the results of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of results more than once.
|
|
2396
2437
|
|
|
2397
2438
|
- **Upstream:** none.
|
|
2398
2439
|
- **Arguments:**
|
|
2399
|
-
- `name` (string): The label of the operator whose
|
|
2400
|
-
- **Returns:** The
|
|
2440
|
+
- `name` (string): The label of the operator whose results to pass on. It must match the label exactly, including case. The labeled operator can come before or after the recall in the statement, but labeled streams that recall each other in a loop are invalid.
|
|
2441
|
+
- **Returns:** The results of the operator that has the given label.
|
|
2401
2442
|
|
|
2402
2443
|
______________________________________________________________________
|
|
2403
2444
|
|
|
@@ -2424,33 +2465,57 @@ ______________________________________________________________________
|
|
|
2424
2465
|
|
|
2425
2466
|
______________________________________________________________________
|
|
2426
2467
|
|
|
2427
|
-
|
|
2468
|
+
<a id="window-tables"></a>
|
|
2428
2469
|
|
|
2429
|
-
|
|
2470
|
+
## Windows
|
|
2471
|
+
|
|
2472
|
+
A window is a span of time. A calling program can apply windows to a statement, usually one span of dates for each person. ConceptQL then ignores the rows of the dataset outside the windows, as though those rows do not exist.
|
|
2473
|
+
|
|
2474
|
+
An algorithm never contains a window, and no algorithm should assume that a window exists. Windows sit outside the algorithm. So any algorithm can later run inside a window with no change to the algorithm. A calling program, such as Jigsaw's export engine, applies the windows of a study to the algorithms that it runs. Then the results stay inside the spans of time of the study.
|
|
2475
|
+
|
|
2476
|
+
The [`time_window`](#time-window-operator) operator is a different thing. It is part of a statement, and it changes the dates of results or makes spans of dates for each person. Its results are still subject to the windows that the calling program applies.
|
|
2477
|
+
|
|
2478
|
+
The calling program wraps the statement like this:
|
|
2430
2479
|
|
|
2431
2480
|
```JSON
|
|
2432
|
-
[ "window", <statement>, {
|
|
2481
|
+
[ "window", <statement>, { <window options> } ]
|
|
2433
2482
|
```
|
|
2434
2483
|
|
|
2435
|
-
|
|
2436
|
-
|
|
2437
|
-
-
|
|
2438
|
-
-
|
|
2484
|
+
These are the window options:
|
|
2485
|
+
|
|
2486
|
+
- `window_table`: a table of windows, with the columns `person_id`, `start_date` and `end_date`. Each row is one window of one person. The value is the name of a database table or a ConceptQL statement.
|
|
2487
|
+
- `start_date` and `end_date`: one date range that is the window of every person. Give both options or neither. One of them without the other makes the statement invalid, and the error names the missing option.
|
|
2488
|
+
- `scope_by`: which dates of a result must be inside the window. With `start_date`, the default, the start_date of the result must be in the window. With `end_date`, the end_date of the result must be in the window. With `date_range`, both dates of the result must be in the window. Any other value makes the statement invalid, and the error names the value.
|
|
2489
|
+
- `adjust_window_start` and `adjust_window_end`: date adjustments, as [`time_window`](#time-window-operator) takes them. They move the start and the end of each window of the window table. They do not change a date range.
|
|
2490
|
+
- `precalculated_window_id`: the window table has its own `window_id` column, and ConceptQL uses it. Without this option, ConceptQL numbers the windows itself.
|
|
2491
|
+
- `person_ids`: a list of person IDs. ConceptQL keeps only the results of these persons.
|
|
2492
|
+
|
|
2493
|
+
Jigsaw's export engine uses `window_table` and `scope_by`. A calling program can give several options, such as a window table and a list of person IDs. Then a result must pass each of them.
|
|
2494
|
+
|
|
2495
|
+
If a window is inside another window, only the options of the inner window apply, and ConceptQL ignores the options of the outer window. A diagram shows the options of the inner window, and only those options can make the statement invalid.
|
|
2496
|
+
|
|
2497
|
+
The windows change a statement in these ways:
|
|
2498
|
+
|
|
2499
|
+
- Each selection operator keeps only the results that are inside a window, as `scope_by` says. Both dates of a window are in the window.
|
|
2500
|
+
- The person operators, such as `person` and `gender`, and the operators `date_range` and `information_periods` ignore the dates of a window. With a window table, they keep each person who has a window. A date range does not change their results.
|
|
2501
|
+
- With a window table, each result gets the column `window_id`, which identifies its window. A result that is in two windows of its person comes out once for each window. Without `precalculated_window_id`, two rows of the window table with the same person and the same dates are one window. A date range adds no `window_id`.
|
|
2502
|
+
- With a window table, operators that work on each person, such as `first`, `episode` and the temporal operators, work on each person and window. For example, `first` keeps the first result of each window.
|
|
2439
2503
|
- If the window table is a ConceptQL statement with errors, the statement that it wraps is invalid. A diagram reports the errors under `window_table`, and counts nothing.
|
|
2504
|
+
- If the window options have an error, such as a lone `start_date` or an unknown `scope_by`, the statement is invalid. A diagram reports the error on the `window` operator, and counts nothing. This is also true when the calling program gives the window options outside the statement.
|
|
2440
2505
|
|
|
2441
2506
|
## Appendix A - Experimental Operators
|
|
2442
2507
|
|
|
2443
|
-
These operators work, but they may change or be removed. They compare the `value_as_number` column, which lab
|
|
2508
|
+
These operators work, but they may change or be removed. They compare the `value_as_number` column, which the results of lab codes (such as `loinc`) carry.
|
|
2444
2509
|
|
|
2445
2510
|
### `Numeric Filter` Operator
|
|
2446
2511
|
|
|
2447
|
-
Passes along the
|
|
2512
|
+
Passes along the results whose value_as_number is in the given range. The results of lab tests, such as those of loinc, carry this value.
|
|
2448
2513
|
|
|
2449
2514
|
- **Upstream:** one operator.
|
|
2450
2515
|
- **Options:**
|
|
2451
2516
|
- `greater_than_or_equal_to` (float): The lowest value_as_number that passes. Without this option, the range has no lower limit.
|
|
2452
2517
|
- `less_than_or_equal_to` (float): The highest value_as_number that passes. Without this option, the range has no upper limit.
|
|
2453
|
-
- **Returns:** The upstream
|
|
2518
|
+
- **Returns:** The upstream results whose value_as_number is in the range. A result that has no value_as_number is not passed along.
|
|
2454
2519
|
|
|
2455
2520
|
______________________________________________________________________
|
|
2456
2521
|
|
|
@@ -2472,12 +2537,12 @@ It is an [inline filter operator](#inline-filter-operators).
|
|
|
2472
2537
|
|
|
2473
2538
|
### `Equal` Operator
|
|
2474
2539
|
|
|
2475
|
-
For each patient, keeps the left-hand
|
|
2540
|
+
For each patient, keeps the left-hand results whose value_as_number equals the value_as_number of a right-hand result. It does not compare dates.
|
|
2476
2541
|
|
|
2477
2542
|
- **Operands:**
|
|
2478
|
-
- `left` (required): The left-hand stream. The operator compares these
|
|
2479
|
-
- `right` (required): The right-hand stream. The operator compares the left-hand
|
|
2480
|
-
- **Returns:** The left-hand
|
|
2543
|
+
- `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
|
|
2544
|
+
- `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
|
|
2545
|
+
- **Returns:** The left-hand results that have a right-hand result of the same patient with the same value_as_number, each result once. Left-hand results without a value_as_number (NULL) are not returned. The right-hand results are not returned.
|
|
2481
2546
|
|
|
2482
2547
|
______________________________________________________________________
|
|
2483
2548
|
|
|
@@ -2564,21 +2629,21 @@ This document is part of the [ConceptQL](https://github.com/outcomesinsights/con
|
|
|
2564
2629
|
|
|
2565
2630
|
The reference section of each operator, the text of the `label` option, the table of all vocabulary operators and Appendix E come from the metadata of the operators (`ConceptQL.metadata`), so the texts in the operator classes are their only copy. A directive in `README.cql.md`, an HTML comment on a line of its own, marks the place of each generated block:
|
|
2566
2631
|
|
|
2567
|
-
- `<!-- conceptql-spec: operator <spec_id> -->` writes the reference section of the operators with that `spec_id`. It comes directly after the heading of the section, whose anchor is the `spec_id`. The section has the operator's description, its upstream operators or its operands (`left` and `right`), its arguments, its options, the
|
|
2632
|
+
- `<!-- conceptql-spec: operator <spec_id> -->` writes the reference section of the operators with that `spec_id`. It comes directly after the heading of the section, whose anchor is the `spec_id`. The section has the operator's description, its upstream operators or its operands (`left` and `right`), its arguments, its options, the results it returns, and its examples. Text about more than one operator, and longer explanations of one operator, follow the directive as markdown.
|
|
2568
2633
|
- `<!-- conceptql-spec: label-option -->` writes the text of the `label` option, which every operator takes.
|
|
2569
2634
|
- `<!-- conceptql-spec: quick-reference -->` writes the table of Appendix E, with one row for each `spec_id`.
|
|
2570
2635
|
- `<!-- conceptql-spec: vocabulary-operators -->` writes the table of [All Vocabulary Operators](#all-vocabulary-operators), with one row for each vocabulary operator: its name, the full name of its vocabulary and its domain. It marks a deprecated operator.
|
|
2571
2636
|
|
|
2572
2637
|
The metadata comes from a ConceptQL database with the data model `gdm`, no data database and no custom vocabularies, so it decides which operators get a section. Each `spec_id` of that metadata needs exactly one directive, and the build stops when one is missing, repeated or not after its heading. The other directives appear exactly once each, and the build also stops on an unknown directive.
|
|
2573
2638
|
|
|
2574
|
-
`just spec` runs `docs/spec/build/run.rb` with the working tree's ConceptQL. For each `ConceptQL` block of `README.cql.md` and each example of a reference section, the build writes the example: the statement as JSON, its diagram, and
|
|
2639
|
+
`just spec` runs `docs/spec/build/run.rb` with the working tree's ConceptQL. For each `ConceptQL` block of `README.cql.md` and each example of a reference section, the build writes the example: the statement as JSON, its diagram, and its first 10 results. It orders the results by `person_id`, `criterion_domain`, start_date, `criterion_table` and `criterion_id`, so each run shows the same results. An example that is not a valid statement stops the build. Then `just spec` formats `README.md` with treefmt. If `README.md` or an image changed, `just spec` lists the changed files and fails, so that you review and commit them. A second run then changes nothing and passes.
|
|
2575
2640
|
|
|
2576
2641
|
The results and the counts come from ConceptQL's DuckDB test data, `synpuf_test_data.duckdb`, with the data model `gdm_wide` and the OHDSI vocabulary tables. `just spec` runs the build through `just test-duckdb`, so it needs what the DuckDB tests need:
|
|
2577
2642
|
|
|
2578
2643
|
- Linux on x86-64. `just test-duckdb` downloads the pinned version of the DuckDB library on the first run.
|
|
2579
2644
|
- The DuckDB test data. `just test-duckdb` copies it from the path in `DUCKDB_TEST_DATA_SOURCE`, and it fails unless the copy matches the checksum in `.github/synpuf_test_data.sha256`.
|
|
2580
2645
|
|
|
2581
|
-
The diagrams are drawn with ConceptQL's diagram renderer, `ConceptQL::DiagramRenderer`, which runs the resvg program. The build needs resvg 0.33.0 exactly, which `mise.toml` pins and `just setup` installs. It fails when resvg is missing or is another version. The renderer labels each line with the
|
|
2646
|
+
The diagrams are drawn with ConceptQL's diagram renderer, `ConceptQL::DiagramRenderer`, which runs the resvg program. The build needs resvg 0.33.0 exactly, which `mise.toml` pins and `just setup` installs. It fails when resvg is missing or is another version. The renderer labels each line with the results (`rows=`) and the people (`n=`) that flow along it, and it lists the options of each operator. The build draws every diagram on each run, and writes each one as `images/<name>.svg`, where the name is the SHA-256 of the statement as canonical JSON. It removes the SVG images that no example uses, and it writes `images/manifest.tsv`, which gives the SHA-256 of the diagram data (with its counts) for each image.
|
|
2582
2647
|
|
|
2583
2648
|
Between releases, the results and the images on the main branch can be out of date. Before a release, `just release-check` runs `just spec-check`. It rebuilds the document and the diagram data, and it draws every image into a temporary directory. It fails when `README.md` does not match the code, or when an image differs from its new drawing, is missing, or is not used. It names each such image.
|
|
2584
2649
|
|
|
@@ -2588,39 +2653,39 @@ The build writes this table from the metadata of the operators. It has one row f
|
|
|
2588
2653
|
|
|
2589
2654
|
| Operator | Jigsaw category | What it does |
|
|
2590
2655
|
| -------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
2591
|
-
| [`Vocabulary` Operator](#vocabulary-operator) | Select by Clinical Codes | Selects the
|
|
2656
|
+
| [`Vocabulary` Operator](#vocabulary-operator) | Select by Clinical Codes | Selects the results whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once. |
|
|
2592
2657
|
| [`gender`](#gender-operator) | Select by Property | Selects the patients with the given genders. |
|
|
2593
2658
|
| [`race`](#race-operator) | Select by Property | Selects the patients with the given races. |
|
|
2594
2659
|
| [`ethnicity`](#ethnicity-operator) | Select by Property | Selects the patients with the given ethnicities. |
|
|
2595
|
-
| [`snf`](#snf-operator) | Select by Property | Selects admissions to skilled nursing facilities (SNF): one
|
|
2596
|
-
| [`hospice`](#hospice-operator) | Select by Property | Selects hospice admissions: one
|
|
2597
|
-
| [`hospitalization`](#hospitalization-operator) | Select by Property | Selects inpatient hospital admissions: one
|
|
2598
|
-
| [`date_range`](#date-range-operator) | Select by Property | Makes one person
|
|
2599
|
-
| [`person`](#person-operator) | Get Related Data | Selects every patient. With one upstream operator, it selects the patients who have a
|
|
2600
|
-
| [`death`](#death-operator) | Get Related Data | Selects every death
|
|
2601
|
-
| [`information_periods`](#information-periods-operator) | Get Related Data | Selects information periods: the spans of time in which the
|
|
2602
|
-
| [`recall`](#recall-operator) | Get Related Data | Passes on the
|
|
2603
|
-
| [`time_window`](#time-window-operator) | Modify Data | Moves the start_date and the end_date of each
|
|
2604
|
-
| [`episode`](#episode-operator) | Modify Data | Merges each patient's
|
|
2605
|
-
| [`union`](#union-operator) | Combine Streams | Combines the
|
|
2606
|
-
| [`co_reported`](#co-reported-operator) | Combine Streams | Passes along the
|
|
2607
|
-
| [`except`](#except-operator) | Filter by Comparing | Removes each left-hand
|
|
2608
|
-
| [`during`](#during-operator) | Filter by Comparing | For each patient, keeps the left-hand
|
|
2609
|
-
| [`contains`](#contains-operator) | Filter by Comparing | For each patient, keeps the left-hand
|
|
2610
|
-
| [`any_overlap`](#any-overlap-operator) | Filter by Comparing | For each patient, keeps the left-hand
|
|
2611
|
-
| [`before`](#before-operator) | Filter by Comparing | For each patient, keeps the left-hand
|
|
2612
|
-
| [`after`](#after-operator) | Filter by Comparing | For each patient, keeps the left-hand
|
|
2613
|
-
| [`trim_date_start`](#trim-date-start-operator) | Filter by Comparing | For each patient, moves the start_date of each left-hand
|
|
2614
|
-
| [`trim_date_end`](#trim-date-end-operator) | Filter by Comparing | For each patient, moves the end_date of each left-hand
|
|
2615
|
-
| [`
|
|
2616
|
-
| [`
|
|
2617
|
-
| [`
|
|
2618
|
-
| [`equal`](#equal-operator) | Filter by Comparing | For each patient, keeps the left-hand
|
|
2619
|
-
| [`occurrence`](#nth-occurrence-operator) | Filter Single Stream | For each person, orders the
|
|
2620
|
-
| [`first`](#first-operator) | Filter Single Stream | For each person, passes along the
|
|
2621
|
-
| [`last`](#last-operator) | Filter Single Stream | For each person, passes along the
|
|
2622
|
-
| [`place_of_service_filter`](#place-of-service-filter-operator) | Filter Single Stream | Passes along the
|
|
2623
|
-
| [`provenance`](#provenance-operator) | Filter Single Stream | Passes along the
|
|
2624
|
-
| [`provider_filter`](#provider-filter-operator) | Filter Single Stream | Passes along the
|
|
2625
|
-
| [`one_in_two_out`](#one-in-two-out-operator) | Filter Single Stream | For each person, passes along the earliest confirmed condition
|
|
2626
|
-
| [`numeric_filter`](#numeric-filter-operator) | Filter Single Stream | Passes along the
|
|
2660
|
+
| [`snf`](#snf-operator) | Select by Property | Selects admissions to skilled nursing facilities (SNF): one result for each admission whose claim came from an SNF file. |
|
|
2661
|
+
| [`hospice`](#hospice-operator) | Select by Property | Selects hospice admissions: one result for each admission whose claim came from a hospice file. |
|
|
2662
|
+
| [`hospitalization`](#hospitalization-operator) | Select by Property | Selects inpatient hospital admissions: one result for each admission whose claim came from an inpatient file. |
|
|
2663
|
+
| [`date_range`](#date-range-operator) | Select by Property | Makes one person result for each patient, with the given start_date and end_date. Use it to compare other results with a fixed range of dates. |
|
|
2664
|
+
| [`person`](#person-operator) | Get Related Data | Selects every patient. With one upstream operator, it selects the patients who have a result in the upstream stream. |
|
|
2665
|
+
| [`death`](#death-operator) | Get Related Data | Selects every death. With one upstream operator, it selects the deaths of the patients who have a result in the upstream stream. |
|
|
2666
|
+
| [`information_periods`](#information-periods-operator) | Get Related Data | Selects information periods: the spans of time in which the dataset has information about a patient, such as periods of insurance enrollment. |
|
|
2667
|
+
| [`recall`](#recall-operator) | Get Related Data | Passes on the results of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of results more than once. |
|
|
2668
|
+
| [`time_window`](#time-window-operator) | Modify Data | Moves the start_date and the end_date of each result. Use it to shift or widen a result's dates before you compare them with other results. |
|
|
2669
|
+
| [`episode`](#episode-operator) | Modify Data | Merges each patient's results into episodes: spans of time in which the results are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug. |
|
|
2670
|
+
| [`union`](#union-operator) | Combine Streams | Combines the results of all its upstream operators into one stream. Each result keeps its own type, so one stream can hold several types. |
|
|
2671
|
+
| [`co_reported`](#co-reported-operator) | Combine Streams | Passes along the results of its upstreams that were reported in the same context as a result of every other upstream. In the GDM, a context is the claim or other source row of the dataset that reported the row behind a result. |
|
|
2672
|
+
| [`except`](#except-operator) | Filter by Comparing | Removes each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same criterion_table, criterion_id, start_date and end_date. |
|
|
2673
|
+
| [`during`](#during-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose date range is inside the date range of a right-hand result. The left-hand result starts on or after the start_date of the right-hand result and ends on or before its end_date. |
|
|
2674
|
+
| [`contains`](#contains-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose date range encloses the date range of a right-hand result. The left-hand result starts on or before the start_date of the right-hand result and ends on or after its end_date. |
|
|
2675
|
+
| [`any_overlap`](#any-overlap-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose date range shares at least one day with the date range of a right-hand result. |
|
|
2676
|
+
| [`before`](#before-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose end_date is before the latest start_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead. |
|
|
2677
|
+
| [`after`](#after-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose start_date is after the earliest end_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead. |
|
|
2678
|
+
| [`trim_date_start`](#trim-date-start-operator) | Filter by Comparing | For each patient, moves the start_date of each left-hand result forward to the end_date of the right-hand result with the latest start_date. A left-hand result that ends before that end_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged. |
|
|
2679
|
+
| [`trim_date_end`](#trim-date-end-operator) | Filter by Comparing | For each patient, moves the end_date of each left-hand result back to the start_date of the right-hand result with the earliest start_date. A left-hand result that starts after that start_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged. |
|
|
2680
|
+
| [`filter`](#filter-operator) | Filter by Comparing | Keeps each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. |
|
|
2681
|
+
| [`match`](#match-operator) | Filter by Comparing | Keeps each left-hand result that is equal to a right-hand result in every column, including label. Two empty (NULL) values count as equal. |
|
|
2682
|
+
| [`person_filter`](#person-filter-operator) | Filter by Comparing | Keeps each left-hand result whose patient has at least one result in the right-hand stream. The type and the dates of the right-hand results do not matter. |
|
|
2683
|
+
| [`equal`](#equal-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose value_as_number equals the value_as_number of a right-hand result. It does not compare dates. |
|
|
2684
|
+
| [`occurrence`](#nth-occurrence-operator) | Filter Single Stream | For each person, orders the results by start_date and passes along the result at the given position. A positive position counts from the earliest result and a negative one from the most recent result. |
|
|
2685
|
+
| [`first`](#first-operator) | Filter Single Stream | For each person, passes along the result with the earliest start_date. |
|
|
2686
|
+
| [`last`](#last-operator) | Filter Single Stream | For each person, passes along the result with the most recent start_date. |
|
|
2687
|
+
| [`place_of_service_filter`](#place-of-service-filter-operator) | Filter Single Stream | Passes along the results whose place of service is one of the given CMS Place of Service codes. A result that has no place of service is not passed along. |
|
|
2688
|
+
| [`provenance`](#provenance-operator) | Filter Single Stream | Passes along the results whose provenance matches at least one of the given keywords. The provenance of a result says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary. |
|
|
2689
|
+
| [`provider_filter`](#provider-filter-operator) | Filter Single Stream | Passes along the results whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value. |
|
|
2690
|
+
| [`one_in_two_out`](#one-in-two-out-operator) | Filter Single Stream | For each person, passes along the earliest confirmed condition result. One inpatient result confirms a condition, and so do two outpatient results whose start dates are a given gap apart. |
|
|
2691
|
+
| [`numeric_filter`](#numeric-filter-operator) | Filter Single Stream | Passes along the results whose value_as_number is in the given range. The results of lab tests, such as those of loinc, carry this value. |
|