conceptql 0.8.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +141 -0
  3. data/docs/spec/README.md +402 -337
  4. data/docs/spec/examples/date-range-operator.json +1 -1
  5. data/docs/spec/examples/death-operator.json +1 -1
  6. data/docs/spec/examples/nth-occurrence-operator.json +1 -1
  7. data/docs/spec/examples/one-in-two-out-operator.json +1 -1
  8. data/docs/spec/examples/person-filter-operator.json +2 -2
  9. data/docs/spec/examples/vocabulary-operator.json +3 -3
  10. data/lib/conceptql/behaviors/metadatable.rb +76 -62
  11. data/lib/conceptql/behaviors/utilizable.rb +1 -1
  12. data/lib/conceptql/database.rb +16 -2
  13. data/lib/conceptql/diagram.rb +2 -2
  14. data/lib/conceptql/lexicon/lexicon_no_db.rb +4 -4
  15. data/lib/conceptql/lexicon.rb +7 -4
  16. data/lib/conceptql/operators/after.rb +5 -4
  17. data/lib/conceptql/operators/any_overlap.rb +4 -3
  18. data/lib/conceptql/operators/before.rb +5 -4
  19. data/lib/conceptql/operators/binary_operator_operator.rb +11 -13
  20. data/lib/conceptql/operators/casting_operator.rb +1 -1
  21. data/lib/conceptql/operators/co_reported.rb +9 -7
  22. data/lib/conceptql/operators/contains.rb +4 -3
  23. data/lib/conceptql/operators/count.rb +3 -3
  24. data/lib/conceptql/operators/date_range.rb +9 -8
  25. data/lib/conceptql/operators/death.rb +4 -4
  26. data/lib/conceptql/operators/dedup.rb +2 -2
  27. data/lib/conceptql/operators/during.rb +4 -3
  28. data/lib/conceptql/operators/episode.rb +9 -9
  29. data/lib/conceptql/operators/equal.rb +3 -2
  30. data/lib/conceptql/operators/ethnicity.rb +7 -3
  31. data/lib/conceptql/operators/except.rb +4 -3
  32. data/lib/conceptql/operators/filter.rb +4 -3
  33. data/lib/conceptql/operators/first.rb +4 -3
  34. data/lib/conceptql/operators/from.rb +2 -1
  35. data/lib/conceptql/operators/gender.rb +4 -3
  36. data/lib/conceptql/operators/hospice.rb +4 -3
  37. data/lib/conceptql/operators/hospitalization.rb +4 -3
  38. data/lib/conceptql/operators/information_periods.rb +5 -4
  39. data/lib/conceptql/operators/last.rb +4 -3
  40. data/lib/conceptql/operators/match.rb +4 -3
  41. data/lib/conceptql/operators/multiple_vocabularies.rb +4 -3
  42. data/lib/conceptql/operators/numeric.rb +4 -4
  43. data/lib/conceptql/operators/numeric_filter.rb +6 -6
  44. data/lib/conceptql/operators/occurrence.rb +25 -18
  45. data/lib/conceptql/operators/one_in_two_out.rb +16 -16
  46. data/lib/conceptql/operators/operator.rb +16 -3
  47. data/lib/conceptql/operators/pass_thru.rb +1 -1
  48. data/lib/conceptql/operators/person.rb +3 -3
  49. data/lib/conceptql/operators/person_filter.rb +6 -5
  50. data/lib/conceptql/operators/place_of_service_filter.rb +7 -7
  51. data/lib/conceptql/operators/provenance.rb +9 -9
  52. data/lib/conceptql/operators/provider_filter.rb +6 -6
  53. data/lib/conceptql/operators/race.rb +4 -3
  54. data/lib/conceptql/operators/recall.rb +6 -5
  55. data/lib/conceptql/operators/snf.rb +4 -3
  56. data/lib/conceptql/operators/sum.rb +6 -5
  57. data/lib/conceptql/operators/time_window.rb +6 -6
  58. data/lib/conceptql/operators/trim_date_end.rb +6 -5
  59. data/lib/conceptql/operators/trim_date_start.rb +6 -5
  60. data/lib/conceptql/operators/union.rb +8 -6
  61. data/lib/conceptql/operators/vocabulary.rb +12 -8
  62. data/lib/conceptql/query.rb +41 -12
  63. data/lib/conceptql/scope.rb +10 -2
  64. data/lib/conceptql/version.rb +1 -1
  65. data/lib/conceptql/visitors/metadata_extractor.rb +3 -1
  66. data/lib/conceptql/vocabularies/drg_systems.rb +16 -14
  67. data/lib/conceptql/vocabularies/read_codes.rb +1 -1
  68. data/lib/conceptql/window.rb +23 -2
  69. data/schemas/gdm.yml +0 -66
  70. data/schemas/gdm_wide.yml +0 -66
  71. metadata +1 -1
data/docs/spec/README.md CHANGED
@@ -84,13 +84,21 @@ ConceptQL is a domain-specific, declarative language. It represents search crite
84
84
  ```YAML
85
85
  # Example 1: A simple example in YAML
86
86
  # The first element is the operator, icd9cm. The second element is a code, 412.
87
- # This statement selects every record with the ICD-9-CM diagnosis code 412.
87
+ # This statement selects every result with the ICD-9-CM diagnosis code 412.
88
88
  ---
89
89
  - icd9cm
90
90
  - '412'
91
91
  ```
92
92
 
93
- The first element of each array is the name of an operator. The other elements are the operator's arguments, its upstream operators, and a hash of options. Codes are strings, so quote them in YAML. Operator names are not case-sensitive.
93
+ A statement is an algorithm written in ConceptQL. It is commonly stored as JSON. Each array of a statement has these parts:
94
+
95
+ - The first element is an operator. An operator is a function that is applied to the dataset or to results. Only a selection operator reads the dataset. Every other operator works on its upstream results.
96
+ - The arguments follow the operator. An argument is an unnamed, positional input. It is a code, such as `412`, or an upstream statement. Many operators take a variable number of arguments.
97
+ - An optional hash at the end holds the options. An option is a named input. Every operator takes the option `label` (see [`label`](#label-option-and-its-features)).
98
+
99
+ The reference section of each operator lists its upstream statements under "Upstream" and its other arguments under "Arguments". Codes are strings, so quote them in YAML. Operator names are not case-sensitive.
100
+
101
+ The words upstream and downstream describe the flow of results through a statement. The upstream of an operator is the results that come into it. An upstream statement is nested inside the array of the operator that takes its results. Downstream is everything in a statement that comes after a given operator. In the second diagram below, the results of `cpt` are the upstream of `first`, and `first` is downstream of `cpt`.
94
102
 
95
103
  ### ConceptQL Diagrams
96
104
 
@@ -121,7 +129,7 @@ ______________________________________________________________________
121
129
 
122
130
  ______________________________________________________________________
123
131
 
124
- Each node in a diagram is an operator. A line between two operators shows that the records of the upper operator flow down into the lower operator. Each line is labeled with the number of records (`rows=`) and people (`n=`) that flow along it in our test data. Here is a slightly larger example:
132
+ Each node in a diagram is an operator. A line between two operators shows that the results of the upper operator flow down into the lower operator. Each line is labeled with the number of results (`rows=`) and people (`n=`) that flow along it in our test dataset. Here is a slightly larger example:
125
133
 
126
134
  ______________________________________________________________________
127
135
 
@@ -152,15 +160,15 @@ The diagram above reads "get all procedures that match CPT 99214 (office visit),
152
160
 
153
161
  Every diagram ends with a line that leads to nothing. The next section explains why.
154
162
 
155
- ### Think of Records as a Stream
163
+ ### Think of Results as a Stream
156
164
 
157
- A ConceptQL diagram has the "leaf" operators at the top and the "trunk" operator at the bottom. Think of the records of a ConceptQL statement as a stream of data. The leaf operators select records from the database, and they act as the sources of the stream. The records flow down through the diagram. Some operators combine streams, and others remove records, until one stream comes out at the bottom of the diagram.
165
+ A ConceptQL diagram has the "leaf" operators at the top and the "trunk" operator at the bottom. Think of the results of a ConceptQL statement as a stream. The leaf operators select rows from the dataset and make results from them, so they act as the sources of the stream. The results flow down through the diagram. Some operators combine streams, and others remove results, until one stream comes out at the bottom of the diagram.
158
166
 
159
- The trailing line in each diagram is a reminder that every ConceptQL statement yields a stream of records.
167
+ The trailing line in each diagram is a reminder that every ConceptQL statement yields a stream of results.
160
168
 
161
169
  ### Streams Have Types
162
170
 
163
- Each record has a type. The type is the `criterion_domain` column of the record. A vocabulary operator takes its type from the vocabulary's domain. For example, ICD-9-CM records are `condition_occurrence` records, and CPT records are `procedure_occurrence` records. Other operators produce a fixed type, such as `person` or `death`.
171
+ Each result has a type. The type is the `criterion_domain` column of the result. A vocabulary operator takes its type from the vocabulary's domain. For example, ICD-9-CM results are `condition_occurrence` results, and CPT results are `procedure_occurrence` results. Other operators produce a fixed type, such as `person` or `death`.
164
172
 
165
173
  The diagrams use color to show types. The node of an operator and the lines that leave it have the color of the types that it outputs:
166
174
 
@@ -177,7 +185,7 @@ The diagrams use color to show types. The node of an operator and the lines that
177
185
 
178
186
  An operator that outputs more than one type has a gray node, and the line that leaves it is drawn as one colored line for each type, side by side.
179
187
 
180
- The type stays with each record as it moves through the operators. Some operators use the type when they compare records. For example, `filter` and `except` match a record only with a record from the same table. Other operators, such as `first` and `person_filter`, ignore the type. Each operator's description says which applies.
188
+ The type stays with each result as it moves through the operators. Some operators use the type when they compare results. For example, `filter` and `except` match a result only with a result from the same table. Other operators, such as `first` and `person_filter`, ignore the type. Each operator's description says which applies.
181
189
 
182
190
  ### Why Types?
183
191
 
@@ -189,19 +197,19 @@ ConceptQL now runs against the [GDM](https://github.com/outcomesinsights/general
189
197
 
190
198
  Thinking in streams is helpful, but sometimes we need to know what happens in the database.
191
199
 
192
- A stream is a set of rows. Each row points back to one row of a GDM table, and it always has these columns:
200
+ A stream is the set of results that passes from one operator to the next. ConceptQL is not a stream-processing system. Here, "stream" means only that set of results. Each result points back to one row of a GDM table, and it always has these columns:
193
201
 
194
202
  - `person_id`: the patient.
195
203
  - `criterion_id`: the ID of the row in its table of origin.
196
204
  - `criterion_table`: the table of origin, such as `clinical_codes`, `patients` or `deaths`.
197
- - `criterion_domain`: the type of the record (see above).
198
- - `start_date` and `end_date`: the dates of the record.
199
- - `source_value` and `source_vocabulary_id`: the code of the record and its vocabulary, when the record has a code.
205
+ - `criterion_domain`: the type of the result (see above).
206
+ - `start_date` and `end_date`: the dates of the result.
207
+ - `source_value` and `source_vocabulary_id`: the code of the result and its vocabulary, when the result has a code.
200
208
  - `label`: the label of the nearest upstream operator that has one (see [`label`](#label-option-and-its-features)).
201
209
 
202
- Some operators add columns. For example, lab records (loinc) add `value_as_number` and unit columns, and the utilization operators add admission and discharge columns.
210
+ Some operators add columns. For example, the results of lab codes (loinc) add `value_as_number` and unit columns, and the utilization operators add admission and discharge columns.
203
211
 
204
- So this statement yields one row for each male patient, pointing at the `patients` table:
212
+ So this statement yields one result for each male patient, pointing at the `patients` table:
205
213
 
206
214
  ______________________________________________________________________
207
215
 
@@ -228,7 +236,7 @@ ______________________________________________________________________
228
236
 
229
237
  ______________________________________________________________________
230
238
 
231
- This statement yields one row for each `clinical_codes` row that has the ICD-9-CM code 250.01:
239
+ This statement yields one result for each `clinical_codes` row that has the ICD-9-CM code 250.01:
232
240
 
233
241
  ______________________________________________________________________
234
242
 
@@ -257,49 +265,49 @@ ______________________________________________________________________
257
265
 
258
266
  It usually helps to think of these queries as a "stream of people" or a "stream of conditions", and not to think about tables and IDs.
259
267
 
260
- A ConceptQL statement yields the rows that pass all of its criteria. What to do with those rows is up to the system that runs the statement. For example, a system that has all 250.01 records could:
268
+ A ConceptQL statement yields the results that pass all of its criteria. What to do with those results is up to the system that runs the statement. For example, a system that has all 250.01 results could:
261
269
 
262
270
  - Find the first and last date of occurrence for each person
263
271
  - Count the occurrences for each person
264
272
  - Count the people with the condition
265
273
  - Count the occurrences in the whole population
266
274
 
267
- That kind of aggregation and analysis is outside the scope of ConceptQL. ConceptQL finds the rows that you are interested in. Other parts of the system decide what to do with them.
275
+ That kind of aggregation and analysis is outside the scope of ConceptQL. ConceptQL finds the results that you are interested in. Other parts of the system decide what to do with them.
268
276
 
269
277
  The rest of this document describes each operator. [Appendix E](#appendix-e---operator-quick-reference) lists every operator in one table.
270
278
 
271
279
  ### Statements That ConceptQL Refuses
272
280
 
273
- ConceptQL checks each statement before it builds SQL for it. A statement with an error is invalid. Examples of errors are an unknown operator, a required option that is missing, and a `recall` of a label that the statement does not have. ConceptQL refuses to build SQL for an invalid statement, and so it does not run it. It reports each error with the name of the operator that has the error.
281
+ ConceptQL checks each statement before it builds SQL for it. A statement with an error is invalid. Examples of errors are an unknown operator, a required option that is missing, a `recall` of a label that the statement does not have, and a code that is a number, such as 412 instead of "412". ConceptQL refuses to build SQL for an invalid statement, and so it does not run it. It reports each error with the name of the operator that has the error.
274
282
 
275
283
  A warning does not make a statement invalid. Examples of warnings are an unknown code, an improperly formatted code and a deprecated operator. A statement with warnings runs. When ConceptQL draws the diagram of a statement, it shows the errors and the warnings of each operator, and it does not refuse an invalid statement.
276
284
 
277
- ConceptQL finds the concept of each code in the vocabulary tables. It uses the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no records, and a statement would return no records with no warning. Without a database, for example when ConceptQL only draws a diagram, it uses no vocabulary tables. The vocabulary tables of a dataset can be trimmed to the codes that the data contains. Then ConceptQL can also read a full set of vocabulary tables, the reference lexicon, but only to tell a code that the dataset does not contain from a code that does not exist. When `LEXICON_URL` is set and the program names no other reference lexicon, the `LEXICON_URL` database is the reference lexicon.
285
+ ConceptQL finds the concept of each code in the vocabulary tables. It uses the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no rows, and a statement would return no results with no warning. Without a database, for example when ConceptQL only draws a diagram, it uses no vocabulary tables. The vocabulary tables of a dataset can be trimmed to the codes that the dataset contains. Then ConceptQL can also read a full set of vocabulary tables, the reference lexicon, but only to tell a code that the dataset does not contain from a code that does not exist. When `LEXICON_URL` is set and the program names no other reference lexicon, the `LEXICON_URL` database is the reference lexicon.
278
286
 
279
287
  ## Selection Operators
280
288
 
281
- Selection operators select records from the database. For example, a search for diagnoses of old myocardial infarction (ICD-9-CM 412) is a selection. Most selection operators are leaf operators: no other operator flows into them. `person` and `death` can also take one upstream operator (see [`person`](#person-operator)).
289
+ Selection operators select rows from the dataset and make results from them. For example, a search for diagnoses of old myocardial infarction (ICD-9-CM 412) is a selection. Most selection operators are leaf operators: no other operator flows into them. `person` and `death` can also take one upstream operator (see [`person`](#person-operator)).
282
290
 
283
- The vocabulary operators select records by their codes. The next section describes them, and [Other Selection Operators](#other-selection-operators) describes the rest.
291
+ The vocabulary operators select rows by their codes. The next section describes them, and [Other Selection Operators](#other-selection-operators) describes the rest.
284
292
 
285
293
  ## `Vocabulary` Operator
286
294
 
287
- Selects the records whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once.
295
+ Selects the results whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once.
288
296
 
289
297
  - **Upstream:** none.
290
298
  - **Arguments:**
291
- - `codes` (codelist): One or more codes of the vocabulary. Write each code as the vocabulary writes it, including any dots: 250.01, not 25001. A code matches exactly, including case, and it does not select its child codes. A code alone of * selects every record of the vocabulary, and there is no other wildcard. A code that does not have the vocabulary's format matches nothing and gives a warning.
292
- - **Returns:** Every record whose code is one of the given codes in the operator's vocabulary. The type of the records depends on the vocabulary, for example condition_occurrence for icd9cm and procedure_occurrence for cpt4. An operator does not follow mappings to other vocabularies.
299
+ - `codes` (codelist): One or more codes of the vocabulary. Write each code as the vocabulary writes it, including any dots: 250.01, not 25001. A code matches exactly, including case, and it does not select its child codes. A code alone of * selects every result of the vocabulary, and there is no other wildcard. A code that does not have the vocabulary's format matches nothing and gives a warning.
300
+ - **Returns:** Every result whose code is one of the given codes in the operator's vocabulary. The type of the results depends on the vocabulary, for example condition_occurrence for icd9cm and procedure_occurrence for cpt4. An operator does not follow mappings to other vocabularies.
293
301
 
294
302
  ______________________________________________________________________
295
303
 
296
- **Example 6 - All records with the ICD-9-CM diagnosis code 412 (old myocardial infarction)**
304
+ **Example 6 - All results with the ICD-9-CM diagnosis code 412 (old myocardial infarction)**
297
305
 
298
306
  ```JSON
299
307
  ["icd9cm","412"]
300
308
  ```
301
309
 
302
- ![All records with the ICD-9-CM diagnosis code 412 (old myocardial infarction)](images/b48f871538250e6a219b3eec4c469815db03de9cfdc382a409de1ba014d60f50.svg)
310
+ ![All results with the ICD-9-CM diagnosis code 412 (old myocardial infarction)](images/b48f871538250e6a219b3eec4c469815db03de9cfdc382a409de1ba014d60f50.svg)
303
311
 
304
312
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
305
313
  | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
@@ -318,13 +326,13 @@ ______________________________________________________________________
318
326
 
319
327
  ______________________________________________________________________
320
328
 
321
- **Example 7 - All MS-DRG 205 records: other respiratory system diagnoses with a major complication or comorbidity**
329
+ **Example 7 - All MS-DRG 205 results: other respiratory system diagnoses with a major complication or comorbidity**
322
330
 
323
331
  ```JSON
324
332
  ["ms_drg","205"]
325
333
  ```
326
334
 
327
- ![All MS-DRG 205 records: other respiratory system diagnoses with a major complication or comorbidity](images/cbf9ae203087a1deb86ecd49670a7883fa238a8912360411f090e1c90b11015d.svg)
335
+ ![All MS-DRG 205 results: other respiratory system diagnoses with a major complication or comorbidity](images/cbf9ae203087a1deb86ecd49670a7883fa238a8912360411f090e1c90b11015d.svg)
328
336
 
329
337
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
330
338
  | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
@@ -336,13 +344,13 @@ ______________________________________________________________________
336
344
 
337
345
  ______________________________________________________________________
338
346
 
339
- **Example 8 - All CMS-DRG 205 records (a liver disorder)**
347
+ **Example 8 - All CMS-DRG 205 results (a liver disorder)**
340
348
 
341
349
  ```JSON
342
350
  ["cms_drg","205"]
343
351
  ```
344
352
 
345
- ![All CMS-DRG 205 records (a liver disorder)](images/8ade3c4d946893bea0b378aa7f732ff2b62f43a149e996dc4783128c54482f9b.svg)
353
+ ![All CMS-DRG 205 results (a liver disorder)](images/8ade3c4d946893bea0b378aa7f732ff2b62f43a149e996dc4783128c54482f9b.svg)
346
354
 
347
355
  No results.
348
356
 
@@ -352,22 +360,22 @@ ConceptQL creates one vocabulary operator for each vocabulary that it knows, so
352
360
 
353
361
  These are the most common vocabulary operators:
354
362
 
355
- | Operator | Alias | Vocabulary | Stream type |
356
- | ------------ | -------------- | -------------------------------------------- | -------------------- |
357
- | cpt4 | cpt | CPT-4 procedure codes | procedure_occurrence |
358
- | hcpcs | | HCPCS procedure codes | procedure_occurrence |
359
- | cpt_or_hcpcs | | CPT-4 and HCPCS together | procedure_occurrence |
360
- | icd9cm | icd9 | ICD-9-CM diagnosis codes | condition_occurrence |
361
- | icd9proc | icd9_procedure | ICD-9-CM procedure codes (volume 3) | procedure_occurrence |
362
- | icd10cm | | ICD-10-CM diagnosis codes | condition_occurrence |
363
- | icd10pcs | icd10_pcs | ICD-10-PCS procedure codes | procedure_occurrence |
364
- | ms_drg | | MS-DRG codes, for events from 2007-10-01 | procedure_occurrence |
365
- | cms_drg | | CMS-DRG codes, for events before 2007-10-01 | procedure_occurrence |
366
- | revenue_code | | UB-04 revenue codes | procedure_occurrence |
367
- | loinc | | LOINC lab codes | observation |
368
- | ndc | | National Drug Codes (11-digit package codes) | drug_exposure |
369
- | rxnorm | | RxNorm codes | drug_exposure |
370
- | snomed | | SNOMED CT codes | condition_occurrence |
363
+ | Operator | Alias | Vocabulary | Stream type |
364
+ | ------------ | -------------- | ----------------------------------------------- | -------------------- |
365
+ | cpt4 | cpt | CPT-4 procedure codes | procedure_occurrence |
366
+ | hcpcs | | HCPCS procedure codes | procedure_occurrence |
367
+ | cpt_or_hcpcs | | CPT-4 and HCPCS together | procedure_occurrence |
368
+ | icd9cm | icd9 | ICD-9-CM diagnosis codes | condition_occurrence |
369
+ | icd9proc | icd9_procedure | ICD-9-CM procedure codes (volume 3) | procedure_occurrence |
370
+ | icd10cm | | ICD-10-CM diagnosis codes | condition_occurrence |
371
+ | icd10pcs | icd10_pcs | ICD-10-PCS procedure codes | procedure_occurrence |
372
+ | ms_drg | | MS-DRG codes, for discharges from 2007-10-01 | procedure_occurrence |
373
+ | cms_drg | | CMS-DRG codes, for discharges before 2007-10-01 | procedure_occurrence |
374
+ | revenue_code | | UB-04 revenue codes | procedure_occurrence |
375
+ | loinc | | LOINC lab codes | observation |
376
+ | ndc | | National Drug Codes (11-digit package codes) | drug_exposure |
377
+ | rxnorm | | RxNorm codes | drug_exposure |
378
+ | snomed | | SNOMED CT codes | condition_occurrence |
371
379
 
372
380
  Examples of how codes match:
373
381
 
@@ -378,26 +386,26 @@ Examples of how codes match:
378
386
  - A code that the vocabulary does not contain gives the warning "unknown code(s)".
379
387
  - When ConceptQL has a full lexicon (the reference lexicon) beside the vocabulary tables of the dataset, a code that only the full lexicon contains gives the warning "code(s) not present in this dataset", not "unknown code(s)". It is a warning, so the statement stays valid on a dataset that lacks some of its codes.
380
388
  - A code that has more than one meaning in the vocabulary gives the warning "code(s) with more than one meaning".
381
- - Some data stores CPT-4 and HCPCS codes only in the merged vocabulary CPT4_HCPCS. On that data, `cpt` and `hcpcs` select nothing, because they read only CPT4 and HCPCS. They give the warning "the data has no records of this vocabulary", which names `cpt_or_hcpcs`. Use `cpt_or_hcpcs`, which reads all three vocabularies.
382
- - When the data lists the vocabularies of its records (the GDM table `etl_vocabulary_ids`) and the operator's vocabulary is not in the list, the operator gives the warning "the data has no records of this vocabulary". Its counts are null, not 0, because the data cannot show whether a patient has the codes. This warning covers all of its codes, so they get no other warning, except the warning for an improperly formatted code.
383
- - `rxnorm` does not find records coded with NDC, because an operator does not follow mappings between vocabularies.
389
+ - Some datasets store CPT-4 and HCPCS codes only in the merged vocabulary CPT4_HCPCS. On such a dataset, `cpt` and `hcpcs` select nothing, because they read only CPT4 and HCPCS. They give the warning "the data has no records of this vocabulary", which names `cpt_or_hcpcs`. Use `cpt_or_hcpcs`, which reads all three vocabularies.
390
+ - When the dataset lists the vocabularies of its rows (the GDM table `etl_vocabulary_ids`) and the operator's vocabulary is not in the list, the operator gives the warning "the data has no records of this vocabulary". Its counts are null, not 0, because the dataset cannot show whether a patient has the codes. This warning covers all of its codes, so they get no other warning, except the warning for an improperly formatted code.
391
+ - `rxnorm` does not find rows coded with NDC, because an operator does not follow mappings between vocabularies.
384
392
 
385
393
  ### DRG Codes
386
394
 
387
- CMS renumbered the Diagnosis-Related Group (DRG) codes on 2007-10-01, when the MS-DRG system replaced the CMS-DRG system. Most codes got a new meaning. For example, DRG 205 meant a liver disorder in CMS-DRG, and it means other respiratory system diagnoses in MS-DRG. One vocabulary, DRG, holds the codes of both systems, and a record does not say which meaning applied. So a statement names the DRG system, and ConceptQL selects each record by its code and by its start_date:
395
+ CMS renumbered the Diagnosis-Related Group (DRG) codes on 2007-10-01, when the MS-DRG system replaced the CMS-DRG system. Most codes got a new meaning. For example, DRG 205 meant a liver disorder in CMS-DRG, and it means other respiratory system diagnoses in MS-DRG. One vocabulary, DRG, holds the codes of both systems, and a row does not say which meaning applied. So a statement names the DRG system, and ConceptQL selects each row by its code and by its start_date:
388
396
 
389
- - `ms_drg` selects the DRG records whose start_date is on or after 2007-10-01.
390
- - `cms_drg` selects the DRG records whose start_date is before 2007-10-01.
397
+ - `ms_drg` selects the DRG rows whose start_date is on or after 2007-10-01.
398
+ - `cms_drg` selects the DRG rows whose start_date is before 2007-10-01.
391
399
 
392
- The examples above show both operators. Our test data has no DRG record before 2007-10-01, so the `cms_drg` example returns no records.
400
+ The examples above show both operators. Our test dataset has no DRG row before 2007-10-01, so the `cms_drg` example returns no results.
393
401
 
394
402
  Both operators check codes as the other vocabulary operators do. A DRG code has 3 digits, with leading zeros. A code that the DRG vocabulary does not have gets the warning "unknown code(s)". These operators never give the warning "code(s) with more than one meaning", because the operator chooses the meaning.
395
403
 
396
- The `drg` operator is deprecated. It selects the DRG records of both systems, whatever their dates. A statement that uses `drg` still runs, but `drg` gets the warning "deprecated operator", and a later release will make it an error. Use `ms_drg` or `cms_drg`.
404
+ The `drg` operator is deprecated. It selects the DRG rows of both systems, whatever their dates. A statement that uses `drg` still runs, but `drg` gets the warning "deprecated operator", and a later release will make it an error. Use `ms_drg` or `cms_drg`.
397
405
 
398
406
  ### All Vocabulary Operators
399
407
 
400
- The build writes this table from the metadata of the operators. It lists the vocabulary operators of a ConceptQL database with no vocabulary tables, ordered by name. Each row gives the full name of the operator's vocabulary and the domain, which is the type of the records that the operator returns. The vocabulary tables of a database can add more operators. A deprecated operator still runs, with a warning.
408
+ The build writes this table from the metadata of the operators. It lists the vocabulary operators of a ConceptQL database with no vocabulary tables, ordered by name. Each row gives the full name of the operator's vocabulary and the domain, which is the type of the results that the operator returns. The vocabulary tables of a database can add more operators. A deprecated operator still runs, with a warning.
401
409
 
402
410
  | Operator | Vocabulary | Domain |
403
411
  | ------------------------ | ------------------------------------------------------------------------------------------------------ | -------------------- |
@@ -406,7 +414,7 @@ The build writes this table from the metadata of the operators. It lists the voc
406
414
  | `atc` | WHO Anatomic Therapeutic Chemical Classification | drug_exposure |
407
415
  | `betos` | BETOS | condition_occurrence |
408
416
  | `clinical_code_type` | Clinical Code Type | condition_occurrence |
409
- | `cms_drg` | CMS Diagnosis-Related Group, events before 2007-10-01 | procedure_occurrence |
417
+ | `cms_drg` | CMS Diagnosis-Related Group, discharges before 2007-10-01 | procedure_occurrence |
410
418
  | `concept_class` | OMOP Concept Class | condition_occurrence |
411
419
  | `condition_type` | OMOP Condition Occurrence Type | condition_occurrence |
412
420
  | `cpt4` | Current Procedural Terminology version 4 (AMA) | procedure_occurrence |
@@ -435,7 +443,7 @@ The build writes this table from the metadata of the operators. It lists the voc
435
443
  | `loinc` | Logical Observation Identifiers Names and Codes (Regenstrief Institute) | observation |
436
444
  | `marcla_obs_type` | Market Clarity Observation Type | observation |
437
445
  | `meas_type` | OMOP Measurement Type | observation |
438
- | `ms_drg` | Medicare Severity Diagnosis-Related Group (CMS), events from 2007-10-01 | procedure_occurrence |
446
+ | `ms_drg` | Medicare Severity Diagnosis-Related Group (CMS), discharges from 2007-10-01 | procedure_occurrence |
439
447
  | `ndc` | National Drug Code (FDA and manufacturers) | drug_exposure |
440
448
  | `note_type` | OMOP Note Type | condition_occurrence |
441
449
  | `nucc` | National Uniform Claim Committee Health Care Provider Taxonomy Code Set (NUCC) | condition_occurrence |
@@ -492,7 +500,7 @@ This section describes the selection operators that are not vocabulary operators
492
500
 
493
501
  ### Person-related Operators
494
502
 
495
- Person operators select records from the `patients` table, which holds the patient demographics. The type of these records is `person`. The start_date and end_date of a person record are both the patient's birth date. [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) explains why that is useful.
503
+ Person operators select rows from the `patients` table, which holds the patient demographics. The type of their results is `person`. The start_date and end_date of a person result are both the patient's birth date. [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) explains why that is useful.
496
504
 
497
505
  #### `Gender` Operator
498
506
 
@@ -501,7 +509,7 @@ Selects the patients with the given genders.
501
509
  - **Upstream:** none.
502
510
  - **Arguments:**
503
511
  - `gender` (string, one of `Male`, `Female`, `Unknown`): One or more genders. A value that starts with M selects the male patients, and a value that starts with F selects the female patients. Any other value, such as Unknown, selects the patients whose gender is neither male nor female, including the patients with no gender. Case does not matter.
504
- - **Returns:** One person record for each patient whose gender is one of the given genders. The start_date and the end_date of each record are the patient's birth date.
512
+ - **Returns:** One person result for each patient whose gender is one of the given genders. The start_date and the end_date of each result are the patient's birth date.
505
513
 
506
514
  ______________________________________________________________________
507
515
 
@@ -560,7 +568,7 @@ Selects the patients with the given races.
560
568
  - **Upstream:** none.
561
569
  - **Arguments:**
562
570
  - `races` (codelist): One or more race names, such as White, or race concept IDs. A name matches the concepts with that name in any vocabulary, and case does not matter. The operator also selects the more detailed races under each given race, and each concept that is a kind of one of the race concepts through an is-a relationship. A value that contains Unknown also selects the patients who have no race or race concept ID 0.
563
- - **Returns:** One person record for each patient whose race is one of the given races. The start_date and the end_date of each record are the patient's birth date.
571
+ - **Returns:** One person result for each patient whose race is one of the given races. The start_date and the end_date of each result are the patient's birth date.
564
572
 
565
573
  ______________________________________________________________________
566
574
 
@@ -594,7 +602,7 @@ Selects the patients with the given ethnicities.
594
602
  - **Upstream:** none.
595
603
  - **Arguments:**
596
604
  - `ethnicity` (string, one of `Hispanic or Latino`, `Not Hispanic or Latino`): One or more ethnicities. A value that starts with Hispanic selects the Hispanic or Latino patients, and a value that starts with Not Hispanic selects the patients who are not Hispanic or Latino. A value that contains Unknown selects the patients who have no ethnicity or ethnicity concept ID 0. Case does not matter, and any other value selects no patients.
597
- - **Returns:** One person record for each patient whose ethnicity is one of the given ethnicities. The start_date and the end_date of each record are the patient's birth date.
605
+ - **Returns:** One person result for each patient whose ethnicity is one of the given ethnicities. The start_date and the end_date of each result are the patient's birth date.
598
606
 
599
607
  ______________________________________________________________________
600
608
 
@@ -618,10 +626,10 @@ ______________________________________________________________________
618
626
 
619
627
  #### `Person` Operator
620
628
 
621
- Selects every patient. With one upstream operator, it selects the patients who have a record in the upstream stream.
629
+ Selects every patient. With one upstream operator, it selects the patients who have a result in the upstream stream.
622
630
 
623
631
  - **Upstream:** none, or one operator.
624
- - **Returns:** One record for each selected patient, from the patients table. The type of the records is person, and their start_date and end_date are both the patient's birth date.
632
+ - **Returns:** One result for each selected patient, from the patients table. The type of the results is person, and their start_date and end_date are both the patient's birth date.
625
633
 
626
634
  ______________________________________________________________________
627
635
 
@@ -677,20 +685,20 @@ ______________________________________________________________________
677
685
 
678
686
  #### `Death` Operator
679
687
 
680
- Selects every death record. With one upstream operator, it selects the deaths of the patients who have a record in the upstream stream.
688
+ Selects every death. With one upstream operator, it selects the deaths of the patients who have a result in the upstream stream.
681
689
 
682
690
  - **Upstream:** none, or one operator.
683
- - **Returns:** One record for each selected death, from the deaths table. The type of the records is death, and their start_date and end_date are both the date of death.
691
+ - **Returns:** One result for each selected death, from the deaths table. The type of the results is death, and their start_date and end_date are both the date of death.
684
692
 
685
693
  ______________________________________________________________________
686
694
 
687
- **Example 15 - All death records**
695
+ **Example 15 - All deaths**
688
696
 
689
697
  ```JSON
690
698
  ["death"]
691
699
  ```
692
700
 
693
- ![All death records](images/e281558c677a04df02e53dcc7fd6497a0d831e4037560cd13942b67ab10f98f4.svg)
701
+ ![All deaths](images/e281558c677a04df02e53dcc7fd6497a0d831e4037560cd13942b67ab10f98f4.svg)
694
702
 
695
703
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
696
704
  | --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ------------ |
@@ -725,12 +733,12 @@ ______________________________________________________________________
725
733
 
726
734
  #### `Information Periods` Operator
727
735
 
728
- Selects information periods: the spans of time in which the data has information about a patient, such as periods of insurance enrollment.
736
+ Selects information periods: the spans of time in which the dataset has information about a patient, such as periods of insurance enrollment.
729
737
 
730
738
  - **Upstream:** none.
731
739
  - **Arguments:**
732
740
  - `information_types` (codelist): Zero or more information types, each as a concept ID (an integer) or a concept code of the GDM_INFORMATION_TYPE vocabulary, such as PARTD (Medicare Part D). A statement may mix both. With no information types, the operator selects every information period. With information types, it selects only the information periods whose type is one of them. An unknown code gets a warning and selects no period.
733
- - **Returns:** One record for each selected information period, from the information_periods table. The type of the records is observation_period.
741
+ - **Returns:** One result for each selected information period, from the information_periods table. The type of the results is observation_period.
734
742
 
735
743
  ______________________________________________________________________
736
744
 
@@ -757,20 +765,30 @@ ______________________________________________________________________
757
765
 
758
766
  ______________________________________________________________________
759
767
 
768
+ An information period is a span of time in which the dataset has a given kind of information about a person. The main kinds are periods of eligibility and periods of enrollment.
769
+
770
+ A person can have several information periods of different types, and the periods can overlap. For example, a person can be enrolled in Medicare Part B for one span and in Part D for a different span. Without Part D enrollment, the dataset gives little insight into the prescription drugs of the person. So the type of a period tells which kinds of information exist for that span. In the example above, each person has several periods that overlap.
771
+
772
+ The CDM of OMOP has a similar table, [`observation_period`](https://ohdsi.github.io/CommonDataModel/cdm54.html#observation_period). The CDM describes it so: "This table contains records which define spans of time during which two conditions are expected to hold: (i) Clinical Events that happened to the Person are recorded in the Event tables, and (ii) absence of records indicate such Events did not occur during this span of time."
773
+
774
+ ConceptQL uses the name "information period", because each span says which information the dataset holds. In study design, an observation period is a choice of the analysis. It is often shorter than the span of information that exists.
775
+
776
+ In general, a period is a span of time in the dataset, and a window is a span of time in a study (see [Windows](#windows)).
777
+
760
778
  ### Utilization Operators
761
779
 
762
- Utilization operators select admissions. Each operator returns one record for each admission whose claim came from the matching kind of file. For example, `hospitalization` uses the claims whose file provenance is inpatient.
780
+ Utilization operators select admissions. Each operator returns one result for each admission whose claim came from the matching kind of file. For example, `hospitalization` uses the claims whose file provenance is inpatient.
763
781
 
764
- The type of these records is `condition_occurrence`, because the GDM has no visit or admission type. So `union` combines them with diagnosis records into one stream. `source_value` usually holds a diagnosis.
782
+ The type of these results is `condition_occurrence`, because the GDM has no visit or admission type. So `union` combines them with diagnosis results into one stream. `source_value` usually holds a diagnosis.
765
783
 
766
- Our test data has no SNF claims and no hospice claims, so the examples of `snf` and `hospice` return no records.
784
+ Our test dataset has no SNF claims and no hospice claims, so the examples of `snf` and `hospice` return no results.
767
785
 
768
786
  #### `SNF` Operator
769
787
 
770
- Selects admissions to skilled nursing facilities (SNF): one record for each admission whose claim came from an SNF file.
788
+ Selects admissions to skilled nursing facilities (SNF): one result for each admission whose claim came from an SNF file.
771
789
 
772
790
  - **Upstream:** none.
773
- - **Returns:** One record for each SNF admission. The type of the records is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each record also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
791
+ - **Returns:** One result for each SNF admission. The type of the results is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each result also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
774
792
 
775
793
  ______________________________________________________________________
776
794
 
@@ -788,10 +806,10 @@ ______________________________________________________________________
788
806
 
789
807
  #### `Hospice` Operator
790
808
 
791
- Selects hospice admissions: one record for each admission whose claim came from a hospice file.
809
+ Selects hospice admissions: one result for each admission whose claim came from a hospice file.
792
810
 
793
811
  - **Upstream:** none.
794
- - **Returns:** One record for each hospice admission. The type of the records is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each record also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
812
+ - **Returns:** One result for each hospice admission. The type of the results is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each result also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
795
813
 
796
814
  ______________________________________________________________________
797
815
 
@@ -809,10 +827,10 @@ ______________________________________________________________________
809
827
 
810
828
  #### `Hospitalization` Operator
811
829
 
812
- Selects inpatient hospital admissions: one record for each admission whose claim came from an inpatient file.
830
+ Selects inpatient hospital admissions: one result for each admission whose claim came from an inpatient file.
813
831
 
814
832
  - **Upstream:** none.
815
- - **Returns:** One record for each inpatient admission. The type of the records is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each record also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
833
+ - **Returns:** One result for each inpatient admission. The type of the results is condition_occurrence, and their criterion_table is collections, the table of claims. Their start_date and end_date are the dates of the claim, and source_value holds the primary code of the admission when it has one. Each result also has admission and discharge columns, such as admission_date, discharge_date and length_of_stay.
816
834
 
817
835
  ______________________________________________________________________
818
836
 
@@ -841,14 +859,14 @@ ______________________________________________________________________
841
859
 
842
860
  ## Set Operators
843
861
 
844
- Streams are sets of records, so ConceptQL has operators that work on sets.
862
+ Streams are sets of results, so ConceptQL has operators that work on sets.
845
863
 
846
864
  ### `Union` Operator
847
865
 
848
- Combines the records of all its upstream operators into one stream. Each record keeps its own type, so one stream can hold several types.
866
+ Combines the results of all its upstream operators into one stream. Each result keeps its own type, so one stream can hold several types.
849
867
 
850
- - **Upstream:** one or more operators.
851
- - **Returns:** Each record of each upstream. A record that comes from more than one upstream comes out once. Two records are the same only when every column is equal, including label.
868
+ - **Upstream:** two or more operators.
869
+ - **Returns:** Each result of each upstream. A result that comes from more than one upstream comes out once. Two results are the same only when every column is equal, including label.
852
870
 
853
871
  ______________________________________________________________________
854
872
 
@@ -925,16 +943,16 @@ ______________________________________________________________________
925
943
 
926
944
  ______________________________________________________________________
927
945
 
928
- Union does not change records of different types. Each later operator decides whether it treats the types separately or together.
946
+ Union does not change results of different types. Each later operator decides whether it treats the types separately or together.
929
947
 
930
948
  ### `Except` Operator
931
949
 
932
- Removes each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same criterion_table, criterion_id, start_date and end_date.
950
+ Removes each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same criterion_table, criterion_id, start_date and end_date.
933
951
 
934
952
  - **Operands:**
935
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
936
- - `right` (required): The right-hand stream. The operator removes each left-hand record that is also in this stream, and does not pass these records on.
937
- - **Returns:** The left-hand records that have no right-hand record with the same criterion_table, criterion_id, start_date and end_date. A left-hand record whose table has no right-hand record passes through unchanged. The right-hand records are not returned.
953
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
954
+ - `right` (required): The right-hand stream. The operator removes each left-hand result that is also in this stream, and does not pass these results on.
955
+ - **Returns:** The left-hand results that have no right-hand result with the same criterion_table, criterion_id, start_date and end_date. A left-hand result whose table has no right-hand result passes through unchanged. The right-hand results are not returned.
938
956
 
939
957
  ______________________________________________________________________
940
958
 
@@ -1011,11 +1029,11 @@ ______________________________________________________________________
1011
1029
 
1012
1030
  ______________________________________________________________________
1013
1031
 
1014
- A left-hand record matches only a right-hand record from the same table, and only when the dates are the same. For example, a record whose dates `time_window` changed no longer matches the original record. The second example above shows that a left-hand record passes through unchanged when no right-hand record comes from its table. The third example has several types in the left-hand stream.
1032
+ A left-hand result matches only a right-hand result from the same table, and only when the dates are the same. For example, a result whose dates `time_window` changed no longer matches the original result. The second example above shows that a left-hand result passes through unchanged when no right-hand result comes from its table. The third example has several types in the left-hand stream.
1015
1033
 
1016
1034
  ### Discussion About Set Operators
1017
1035
 
1018
- Why can one stream hold records of several types? Because operators such as `first` then act across all the types. For example, the statement below finds the first MI or death for each patient. Add a few more criteria, and you can find the first censoring event for each patient.
1036
+ Why can one stream hold results of several types? Because operators such as `first` then act across all the types. For example, the statement below finds the first MI or death for each patient. Add a few more criteria, and you can find the first censoring event for each patient.
1019
1037
 
1020
1038
  ______________________________________________________________________
1021
1039
 
@@ -1044,28 +1062,28 @@ ______________________________________________________________________
1044
1062
 
1045
1063
  ## Time-oriented Operators
1046
1064
 
1047
- Every record has a start_date and an end_date. All temporal comparisons use these two columns. Each record takes its dates from its row in its table of origin.
1065
+ Every result has a start_date and an end_date. All temporal comparisons use these two columns. Each result takes its dates from its row in its table of origin.
1048
1066
 
1049
- If the table of origin has only one date, the record uses that date as both its start_date and its end_date. For example, a death record uses the `date` column of its row in `deaths`.
1067
+ If the table of origin has only one date, the result uses that date as both its start_date and its end_date. For example, a death result uses the `date` column of its row in `deaths`.
1050
1068
 
1051
- Person records are a special case. A person record uses the patient's birth date as its start_date and end_date. This may sound strange, but [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) shows why it is useful.
1069
+ Person results are a special case. A person result uses the patient's birth date as its start_date and end_date. This may sound strange, but [Temporal Operators and Person Streams](#temporal-operators-and-person-streams) shows why it is useful.
1052
1070
 
1053
1071
  ## Relative Temporal Operators
1054
1072
 
1055
- For each patient, we may want only the chronologically first or last record, or the second, or the second-to-last. Relative temporal operators provide this filtering. They order each patient's records by start_date. When two records have the same start_date, the record with the lower `criterion_id` comes first.
1073
+ For each patient, we may want only the chronologically first or last result, or the second, or the second-to-last. Relative temporal operators provide this filtering. They order each patient's results by start_date. When two results have the same start_date, the result with the lower `criterion_id` comes first.
1056
1074
 
1057
1075
  ### `Nth Occurrence` Operator
1058
1076
 
1059
- For each person, orders the records by start_date and passes along the record at the given position. A positive position counts from the earliest record and a negative one from the most recent record.
1077
+ For each person, orders the results by start_date and passes along the result at the given position. A positive position counts from the earliest result and a negative one from the most recent result.
1060
1078
 
1061
1079
  - **Upstream:** one operator.
1062
1080
  - **Arguments:**
1063
- - `occurrence` (integer): The position of the record to pass along, as an integer. A positive number counts from the earliest record, so 1 is the first record and 2 is the second. A negative number counts from the most recent record, so -1 is the last record and -3 is the third from the last. 0 selects nothing.
1081
+ - `occurrence` (integer): The position of the result to pass along, as an integer. A positive number counts from the earliest result, so 1 is the first result and 2 is the second. A negative number counts from the most recent result, so -1 is the last result and -3 is the third from the last. 0 selects nothing.
1064
1082
  - **Options:**
1065
- - `at_least` (string): A date adjustment, such as 30d. For each record, the operator counts that record and the later records that start at least this long after it ends, and passes along the record at the given position. So a person can have more than one record in the result.
1066
- - `within` (string): A date adjustment, such as 365d. For each record, the operator counts that record and the later records that start at most this long after it ends, and passes along the record at the given position. So a person can have more than one record in the result.
1067
- - `group_by_date` (boolean): Treats the records of a person on one start_date as one occurrence, so the position counts dates. From the date at that position, the operator passes along the record with the lowest criterion_id, or the highest criterion_id for a negative position.
1068
- - **Returns:** One record per person: the record at the given position. A person who has fewer records than the position asks for is not passed along. With at_least or within, a person can have more than one record.
1083
+ - `at_least` (string): A date adjustment, such as 30d. For each result, the operator counts that result and the later results that start at least this long after it ends, and passes along the result at the given position. So a person can have more than one result.
1084
+ - `within` (string): A date adjustment, such as 365d. For each result, the operator counts that result and the later results that start at most this long after it ends, and passes along the result at the given position. So a person can have more than one result.
1085
+ - `group_by_date` (boolean): Treats the results of a person on one start_date as one occurrence, so the position counts dates. From the date at that position, the operator passes along the result with the lowest criterion_id, or the highest criterion_id for a negative position.
1086
+ - **Returns:** One result per person: the result at the given position. A person who has fewer results than the position asks for is not passed along. With at_least or within, a person can have more than one result.
1069
1087
 
1070
1088
  ______________________________________________________________________
1071
1089
 
@@ -1094,13 +1112,13 @@ ______________________________________________________________________
1094
1112
 
1095
1113
  ______________________________________________________________________
1096
1114
 
1097
- **Example 29 - For each patient, the first record on the second day with any of these diagnoses**
1115
+ **Example 29 - For each patient, the first result on the second day with any of these diagnoses**
1098
1116
 
1099
1117
  ```JSON
1100
1118
  ["occurrence",2,["icd9","410.00","410.01","250.00","250.01"],{"group_by_date":true}]
1101
1119
  ```
1102
1120
 
1103
- ![For each patient, the first record on the second day with any of these diagnoses](images/15987c0030e33d5ac57289ff7018fec87a2cc1252c4467719510035227fc91bc.svg)
1121
+ ![For each patient, the first result on the second day with any of these diagnoses](images/15987c0030e33d5ac57289ff7018fec87a2cc1252c4467719510035227fc91bc.svg)
1104
1122
 
1105
1123
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
1106
1124
  | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
@@ -1121,10 +1139,10 @@ ______________________________________________________________________
1121
1139
 
1122
1140
  ### `First` Operator
1123
1141
 
1124
- For each person, passes along the record with the earliest start_date.
1142
+ For each person, passes along the result with the earliest start_date.
1125
1143
 
1126
1144
  - **Upstream:** one operator.
1127
- - **Returns:** One record per person: the upstream record with the earliest start_date. When several records have that start_date, the one with the lowest criterion_id.
1145
+ - **Returns:** One result per person: the upstream result with the earliest start_date. When several results have that start_date, the one with the lowest criterion_id.
1128
1146
 
1129
1147
  ______________________________________________________________________
1130
1148
 
@@ -1155,10 +1173,10 @@ ______________________________________________________________________
1155
1173
 
1156
1174
  ### `Last` Operator
1157
1175
 
1158
- For each person, passes along the record with the most recent start_date.
1176
+ For each person, passes along the result with the most recent start_date.
1159
1177
 
1160
1178
  - **Upstream:** one operator.
1161
- - **Returns:** One record per person: the upstream record with the latest start_date. When several records have that start_date, the one with the highest criterion_id.
1179
+ - **Returns:** One result per person: the upstream result with the latest start_date. When several results have that start_date, the one with the highest criterion_id.
1162
1180
 
1163
1181
  ______________________________________________________________________
1164
1182
 
@@ -1193,13 +1211,13 @@ To represent a fixed range of dates, use the `date_range` operator.
1193
1211
 
1194
1212
  ### `Date Range` Operator
1195
1213
 
1196
- Makes one person record for each patient, with the given start_date and end_date. Use it to compare other records with a fixed range of dates.
1214
+ Makes one person result for each patient, with the given start_date and end_date. Use it to compare other results with a fixed range of dates.
1197
1215
 
1198
1216
  - **Upstream:** none.
1199
1217
  - **Options:**
1200
- - `start` (string, required): The start_date of each record. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive.
1201
- - `end` (string, required): The end_date of each record. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive. For a single day, give the same date as the start option.
1202
- - **Returns:** One record for each patient. The type of the records is person, and each record has the start_date and the end_date that the options give.
1218
+ - `start` (string, required): The start_date of each result. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive.
1219
+ - `end` (string, required): The end_date of each result. It is a date in YYYY-MM-DD format, START or END. START is the earliest start date of any information period in the database, and END is the latest end date. START and END are not case-sensitive. For a single day, give the same date as the start option.
1220
+ - **Returns:** One result for each patient. The type of the results is person, and each result has the start_date and the end_date that the options give.
1203
1221
 
1204
1222
  ______________________________________________________________________
1205
1223
 
@@ -1228,13 +1246,13 @@ ______________________________________________________________________
1228
1246
 
1229
1247
  ______________________________________________________________________
1230
1248
 
1231
- **Example 33 - Each patient, with the dates from the start to the end of the data**
1249
+ **Example 33 - Each patient, with the dates from the start to the end of the dataset**
1232
1250
 
1233
1251
  ```JSON
1234
1252
  ["date_range",{"start":"START","end":"END"}]
1235
1253
  ```
1236
1254
 
1237
- ![Each patient, with the dates from the start to the end of the data](images/bac60c695e4b466108923e828da1c6b2a0fda21b11ec496d32fa23300c3a4c49.svg)
1255
+ ![Each patient, with the dates from the start to the end of the dataset](images/bac60c695e4b466108923e828da1c6b2a0fda21b11ec496d32fa23300c3a4c49.svg)
1238
1256
 
1239
1257
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
1240
1258
  | --------- | ------------ | --------------- | ---------------- | ---------- | ---------- | ---------------- |
@@ -1255,13 +1273,13 @@ The range includes both dates. A `YYYY-MM-DD` date has a four-digit year, a two-
1255
1273
 
1256
1274
  ## Temporal Comparison Operators
1257
1275
 
1258
- As described above, each record has a start date and an end date, which define its own date range. Temporal operators filter streams by comparing these date ranges.
1276
+ As described above, each result has a start date and an end date, which define its own date range. Temporal operators filter streams by comparing these date ranges.
1259
1277
 
1260
- A temporal operator takes two streams: the left-hand records (LHR) and the right-hand records (RHR). It compares each patient's LHR with the same patient's RHR, and it passes along the LHR that pass the comparison. It discards the RHR after the comparison.
1278
+ A temporal operator takes two streams: the left-hand results (LHR) and the right-hand results (RHR). It compares each patient's LHR with the same patient's RHR, and it passes along the LHR that pass the comparison. It discards the RHR after the comparison.
1261
1279
 
1262
1280
  - Each LHR is passed along at most once, however many RHR it matches.
1263
1281
  - A patient who has LHR but no RHR loses all of their LHR.
1264
- - A record that is in both streams can match itself. For example, a record is during itself.
1282
+ - A result that is in both streams can match itself. For example, a result is during itself.
1265
1283
 
1266
1284
  Edge behaviors: `during`, `contains` and `any_overlap` compare each LHR with every RHR of the patient. `before` and `after` do not. Without the `within` option, `before` compares each LHR only with the latest start_date of the patient's RHR, and `after` compares each LHR only with the earliest end_date of the patient's RHR. [`before`](#before-operator) and [`after`](#after-operator) explain why, and [Working Around `Before`/`After` Behavior](#working-around-beforeafter-behavior) shows how to choose another RHR.
1267
1285
 
@@ -1269,14 +1287,14 @@ Each operator compares the dates in a different way. All of them take [some opti
1269
1287
 
1270
1288
  ### `During` Operator
1271
1289
 
1272
- For each patient, keeps the left-hand records whose date range is inside the date range of a right-hand record. The left-hand record starts on or after the start_date of the right-hand record and ends on or before its end_date.
1290
+ For each patient, keeps the left-hand results whose date range is inside the date range of a right-hand result. The left-hand result starts on or after the start_date of the right-hand result and ends on or before its end_date.
1273
1291
 
1274
1292
  - **Operands:**
1275
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
1276
- - `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
1293
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
1294
+ - `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
1277
1295
  - **Options:**
1278
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand record is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
1279
- - **Returns:** The left-hand records that are inside the date range of at least one right-hand record of the same patient, each record once. The right-hand records are not returned.
1296
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand result is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
1297
+ - **Returns:** The left-hand results that are inside the date range of at least one right-hand result of the same patient, each result once. The right-hand results are not returned.
1280
1298
 
1281
1299
  ______________________________________________________________________
1282
1300
 
@@ -1305,14 +1323,14 @@ ______________________________________________________________________
1305
1323
 
1306
1324
  ### `Contains` Operator
1307
1325
 
1308
- For each patient, keeps the left-hand records whose date range encloses the date range of a right-hand record. The left-hand record starts on or before the start_date of the right-hand record and ends on or after its end_date.
1326
+ For each patient, keeps the left-hand results whose date range encloses the date range of a right-hand result. The left-hand result starts on or before the start_date of the right-hand result and ends on or after its end_date.
1309
1327
 
1310
1328
  - **Operands:**
1311
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
1312
- - `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
1329
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
1330
+ - `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
1313
1331
  - **Options:**
1314
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each left-hand record is widened by this amount in each direction: its start_date moves back and its end_date moves forward. The returned records keep their own dates. Use a positive amount.
1315
- - **Returns:** The left-hand records that enclose the date range of at least one right-hand record of the same patient, each record once. The right-hand records are not returned.
1332
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each left-hand result is widened by this amount in each direction: its start_date moves back and its end_date moves forward. The returned results keep their own dates. Use a positive amount.
1333
+ - **Returns:** The left-hand results that enclose the date range of at least one right-hand result of the same patient, each result once. The right-hand results are not returned.
1316
1334
 
1317
1335
  ______________________________________________________________________
1318
1336
 
@@ -1332,14 +1350,14 @@ ______________________________________________________________________
1332
1350
 
1333
1351
  ### `Any Overlap` Operator
1334
1352
 
1335
- For each patient, keeps the left-hand records whose date range shares at least one day with the date range of a right-hand record.
1353
+ For each patient, keeps the left-hand results whose date range shares at least one day with the date range of a right-hand result.
1336
1354
 
1337
1355
  - **Operands:**
1338
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
1339
- - `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
1356
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
1357
+ - `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
1340
1358
  - **Options:**
1341
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand record is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
1342
- - **Returns:** The left-hand records that overlap the date range of at least one right-hand record of the same patient, each record once. The right-hand records are not returned.
1359
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. Before the comparison, the date range of each right-hand result is widened by this amount in each direction: its start_date moves back and its end_date moves forward. Use a positive amount.
1360
+ - **Returns:** The left-hand results that overlap the date range of at least one right-hand result of the same patient, each result once. The right-hand results are not returned.
1343
1361
 
1344
1362
  ______________________________________________________________________
1345
1363
 
@@ -1358,21 +1376,21 @@ ______________________________________________________________________
1358
1376
 
1359
1377
  ______________________________________________________________________
1360
1378
 
1361
- This diagram shows each kind of L record that has "any overlap" with an R record:
1379
+ This diagram shows each kind of L result that has "any overlap" with an R result:
1362
1380
 
1363
1381
  ![](images/any_overlap.png)
1364
1382
 
1365
1383
  ### `Before` Operator
1366
1384
 
1367
- For each patient, keeps the left-hand records whose end_date is before the latest start_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead.
1385
+ For each patient, keeps the left-hand results whose end_date is before the latest start_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead.
1368
1386
 
1369
1387
  - **Operands:**
1370
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
1371
- - `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
1388
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
1389
+ - `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
1372
1390
  - **Options:**
1373
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand record must also end no more than this amount before the start_date of the right-hand record. With this option, each left-hand record is compared with every right-hand record of the patient, not only with the one that starts last. Use a positive amount.
1374
- - `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand record must end more than this amount before the start_date of the right-hand record. Use a positive amount.
1375
- - **Returns:** The left-hand records that end before a right-hand record of the same patient starts, each record once. The right-hand records are not returned.
1391
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand result must also end no more than this amount before the start_date of the right-hand result. With this option, each left-hand result is compared with every right-hand result of the patient, not only with the one that starts last. Use a positive amount.
1392
+ - `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand result must end more than this amount before the start_date of the right-hand result. Use a positive amount.
1393
+ - **Returns:** The left-hand results that end before a right-hand result of the same patient starts, each result once. The right-hand results are not returned.
1376
1394
 
1377
1395
  ______________________________________________________________________
1378
1396
 
@@ -1442,19 +1460,19 @@ ______________________________________________________________________
1442
1460
 
1443
1461
  The comparison is strict, so an LHR that ends on the day the RHR starts does not pass.
1444
1462
 
1445
- This rule needs some explanation. Imagine the events A-A-B-A-B-A. How many A's come before a B? One could argue that only the first two A's qualify, because they come before the first B. But three A's come before some B. The `before` operator uses the second answer, so it compares every LHR with the start_date of the patient's last RHR.
1463
+ This rule needs some explanation. Imagine the results A-A-B-A-B-A. How many A's come before a B? One could argue that only the first two A's qualify, because they come before the first B. But three A's come before some B. The `before` operator uses the second answer, so it compares every LHR with the start_date of the patient's last RHR.
1446
1464
 
1447
1465
  ### `After` Operator
1448
1466
 
1449
- For each patient, keeps the left-hand records whose start_date is after the earliest end_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead.
1467
+ For each patient, keeps the left-hand results whose start_date is after the earliest end_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead.
1450
1468
 
1451
1469
  - **Operands:**
1452
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
1453
- - `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
1470
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
1471
+ - `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
1454
1472
  - **Options:**
1455
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand record must also start no more than this amount after the end_date of the right-hand record. With this option, each left-hand record is compared with every right-hand record of the patient, not only with the one that ends first. Use a positive amount.
1456
- - `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand record must start more than this amount after the end_date of the right-hand record. Use a positive amount.
1457
- - **Returns:** The left-hand records that start after a right-hand record of the same patient ends, each record once. The right-hand records are not returned.
1473
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d. The left-hand result must also start no more than this amount after the end_date of the right-hand result. With this option, each left-hand result is compared with every right-hand result of the patient, not only with the one that ends first. Use a positive amount.
1474
+ - `at_least` (string): A date adjustment, such as 7d or 1w. The left-hand result must start more than this amount after the end_date of the right-hand result. Use a positive amount.
1475
+ - **Returns:** The left-hand results that start after a right-hand result of the same patient ends, each result once. The right-hand results are not returned.
1458
1476
 
1459
1477
  ______________________________________________________________________
1460
1478
 
@@ -1539,7 +1557,7 @@ ______________________________________________________________________
1539
1557
 
1540
1558
  The comparison is strict, so an LHR that starts on the day the RHR ends does not pass.
1541
1559
 
1542
- Imagine the events A-A-B-A-B-A again. How many A's come after a B? One could argue that only the last A qualifies, because it comes after the last B. But two A's come after some B. The `after` operator uses the second answer, so it compares every LHR with the earliest end_date of the patient's RHR.
1560
+ Imagine the results A-A-B-A-B-A again. How many A's come after a B? One could argue that only the last A qualifies, because it comes after the last B. But two A's come after some B. The `after` operator uses the second answer, so it compares every LHR with the earliest end_date of the patient's RHR.
1543
1561
 
1544
1562
  ### Working Around `Before`/`After` Behavior
1545
1563
 
@@ -1549,13 +1567,13 @@ If you want a different RHR for the comparison, use a relative temporal operator
1549
1567
 
1550
1568
  ### `Time Window` Operator
1551
1569
 
1552
- Moves the start_date and the end_date of each record. Use it to shift or widen a record's dates before you compare them with other records.
1570
+ Moves the start_date and the end_date of each result. Use it to shift or widen a result's dates before you compare them with other results.
1553
1571
 
1554
1572
  - **Upstream:** one operator.
1555
1573
  - **Options:**
1556
- - `start` (string): How to change the start_date. If you omit it, the start_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the record's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
1557
- - `end` (string): How to change the end_date. If you omit it, the end_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the record's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
1558
- - **Returns:** One record for each upstream record, with the start_date and the end_date that the options give. The other columns do not change.
1574
+ - `start` (string): How to change the start_date. If you omit it, the start_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the result's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
1575
+ - `end` (string): How to change the end_date. If you omit it, the end_date does not change. The value is an amount of time, made of one or more numbers, each with a unit: d (days), w (weeks), m (months) or y (years). Each part can have a - or + sign, a bare number means days, and a unit without a number means 1 of it. For example, -1m is 1 month back, and 10d3m is 3 months and 10 days. An empty value or 0 leaves the date unchanged. The value start or end uses the result's start_date or end_date, and a date in YYYY-MM-DD format sets the date to that date.
1576
+ - **Returns:** One result for each upstream result, with the start_date and the end_date that the options give. The other columns do not change.
1559
1577
 
1560
1578
  ______________________________________________________________________
1561
1579
 
@@ -1684,14 +1702,14 @@ More examples of date adjustments:
1684
1702
 
1685
1703
  ### `Trim Date Start` Operator
1686
1704
 
1687
- For each patient, moves the start_date of each left-hand record forward to the end_date of the right-hand record with the latest start_date. A left-hand record that ends before that end_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged.
1705
+ For each patient, moves the start_date of each left-hand result forward to the end_date of the right-hand result with the latest start_date. A left-hand result that ends before that end_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged.
1688
1706
 
1689
1707
  - **Operands:**
1690
- - `left` (required): The left-hand stream. The operator moves the start_date of these records and passes them on.
1691
- - `right` (required): The right-hand stream. For each patient, the end_date of the record with the latest start_date sets the earliest start_date of the left-hand records. These records are not passed on.
1708
+ - `left` (required): The left-hand stream. The operator moves the start_date of these results and passes them on.
1709
+ - `right` (required): The right-hand stream. For each patient, the end_date of the result with the latest start_date sets the earliest start_date of the left-hand results. These results are not passed on.
1692
1710
  - **Options:**
1693
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is added to the end_date of the right-hand record before the comparison. The adjusted date is the one that moves the start_date of the left-hand record.
1694
- - **Returns:** The left-hand records, each with its start_date set to the later of its own start_date and the end_date of the patient's right-hand record. Left-hand records that end before that end_date are not returned. The right-hand records are not returned.
1711
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is added to the end_date of the right-hand result before the comparison. The adjusted date is the one that moves the start_date of the left-hand result.
1712
+ - **Returns:** The left-hand results, each with its start_date set to the later of its own start_date and the end_date of the patient's right-hand result. Left-hand results that end before that end_date are not returned. The right-hand results are not returned.
1695
1713
 
1696
1714
  ______________________________________________________________________
1697
1715
 
@@ -1718,7 +1736,7 @@ ______________________________________________________________________
1718
1736
 
1719
1737
  ______________________________________________________________________
1720
1738
 
1721
- Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's records. There are four cases:
1739
+ Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's results. There are four cases:
1722
1740
 
1723
1741
  - The patient has no RHR.
1724
1742
  - The LHR passes through unchanged.
@@ -1731,14 +1749,14 @@ Like a [temporal comparison operator](#temporal-comparison-operators), it takes
1731
1749
 
1732
1750
  ### `Trim Date End` Operator
1733
1751
 
1734
- For each patient, moves the end_date of each left-hand record back to the start_date of the right-hand record with the earliest start_date. A left-hand record that starts after that start_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged.
1752
+ For each patient, moves the end_date of each left-hand result back to the start_date of the right-hand result with the earliest start_date. A left-hand result that starts after that start_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged.
1735
1753
 
1736
1754
  - **Operands:**
1737
- - `left` (required): The left-hand stream. The operator moves the end_date of these records and passes them on.
1738
- - `right` (required): The right-hand stream. For each patient, the earliest start_date of these records sets the latest end_date of the left-hand records. These records are not passed on.
1755
+ - `left` (required): The left-hand stream. The operator moves the end_date of these results and passes them on.
1756
+ - `right` (required): The right-hand stream. For each patient, the earliest start_date of these results sets the latest end_date of the left-hand results. These results are not passed on.
1739
1757
  - **Options:**
1740
- - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is subtracted from the start_date of the right-hand record before the comparison. The adjusted date is the one that moves the end_date of the left-hand record.
1741
- - **Returns:** The left-hand records, each with its end_date set to the earlier of its own end_date and the start_date of the patient's right-hand record. Left-hand records that start after that start_date are not returned. The right-hand records are not returned.
1758
+ - `within` (string): A date adjustment, such as 30d, 2m or 1y-3d, that is subtracted from the start_date of the right-hand result before the comparison. The adjusted date is the one that moves the end_date of the left-hand result.
1759
+ - **Returns:** The left-hand results, each with its end_date set to the earlier of its own end_date and the start_date of the patient's right-hand result. Left-hand results that start after that start_date are not returned. The right-hand results are not returned.
1742
1760
 
1743
1761
  ______________________________________________________________________
1744
1762
 
@@ -1765,7 +1783,7 @@ ______________________________________________________________________
1765
1783
 
1766
1784
  ______________________________________________________________________
1767
1785
 
1768
- Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's records. There are four cases:
1786
+ Like a [temporal comparison operator](#temporal-comparison-operators), it takes left-hand and right-hand streams and compares each patient's results. There are four cases:
1769
1787
 
1770
1788
  - The patient has no RHR.
1771
1789
  - The LHR passes through unchanged.
@@ -1787,7 +1805,7 @@ Systems that build statements, such as Jigsaw's export engine, give `trim_date_s
1787
1805
  - The `within` option moves each RHR date before the comparison, as above.
1788
1806
  - An LHR is discarded only when it has RHR (of its patient and its `join_columns` values) and none of them qualifies. An LHR without RHR passes through unchanged.
1789
1807
 
1790
- These systems often join on the column `uuid`. A selection operator, such as a vocabulary operator, takes the option `uuid` (`{"uuid": true}`), which adds the column `uuid` to its records. The uuid is the `person_id`, `criterion_id`, `criterion_table`, start_date and end_date of the record, joined with `/`, such as `17/27269/observations/2008-08-25/2008-08-25`. An empty part is an empty string, so a record without an end_date has a uuid such as `1/2/observations/2008-03-10/`. The uuid keeps its value when a later operator, such as `time_window`, moves the dates. So two streams made from the same records can be joined record by record.
1808
+ These systems often join on the column `uuid`. A selection operator, such as a vocabulary operator, takes the option `uuid` (`{"uuid": true}`), which adds the column `uuid` to its results. The uuid is the `person_id`, `criterion_id`, `criterion_table`, start_date and end_date of the result, joined with `/`, such as `17/27269/observations/2008-08-25/2008-08-25`. An empty part is an empty string, so a result without an end_date has a uuid such as `1/2/observations/2008-03-10/`. The uuid keeps its value when a later operator, such as `time_window`, moves the dates. So two streams made from the same results can be joined result by result.
1791
1809
 
1792
1810
  This example makes a one-year window after each MI. Then it trims each window to end 30 days after its own MI. The `join_columns` option makes sure that no other MI of the same patient trims the window:
1793
1811
 
@@ -1831,23 +1849,23 @@ Walk-through of the second example of [`after`](#after-operator), which finds th
1831
1849
  Walk-through of the third example of [`before`](#before-operator), which asks whether anyone dies within a year of a heart attack:
1832
1850
 
1833
1851
  - Select heart attack diagnoses as the LHR.
1834
- - Select death records as the RHR.
1852
+ - Select deaths as the RHR.
1835
1853
  - Keep the LHR whose end_date is before the date of death, and no more than 1 year before it.
1836
1854
 
1837
1855
  The third example of [`after`](#after-operator) shows how `at_least` works.
1838
1856
 
1839
1857
  ### Temporal Operators and Person Streams
1840
1858
 
1841
- The start and end dates of a person record are the patient's birth date. So a person stream is rarely useful as the left-hand stream of a temporal operator. But it is useful as the right-hand stream. If we move the birth date with `time_window`, we can filter by the patient's age. The last example of [`after`](#after-operator) selects the MIs that occurred after a male patient's 50th birthday.
1859
+ The start and end dates of a person result are the patient's birth date. So a person stream is rarely useful as the left-hand stream of a temporal operator. But it is useful as the right-hand stream. If we move the birth date with `time_window`, we can filter by the patient's age. The last example of [`after`](#after-operator) selects the MIs that occurred after a male patient's 50th birthday.
1842
1860
 
1843
1861
  ## `Episode` Operator
1844
1862
 
1845
- Merges each patient's records into episodes: spans of time in which the records are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug.
1863
+ Merges each patient's results into episodes: spans of time in which the results are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug.
1846
1864
 
1847
1865
  - **Upstream:** one operator.
1848
1866
  - **Options:**
1849
- - `gap_of` (integer): The largest gap in days that stays in the same episode. The operator orders each patient's records by start_date and then by end_date. A record stays in the current episode when its start_date is no more than this many days after the latest end_date of the earlier records, not only of the record just before. Otherwise it starts a new episode. A record without an end_date counts as ending on its start_date. The default is 0.
1850
- - **Returns:** One record for each episode. Its start_date is the earliest start_date in the episode, and its end_date is the latest end_date. Its criterion_table and criterion_domain are episode, and its criterion_id is 0, because an episode merges several records.
1867
+ - `gap_of` (integer): The largest gap in days that stays in the same episode. The operator orders each patient's results by start_date and then by end_date. A result stays in the current episode when its start_date is no more than this many days after the latest end_date of the earlier results, not only of the result just before. Otherwise it starts a new episode. A result without an end_date counts as ending on its start_date. The default is 0.
1868
+ - **Returns:** One result for each episode. Its start_date is the earliest start_date in the episode, and its end_date is the latest end_date. Its criterion_table and criterion_domain are episode, and its criterion_id is 0, because an episode merges several results.
1851
1869
 
1852
1870
  ______________________________________________________________________
1853
1871
 
@@ -1874,20 +1892,20 @@ ______________________________________________________________________
1874
1892
 
1875
1893
  ______________________________________________________________________
1876
1894
 
1877
- A long record keeps the episode open: the records 2020-01-01 to 2020-03-01 and 2020-01-05 to 2020-01-10 and 2020-02-15 to 2020-02-20 make one episode with `gap_of` 0, because the first record ends after the third record starts. An episode stream is often most useful as the right-hand stream of a [temporal operator](#temporal-comparison-operators).
1895
+ A long result keeps the episode open: the results 2020-01-01 to 2020-03-01 and 2020-01-05 to 2020-01-10 and 2020-02-15 to 2020-02-20 make one episode with `gap_of` 0, because the first result ends after the third result starts. An episode stream is often most useful as the right-hand stream of a [temporal operator](#temporal-comparison-operators).
1878
1896
 
1879
1897
  ## Inline-Filter Operators
1880
1898
 
1881
- These operators filter one incoming stream and pass along the records that remain.
1899
+ These operators filter one incoming stream and pass along the results that remain.
1882
1900
 
1883
1901
  ### `Place of Service Filter` Operator
1884
1902
 
1885
- Passes along the records whose place of service is one of the given CMS Place of Service codes. A record that has no place of service is not passed along.
1903
+ Passes along the results whose place of service is one of the given CMS Place of Service codes. A result that has no place of service is not passed along.
1886
1904
 
1887
1905
  - **Upstream:** one operator.
1888
1906
  - **Arguments:**
1889
- - `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a record whose place of service is any code of that vocabulary. An unknown code gets a warning and matches no record.
1890
- - **Returns:** The upstream records whose place of service is one of the given codes.
1907
+ - `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a result whose place of service is any code of that vocabulary. An unknown code gets a warning and matches no result.
1908
+ - **Returns:** The upstream results whose place of service is one of the given codes.
1891
1909
 
1892
1910
  ______________________________________________________________________
1893
1911
 
@@ -1903,16 +1921,16 @@ No results.
1903
1921
 
1904
1922
  ______________________________________________________________________
1905
1923
 
1906
- Claims often report the place of service of an event with a CMS place of service code. Our test data has no place of service information, so the example returns no records.
1924
+ Claims often report the place of service with a CMS place of service code. Our test dataset has no place of service information, so the example returns no results.
1907
1925
 
1908
1926
  ### `Provenance` Operator
1909
1927
 
1910
- Passes along the records whose provenance matches at least one of the given keywords. The provenance of a record says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary.
1928
+ Passes along the results whose provenance matches at least one of the given keywords. The provenance of a result says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary.
1911
1929
 
1912
1930
  - **Upstream:** one operator.
1913
1931
  - **Arguments:**
1914
- - `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a record came from. A code keyword, such as primary or first, says where the code was on the claim. A plain keyword matches a record when its file provenance or its code provenance matches it. A keyword also matches the provenances under it, so inpatient includes snf. A keyword of the form file:code, such as inpatient:primary, needs both the file and the code provenance to match. The keyword * matches a record that has any provenance. A statement with no keyword is invalid. An unknown keyword gets a warning and matches no record.
1915
- - **Returns:** The upstream records whose provenance matches at least one of the keywords.
1932
+ - `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a result came from. A code keyword, such as primary or first, says where the code was on the claim. A plain keyword matches a result when its file provenance or its code provenance matches it. A keyword also matches the provenances under it, so inpatient includes snf. A keyword of the form file:code, such as inpatient:primary, needs both the file and the code provenance to match. The keyword * matches a result that has any provenance. A statement with no keyword is invalid. An unknown keyword gets a warning and matches no result.
1933
+ - **Returns:** The upstream results whose provenance matches at least one of the keywords.
1916
1934
 
1917
1935
  ______________________________________________________________________
1918
1936
 
@@ -1939,24 +1957,47 @@ ______________________________________________________________________
1939
1957
 
1940
1958
  ______________________________________________________________________
1941
1959
 
1942
- Provenance is information about the origin of claims data. It is sometimes an important factor in research algorithms. For example, authors often describe "inpatient records", by which they mean claims from the inpatient file of a claims dataset. The [GDM](https://github.com/outcomesinsights/generalized_data_model#contexts) records the provenance of the data during ETL.
1960
+ Provenance is information about the origin of a row of claims data. It is sometimes an important factor in research algorithms. There are two kinds of provenance:
1961
+
1962
+ - File provenance is the file of the dataset that a row came from. Examples are the MedPAR file and the carrier claims file of Medicare data.
1963
+ - Code provenance is the role of a code on its claim. Examples are a primary diagnosis and an admitting diagnosis.
1964
+
1965
+ The [GDM](https://github.com/outcomesinsights/generalized_data_model#contexts) records both kinds during ETL. The file provenance of a row is the `source_type_concept_id` of its context. The code provenance of a row is its `provenance_concept_id`. ConceptQL names these two columns `file_provenance_type` and `code_provenance_type`.
1966
+
1967
+ Each keyword names a concept in a provenance vocabulary. A file keyword names a concept of the file provenance vocabulary, and a code keyword names a concept of the code provenance vocabulary. A keyword matches its concept and each descendant of that concept in the `concept_ancestor` table of the dataset. For example, in our test dataset, the concept of the MedPAR file of inpatient stays is a descendant of `inpatient`.
1943
1968
 
1944
1969
  These are the keywords of each kind:
1945
1970
 
1946
1971
  - File keywords: `inpatient`, `outpatient`, `carrier_claim`, `claim`, `dme`, `hha`, `hospice`, `snf`, `prescription_drug` and `demographics`.
1947
1972
  - Code keywords: `primary`, `primary_or_first`, `first`, `second`, `admitting`, `discharge`, `problem_list`, `chief_complaint` and `unspecified`.
1948
1973
 
1949
- Besides `inpatient`, which includes `snf`, `outpatient` includes `hha` and `hospice`, and `claim` includes every kind of claim file.
1974
+ The hierarchy of these concepts is part of the vocabulary tables of the dataset. It is not a rule of ConceptQL. In the vocabulary tables of our test dataset, `inpatient` includes `snf`, and `outpatient` includes `hha` and `hospice`. There, `carrier_claim` includes `dme`, and `claim` includes each claim file of the test dataset.
1975
+
1976
+ #### Inpatient and Outpatient Settings
1977
+
1978
+ Authors often describe "inpatient records" or "outpatient visits". In ConceptQL terms, the authors mean the results whose rows came from an inpatient setting. An inpatient setting is meant to be a stay after admission to a hospital. In practice, the line between the inpatient and the outpatient setting is poorly defined. These are some of the hard cases:
1979
+
1980
+ - In some data, the file decides. For example, the MedPAR file of Medicare data holds the inpatient stays.
1981
+ - A carrier claim can be for a service given in a hospital.
1982
+ - An emergency room visit can become an admission. Then the visit has an outpatient part and an inpatient part.
1983
+ - The place of service code of a claim may or may not decide the setting.
1984
+ - A stay in a skilled nursing facility may or may not count as inpatient.
1985
+
1986
+ Outpatient visits have the same problems.
1987
+
1988
+ ConceptQL does not decide these cases. The dataset declares what counts as inpatient and what counts as outpatient, through a mapping. The GDM stores the concrete facts of the dataset, such as the file that a row came from and the place of service. The mapping, not the ETL, decides which of these rows are inpatient. So the setting is a decision of the analysis. A different analysis can use a different mapping, with no new ETL.
1989
+
1990
+ The mapping is the hierarchy of the file and code provenance concepts in the vocabulary tables of the dataset. For example, our test dataset maps its inpatient claims file to `inpatient` and its outpatient claims file to `outpatient`. [`one_in_two_out`](#one-in-two-out-operator) uses the keywords `inpatient`, `outpatient` and `carrier_claim`. As plain keywords, they match either kind of provenance. In our test dataset, only the file provenance matches them, because no code provenance of the dataset is a descendant of these keywords. The [utilization operators](#utilization-operators), such as `hospitalization`, use only the file provenance.
1950
1991
 
1951
1992
  ### `Provider Filter` Operator
1952
1993
 
1953
- Passes along the records whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value.
1994
+ Passes along the results whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value.
1954
1995
 
1955
1996
  - **Upstream:** one operator.
1956
1997
  - **Options:**
1957
1998
  - `specialties` (string, required): A comma-separated list of specialty concept IDs, such as 38003868 (Endocrinology, Diabetes & Metabolism Physician), or * for any specialty. A provider matches when the specialty recorded for the provider on the claim, or the provider's own specialty, is in the list.
1958
1999
  - `roles` (string, required): A comma-separated list of role concept IDs of the GDM_ROLE vocabulary, such as 100000067 (Attending physician) or 100176860 (Billing provider), or * for any role.
1959
- - **Returns:** The upstream records that have a provider who matches both the specialties and the roles. A record can have more than one provider, so a record comes out once for each provider who matches. Each record gets the columns provider_id, context_specialty_concept_id, practitioner_specialty_concept_id and role_type_concept_id.
2000
+ - **Returns:** The upstream results that have a provider who matches both the specialties and the roles. A result can have more than one provider, so a result comes out once for each provider who matches. Each result gets the columns provider_id, context_specialty_concept_id, practitioner_specialty_concept_id and role_type_concept_id.
1960
2001
 
1961
2002
  ______________________________________________________________________
1962
2003
 
@@ -2005,25 +2046,25 @@ ______________________________________________________________________
2005
2046
 
2006
2047
  ### `One In Two Out` Operator
2007
2048
 
2008
- For each person, passes along the earliest confirmed condition record. One inpatient record confirms a condition, and so do two outpatient records whose start dates are a given gap apart.
2049
+ For each person, passes along the earliest confirmed condition result. One inpatient result confirms a condition, and so do two outpatient results whose start dates are a given gap apart.
2009
2050
 
2010
2051
  - **Upstream:** one operator.
2011
2052
  - **Options:**
2012
- - `inpatient_length_of_stay` (integer): An inpatient stay must be longer than this many days: an inpatient record counts only when its discharge date is more than this many days after its admission date. 0 or no value means no length-of-stay check.
2013
- - `outpatient_minimum_gap` (string): A date adjustment, such as 30d. The confirming outpatient record must start at least this long after the initial outpatient record. Without this option, the gap is 30 days.
2014
- - `outpatient_maximum_gap` (string): A date adjustment, such as 365d. The confirming outpatient record must start no more than this long after the initial outpatient record. Without this option, there is no maximum gap.
2015
- - `outpatient_event_to_return` (string, one of `Initial Event`, `Confirming Event`): Which outpatient record the operator passes on when two outpatient records confirm the condition: the initial record or the confirming record. Without this option, or with any value other than Initial Event, it passes on the confirming record.
2016
- - **Returns:** One record per person: the earliest confirmed condition record. A returned inpatient record has the admission date as its start_date and the discharge date as its end_date, when the claim has them. Only condition records with inpatient, outpatient or carrier_claim provenance can be confirmed.
2053
+ - `inpatient_length_of_stay` (integer): An inpatient stay must be longer than this many days: an inpatient result counts only when its discharge date is more than this many days after its admission date. 0 or no value means no length-of-stay check.
2054
+ - `outpatient_minimum_gap` (string): A date adjustment, such as 30d. The confirming outpatient result must start at least this long after the initial outpatient result. Without this option, the gap is 30 days.
2055
+ - `outpatient_maximum_gap` (string): A date adjustment, such as 365d. The confirming outpatient result must start no more than this long after the initial outpatient result. Without this option, there is no maximum gap.
2056
+ - `outpatient_event_to_return` (string, one of `Initial Event`, `Confirming Event`): Which outpatient result the operator passes on when two outpatient results confirm the condition: the initial result or the confirming result. Without this option, or with any value other than Initial Event, it passes on the confirming result.
2057
+ - **Returns:** One result per person: the earliest confirmed condition result. A returned inpatient result has the admission date as its start_date and the discharge date as its end_date, when the claim has them. Only condition results with inpatient, outpatient or carrier_claim provenance can be confirmed.
2017
2058
 
2018
2059
  ______________________________________________________________________
2019
2060
 
2020
- **Example 57 - First diabetes diagnosis, either found once in an inpatient record, or twice in outpatient records at least 30 days apart**
2061
+ **Example 57 - First diabetes diagnosis, either one inpatient result, or two outpatient results at least 30 days apart**
2021
2062
 
2022
2063
  ```JSON
2023
2064
  ["one_in_two_out",["icd9","250.00"],{"outpatient_minimum_gap":"30d","outpatient_event_to_return":"Initial Event"}]
2024
2065
  ```
2025
2066
 
2026
- ![First diabetes diagnosis, either found once in an inpatient record, or twice in outpatient records at least 30 days apart](images/b960e22cfff80e63e8cc513360fa42bd42c4227b922685008febdc033fb2d32b.svg)
2067
+ ![First diabetes diagnosis, either one inpatient result, or two outpatient results at least 30 days apart](images/b960e22cfff80e63e8cc513360fa42bd42c4227b922685008febdc033fb2d32b.svg)
2027
2068
 
2028
2069
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2029
2070
  | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
@@ -2040,31 +2081,107 @@ ______________________________________________________________________
2040
2081
 
2041
2082
  ______________________________________________________________________
2042
2083
 
2043
- The operator decides between inpatient and outpatient by the provenance of each record (see [`provenance`](#provenance-operator)):
2084
+ The operator decides between inpatient and outpatient by the provenance of each result (see [`provenance`](#provenance-operator)):
2044
2085
 
2045
- - A record with `inpatient` provenance (which includes SNF) is an inpatient record.
2046
- - A record with `outpatient` or `carrier_claim` provenance is an outpatient record.
2047
- - Other records, and records whose type is not `condition_occurrence`, are removed.
2086
+ - A result with `inpatient` provenance (which includes SNF in our test dataset) is an inpatient result.
2087
+ - A result with `outpatient` or `carrier_claim` provenance is an outpatient result.
2088
+ - Other results, and results whose type is not `condition_occurrence`, are removed.
2048
2089
 
2049
- The place of service of a record does not matter. The two outpatient records need not have the same code. Any two records from the upstream stream can confirm each other. For outpatient records, the operator compares start_dates.
2090
+ The place of service of a result does not matter. The two outpatient results need not have the same code. Any two results from the upstream stream can confirm each other. For outpatient results, the operator compares start_dates.
2050
2091
 
2051
2092
  More about the options:
2052
2093
 
2053
2094
  - `outpatient_maximum_gap` helps with acute diseases. For example, to confirm the flu, set a maximum gap so that the two diagnoses come from the same illness. Jigsaw fills in 365 days.
2054
2095
  - `outpatient_event_to_return`: use `Initial Event` in most cases, for example when the algorithm records an exposure of interest or an outcome. Use `Confirming Event` when the algorithm finds an index event. The initial event would introduce immortal time bias. Jigsaw fills in `Initial Event`, but a statement that omits the option gets `Confirming Event`. So always set this option.
2055
2096
 
2097
+ ## Comparison Operators
2098
+
2099
+ These operators compare the results of incoming streams to produce a set of results.
2100
+
2101
+ ### `Filter` Operator
2102
+
2103
+ Keeps each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date.
2104
+
2105
+ - **Operands:**
2106
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
2107
+ - `right` (required): The right-hand stream. The operator keeps each left-hand result that is also in this stream, and does not pass these results on.
2108
+ - **Returns:** The left-hand results that have a right-hand result with the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. The right-hand results are not returned.
2109
+
2110
+ ______________________________________________________________________
2111
+
2112
+ **Example 58 - MIs that are a patient's first diagnosis of MI or diabetes**
2113
+
2114
+ ```JSON
2115
+ ["filter",{"left":["icd9","412"],"right":["first",["union",["icd9","412"],["icd9","250.01"]]]}]
2116
+ ```
2117
+
2118
+ ![MIs that are a patient's first diagnosis of MI or diabetes](images/9764fa797ec9e7cf4eacc04cd15d83453ecb499ff0b5a440e190f8be3f2624c0.svg)
2119
+
2120
+ | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2121
+ | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
2122
+ | 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
2123
+ | 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
2124
+ | 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
2125
+ | 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
2126
+ | 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
2127
+ | 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
2128
+ | 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
2129
+ | 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
2130
+ | 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
2131
+ | 90 | 33940 | observations | condition_occurrence | 2010-05-26 | 2010-05-26 | 412 |
2132
+
2133
+ ______________________________________________________________________
2134
+
2135
+ It is an intersection of the two streams. Filter compares results, not date ranges. A result whose dates `time_window` changed no longer matches the original result.
2136
+
2137
+ ### `Match` Operator
2138
+
2139
+ Keeps each left-hand result that is equal to a right-hand result in every column, including label. Two empty (NULL) values count as equal.
2140
+
2141
+ - **Operands:**
2142
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
2143
+ - `right` (required): The right-hand stream. The operator keeps each left-hand result that is equal to a result of this stream, and does not pass these results on.
2144
+ - **Returns:** The left-hand results that have a right-hand result with the same value in every column. The right-hand results are not returned.
2145
+
2146
+ ______________________________________________________________________
2147
+
2148
+ **Example 59 - MIs that are a patient's first MI**
2149
+
2150
+ ```JSON
2151
+ ["match",{"left":["icd9","412"],"right":["first",["icd9","412"]]}]
2152
+ ```
2153
+
2154
+ ![MIs that are a patient's first MI](images/5d5085351acc8c6776b7df9e62c3143a65f00fffb715e7929e65d8aad0509890.svg)
2155
+
2156
+ | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2157
+ | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
2158
+ | 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
2159
+ | 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
2160
+ | 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
2161
+ | 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
2162
+ | 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
2163
+ | 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
2164
+ | 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
2165
+ | 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
2166
+ | 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
2167
+ | 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
2168
+
2169
+ ______________________________________________________________________
2170
+
2171
+ It is stricter than [`filter`](#filter-operator), so most users want `filter`. Systems that build statements, such as Jigsaw's export engine, use match with options that Jigsaw does not show.
2172
+
2056
2173
  ### `Person Filter` Operator
2057
2174
 
2058
- Keeps each left-hand record whose patient has at least one record in the right-hand stream. The type and the dates of the right-hand records do not matter.
2175
+ Keeps each left-hand result whose patient has at least one result in the right-hand stream. The type and the dates of the right-hand results do not matter.
2059
2176
 
2060
2177
  - **Operands:**
2061
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
2062
- - `right` (required): The right-hand stream. The operator keeps the left-hand records of each patient who has a record in this stream, and does not pass these records on.
2063
- - **Returns:** The left-hand records of every type whose patient has at least one right-hand record. The right-hand records are not returned.
2178
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
2179
+ - `right` (required): The right-hand stream. The operator keeps the left-hand results of each patient who has a result in this stream, and does not pass these results on.
2180
+ - **Returns:** The left-hand results of every type whose patient has at least one right-hand result. The right-hand results are not returned.
2064
2181
 
2065
2182
  ______________________________________________________________________
2066
2183
 
2067
- **Example 58 - All MI Conditions for people who are male**
2184
+ **Example 60 - All MI Conditions for people who are male**
2068
2185
 
2069
2186
  ```JSON
2070
2187
  ["person_filter",{"left":["icd9","412"],"right":["gender","Male"]}]
@@ -2089,13 +2206,13 @@ ______________________________________________________________________
2089
2206
 
2090
2207
  ______________________________________________________________________
2091
2208
 
2092
- **Example 59 - All MI Conditions for people who had an office visit at some point in the data**
2209
+ **Example 61 - All MI Conditions for people who had an office visit at some point in the dataset**
2093
2210
 
2094
2211
  ```JSON
2095
2212
  ["person_filter",{"left":["icd9","412"],"right":["cpt","99214"]}]
2096
2213
  ```
2097
2214
 
2098
- ![All MI Conditions for people who had an office visit at some point in the data](images/cc16bcf500d096db934dc11e48140704bf8b255d116e52aaf9bde84387f818a1.svg)
2215
+ ![All MI Conditions for people who had an office visit at some point in the dataset](images/cc16bcf500d096db934dc11e48140704bf8b255d116e52aaf9bde84387f818a1.svg)
2099
2216
 
2100
2217
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2101
2218
  | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
@@ -2114,7 +2231,7 @@ ______________________________________________________________________
2114
2231
 
2115
2232
  ______________________________________________________________________
2116
2233
 
2117
- **Example 60 - The same result as the example above, with the office visits converted to patients first**
2234
+ **Example 62 - The same result as the example above, with the office visits converted to patients first**
2118
2235
 
2119
2236
  ```JSON
2120
2237
  ["person_filter",{"left":["icd9","412"],"right":["person",["cpt","99214"]]}]
@@ -2139,13 +2256,13 @@ ______________________________________________________________________
2139
2256
 
2140
2257
  ______________________________________________________________________
2141
2258
 
2142
- **Example 61 - All MI Conditions for people who are Male OR had an office visit at some point in the data**
2259
+ **Example 63 - All MI Conditions for people who are Male OR had an office visit at some point in the dataset**
2143
2260
 
2144
2261
  ```JSON
2145
2262
  ["person_filter",{"left":["icd9","412"],"right":["union",["cpt","99214"],["gender","Male"]]}]
2146
2263
  ```
2147
2264
 
2148
- ![All MI Conditions for people who are Male OR had an office visit at some point in the data](images/72bbf4f2895aa831fd4fc3a86969b5266aaac3c2e856cb3281f74780f0d4b927.svg)
2265
+ ![All MI Conditions for people who are Male OR had an office visit at some point in the dataset](images/72bbf4f2895aa831fd4fc3a86969b5266aaac3c2e856cb3281f74780f0d4b927.svg)
2149
2266
 
2150
2267
  | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2151
2268
  | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
@@ -2164,7 +2281,7 @@ ______________________________________________________________________
2164
2281
 
2165
2282
  ______________________________________________________________________
2166
2283
 
2167
- **Example 62 - Yields two types: all MI Conditions for people who are Male, and all office visit Procedures for people who are Male**
2284
+ **Example 64 - Yields two types: all MI Conditions for people who are Male, and all office visit Procedures for people who are Male**
2168
2285
 
2169
2286
  ```JSON
2170
2287
  ["person_filter",{"left":["union",["icd9","412"],["cpt","99214"]],"right":["gender","Male"]}]
@@ -2187,90 +2304,14 @@ ______________________________________________________________________
2187
2304
 
2188
2305
  ______________________________________________________________________
2189
2306
 
2190
- For example, to find all MIs of males, use `person_filter`. The right-hand stream need not be a person stream, because any record of a patient counts. The left-hand stream can also have several types, as the last example shows.
2191
-
2192
- ## Comparison Operators
2193
-
2194
- These operators compare records of incoming streams to produce a set of records.
2195
-
2196
- ### `Filter` Operator
2197
-
2198
- Keeps each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date.
2199
-
2200
- - **Operands:**
2201
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
2202
- - `right` (required): The right-hand stream. The operator keeps each left-hand record that is also in this stream, and does not pass these records on.
2203
- - **Returns:** The left-hand records that have a right-hand record with the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. The right-hand records are not returned.
2204
-
2205
- ______________________________________________________________________
2206
-
2207
- **Example 63 - MIs that are a patient's first diagnosis of MI or diabetes**
2208
-
2209
- ```JSON
2210
- ["filter",{"left":["icd9","412"],"right":["first",["union",["icd9","412"],["icd9","250.01"]]]}]
2211
- ```
2212
-
2213
- ![MIs that are a patient's first diagnosis of MI or diabetes](images/9764fa797ec9e7cf4eacc04cd15d83453ecb499ff0b5a440e190f8be3f2624c0.svg)
2214
-
2215
- | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2216
- | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
2217
- | 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
2218
- | 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
2219
- | 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
2220
- | 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
2221
- | 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
2222
- | 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
2223
- | 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
2224
- | 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
2225
- | 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
2226
- | 90 | 33940 | observations | condition_occurrence | 2010-05-26 | 2010-05-26 | 412 |
2227
-
2228
- ______________________________________________________________________
2229
-
2230
- It is an intersection of the two streams. Filter compares records, not date ranges. A record whose dates `time_window` changed no longer matches the original record.
2231
-
2232
- ### `Match` Operator
2233
-
2234
- Keeps each left-hand record that is equal to a right-hand record in every column, including label. Two empty (NULL) values count as equal.
2235
-
2236
- - **Operands:**
2237
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
2238
- - `right` (required): The right-hand stream. The operator keeps each left-hand record that is equal to a record of this stream, and does not pass these records on.
2239
- - **Returns:** The left-hand records that have a right-hand record with the same value in every column. The right-hand records are not returned.
2240
-
2241
- ______________________________________________________________________
2242
-
2243
- **Example 64 - MIs that are a patient's first MI**
2244
-
2245
- ```JSON
2246
- ["match",{"left":["icd9","412"],"right":["first",["icd9","412"]]}]
2247
- ```
2248
-
2249
- ![MIs that are a patient's first MI](images/5d5085351acc8c6776b7df9e62c3143a65f00fffb715e7929e65d8aad0509890.svg)
2250
-
2251
- | person_id | criterion_id | criterion_table | criterion_domain | start_date | end_date | source_value |
2252
- | --------- | ------------ | --------------- | -------------------- | ---------- | ---------- | ------------ |
2253
- | 11 | 82614 | observations | condition_occurrence | 2008-10-29 | 2008-10-29 | 412 |
2254
- | 17 | 27269 | observations | condition_occurrence | 2008-08-25 | 2008-08-25 | 412 |
2255
- | 38 | 40556 | observations | condition_occurrence | 2010-02-12 | 2010-02-12 | 412 |
2256
- | 54 | 74606 | observations | condition_occurrence | 2008-06-05 | 2008-06-05 | 412 |
2257
- | 60 | 986 | observations | condition_occurrence | 2009-07-19 | 2009-07-22 | 412 |
2258
- | 66 | 16171 | observations | condition_occurrence | 2009-07-25 | 2009-07-25 | 412 |
2259
- | 73 | 52843 | observations | condition_occurrence | 2008-11-16 | 2008-11-16 | 412 |
2260
- | 77 | 58165 | observations | condition_occurrence | 2010-10-06 | 2010-10-06 | 412 |
2261
- | 81 | 1405 | observations | condition_occurrence | 2009-01-28 | 2009-01-30 | 412 |
2262
- | 88 | 1572 | observations | condition_occurrence | 2009-01-03 | 2009-01-09 | 412 |
2263
-
2264
- ______________________________________________________________________
2265
-
2266
- It is stricter than [`filter`](#filter-operator), so most users want `filter`. Systems that build statements, such as Jigsaw's export engine, use match with options that Jigsaw does not show.
2307
+ For example, to find all MIs of males, use `person_filter`. The right-hand stream need not be a person stream, because any result of a patient counts. The left-hand stream can also have several types, as the last example shows.
2267
2308
 
2268
2309
  ### `Co-Reported` Operator
2269
2310
 
2270
- Passes along the records of its upstreams that were reported in the same context as a record of every other upstream. In the GDM, a context is the claim or other source record that reported the record.
2311
+ Passes along the results of its upstreams that were reported in the same context as a result of every other upstream. In the GDM, a context is the claim or other source row of the dataset that reported the row behind a result.
2271
2312
 
2272
- - **Upstream:** one or more operators.
2273
- - **Returns:** The records of each upstream whose context also has a record of every other upstream. The operator does not compare dates. If one upstream has no record in a context, the records of all upstreams in that context are not passed along. Person and death records have no context, so they never pass.
2313
+ - **Upstream:** two or more operators.
2314
+ - **Returns:** The results of each upstream whose context also has a result of every other upstream. The operator does not compare dates. If one upstream has no result in a context, the results of all upstreams in that context are not passed along. Person and death results have no context, so they never pass.
2274
2315
 
2275
2316
  ______________________________________________________________________
2276
2317
 
@@ -2303,7 +2344,7 @@ Claims data often reports diagnoses on the same claim as a procedure. This is do
2303
2344
 
2304
2345
  Every ConceptQL operator takes a `label` option, so the reference sections of the operators do not list it:
2305
2346
 
2306
- A short description of the operator's records. The operator writes it into the label column of each of its records, and a recall operator selects those records by it. Each label in a statement must be different.
2347
+ A short description of the operator's results. The operator writes it into the label column of each of its results, and a recall operator selects those results by it. Each label in a statement must be different.
2307
2348
 
2308
2349
  An operator without a label passes the upstream label through.
2309
2350
 
@@ -2392,12 +2433,12 @@ A label must be a string.
2392
2433
 
2393
2434
  ### `Recall` Operator
2394
2435
 
2395
- Passes on the records of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of records more than once.
2436
+ Passes on the results of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of results more than once.
2396
2437
 
2397
2438
  - **Upstream:** none.
2398
2439
  - **Arguments:**
2399
- - `name` (string): The label of the operator whose records to pass on. It must match the label exactly, including case. The labeled operator can come before or after the recall in the statement, but labeled streams that recall each other in a loop are invalid.
2400
- - **Returns:** The records of the operator that has the given label.
2440
+ - `name` (string): The label of the operator whose results to pass on. It must match the label exactly, including case. The labeled operator can come before or after the recall in the statement, but labeled streams that recall each other in a loop are invalid.
2441
+ - **Returns:** The results of the operator that has the given label.
2401
2442
 
2402
2443
  ______________________________________________________________________
2403
2444
 
@@ -2424,33 +2465,57 @@ ______________________________________________________________________
2424
2465
 
2425
2466
  ______________________________________________________________________
2426
2467
 
2427
- ## Window Tables
2468
+ <a id="window-tables"></a>
2428
2469
 
2429
- A system can run a statement inside windows. A window is a date range of one patient. Jigsaw's export engine uses windows. The system gives the windows in a window table, and it wraps the statement like this:
2470
+ ## Windows
2471
+
2472
+ A window is a span of time. A calling program can apply windows to a statement, usually one span of dates for each person. ConceptQL then ignores the rows of the dataset outside the windows, as though those rows do not exist.
2473
+
2474
+ An algorithm never contains a window, and no algorithm should assume that a window exists. Windows sit outside the algorithm. So any algorithm can later run inside a window with no change to the algorithm. A calling program, such as Jigsaw's export engine, applies the windows of a study to the algorithms that it runs. Then the results stay inside the spans of time of the study.
2475
+
2476
+ The [`time_window`](#time-window-operator) operator is a different thing. It is part of a statement, and it changes the dates of results or makes spans of dates for each person. Its results are still subject to the windows that the calling program applies.
2477
+
2478
+ The calling program wraps the statement like this:
2430
2479
 
2431
2480
  ```JSON
2432
- [ "window", <statement>, { "window_table": <window table> } ]
2481
+ [ "window", <statement>, { <window options> } ]
2433
2482
  ```
2434
2483
 
2435
- - The window table has the columns `person_id`, `start_date` and `end_date`. It is the name of a database table or a ConceptQL statement.
2436
- - Each selection operator keeps only the records whose patient has a window and whose start_date is in that window. Both dates of the window are in the window. The person operators, such as `person` and `gender`, keep each patient who has a window, whatever the dates.
2437
- - Each record gets the column `window_id`, which identifies its window. A record that is in two windows of its patient comes out once for each window.
2438
- - Operators that work on each patient, such as `first`, `episode` and the temporal operators, work on each patient and window. For example, `first` keeps the first record of each window.
2484
+ These are the window options:
2485
+
2486
+ - `window_table`: a table of windows, with the columns `person_id`, `start_date` and `end_date`. Each row is one window of one person. The value is the name of a database table or a ConceptQL statement.
2487
+ - `start_date` and `end_date`: one date range that is the window of every person. Give both options or neither. One of them without the other makes the statement invalid, and the error names the missing option.
2488
+ - `scope_by`: which dates of a result must be inside the window. With `start_date`, the default, the start_date of the result must be in the window. With `end_date`, the end_date of the result must be in the window. With `date_range`, both dates of the result must be in the window. Any other value makes the statement invalid, and the error names the value.
2489
+ - `adjust_window_start` and `adjust_window_end`: date adjustments, as [`time_window`](#time-window-operator) takes them. They move the start and the end of each window of the window table. They do not change a date range.
2490
+ - `precalculated_window_id`: the window table has its own `window_id` column, and ConceptQL uses it. Without this option, ConceptQL numbers the windows itself.
2491
+ - `person_ids`: a list of person IDs. ConceptQL keeps only the results of these persons.
2492
+
2493
+ Jigsaw's export engine uses `window_table` and `scope_by`. A calling program can give several options, such as a window table and a list of person IDs. Then a result must pass each of them.
2494
+
2495
+ If a window is inside another window, only the options of the inner window apply, and ConceptQL ignores the options of the outer window. A diagram shows the options of the inner window, and only those options can make the statement invalid.
2496
+
2497
+ The windows change a statement in these ways:
2498
+
2499
+ - Each selection operator keeps only the results that are inside a window, as `scope_by` says. Both dates of a window are in the window.
2500
+ - The person operators, such as `person` and `gender`, and the operators `date_range` and `information_periods` ignore the dates of a window. With a window table, they keep each person who has a window. A date range does not change their results.
2501
+ - With a window table, each result gets the column `window_id`, which identifies its window. A result that is in two windows of its person comes out once for each window. Without `precalculated_window_id`, two rows of the window table with the same person and the same dates are one window. A date range adds no `window_id`.
2502
+ - With a window table, operators that work on each person, such as `first`, `episode` and the temporal operators, work on each person and window. For example, `first` keeps the first result of each window.
2439
2503
  - If the window table is a ConceptQL statement with errors, the statement that it wraps is invalid. A diagram reports the errors under `window_table`, and counts nothing.
2504
+ - If the window options have an error, such as a lone `start_date` or an unknown `scope_by`, the statement is invalid. A diagram reports the error on the `window` operator, and counts nothing. This is also true when the calling program gives the window options outside the statement.
2440
2505
 
2441
2506
  ## Appendix A - Experimental Operators
2442
2507
 
2443
- These operators work, but they may change or be removed. They compare the `value_as_number` column, which lab records (such as `loinc`) carry.
2508
+ These operators work, but they may change or be removed. They compare the `value_as_number` column, which the results of lab codes (such as `loinc`) carry.
2444
2509
 
2445
2510
  ### `Numeric Filter` Operator
2446
2511
 
2447
- Passes along the records whose value_as_number is in the given range. Lab records, such as those of loinc, carry this value.
2512
+ Passes along the results whose value_as_number is in the given range. The results of lab tests, such as those of loinc, carry this value.
2448
2513
 
2449
2514
  - **Upstream:** one operator.
2450
2515
  - **Options:**
2451
2516
  - `greater_than_or_equal_to` (float): The lowest value_as_number that passes. Without this option, the range has no lower limit.
2452
2517
  - `less_than_or_equal_to` (float): The highest value_as_number that passes. Without this option, the range has no upper limit.
2453
- - **Returns:** The upstream records whose value_as_number is in the range. A record that has no value_as_number is not passed along.
2518
+ - **Returns:** The upstream results whose value_as_number is in the range. A result that has no value_as_number is not passed along.
2454
2519
 
2455
2520
  ______________________________________________________________________
2456
2521
 
@@ -2472,12 +2537,12 @@ It is an [inline filter operator](#inline-filter-operators).
2472
2537
 
2473
2538
  ### `Equal` Operator
2474
2539
 
2475
- For each patient, keeps the left-hand records whose value_as_number equals the value_as_number of a right-hand record. It does not compare dates.
2540
+ For each patient, keeps the left-hand results whose value_as_number equals the value_as_number of a right-hand result. It does not compare dates.
2476
2541
 
2477
2542
  - **Operands:**
2478
- - `left` (required): The left-hand stream. The operator compares these records with the right-hand stream and passes on the ones that meet its condition.
2479
- - `right` (required): The right-hand stream. The operator compares the left-hand records with these records and does not pass them on.
2480
- - **Returns:** The left-hand records that have a right-hand record of the same patient with the same value_as_number, each record once. Left-hand records without a value_as_number (NULL) are not returned. The right-hand records are not returned.
2543
+ - `left` (required): The left-hand stream. The operator compares these results with the right-hand stream and passes on the ones that meet its condition.
2544
+ - `right` (required): The right-hand stream. The operator compares the left-hand results with these results and does not pass them on.
2545
+ - **Returns:** The left-hand results that have a right-hand result of the same patient with the same value_as_number, each result once. Left-hand results without a value_as_number (NULL) are not returned. The right-hand results are not returned.
2481
2546
 
2482
2547
  ______________________________________________________________________
2483
2548
 
@@ -2564,21 +2629,21 @@ This document is part of the [ConceptQL](https://github.com/outcomesinsights/con
2564
2629
 
2565
2630
  The reference section of each operator, the text of the `label` option, the table of all vocabulary operators and Appendix E come from the metadata of the operators (`ConceptQL.metadata`), so the texts in the operator classes are their only copy. A directive in `README.cql.md`, an HTML comment on a line of its own, marks the place of each generated block:
2566
2631
 
2567
- - `<!-- conceptql-spec: operator <spec_id> -->` writes the reference section of the operators with that `spec_id`. It comes directly after the heading of the section, whose anchor is the `spec_id`. The section has the operator's description, its upstream operators or its operands (`left` and `right`), its arguments, its options, the records it returns, and its examples. Text about more than one operator, and longer explanations of one operator, follow the directive as markdown.
2632
+ - `<!-- conceptql-spec: operator <spec_id> -->` writes the reference section of the operators with that `spec_id`. It comes directly after the heading of the section, whose anchor is the `spec_id`. The section has the operator's description, its upstream operators or its operands (`left` and `right`), its arguments, its options, the results it returns, and its examples. Text about more than one operator, and longer explanations of one operator, follow the directive as markdown.
2568
2633
  - `<!-- conceptql-spec: label-option -->` writes the text of the `label` option, which every operator takes.
2569
2634
  - `<!-- conceptql-spec: quick-reference -->` writes the table of Appendix E, with one row for each `spec_id`.
2570
2635
  - `<!-- conceptql-spec: vocabulary-operators -->` writes the table of [All Vocabulary Operators](#all-vocabulary-operators), with one row for each vocabulary operator: its name, the full name of its vocabulary and its domain. It marks a deprecated operator.
2571
2636
 
2572
2637
  The metadata comes from a ConceptQL database with the data model `gdm`, no data database and no custom vocabularies, so it decides which operators get a section. Each `spec_id` of that metadata needs exactly one directive, and the build stops when one is missing, repeated or not after its heading. The other directives appear exactly once each, and the build also stops on an unknown directive.
2573
2638
 
2574
- `just spec` runs `docs/spec/build/run.rb` with the working tree's ConceptQL. For each `ConceptQL` block of `README.cql.md` and each example of a reference section, the build writes the example: the statement as JSON, its diagram, and the first 10 records of its result. It orders the records by `person_id`, `criterion_domain`, start_date, `criterion_table` and `criterion_id`, so each run shows the same records. An example that is not a valid statement stops the build. Then `just spec` formats `README.md` with treefmt. If `README.md` or an image changed, `just spec` lists the changed files and fails, so that you review and commit them. A second run then changes nothing and passes.
2639
+ `just spec` runs `docs/spec/build/run.rb` with the working tree's ConceptQL. For each `ConceptQL` block of `README.cql.md` and each example of a reference section, the build writes the example: the statement as JSON, its diagram, and its first 10 results. It orders the results by `person_id`, `criterion_domain`, start_date, `criterion_table` and `criterion_id`, so each run shows the same results. An example that is not a valid statement stops the build. Then `just spec` formats `README.md` with treefmt. If `README.md` or an image changed, `just spec` lists the changed files and fails, so that you review and commit them. A second run then changes nothing and passes.
2575
2640
 
2576
2641
  The results and the counts come from ConceptQL's DuckDB test data, `synpuf_test_data.duckdb`, with the data model `gdm_wide` and the OHDSI vocabulary tables. `just spec` runs the build through `just test-duckdb`, so it needs what the DuckDB tests need:
2577
2642
 
2578
2643
  - Linux on x86-64. `just test-duckdb` downloads the pinned version of the DuckDB library on the first run.
2579
2644
  - The DuckDB test data. `just test-duckdb` copies it from the path in `DUCKDB_TEST_DATA_SOURCE`, and it fails unless the copy matches the checksum in `.github/synpuf_test_data.sha256`.
2580
2645
 
2581
- The diagrams are drawn with ConceptQL's diagram renderer, `ConceptQL::DiagramRenderer`, which runs the resvg program. The build needs resvg 0.33.0 exactly, which `mise.toml` pins and `just setup` installs. It fails when resvg is missing or is another version. The renderer labels each line with the records (`rows=`) and the people (`n=`) that flow along it, and it lists the options of each operator. The build draws every diagram on each run, and writes each one as `images/<name>.svg`, where the name is the SHA-256 of the statement as canonical JSON. It removes the SVG images that no example uses, and it writes `images/manifest.tsv`, which gives the SHA-256 of the diagram data (with its counts) for each image.
2646
+ The diagrams are drawn with ConceptQL's diagram renderer, `ConceptQL::DiagramRenderer`, which runs the resvg program. The build needs resvg 0.33.0 exactly, which `mise.toml` pins and `just setup` installs. It fails when resvg is missing or is another version. The renderer labels each line with the results (`rows=`) and the people (`n=`) that flow along it, and it lists the options of each operator. The build draws every diagram on each run, and writes each one as `images/<name>.svg`, where the name is the SHA-256 of the statement as canonical JSON. It removes the SVG images that no example uses, and it writes `images/manifest.tsv`, which gives the SHA-256 of the diagram data (with its counts) for each image.
2582
2647
 
2583
2648
  Between releases, the results and the images on the main branch can be out of date. Before a release, `just release-check` runs `just spec-check`. It rebuilds the document and the diagram data, and it draws every image into a temporary directory. It fails when `README.md` does not match the code, or when an image differs from its new drawing, is missing, or is not used. It names each such image.
2584
2649
 
@@ -2588,39 +2653,39 @@ The build writes this table from the metadata of the operators. It has one row f
2588
2653
 
2589
2654
  | Operator | Jigsaw category | What it does |
2590
2655
  | -------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
2591
- | [`Vocabulary` Operator](#vocabulary-operator) | Select by Clinical Codes | Selects the records whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once. |
2656
+ | [`Vocabulary` Operator](#vocabulary-operator) | Select by Clinical Codes | Selects the results whose code is one of the given codes in a vocabulary, such as ICD-9-CM diagnoses or CPT procedures. Each vocabulary has its own operator, and a few operators search several vocabularies at once. |
2592
2657
  | [`gender`](#gender-operator) | Select by Property | Selects the patients with the given genders. |
2593
2658
  | [`race`](#race-operator) | Select by Property | Selects the patients with the given races. |
2594
2659
  | [`ethnicity`](#ethnicity-operator) | Select by Property | Selects the patients with the given ethnicities. |
2595
- | [`snf`](#snf-operator) | Select by Property | Selects admissions to skilled nursing facilities (SNF): one record for each admission whose claim came from an SNF file. |
2596
- | [`hospice`](#hospice-operator) | Select by Property | Selects hospice admissions: one record for each admission whose claim came from a hospice file. |
2597
- | [`hospitalization`](#hospitalization-operator) | Select by Property | Selects inpatient hospital admissions: one record for each admission whose claim came from an inpatient file. |
2598
- | [`date_range`](#date-range-operator) | Select by Property | Makes one person record for each patient, with the given start_date and end_date. Use it to compare other records with a fixed range of dates. |
2599
- | [`person`](#person-operator) | Get Related Data | Selects every patient. With one upstream operator, it selects the patients who have a record in the upstream stream. |
2600
- | [`death`](#death-operator) | Get Related Data | Selects every death record. With one upstream operator, it selects the deaths of the patients who have a record in the upstream stream. |
2601
- | [`information_periods`](#information-periods-operator) | Get Related Data | Selects information periods: the spans of time in which the data has information about a patient, such as periods of insurance enrollment. |
2602
- | [`recall`](#recall-operator) | Get Related Data | Passes on the records of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of records more than once. |
2603
- | [`time_window`](#time-window-operator) | Modify Data | Moves the start_date and the end_date of each record. Use it to shift or widen a record's dates before you compare them with other records. |
2604
- | [`episode`](#episode-operator) | Modify Data | Merges each patient's records into episodes: spans of time in which the records are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug. |
2605
- | [`union`](#union-operator) | Combine Streams | Combines the records of all its upstream operators into one stream. Each record keeps its own type, so one stream can hold several types. |
2606
- | [`co_reported`](#co-reported-operator) | Combine Streams | Passes along the records of its upstreams that were reported in the same context as a record of every other upstream. In the GDM, a context is the claim or other source record that reported the record. |
2607
- | [`except`](#except-operator) | Filter by Comparing | Removes each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same criterion_table, criterion_id, start_date and end_date. |
2608
- | [`during`](#during-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose date range is inside the date range of a right-hand record. The left-hand record starts on or after the start_date of the right-hand record and ends on or before its end_date. |
2609
- | [`contains`](#contains-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose date range encloses the date range of a right-hand record. The left-hand record starts on or before the start_date of the right-hand record and ends on or after its end_date. |
2610
- | [`any_overlap`](#any-overlap-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose date range shares at least one day with the date range of a right-hand record. |
2611
- | [`before`](#before-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose end_date is before the latest start_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead. |
2612
- | [`after`](#after-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose start_date is after the earliest end_date of the patient's right-hand records. With the within option, each left-hand record is compared with every right-hand record of the patient instead. |
2613
- | [`trim_date_start`](#trim-date-start-operator) | Filter by Comparing | For each patient, moves the start_date of each left-hand record forward to the end_date of the right-hand record with the latest start_date. A left-hand record that ends before that end_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged. |
2614
- | [`trim_date_end`](#trim-date-end-operator) | Filter by Comparing | For each patient, moves the end_date of each left-hand record back to the start_date of the right-hand record with the earliest start_date. A left-hand record that starts after that start_date is removed, and the left-hand records of a patient without right-hand records pass through unchanged. |
2615
- | [`person_filter`](#person-filter-operator) | Filter by Comparing | Keeps each left-hand record whose patient has at least one record in the right-hand stream. The type and the dates of the right-hand records do not matter. |
2616
- | [`filter`](#filter-operator) | Filter by Comparing | Keeps each left-hand record that also appears in the right-hand stream. Two records are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. |
2617
- | [`match`](#match-operator) | Filter by Comparing | Keeps each left-hand record that is equal to a right-hand record in every column, including label. Two empty (NULL) values count as equal. |
2618
- | [`equal`](#equal-operator) | Filter by Comparing | For each patient, keeps the left-hand records whose value_as_number equals the value_as_number of a right-hand record. It does not compare dates. |
2619
- | [`occurrence`](#nth-occurrence-operator) | Filter Single Stream | For each person, orders the records by start_date and passes along the record at the given position. A positive position counts from the earliest record and a negative one from the most recent record. |
2620
- | [`first`](#first-operator) | Filter Single Stream | For each person, passes along the record with the earliest start_date. |
2621
- | [`last`](#last-operator) | Filter Single Stream | For each person, passes along the record with the most recent start_date. |
2622
- | [`place_of_service_filter`](#place-of-service-filter-operator) | Filter Single Stream | Passes along the records whose place of service is one of the given CMS Place of Service codes. A record that has no place of service is not passed along. |
2623
- | [`provenance`](#provenance-operator) | Filter Single Stream | Passes along the records whose provenance matches at least one of the given keywords. The provenance of a record says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary. |
2624
- | [`provider_filter`](#provider-filter-operator) | Filter Single Stream | Passes along the records whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value. |
2625
- | [`one_in_two_out`](#one-in-two-out-operator) | Filter Single Stream | For each person, passes along the earliest confirmed condition record. One inpatient record confirms a condition, and so do two outpatient records whose start dates are a given gap apart. |
2626
- | [`numeric_filter`](#numeric-filter-operator) | Filter Single Stream | Passes along the records whose value_as_number is in the given range. Lab records, such as those of loinc, carry this value. |
2660
+ | [`snf`](#snf-operator) | Select by Property | Selects admissions to skilled nursing facilities (SNF): one result for each admission whose claim came from an SNF file. |
2661
+ | [`hospice`](#hospice-operator) | Select by Property | Selects hospice admissions: one result for each admission whose claim came from a hospice file. |
2662
+ | [`hospitalization`](#hospitalization-operator) | Select by Property | Selects inpatient hospital admissions: one result for each admission whose claim came from an inpatient file. |
2663
+ | [`date_range`](#date-range-operator) | Select by Property | Makes one person result for each patient, with the given start_date and end_date. Use it to compare other results with a fixed range of dates. |
2664
+ | [`person`](#person-operator) | Get Related Data | Selects every patient. With one upstream operator, it selects the patients who have a result in the upstream stream. |
2665
+ | [`death`](#death-operator) | Get Related Data | Selects every death. With one upstream operator, it selects the deaths of the patients who have a result in the upstream stream. |
2666
+ | [`information_periods`](#information-periods-operator) | Get Related Data | Selects information periods: the spans of time in which the dataset has information about a patient, such as periods of insurance enrollment. |
2667
+ | [`recall`](#recall-operator) | Get Related Data | Passes on the results of the operator in the statement that has the given label. Use it to break a complex statement into parts, or to use a stream of results more than once. |
2668
+ | [`time_window`](#time-window-operator) | Modify Data | Moves the start_date and the end_date of each result. Use it to shift or widen a result's dates before you compare them with other results. |
2669
+ | [`episode`](#episode-operator) | Modify Data | Merges each patient's results into episodes: spans of time in which the results are close together. For example, it can turn a set of prescriptions into the spans of time in which a patient takes a drug. |
2670
+ | [`union`](#union-operator) | Combine Streams | Combines the results of all its upstream operators into one stream. Each result keeps its own type, so one stream can hold several types. |
2671
+ | [`co_reported`](#co-reported-operator) | Combine Streams | Passes along the results of its upstreams that were reported in the same context as a result of every other upstream. In the GDM, a context is the claim or other source row of the dataset that reported the row behind a result. |
2672
+ | [`except`](#except-operator) | Filter by Comparing | Removes each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same criterion_table, criterion_id, start_date and end_date. |
2673
+ | [`during`](#during-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose date range is inside the date range of a right-hand result. The left-hand result starts on or after the start_date of the right-hand result and ends on or before its end_date. |
2674
+ | [`contains`](#contains-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose date range encloses the date range of a right-hand result. The left-hand result starts on or before the start_date of the right-hand result and ends on or after its end_date. |
2675
+ | [`any_overlap`](#any-overlap-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose date range shares at least one day with the date range of a right-hand result. |
2676
+ | [`before`](#before-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose end_date is before the latest start_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead. |
2677
+ | [`after`](#after-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose start_date is after the earliest end_date of the patient's right-hand results. With the within option, each left-hand result is compared with every right-hand result of the patient instead. |
2678
+ | [`trim_date_start`](#trim-date-start-operator) | Filter by Comparing | For each patient, moves the start_date of each left-hand result forward to the end_date of the right-hand result with the latest start_date. A left-hand result that ends before that end_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged. |
2679
+ | [`trim_date_end`](#trim-date-end-operator) | Filter by Comparing | For each patient, moves the end_date of each left-hand result back to the start_date of the right-hand result with the earliest start_date. A left-hand result that starts after that start_date is removed, and the left-hand results of a patient without right-hand results pass through unchanged. |
2680
+ | [`filter`](#filter-operator) | Filter by Comparing | Keeps each left-hand result that also appears in the right-hand stream. Two results are the same when they have the same person_id, criterion_id, criterion_table, criterion_domain, start_date and end_date. |
2681
+ | [`match`](#match-operator) | Filter by Comparing | Keeps each left-hand result that is equal to a right-hand result in every column, including label. Two empty (NULL) values count as equal. |
2682
+ | [`person_filter`](#person-filter-operator) | Filter by Comparing | Keeps each left-hand result whose patient has at least one result in the right-hand stream. The type and the dates of the right-hand results do not matter. |
2683
+ | [`equal`](#equal-operator) | Filter by Comparing | For each patient, keeps the left-hand results whose value_as_number equals the value_as_number of a right-hand result. It does not compare dates. |
2684
+ | [`occurrence`](#nth-occurrence-operator) | Filter Single Stream | For each person, orders the results by start_date and passes along the result at the given position. A positive position counts from the earliest result and a negative one from the most recent result. |
2685
+ | [`first`](#first-operator) | Filter Single Stream | For each person, passes along the result with the earliest start_date. |
2686
+ | [`last`](#last-operator) | Filter Single Stream | For each person, passes along the result with the most recent start_date. |
2687
+ | [`place_of_service_filter`](#place-of-service-filter-operator) | Filter Single Stream | Passes along the results whose place of service is one of the given CMS Place of Service codes. A result that has no place of service is not passed along. |
2688
+ | [`provenance`](#provenance-operator) | Filter Single Stream | Passes along the results whose provenance matches at least one of the given keywords. The provenance of a result says which kind of file it came from, such as inpatient, and where its code was on the claim, such as primary. |
2689
+ | [`provider_filter`](#provider-filter-operator) | Filter Single Stream | Passes along the results whose provider matches the given specialties and roles. Each option is a comma-separated list of concept IDs, and * matches any value. |
2690
+ | [`one_in_two_out`](#one-in-two-out-operator) | Filter Single Stream | For each person, passes along the earliest confirmed condition result. One inpatient result confirms a condition, and so do two outpatient results whose start dates are a given gap apart. |
2691
+ | [`numeric_filter`](#numeric-filter-operator) | Filter Single Stream | Passes along the results whose value_as_number is in the given range. The results of lab tests, such as those of loinc, carry this value. |