conceptql 0.8.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +141 -0
  3. data/docs/spec/README.md +402 -337
  4. data/docs/spec/examples/date-range-operator.json +1 -1
  5. data/docs/spec/examples/death-operator.json +1 -1
  6. data/docs/spec/examples/nth-occurrence-operator.json +1 -1
  7. data/docs/spec/examples/one-in-two-out-operator.json +1 -1
  8. data/docs/spec/examples/person-filter-operator.json +2 -2
  9. data/docs/spec/examples/vocabulary-operator.json +3 -3
  10. data/lib/conceptql/behaviors/metadatable.rb +76 -62
  11. data/lib/conceptql/behaviors/utilizable.rb +1 -1
  12. data/lib/conceptql/database.rb +16 -2
  13. data/lib/conceptql/diagram.rb +2 -2
  14. data/lib/conceptql/lexicon/lexicon_no_db.rb +4 -4
  15. data/lib/conceptql/lexicon.rb +7 -4
  16. data/lib/conceptql/operators/after.rb +5 -4
  17. data/lib/conceptql/operators/any_overlap.rb +4 -3
  18. data/lib/conceptql/operators/before.rb +5 -4
  19. data/lib/conceptql/operators/binary_operator_operator.rb +11 -13
  20. data/lib/conceptql/operators/casting_operator.rb +1 -1
  21. data/lib/conceptql/operators/co_reported.rb +9 -7
  22. data/lib/conceptql/operators/contains.rb +4 -3
  23. data/lib/conceptql/operators/count.rb +3 -3
  24. data/lib/conceptql/operators/date_range.rb +9 -8
  25. data/lib/conceptql/operators/death.rb +4 -4
  26. data/lib/conceptql/operators/dedup.rb +2 -2
  27. data/lib/conceptql/operators/during.rb +4 -3
  28. data/lib/conceptql/operators/episode.rb +9 -9
  29. data/lib/conceptql/operators/equal.rb +3 -2
  30. data/lib/conceptql/operators/ethnicity.rb +7 -3
  31. data/lib/conceptql/operators/except.rb +4 -3
  32. data/lib/conceptql/operators/filter.rb +4 -3
  33. data/lib/conceptql/operators/first.rb +4 -3
  34. data/lib/conceptql/operators/from.rb +2 -1
  35. data/lib/conceptql/operators/gender.rb +4 -3
  36. data/lib/conceptql/operators/hospice.rb +4 -3
  37. data/lib/conceptql/operators/hospitalization.rb +4 -3
  38. data/lib/conceptql/operators/information_periods.rb +5 -4
  39. data/lib/conceptql/operators/last.rb +4 -3
  40. data/lib/conceptql/operators/match.rb +4 -3
  41. data/lib/conceptql/operators/multiple_vocabularies.rb +4 -3
  42. data/lib/conceptql/operators/numeric.rb +4 -4
  43. data/lib/conceptql/operators/numeric_filter.rb +6 -6
  44. data/lib/conceptql/operators/occurrence.rb +25 -18
  45. data/lib/conceptql/operators/one_in_two_out.rb +16 -16
  46. data/lib/conceptql/operators/operator.rb +16 -3
  47. data/lib/conceptql/operators/pass_thru.rb +1 -1
  48. data/lib/conceptql/operators/person.rb +3 -3
  49. data/lib/conceptql/operators/person_filter.rb +6 -5
  50. data/lib/conceptql/operators/place_of_service_filter.rb +7 -7
  51. data/lib/conceptql/operators/provenance.rb +9 -9
  52. data/lib/conceptql/operators/provider_filter.rb +6 -6
  53. data/lib/conceptql/operators/race.rb +4 -3
  54. data/lib/conceptql/operators/recall.rb +6 -5
  55. data/lib/conceptql/operators/snf.rb +4 -3
  56. data/lib/conceptql/operators/sum.rb +6 -5
  57. data/lib/conceptql/operators/time_window.rb +6 -6
  58. data/lib/conceptql/operators/trim_date_end.rb +6 -5
  59. data/lib/conceptql/operators/trim_date_start.rb +6 -5
  60. data/lib/conceptql/operators/union.rb +8 -6
  61. data/lib/conceptql/operators/vocabulary.rb +12 -8
  62. data/lib/conceptql/query.rb +41 -12
  63. data/lib/conceptql/scope.rb +10 -2
  64. data/lib/conceptql/version.rb +1 -1
  65. data/lib/conceptql/visitors/metadata_extractor.rb +3 -1
  66. data/lib/conceptql/vocabularies/drg_systems.rb +16 -14
  67. data/lib/conceptql/vocabularies/read_codes.rb +1 -1
  68. data/lib/conceptql/window.rb +23 -2
  69. data/schemas/gdm.yml +0 -66
  70. data/schemas/gdm_wide.yml +0 -66
  71. metadata +1 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 296389d035ad781649b32662a3fafba66aa209dea6c76fe039b372463854293f
4
- data.tar.gz: 9e7225d0b68c8dd72a7b6b6ade8f3ffdb67b3b1c8cd6507e53e8d84a1fa0a6bc
3
+ metadata.gz: 01bc0bcc6e8ae91743cf821c431cabfe6087fa71b592e243e8185117e1674e13
4
+ data.tar.gz: 1b418ae79bda1cb8a4ce353b4f4de9b57019a22db665dabe05b0a2900135d377
5
5
  SHA512:
6
- metadata.gz: fb10f21a65e9d4f3dfebd0fa559d1e8c5465d8e69ecf281d62494c981d51c5c70054e8e1b04782d0a15c1ebda762aef1b36e477d1a667cce266de5d8ce053a06
7
- data.tar.gz: ceb2fd1d113b3fc37af25bb613e1405ad250a7d6452c2e710020df2fc9341c4398524add0f1b0344834e74348b6f78145a91b43e73f6347edeb6c01f159e3267
6
+ metadata.gz: 6059b619e417fd0e8466f4154ecb16862006c4a7e02c2e67ca07da2407b424630c3c5d497301e6402dd54ea5e925bd79d97cb88f9bd56c6339dd75bd16371118
7
+ data.tar.gz: 8ba160a3456ba4725460ec5f3777c2f7f31beab01e1012a7cf408cd586834aa6e2b034519abca22a861a3e43065dd0ef145582739fe3ba173047df8a2487855d
data/CHANGELOG.md CHANGED
@@ -4,6 +4,147 @@ All notable changes to this project will be documented in this file.
4
4
 
5
5
  ## [Unreleased]
6
6
 
7
+ ## [0.9.0] - 2026-11-13
8
+
9
+ ### Added
10
+
11
+ - Each operator in `ConceptQL.metadata` has seven facets as top-level keys: `upstreams`,
12
+ `reads_data`, `timeless`, `takes_codes`, `uses_provenance`, `utilization` and
13
+ `temporal_comparison`. Their values come from the code (the upstream validations, the
14
+ behavior modules and the superclasses), so no operator declares them.
15
+ `ConceptQL::Metadatable::FACET_KEYS` lists the keys. `docs/metadata.md` says what each one
16
+ means.
17
+ - Each operator in `ConceptQL.metadata` has the top-level key `effects`: a list of strings
18
+ that says what the operator does to results. The values are `select`, `keep`, `change`,
19
+ `combine`, `aggregate`, `cast` and `refer`, and `docs/metadata.md` says what each one
20
+ means. Each operator class declares its effects with `effects :keep, :change`, and a
21
+ subclass inherits them, so every vocabulary operator is `select`. An unknown value raises
22
+ `ConceptQL::Metadatable::UnknownEffectError` when the class loads. `effects` is the last
23
+ key of `ConceptQL::Metadatable::FACET_KEYS`.
24
+ - Each operator in `ConceptQL.metadata` has the key `diagram_base`: the symbol that a diagram
25
+ draws for it (`:selection`, `:filter`, `:set`, `:temporal` or `:cast`). It is not a facet.
26
+ - The spec build (`just spec`, `just spec-check`) checks the class terms of the spec against
27
+ the facets. An operator under a `##` heading of a class (for example "Temporal Comparison
28
+ Operators"), or in a section that says "It is a [term](...)", must meet the rule of that
29
+ class, or the build fails and names the operator, the term and its facets. "Set operator"
30
+ and "relative temporal operator" have no rule. `person_filter` moves from Inline-Filter
31
+ Operators to Comparison Operators in the spec, because it takes left and right streams.
32
+
33
+ ### Removed
34
+
35
+ - **Breaking:** the operator metadata keys `basic_type`, `arity`, `min_upstreams` and
36
+ `max_upstreams`. The `upstreams` facet replaces `min_upstreams` and `max_upstreams`, and
37
+ says which upstreams the validations accept. `diagram_base` replaces `basic_type`, with
38
+ the same values, and `arity`, which was a number made from `basic_type`. The class methods
39
+ `arity`, `allows_one_upstream` and `allows_many_upstreams` are gone too. A reader of
40
+ `ConceptQL.metadata` must read the new keys.
41
+ - The GDM vocabulary tables `concepts`, `vocabularies` and `mappings` are no longer in the
42
+ shipped table descriptions `schemas/gdm.yml` and `schemas/gdm_wide.yml`. No lexicon reads
43
+ them since 0.8.0. The SQL and the annotations that a Sequel mock database builds do not
44
+ change.
45
+
46
+ ### Changed
47
+
48
+ - **Breaking:** an operator class declares its diagram symbol with `diagram_base :set`. The
49
+ class method `basic_type` is gone.
50
+ - `ConceptQL::Diagram` (the render tree and the Jigsaw JSON), `Query#extract_metadata` and
51
+ the spec build read `diagram_base` and the `upstreams` facet, not the old keys. Every
52
+ diagram and every value of `codeset_only` is the same as before.
53
+ - The spec (`docs/spec/README.md`) explains the parts of a statement (operator, arguments,
54
+ options, upstream and downstream), file and code provenance, inpatient and outpatient
55
+ settings, and information periods. The section "Window Tables" is now "Windows". It says
56
+ why windows sit outside an algorithm, how they differ from `time_window`, and every
57
+ window option: `window_table`, `start_date` and `end_date`, `scope_by`,
58
+ `adjust_window_start` and `adjust_window_end`, `precalculated_window_id` and `person_ids`.
59
+ Before, it said only that a result must start inside its window, which is true only for
60
+ the default `scope_by`. The old anchor `#window-tables` still works.
61
+ - The spec and the operator texts in `ConceptQL.metadata` (`desc`, `returns`, the meanings
62
+ of the arguments and the options, and the example titles) use the terms of the glossary.
63
+ A ConceptQL row is a "result", a row of a dataset table is a "row", and a "stream" is the
64
+ set of results that passes from one operator to the next. Before, the texts said "record"
65
+ for both. The heading "Think of Records as a Stream" is now "Think of Results as a
66
+ Stream", and "What *are* Streams Really?" says that ConceptQL is not a stream-processing
67
+ system. The standard description of a vocabulary operator is now "Selects results based
68
+ on the ... vocabulary.". No warning or error message changed: the warning "the data has
69
+ no records of this vocabulary" keeps its text, because Jigsaw's tests match it.
70
+ - The deprecation message of `count`, which a calling program can show, now says "To count
71
+ the results of each person". Before, it said "To count the events of each person". The
72
+ rest of the message is the same.
73
+ - The long names of `ms_drg` and `cms_drg` (`long_name` in `ConceptQL.metadata` and the
74
+ vocabulary table of the spec) say "discharges from 2007-10-01" and "discharges before
75
+ 2007-10-01", because a DRG applies to a discharge. Before, they said "events".
76
+ - The deprecation warning of `drg` (`ConceptQL::Vocabularies::DrgSystems::DRG_DEPRECATION`)
77
+ says "drg selects the results of both" and "discharges from 2007-10-01" or "discharges
78
+ before 2007-10-01", the same as the long names. Before, it said "events" in all three
79
+ places. The rest of the message is the same.
80
+
81
+ ### Fixed
82
+
83
+ - The reference sections of `union` and `co_reported` in `docs/spec/README.md` say that
84
+ they take two or more upstream operators. Before, they said one or more.
85
+ - `Query#events_per_patient` is `:multiple` for `occurrence` with `at_least` or `within`,
86
+ because the operator can then pass along several records of one person. Before, it was
87
+ always `:single`, so a Jigsaw export of such an algorithm said that each patient had at
88
+ most one event. `occurrence` without these options, `first` and `last` stay `:single`.
89
+ - `Query#events_per_patient` is `:multiple` for `union` and `co_reported` with two or more
90
+ upstreams, because each upstream passes along its own records. Before, it was `:single`
91
+ when every upstream was `:single`: on the DuckDB fixture, a `union` of two `first`
92
+ operators gave 36 persons more than one row. With one upstream, they still report the
93
+ value of that upstream.
94
+ - `ethnicity` applies the windows of a window table, as `gender` and `race` do. It keeps
95
+ each patient who has a window, whatever the dates, and each record gets its `window_id`.
96
+ Before, it ignored the windows: it kept the patients with no window, and their records
97
+ had no `window_id`.
98
+ - `Query#extract_metadata` works on a binary operator (`after`, `before`, `during`, ...)
99
+ whose `left` or `right` is missing or is not a statement. It lists only the operands
100
+ that exist. Before, it raised `NoMethodError`. The statement is still invalid, with the
101
+ same errors.
102
+ - `Query#events_per_patient` is `:multiple` and `Query#multiple_vocabularies` is `false` for
103
+ a binary operator whose `left` is missing or is not a statement. These are the values of
104
+ an operator with no upstream. Before, both raised `NoMethodError`. The statement is still
105
+ invalid, with the same errors.
106
+ - A window with only one of `start_date` and `end_date` makes the statement invalid. The
107
+ error is on the `window` operator ("date range needs start_date and end_date"), and its
108
+ detail names the missing option. Before, ConceptQL ignored the lone date with no warning,
109
+ so the statement returned results from all dates.
110
+ - A window with a `scope_by` other than `start_date`, `end_date` or `date_range` makes the
111
+ statement invalid. The error is on the `window` operator ("scope_by must be start_date,
112
+ end_date or date_range"), and its detail is the value. Before, building the SQL raised a
113
+ plain `RuntimeError`, so `Query#errors` and the diagram did not show it. A `scope_by` of
114
+ `nil` is the default, `start_date`. Before, it raised `NoMethodError`.
115
+ - Both checks read the window statement and the `scope_opts` of a calling program. When the
116
+ window options have an error, `annotate` puts the statement inside a `window` operator
117
+ that holds the error, and it counts nothing, as for an invalid window table statement.
118
+ `Query#sql` raises `ConceptQL::InvalidStatementError` with the same errors.
119
+ - Window options with String keys in the `scope_opts` of `Database#query` and
120
+ `Database.new` (and in their `window_opts`) take effect, as Symbol keys do. A calling
121
+ program that builds its options from JSON has String keys. Before, ConceptQL ignored
122
+ them with no error, so the statement ran over all dates and persons. Both window checks
123
+ above also read String keys.
124
+ - For a window inside a window, the diagram (`Query#annotate`) shows the options of the
125
+ inner window, which are the options that the SQL uses. Before, it showed the options of
126
+ both windows merged, so an option of the outer window (for example `scope_by`) appeared
127
+ in the diagram but did not run. The window checks read the options of the inner window,
128
+ so a lone date in the outer window is not an error. The SQL does not change.
129
+ - A `window_table` that is a ConceptQL statement works in the `scope_opts` of a calling
130
+ program, at the top or in `window_opts`. In both places, it gives the SQL, the rows and
131
+ the diagram of the same statement in a `window` statement. Before, building the SQL raised
132
+ `NoMethodError`, because only a `window` statement gave the window table the Database
133
+ that queries its statement. Its errors and the window checks apply, as before.
134
+ - A `window_table` at the top of the `scope_opts` of a calling program works as one in
135
+ `window_opts` does: each result gets its `window_id`, and the operators that work on each
136
+ person, such as `first`, work on each person and window. Before, a window table there (a
137
+ table name or a dataset) only filtered the results. The results had no `window_id`, and
138
+ `first` kept one result for each person, not one for each window.
139
+ - A code that is not a string, such as the number 412 in `["icd9", 412]`, is an error of
140
+ its vocabulary operator, with the message "codes must be strings" and the code
141
+ (`ConceptQL::Operators::Operator::NON_STRING_CODE`). The statement is invalid, so
142
+ ConceptQL builds no SQL for it, and the diagram shows the error. Before, the statement
143
+ was valid, and its SQL, its counts and its code list raised a database error: on
144
+ Postgres "operator does not exist: text = integer", and on DuckDB a conversion error.
145
+ This applies to every vocabulary operator, including the operators that search several
146
+ vocabularies, such as `cpt_or_hcpcs`.
147
+
7
148
  ## [0.8.0] - 2026-11-10
8
149
 
9
150
  ### Added