conceptql 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +141 -0
- data/docs/spec/README.md +402 -337
- data/docs/spec/examples/date-range-operator.json +1 -1
- data/docs/spec/examples/death-operator.json +1 -1
- data/docs/spec/examples/nth-occurrence-operator.json +1 -1
- data/docs/spec/examples/one-in-two-out-operator.json +1 -1
- data/docs/spec/examples/person-filter-operator.json +2 -2
- data/docs/spec/examples/vocabulary-operator.json +3 -3
- data/lib/conceptql/behaviors/metadatable.rb +76 -62
- data/lib/conceptql/behaviors/utilizable.rb +1 -1
- data/lib/conceptql/database.rb +16 -2
- data/lib/conceptql/diagram.rb +2 -2
- data/lib/conceptql/lexicon/lexicon_no_db.rb +4 -4
- data/lib/conceptql/lexicon.rb +7 -4
- data/lib/conceptql/operators/after.rb +5 -4
- data/lib/conceptql/operators/any_overlap.rb +4 -3
- data/lib/conceptql/operators/before.rb +5 -4
- data/lib/conceptql/operators/binary_operator_operator.rb +11 -13
- data/lib/conceptql/operators/casting_operator.rb +1 -1
- data/lib/conceptql/operators/co_reported.rb +9 -7
- data/lib/conceptql/operators/contains.rb +4 -3
- data/lib/conceptql/operators/count.rb +3 -3
- data/lib/conceptql/operators/date_range.rb +9 -8
- data/lib/conceptql/operators/death.rb +4 -4
- data/lib/conceptql/operators/dedup.rb +2 -2
- data/lib/conceptql/operators/during.rb +4 -3
- data/lib/conceptql/operators/episode.rb +9 -9
- data/lib/conceptql/operators/equal.rb +3 -2
- data/lib/conceptql/operators/ethnicity.rb +7 -3
- data/lib/conceptql/operators/except.rb +4 -3
- data/lib/conceptql/operators/filter.rb +4 -3
- data/lib/conceptql/operators/first.rb +4 -3
- data/lib/conceptql/operators/from.rb +2 -1
- data/lib/conceptql/operators/gender.rb +4 -3
- data/lib/conceptql/operators/hospice.rb +4 -3
- data/lib/conceptql/operators/hospitalization.rb +4 -3
- data/lib/conceptql/operators/information_periods.rb +5 -4
- data/lib/conceptql/operators/last.rb +4 -3
- data/lib/conceptql/operators/match.rb +4 -3
- data/lib/conceptql/operators/multiple_vocabularies.rb +4 -3
- data/lib/conceptql/operators/numeric.rb +4 -4
- data/lib/conceptql/operators/numeric_filter.rb +6 -6
- data/lib/conceptql/operators/occurrence.rb +25 -18
- data/lib/conceptql/operators/one_in_two_out.rb +16 -16
- data/lib/conceptql/operators/operator.rb +16 -3
- data/lib/conceptql/operators/pass_thru.rb +1 -1
- data/lib/conceptql/operators/person.rb +3 -3
- data/lib/conceptql/operators/person_filter.rb +6 -5
- data/lib/conceptql/operators/place_of_service_filter.rb +7 -7
- data/lib/conceptql/operators/provenance.rb +9 -9
- data/lib/conceptql/operators/provider_filter.rb +6 -6
- data/lib/conceptql/operators/race.rb +4 -3
- data/lib/conceptql/operators/recall.rb +6 -5
- data/lib/conceptql/operators/snf.rb +4 -3
- data/lib/conceptql/operators/sum.rb +6 -5
- data/lib/conceptql/operators/time_window.rb +6 -6
- data/lib/conceptql/operators/trim_date_end.rb +6 -5
- data/lib/conceptql/operators/trim_date_start.rb +6 -5
- data/lib/conceptql/operators/union.rb +8 -6
- data/lib/conceptql/operators/vocabulary.rb +12 -8
- data/lib/conceptql/query.rb +41 -12
- data/lib/conceptql/scope.rb +10 -2
- data/lib/conceptql/version.rb +1 -1
- data/lib/conceptql/visitors/metadata_extractor.rb +3 -1
- data/lib/conceptql/vocabularies/drg_systems.rb +16 -14
- data/lib/conceptql/vocabularies/read_codes.rb +1 -1
- data/lib/conceptql/window.rb +23 -2
- data/schemas/gdm.yml +0 -66
- data/schemas/gdm_wide.yml +0 -66
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 01bc0bcc6e8ae91743cf821c431cabfe6087fa71b592e243e8185117e1674e13
|
|
4
|
+
data.tar.gz: 1b418ae79bda1cb8a4ce353b4f4de9b57019a22db665dabe05b0a2900135d377
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6059b619e417fd0e8466f4154ecb16862006c4a7e02c2e67ca07da2407b424630c3c5d497301e6402dd54ea5e925bd79d97cb88f9bd56c6339dd75bd16371118
|
|
7
|
+
data.tar.gz: 8ba160a3456ba4725460ec5f3777c2f7f31beab01e1012a7cf408cd586834aa6e2b034519abca22a861a3e43065dd0ef145582739fe3ba173047df8a2487855d
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,147 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
## [Unreleased]
|
|
6
6
|
|
|
7
|
+
## [0.9.0] - 2026-11-13
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- Each operator in `ConceptQL.metadata` has seven facets as top-level keys: `upstreams`,
|
|
12
|
+
`reads_data`, `timeless`, `takes_codes`, `uses_provenance`, `utilization` and
|
|
13
|
+
`temporal_comparison`. Their values come from the code (the upstream validations, the
|
|
14
|
+
behavior modules and the superclasses), so no operator declares them.
|
|
15
|
+
`ConceptQL::Metadatable::FACET_KEYS` lists the keys. `docs/metadata.md` says what each one
|
|
16
|
+
means.
|
|
17
|
+
- Each operator in `ConceptQL.metadata` has the top-level key `effects`: a list of strings
|
|
18
|
+
that says what the operator does to results. The values are `select`, `keep`, `change`,
|
|
19
|
+
`combine`, `aggregate`, `cast` and `refer`, and `docs/metadata.md` says what each one
|
|
20
|
+
means. Each operator class declares its effects with `effects :keep, :change`, and a
|
|
21
|
+
subclass inherits them, so every vocabulary operator is `select`. An unknown value raises
|
|
22
|
+
`ConceptQL::Metadatable::UnknownEffectError` when the class loads. `effects` is the last
|
|
23
|
+
key of `ConceptQL::Metadatable::FACET_KEYS`.
|
|
24
|
+
- Each operator in `ConceptQL.metadata` has the key `diagram_base`: the symbol that a diagram
|
|
25
|
+
draws for it (`:selection`, `:filter`, `:set`, `:temporal` or `:cast`). It is not a facet.
|
|
26
|
+
- The spec build (`just spec`, `just spec-check`) checks the class terms of the spec against
|
|
27
|
+
the facets. An operator under a `##` heading of a class (for example "Temporal Comparison
|
|
28
|
+
Operators"), or in a section that says "It is a [term](...)", must meet the rule of that
|
|
29
|
+
class, or the build fails and names the operator, the term and its facets. "Set operator"
|
|
30
|
+
and "relative temporal operator" have no rule. `person_filter` moves from Inline-Filter
|
|
31
|
+
Operators to Comparison Operators in the spec, because it takes left and right streams.
|
|
32
|
+
|
|
33
|
+
### Removed
|
|
34
|
+
|
|
35
|
+
- **Breaking:** the operator metadata keys `basic_type`, `arity`, `min_upstreams` and
|
|
36
|
+
`max_upstreams`. The `upstreams` facet replaces `min_upstreams` and `max_upstreams`, and
|
|
37
|
+
says which upstreams the validations accept. `diagram_base` replaces `basic_type`, with
|
|
38
|
+
the same values, and `arity`, which was a number made from `basic_type`. The class methods
|
|
39
|
+
`arity`, `allows_one_upstream` and `allows_many_upstreams` are gone too. A reader of
|
|
40
|
+
`ConceptQL.metadata` must read the new keys.
|
|
41
|
+
- The GDM vocabulary tables `concepts`, `vocabularies` and `mappings` are no longer in the
|
|
42
|
+
shipped table descriptions `schemas/gdm.yml` and `schemas/gdm_wide.yml`. No lexicon reads
|
|
43
|
+
them since 0.8.0. The SQL and the annotations that a Sequel mock database builds do not
|
|
44
|
+
change.
|
|
45
|
+
|
|
46
|
+
### Changed
|
|
47
|
+
|
|
48
|
+
- **Breaking:** an operator class declares its diagram symbol with `diagram_base :set`. The
|
|
49
|
+
class method `basic_type` is gone.
|
|
50
|
+
- `ConceptQL::Diagram` (the render tree and the Jigsaw JSON), `Query#extract_metadata` and
|
|
51
|
+
the spec build read `diagram_base` and the `upstreams` facet, not the old keys. Every
|
|
52
|
+
diagram and every value of `codeset_only` is the same as before.
|
|
53
|
+
- The spec (`docs/spec/README.md`) explains the parts of a statement (operator, arguments,
|
|
54
|
+
options, upstream and downstream), file and code provenance, inpatient and outpatient
|
|
55
|
+
settings, and information periods. The section "Window Tables" is now "Windows". It says
|
|
56
|
+
why windows sit outside an algorithm, how they differ from `time_window`, and every
|
|
57
|
+
window option: `window_table`, `start_date` and `end_date`, `scope_by`,
|
|
58
|
+
`adjust_window_start` and `adjust_window_end`, `precalculated_window_id` and `person_ids`.
|
|
59
|
+
Before, it said only that a result must start inside its window, which is true only for
|
|
60
|
+
the default `scope_by`. The old anchor `#window-tables` still works.
|
|
61
|
+
- The spec and the operator texts in `ConceptQL.metadata` (`desc`, `returns`, the meanings
|
|
62
|
+
of the arguments and the options, and the example titles) use the terms of the glossary.
|
|
63
|
+
A ConceptQL row is a "result", a row of a dataset table is a "row", and a "stream" is the
|
|
64
|
+
set of results that passes from one operator to the next. Before, the texts said "record"
|
|
65
|
+
for both. The heading "Think of Records as a Stream" is now "Think of Results as a
|
|
66
|
+
Stream", and "What *are* Streams Really?" says that ConceptQL is not a stream-processing
|
|
67
|
+
system. The standard description of a vocabulary operator is now "Selects results based
|
|
68
|
+
on the ... vocabulary.". No warning or error message changed: the warning "the data has
|
|
69
|
+
no records of this vocabulary" keeps its text, because Jigsaw's tests match it.
|
|
70
|
+
- The deprecation message of `count`, which a calling program can show, now says "To count
|
|
71
|
+
the results of each person". Before, it said "To count the events of each person". The
|
|
72
|
+
rest of the message is the same.
|
|
73
|
+
- The long names of `ms_drg` and `cms_drg` (`long_name` in `ConceptQL.metadata` and the
|
|
74
|
+
vocabulary table of the spec) say "discharges from 2007-10-01" and "discharges before
|
|
75
|
+
2007-10-01", because a DRG applies to a discharge. Before, they said "events".
|
|
76
|
+
- The deprecation warning of `drg` (`ConceptQL::Vocabularies::DrgSystems::DRG_DEPRECATION`)
|
|
77
|
+
says "drg selects the results of both" and "discharges from 2007-10-01" or "discharges
|
|
78
|
+
before 2007-10-01", the same as the long names. Before, it said "events" in all three
|
|
79
|
+
places. The rest of the message is the same.
|
|
80
|
+
|
|
81
|
+
### Fixed
|
|
82
|
+
|
|
83
|
+
- The reference sections of `union` and `co_reported` in `docs/spec/README.md` say that
|
|
84
|
+
they take two or more upstream operators. Before, they said one or more.
|
|
85
|
+
- `Query#events_per_patient` is `:multiple` for `occurrence` with `at_least` or `within`,
|
|
86
|
+
because the operator can then pass along several records of one person. Before, it was
|
|
87
|
+
always `:single`, so a Jigsaw export of such an algorithm said that each patient had at
|
|
88
|
+
most one event. `occurrence` without these options, `first` and `last` stay `:single`.
|
|
89
|
+
- `Query#events_per_patient` is `:multiple` for `union` and `co_reported` with two or more
|
|
90
|
+
upstreams, because each upstream passes along its own records. Before, it was `:single`
|
|
91
|
+
when every upstream was `:single`: on the DuckDB fixture, a `union` of two `first`
|
|
92
|
+
operators gave 36 persons more than one row. With one upstream, they still report the
|
|
93
|
+
value of that upstream.
|
|
94
|
+
- `ethnicity` applies the windows of a window table, as `gender` and `race` do. It keeps
|
|
95
|
+
each patient who has a window, whatever the dates, and each record gets its `window_id`.
|
|
96
|
+
Before, it ignored the windows: it kept the patients with no window, and their records
|
|
97
|
+
had no `window_id`.
|
|
98
|
+
- `Query#extract_metadata` works on a binary operator (`after`, `before`, `during`, ...)
|
|
99
|
+
whose `left` or `right` is missing or is not a statement. It lists only the operands
|
|
100
|
+
that exist. Before, it raised `NoMethodError`. The statement is still invalid, with the
|
|
101
|
+
same errors.
|
|
102
|
+
- `Query#events_per_patient` is `:multiple` and `Query#multiple_vocabularies` is `false` for
|
|
103
|
+
a binary operator whose `left` is missing or is not a statement. These are the values of
|
|
104
|
+
an operator with no upstream. Before, both raised `NoMethodError`. The statement is still
|
|
105
|
+
invalid, with the same errors.
|
|
106
|
+
- A window with only one of `start_date` and `end_date` makes the statement invalid. The
|
|
107
|
+
error is on the `window` operator ("date range needs start_date and end_date"), and its
|
|
108
|
+
detail names the missing option. Before, ConceptQL ignored the lone date with no warning,
|
|
109
|
+
so the statement returned results from all dates.
|
|
110
|
+
- A window with a `scope_by` other than `start_date`, `end_date` or `date_range` makes the
|
|
111
|
+
statement invalid. The error is on the `window` operator ("scope_by must be start_date,
|
|
112
|
+
end_date or date_range"), and its detail is the value. Before, building the SQL raised a
|
|
113
|
+
plain `RuntimeError`, so `Query#errors` and the diagram did not show it. A `scope_by` of
|
|
114
|
+
`nil` is the default, `start_date`. Before, it raised `NoMethodError`.
|
|
115
|
+
- Both checks read the window statement and the `scope_opts` of a calling program. When the
|
|
116
|
+
window options have an error, `annotate` puts the statement inside a `window` operator
|
|
117
|
+
that holds the error, and it counts nothing, as for an invalid window table statement.
|
|
118
|
+
`Query#sql` raises `ConceptQL::InvalidStatementError` with the same errors.
|
|
119
|
+
- Window options with String keys in the `scope_opts` of `Database#query` and
|
|
120
|
+
`Database.new` (and in their `window_opts`) take effect, as Symbol keys do. A calling
|
|
121
|
+
program that builds its options from JSON has String keys. Before, ConceptQL ignored
|
|
122
|
+
them with no error, so the statement ran over all dates and persons. Both window checks
|
|
123
|
+
above also read String keys.
|
|
124
|
+
- For a window inside a window, the diagram (`Query#annotate`) shows the options of the
|
|
125
|
+
inner window, which are the options that the SQL uses. Before, it showed the options of
|
|
126
|
+
both windows merged, so an option of the outer window (for example `scope_by`) appeared
|
|
127
|
+
in the diagram but did not run. The window checks read the options of the inner window,
|
|
128
|
+
so a lone date in the outer window is not an error. The SQL does not change.
|
|
129
|
+
- A `window_table` that is a ConceptQL statement works in the `scope_opts` of a calling
|
|
130
|
+
program, at the top or in `window_opts`. In both places, it gives the SQL, the rows and
|
|
131
|
+
the diagram of the same statement in a `window` statement. Before, building the SQL raised
|
|
132
|
+
`NoMethodError`, because only a `window` statement gave the window table the Database
|
|
133
|
+
that queries its statement. Its errors and the window checks apply, as before.
|
|
134
|
+
- A `window_table` at the top of the `scope_opts` of a calling program works as one in
|
|
135
|
+
`window_opts` does: each result gets its `window_id`, and the operators that work on each
|
|
136
|
+
person, such as `first`, work on each person and window. Before, a window table there (a
|
|
137
|
+
table name or a dataset) only filtered the results. The results had no `window_id`, and
|
|
138
|
+
`first` kept one result for each person, not one for each window.
|
|
139
|
+
- A code that is not a string, such as the number 412 in `["icd9", 412]`, is an error of
|
|
140
|
+
its vocabulary operator, with the message "codes must be strings" and the code
|
|
141
|
+
(`ConceptQL::Operators::Operator::NON_STRING_CODE`). The statement is invalid, so
|
|
142
|
+
ConceptQL builds no SQL for it, and the diagram shows the error. Before, the statement
|
|
143
|
+
was valid, and its SQL, its counts and its code list raised a database error: on
|
|
144
|
+
Postgres "operator does not exist: text = integer", and on DuckDB a conversion error.
|
|
145
|
+
This applies to every vocabulary operator, including the operators that search several
|
|
146
|
+
vocabularies, such as `cpt_or_hcpcs`.
|
|
147
|
+
|
|
7
148
|
## [0.8.0] - 2026-11-10
|
|
8
149
|
|
|
9
150
|
### Added
|