conceptql 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5201b0144372677675ffcd8abea73662b8e79544ffacfbbd1a61e39a9923d2da
-  data.tar.gz: b81fc6806c7aae55562b073e33b2f6704f36fd6cd57266ffe6b8055df6c6baa0
+  metadata.gz: 765ed8955407259730aefd31f2062c0414dfcada248f171d9f0b903d6e304ca8
+  data.tar.gz: 8873e43bb80c25b5ef6f12c2c1cd3665e3c9bf0422bfc7f83d32de3f357be65f
 SHA512:
-  metadata.gz: 2a49e99ec315bac889401893e68a83de444ab6ac3307e3ae08897a07f7cd59f9a60e3f4f712cff0d43e4d50394d6ff921900e7c23f06f86fdd6134f691fd1d1f
-  data.tar.gz: bf260ffadecac3a5e0dbd8aeb6eb932258fdebcc709363be34704b8f74abde9bd092be0f5710208b4a84d362758087c0c7fb4bf72871f2bafce8fb9496d5fbf9
+  metadata.gz: c25623713499c6538146a8321d03081703f5b09012f53ab7ec54989e617e63c35f3e1912803030a30280d16bfd6951961bbeb44602a62afcfca5af688aebade1
+  data.tar.gz: e45862d4968dc501e3d42b59b83f404c36b48ccc8691986100771b899715889cf5e9731329e56bf84f9ee55f487db32ba0c19e8fae998857b3cbfb41c0165301

data/CHANGELOG.md

@@ -4,6 +4,70 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-10-29
+
+### Removed
+
+- **Breaking:** the `omopv4_plus` data model (OMOP CDM v4) is gone. ConceptQL now supports
+  only `gdm` and `gdm_wide`. A `ConceptQL::Database` created with `data_model: :omopv4_plus`
+  raises `ConceptQL::DataModel::UnsupportedError`, which says that the model was removed.
+  The OMOP query modifiers, `schemas/omopv4_plus.yml`, the OMOP part of the `read` operator,
+  the OMOP and cost vocabulary behaviors, the `omopv4_vocabulary_id` column of
+  `config/vocabularies.csv` and the rake task `test_omopv4_plus` are removed with it. The
+  OHDSI vocabularies (`ohdsi_vocabs`) are not affected. The SQL for `gdm` and `gdm_wide` does
+  not change.
+
+### Added
+
+- The gem ships the specification, `doc/spec.md`. `ConceptQL.spec_path` returns its path, so
+  applications can serve it without building a path into the gem directory. The Jigsaw
+  diagram editor serves it to agents over MCP.
+- Annotate warns about a selected code that has more than one meaning in the vocabulary, with
+  the warning "code(s) with more than one meaning". Each item names the code and all of its
+  distinct names, for example DRG 410, which meant chemotherapy until 2007-09-30 and a biliary
+  tract procedure after that. `Lexicon#code_meanings` is the new cached lookup behind it.
+
+### Changed
+
+- The `co_reported` description now says that a source record is kept only if every upstream
+  stream has an event in it, and that the events of all streams in any other record are
+  dropped without a warning. It no longer mentions OMOP visits. The spec has a new `co_reported` section that says the same. The
+  operator's behavior does not change. Applications that show operator descriptions (the
+  Jigsaw diagram editor) show the new text.
+- The specification (`doc/spec.md` and its source `doc/spec.md.cql`) now describes GDM
+  instead of OMOP CDM v4. Every example statement now validates on `gdm_wide`. Each example
+  table comes from the DuckDB test fixture, with the columns `criterion_table` and
+  `criterion_domain` instead of `criterion_type`. The diagrams are drawn again from the same
+  data. Examples that used removed operators (`intersect`, `complement`, `primary_diagnosis`,
+  `place_of_service_code`, `visit_occurrence`, `procedure_cost`, `payer`, `value`,
+  `greater_than`) now use `filter`, `provenance`, `hospitalization`, `co_reported` and
+  `numeric_filter`, or are removed. The removed sections are "Intersect", "Complement",
+  "Casting to a visit_occurrence", "Cast all the Things!", the intersect question in
+  "Discussion about Set Operators", and "How to Handle fact_relationship Table from CDMv5".
+  The new sections are "Filter" and "Casting to death". The Appendix C proposal "Filter
+  Operator" is now "`as` Option for Filter", because the filter operator exists. "Time Windows" is now a `##` section,
+  so that no section is larger than about 23 KB.
+
+## [0.4.1] - 2026-10-28
+
+### Changed
+
+- `Query#sql(:formatted)` formats SQL inside the Ruby process with the new runtime dependency
+  sequel-sqlformat (`~> 0.1`), a binding to the Rust `sqlformat` crate. It no longer starts
+  sql-formatter, pg_format or sqlformat once per statement. A large algorithm (78 statements)
+  now formats in about 0.13 s instead of about 8.3 s. Formatting changes only whitespace and
+  keyword case, for Postgres, DuckDB and Spark SQL. sequel-sqlformat ships precompiled gems
+  for Linux and macOS, so no Rust toolchain is needed.
+- The development dependency sequel-duckdb is now `~> 0.4.2` (was `~> 0.4.1`). 0.4.2 fixes
+  `create_or_replace_view` and drops support for the duckdb gem 1.5.x, which conceptql
+  already did not use.
+
+### Removed
+
+- `ConceptQL::SqlFormatters`, every adapter's `preferred_formatter`, and
+  `ConceptQL::Utils.timed_capture`. The external formatter programs are no longer used.
+- The test Docker image no longer installs the `sqlformat` package.
+
 ## [0.4.0] - 2026-10-26
 
 ### Added

data/config/vocabularies.csv

@@ -1,87 +1,87 @@
-id,omopv4_vocabulary_id,vocabulary_full_name,vocabulary_short_name,domain,hidden,format_regexp,aliases,requires_lexicon
-ADMSRCE,1000000,ADMSRCE,,,,,,
-ADMTYPE,1000001,ADMTYPE,,,,,,
-ATC,21,WHO Anatomic Therapeutic Chemical Classification,WHO ATC,drug_exposure,,"^[ABCDGHJLMNPRSVabcdghjlmnprsv](\d{2}([A-Za-z]{1,2}(\d{2})?)?)?$",,
-BETOS,1000002,BETOS,,,,,,
-Clinical Code Type,1000004,Clinical Code Type,,,,,,
-Concept Class,68,OMOP Concept Class,,,Y,,,
-Condition Type,37,OMOP Condition Occurrence Type,,condition_occurrence,Y,,,
-CPT4,4,Current Procedural Terminology version 4 (AMA),CPT,procedure_occurrence,,"^\w\d{3}\w$",cpt,
-CPT4_HCPCS,,CPT4 and HCPCS merged (OI supplemental),CPT4/HCPCS,procedure_occurrence,,"^\w\d{3}\w$",,Y
-CS_Schema,1000003,CS_Schema,,,,,,
-Currency,65,International Currency Symbol (ISO 4217),,,Y,,,
-Death Type,45,OMOP Death Type,,death,Y,,,
-Device Type,63,OMOP Device Type,,observation,Y,,,
-Domain,59,OMOP Domain,,,Y,,,
-DRG,40,Diagnosis-related group (CMS),DRG,procedure_occurrence,,"^\d{3}$",,
-DrugTextString,26400,Drug Text String,Drug Text String,drug_exposure,,,,
-Drug Type,36,OMOP Drug Exposure Type,,drug_exposure,Y,,,
-DURATION_UNIT,1000005,DURATION_UNIT,,,,,,
-Ethnicity,44,OMOP Ethnicity,,person,Y,,,
-Gemscript,56,,Gemscript,drug_exposure,,"^\d{8}$",,
-Gender,12,OMOP Gender,,person,Y,,,
-HCFASPEC,1000006,HCFASPEC,,,,,,
-HCFATYPE,1000007,HCFATYPE,,,,,,
-HCPCS,5,Healthcare Common Procedure Coding System (CMS),HCPCS,procedure_occurrence,,"^[A-Za-z]\d{4}$",,
-ICD03_Morphology,1000008,ICD03_Morphology,,,,"^\d{4}/\d$",,
-ICD03_Topography,1000009,ICD03_Topography,,,,"^[A-Za-z]\d{2}(\.\d{1,2})?$",,
-ICD10,34,"International Classification of Diseases, Tenth Revision (WHO)",ICD-10 WHO,condition_occurrence,,"^([A-Za-z][0-9][0-9A-Za-z](\.\w{1,4})?|[A-Za-z][0-9][0-9A-Za-z]-[A-Za-z][0-9][0-9A-Za-z]|(?:I|II|III|IV|V|VI|VII|VIII|IX|X|XI|XII|XIII|XIV|XV|XVI|XVII|XVIII|XIX|XX|XXI|XXII))$",,
-ICD10PCS,35,"ICD-10 Procedure Coding System (CMS)",ICD-10 PCS,procedure_occurrence,,"^[0-9BCDFGHX][0-9A-HJ-NP-Z]{6}$",icd10_pcs,
-ICD10CM,70,"International Classification of Diseases, Tenth Revision, Clinical Modification (NCHS)",ICD-10 CM,condition_occurrence,,"^(?:QA[0-9]|[A-Za-z][0-9][0-9ABab])(\.[0-9A-Za-z]{1,4})?$",,
-ICD9CM,2,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 1 and 2 (NCHS)",ICD-9 CM,condition_occurrence,,"^(V\d{2}(\.\d{1,2})?|\d{3}(\.\d{1,2})?|E\d{3}(\.\d)?)$",icd9,
-ICD9Proc,3,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 3 (NCHS)",ICD-9 Proc,procedure_occurrence,,"^\d{2}(\.\d{1,2})?$",icd9_procedure,
-LOINC,6,Logical Observation Identifiers Names and Codes (Regenstrief Institute),,observation,,"^\w{3,7}-\w$",,
-GDM_CEN_TRACT_MEAS,0,Census Tract Measures from SEER,,observation,,,,
-SEER_YOST_ACS_06_10,0,SEER's Yost Index calculated from ACS 2006-2010,,observation,,,,
-SEER_YOST_ACS_10_14,0,SEER's Yost Index calculated from ACS 2010-2014,,observation,,,,
-SEER_YOST_ACS_13_17,0,SEER's Yost Index calculated from ACS 2013-2017,,observation,,,,
-MARCLA_OBS_TYPE,0,Market Clarity Observation Type,,observation,,,,
-Meas Type,64,OMOP Measurement Type,,observation,Y,,,
-NDC,9,National Drug Code (FDA and manufacturers),,drug_exposure,,"^(\d{5}\w{4}|\d{5}\w{6})$",,
-Note Type,58,OMOP Note Type,,,Y,,,
-NUCC,47,National Uniform Claim Committee Health Care Provider Taxonomy Code Set (NUCC),,,Y,,,
-Observation Type,39,OMOP Observation Type,,observation,Y,,,
-Obs Period Type,61,OMOP Observation Period Type,,observation_period,Y,,,
-OI_FILE_TYPE,1000010,OI_FILE_TYPE,,,,,,
-OI_INFORMATION_TYPE,1000011,OI_INFORMATION_TYPE,,,,,,
-OI_PROV,1000012,OI_PROV,,,,,,
-Place of Service,14,Place of Service Codes for Professional Claims (CMS),,visit_occurrence,Y,,,
-PLCSRVC,1000013,PLCSRVC,,,,,,
-Procedure Type,38,OMOP Procedure Occurrence Type,,procedure_occurrence,Y,,,
-PROVIDER_FACILITY_TYPE,1000014,PROVIDER_FACILITY_TYPE,,,,,,
-Race,13,Race and Ethnicity Code Set (USBC),,person,Y,,,
-Relationship,66,OMOP Relationship,,,Y,,,
-Revenue Code,43,UB04/CMS1450 Revenue Codes (CMS),Revenue Code,procedure_occurrence,,,revenue_code,
-RxNorm,8,RxNorm (NLM),,drug_exposure,,"^\d{2,8}$",,
-SEER_ADJAJC6M,1000015,SEER_ADJAJC6M,,,,,,
-SEER_ADJAJC6N,1000016,SEER_ADJAJC6N,,,,,,
-SEER_ADJAJC6T,1000017,SEER_ADJAJC6T,,,,,,
-SEER_BEH,1000018,SEER_BEH,,,,,,
-SEER_CENSUS_POV_IND,1000019,SEER_CENSUS_POV_IND,,,,,,
-SEER_DAJCC7,1000020,SEER_DAJCC7,,,,,,
-SEER_DAJCCSTG,1000021,SEER_DAJCCSTG,,,,,,
-SEER_DXCONF,1000022,SEER_DXCONF,,,,,,
-SEER_GRADE,1000023,SEER_GRADE,,,,,,
-SEER_LAT,1000024,SEER_LAT,,,,,,
-SEER_MARST,1000025,SEER_MARST,,,,,,
-SEER_MSEX,1000026,SEER_MSEX,,,,,,
-SEER_NHIADE,1000027,SEER_NHIADE,,,,,,
-SEER_NOSRG,1000028,SEER_NOSRG,,,,,,
-SEER_RAD,1000029,SEER_RAD,,,,,,
-SEER_RADSURG,1000030,SEER_RADSURG,,,,,,
-SEER_SEQ,1000031,SEER_SEQ,,,,,,
-SEER_SITERWHO,1000032,SEER_SITERWHO,,,,,,
-SEER_SRACE,1000033,SEER_SRACE,,,,,,
-SEER_SRC,1000034,SEER_SRC,,,,,,
-SEER_SXPRIF,1000035,SEER_SXPRIF,,,,,,
-SEER_SXSITF,1000036,SEER_SXSITF,,,,,,
-SEER_TYPEFU,1000037,SEER_TYPEFU,,,,,,
-SEER_URBRUR,1000038,SEER_URBRUR,,,,,,
-SNOMED,1,SNOMED-CT,SNOMED-CT,condition_occurrence,,"^\d{6,18}$",,
-Specialty,48,Medicare provider/supplier specialty codes (CMS),,provider,Y,,,
-STUS_CD,1000039,STUS_CD,,,,,,
-TYPESRVC,1000040,TYPESRVC,,,,,,
-UCUM,11,Unified Code for Units of Measure (Regenstrief Institute),,,Y,,,
-Visit,24,OMOP Visit,,visit_occurrence,Y,,,
-Visit Type,62,OMOP Visit Type,,visit_occurrence,Y,,,
-Vocabulary,67,OMOP Vocabulary,,,Y,,,
+id,vocabulary_full_name,vocabulary_short_name,domain,hidden,format_regexp,aliases,requires_lexicon
+ADMSRCE,ADMSRCE,,,,,,
+ADMTYPE,ADMTYPE,,,,,,
+ATC,WHO Anatomic Therapeutic Chemical Classification,WHO ATC,drug_exposure,,"^[ABCDGHJLMNPRSVabcdghjlmnprsv](\d{2}([A-Za-z]{1,2}(\d{2})?)?)?$",,
+BETOS,BETOS,,,,,,
+Clinical Code Type,Clinical Code Type,,,,,,
+Concept Class,OMOP Concept Class,,,Y,,,
+Condition Type,OMOP Condition Occurrence Type,,condition_occurrence,Y,,,
+CPT4,Current Procedural Terminology version 4 (AMA),CPT,procedure_occurrence,,"^\w\d{3}\w$",cpt,
+CPT4_HCPCS,CPT4 and HCPCS merged (OI supplemental),CPT4/HCPCS,procedure_occurrence,,"^\w\d{3}\w$",,Y
+CS_Schema,CS_Schema,,,,,,
+Currency,International Currency Symbol (ISO 4217),,,Y,,,
+Death Type,OMOP Death Type,,death,Y,,,
+Device Type,OMOP Device Type,,observation,Y,,,
+Domain,OMOP Domain,,,Y,,,
+DRG,Diagnosis-related group (CMS),DRG,procedure_occurrence,,"^\d{3}$",,
+DrugTextString,Drug Text String,Drug Text String,drug_exposure,,,,
+Drug Type,OMOP Drug Exposure Type,,drug_exposure,Y,,,
+DURATION_UNIT,DURATION_UNIT,,,,,,
+Ethnicity,OMOP Ethnicity,,person,Y,,,
+Gemscript,,Gemscript,drug_exposure,,"^\d{8}$",,
+Gender,OMOP Gender,,person,Y,,,
+HCFASPEC,HCFASPEC,,,,,,
+HCFATYPE,HCFATYPE,,,,,,
+HCPCS,Healthcare Common Procedure Coding System (CMS),HCPCS,procedure_occurrence,,"^[A-Za-z]\d{4}$",,
+ICD03_Morphology,ICD03_Morphology,,,,"^\d{4}/\d$",,
+ICD03_Topography,ICD03_Topography,,,,"^[A-Za-z]\d{2}(\.\d{1,2})?$",,
+ICD10,"International Classification of Diseases, Tenth Revision (WHO)",ICD-10 WHO,condition_occurrence,,"^([A-Za-z][0-9][0-9A-Za-z](\.\w{1,4})?|[A-Za-z][0-9][0-9A-Za-z]-[A-Za-z][0-9][0-9A-Za-z]|(?:I|II|III|IV|V|VI|VII|VIII|IX|X|XI|XII|XIII|XIV|XV|XVI|XVII|XVIII|XIX|XX|XXI|XXII))$",,
+ICD10PCS,"ICD-10 Procedure Coding System (CMS)",ICD-10 PCS,procedure_occurrence,,"^[0-9BCDFGHX][0-9A-HJ-NP-Z]{6}$",icd10_pcs,
+ICD10CM,"International Classification of Diseases, Tenth Revision, Clinical Modification (NCHS)",ICD-10 CM,condition_occurrence,,"^(?:QA[0-9]|[A-Za-z][0-9][0-9ABab])(\.[0-9A-Za-z]{1,4})?$",,
+ICD9CM,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 1 and 2 (NCHS)",ICD-9 CM,condition_occurrence,,"^(V\d{2}(\.\d{1,2})?|\d{3}(\.\d{1,2})?|E\d{3}(\.\d)?)$",icd9,
+ICD9Proc,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 3 (NCHS)",ICD-9 Proc,procedure_occurrence,,"^\d{2}(\.\d{1,2})?$",icd9_procedure,
+LOINC,Logical Observation Identifiers Names and Codes (Regenstrief Institute),,observation,,"^\w{3,7}-\w$",,
+GDM_CEN_TRACT_MEAS,Census Tract Measures from SEER,,observation,,,,
+SEER_YOST_ACS_06_10,SEER's Yost Index calculated from ACS 2006-2010,,observation,,,,
+SEER_YOST_ACS_10_14,SEER's Yost Index calculated from ACS 2010-2014,,observation,,,,
+SEER_YOST_ACS_13_17,SEER's Yost Index calculated from ACS 2013-2017,,observation,,,,
+MARCLA_OBS_TYPE,Market Clarity Observation Type,,observation,,,,
+Meas Type,OMOP Measurement Type,,observation,Y,,,
+NDC,National Drug Code (FDA and manufacturers),,drug_exposure,,"^(\d{5}\w{4}|\d{5}\w{6})$",,
+Note Type,OMOP Note Type,,,Y,,,
+NUCC,National Uniform Claim Committee Health Care Provider Taxonomy Code Set (NUCC),,,Y,,,
+Observation Type,OMOP Observation Type,,observation,Y,,,
+Obs Period Type,OMOP Observation Period Type,,observation_period,Y,,,
+OI_FILE_TYPE,OI_FILE_TYPE,,,,,,
+OI_INFORMATION_TYPE,OI_INFORMATION_TYPE,,,,,,
+OI_PROV,OI_PROV,,,,,,
+Place of Service,Place of Service Codes for Professional Claims (CMS),,visit_occurrence,Y,,,
+PLCSRVC,PLCSRVC,,,,,,
+Procedure Type,OMOP Procedure Occurrence Type,,procedure_occurrence,Y,,,
+PROVIDER_FACILITY_TYPE,PROVIDER_FACILITY_TYPE,,,,,,
+Race,Race and Ethnicity Code Set (USBC),,person,Y,,,
+Relationship,OMOP Relationship,,,Y,,,
+Revenue Code,UB04/CMS1450 Revenue Codes (CMS),Revenue Code,procedure_occurrence,,,revenue_code,
+RxNorm,RxNorm (NLM),,drug_exposure,,"^\d{2,8}$",,
+SEER_ADJAJC6M,SEER_ADJAJC6M,,,,,,
+SEER_ADJAJC6N,SEER_ADJAJC6N,,,,,,
+SEER_ADJAJC6T,SEER_ADJAJC6T,,,,,,
+SEER_BEH,SEER_BEH,,,,,,
+SEER_CENSUS_POV_IND,SEER_CENSUS_POV_IND,,,,,,
+SEER_DAJCC7,SEER_DAJCC7,,,,,,
+SEER_DAJCCSTG,SEER_DAJCCSTG,,,,,,
+SEER_DXCONF,SEER_DXCONF,,,,,,
+SEER_GRADE,SEER_GRADE,,,,,,
+SEER_LAT,SEER_LAT,,,,,,
+SEER_MARST,SEER_MARST,,,,,,
+SEER_MSEX,SEER_MSEX,,,,,,
+SEER_NHIADE,SEER_NHIADE,,,,,,
+SEER_NOSRG,SEER_NOSRG,,,,,,
+SEER_RAD,SEER_RAD,,,,,,
+SEER_RADSURG,SEER_RADSURG,,,,,,
+SEER_SEQ,SEER_SEQ,,,,,,
+SEER_SITERWHO,SEER_SITERWHO,,,,,,
+SEER_SRACE,SEER_SRACE,,,,,,
+SEER_SRC,SEER_SRC,,,,,,
+SEER_SXPRIF,SEER_SXPRIF,,,,,,
+SEER_SXSITF,SEER_SXSITF,,,,,,
+SEER_TYPEFU,SEER_TYPEFU,,,,,,
+SEER_URBRUR,SEER_URBRUR,,,,,,
+SNOMED,SNOMED-CT,SNOMED-CT,condition_occurrence,,"^\d{6,18}$",,
+Specialty,Medicare provider/supplier specialty codes (CMS),,provider,Y,,,
+STUS_CD,STUS_CD,,,,,,
+TYPESRVC,TYPESRVC,,,,,,
+UCUM,Unified Code for Units of Measure (Regenstrief Institute),,,Y,,,
+Visit,OMOP Visit,,visit_occurrence,Y,,,
+Visit Type,OMOP Visit Type,,visit_occurrence,Y,,,
+Vocabulary,OMOP Vocabulary,,,Y,,,