conceptql 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: bb925d74153217fcb29ee4c4e5874827005246be
-  data.tar.gz: 41ae6d964b8a6ae82b37098adcda8548917543bd
+SHA256:
+  metadata.gz: 5201b0144372677675ffcd8abea73662b8e79544ffacfbbd1a61e39a9923d2da
+  data.tar.gz: b81fc6806c7aae55562b073e33b2f6704f36fd6cd57266ffe6b8055df6c6baa0
 SHA512:
-  metadata.gz: f62d0381dbd1ceabed67e9fff58d95975bf4cff27a957f03aa0ba495ca7726b22af3727d26acd4f683ca048a17dac858dffd2bef7f2e0707cae7e5cfd29e6677
-  data.tar.gz: 004743713781e5ea7cf7281ea071597ac8fa0959714571639d97d69a9fdd67b03692b10b9e1c77802230f57149212e9daf5077459d02db86814759051b44cc62
+  metadata.gz: 2a49e99ec315bac889401893e68a83de444ab6ac3307e3ae08897a07f7cd59f9a60e3f4f712cff0d43e4d50394d6ff921900e7c23f06f86fdd6134f691fd1d1f
+  data.tar.gz: bf260ffadecac3a5e0dbd8aeb6eb932258fdebcc709363be34704b8f74abde9bd092be0f5710208b4a84d362758087c0c7fb4bf72871f2bafce8fb9496d5fbf9

data/CHANGELOG.md

@@ -1,30 +1,202 @@
 # Changelog
+
 All notable changes to this project will be documented in this file.
 
-## Unreleased
+## [Unreleased]
+
+## [0.4.0] - 2026-10-26
+
+### Added
+
+- `conceptql render_json STATEMENT_FILE [--counts] [--data-model=...]` prints statements as a
+  `conceptql-diagram/v1` render tree for the diagram renderer (`ConceptQL::Diagram`), plus
+  `ConceptQL::Diagram::JigsawTree`, a port of the diagram editor's ConceptqlAdapter
+- `ConceptQL::Database#operators(data_model = own)`: each Database's own operator registry, with
+  vocabulary operators built from that Database's lexicon and the built-in operators merged over
+  them. Every operator lookup goes through it.
+- `ConceptQL::Database.new(db, name_concepts_table: name)` and
+  `ConceptQL::Database#create_name_concepts_table`: the display-name joins (drug name and dose
+  units, lab units and result concept, admission source and discharge location) read a table of
+  only the concept rows that the data references, instead of the whole concept table. The caller
+  builds the table once, before its queries run, and drops it afterwards. Off by default.
+- `ConceptQL::Database.new(db, share_utilization: true)` and `ConceptQL::Database#shared_tables`:
+  the statements of one Database share each hospitalization, SNF and hospice list (the
+  `Utilizable` operators) as a table named `<table_prefix>_util_<hash>`, in `scratch_database`
+  when that is set. While the table does not exist, `Query#sql_statements` lists it as an entry
+  before the entries that read it, so a caller that runs the entries in order (as CREATE TABLE AS,
+  then the query) creates it first. A caller that reads rows through conceptql gets the table
+  created at the first read. Building SQL creates no table. Two threads that generate SQL on one
+  Database at the same time can both list the same table: this is not handled. The key is the SQL
+  of the list, so each window gets its own table, and a list shares only when its lookup ids come
+  in one order: use the option with `cache_lookups: true`. The option needs a `table_prefix` in
+  `scope_opts`. conceptql never drops these tables: the caller drops `shared_tables`. Off by
+  default. On DuckDB, a t_shank prototype of this change cut one export with 40 hospitalization
+  criteria from 6.85 to 5.10 minutes. Spark and Postgres timing are not measured.
+- `ConceptQL::Database.new(db, cache_lookups: true)`: each vocabulary lookup reaches the database
+  once per distinct argument list for the life of the Database. This includes the empty-vocabulary
+  check behind the unknown-code warnings, the Race operator's name lookup, the Provenance
+  operator's keyword check, and the checks that the data model's source tables and the concept
+  table exist (once per table and database). Off by default.
+- Tag-based releases: pushing a `v*` tag runs `.github/workflows/release.yml`, which publishes the
+  gem to rubygems.org through RubyGems trusted publishing (no stored API key). It refuses unless the
+  tag equals `v` + `ConceptQL::VERSION`, the commit is on `main`, and the `Run Tests` workflow passed
+  on a push to `main` for that commit. Before publishing, it installs the built gem into an empty
+  `GEM_HOME` and checks that it loads and ships `config/` and `schemas/`. After publishing, it checks
+  that rubygems.org lists the version. See "Releasing" in `CLAUDE.md`.
+
+### Fixed
+
+- `ConceptQL.metadata` marks every option that an operator requires as `required`, not only the
+  first. `right` on each two-stream operator, `end` on `date_range` and `roles` on
+  `provider_filter` were not marked. Marking a subclass's options no longer changes its parent
+  class's option hashes
+- `trim_date_end` and `trim_date_start` accept `join_columns` from a JSON statement. The column
+  names arrived as strings and became string literals in the SQL, so the query failed
+- `Query#code_list` on a statement with validation errors raises `Query::InvalidStatementError`
+  (a `Query::QueryError` that names the errors) instead of NoMethodError or a partial code list
+- `require 'conceptql'` no longer connects to a database. It used to connect to whatever
+  Sequelizer was configured for, to register an operator per vocabulary, so it crashed with no
+  database configured and, on DuckDB, a second process could not load the library while another
+  held the file
+- Bug with PlaceOfServiceFilter and other procedure_cost-based operators (#84)
+- Spark CI downloads its test data from the repo variable `SPARK_TEST_DATA_URL` and checks it
+  against `.github/synpuf_test_data.tgz.sha256` before it extracts it. The hardcoded Dropbox link
+  it used was dead
+- CLI: `conceptql metadata` raised ArgumentError when it wrote metadata.json, and
+  `CONCEPTQL_PARQUET_TEST_DIR` raised NoMethodError (`String#glob`) for every command that
+  connects to a database. It now makes the parquet files of each directory in its glob ready.
+  `conceptql scope_annotate` ignored the connection options, and `conceptql help` listed
+  `simple_visit` as `extract_metadata`
+- A vocabulary statement with `*` no longer warns about the format of its other codes. The check
+  for `*` never ran, because `Vocabulary#select_all?` is private
+- `occurrence` with a blank or whitespace-only `at_least` or `within` (as an editor can send for
+  an empty field) gives the same rows as without the option. It used to take the timed query,
+  which returns other rows
+- `match` without `only_columns` compared every column with `=`, so two NULL values never matched
+  and a stream matched with itself kept nothing. It now compares the key columns (`person_id`,
+  `criterion_id`, `criterion_table`, `criterion_domain`, `start_date`, `uuid`, `window_id`) with
+  `=` and the other columns NULL-safe (`IS NOT DISTINCT FROM`, `<=>` on Spark). `match` also
+  ignored `join_columns`. It now compares only the person and the listed columns, as the other
+  two-stream operators do, and rejects `join_columns` next to `only_columns` or `except_columns`
+- An option that lists columns (`only_columns`, `except_columns` and `join_columns` on `match`,
+  and `join_columns` on the temporal operators and `trim_date_start`/`trim_date_end`) also
+  accepts a comma-separated string, such as `"uuid, start_date"`. A blank string is an unset
+  option. Before, a string raised NoMethodError
+- `match` with `invert_match` set to the string `"false"` (in any case), or to an empty string,
+  no longer inverts the match. Before, any value other than `false` and `nil` inverted it
+
+### Changed
+
+- Set RUNTIME_FILTER_MODE only if asked
+- `ConceptQL::Operators.operators` is removed. Built-in operators are in
+  `ConceptQL::Operators.static_operators`; vocabulary operators come from each Database
+  (`ConceptQL::Database#operators`). `ConceptQL::Database.lexicon` (class method) is removed;
+  use `Database#lexicon`
+- The sequelizer dependency now admits 0.4.x (`>= 0.3.0, < 0.5`). The previous `< 0.4` bound
+  rejected the released sequelizer 0.4.0. Gemfile.lock now resolves sequelizer 0.4.0.
+- Development dependencies are updated to their newest releases (2026-10-26), including the
+  duckdb gem 2.0.0.1, pg 1.7.0, rubocop 1.92.1, and the newest sequelizer, sequel-duckdb 0.4.1
+  and sequel-hexspace 2.1.0. The test suites pass on Postgres and DuckDB. `just test-duckdb`
+  pins the libduckdb library to 1.5.6 (override with `LIBDUCKDB_VERSION`). GitHub CI runs the
+  DuckDB suite against libduckdb 1.5.6 and 2.0.0
+- `just release-check` fails when a dependency is behind its newest allowed release. The release
+  procedure runs it first
+- Docker test runs use the ghcr `test_data.jumper` image pinned by digest, the same one as CI, and
+  run as the calling user, so they leave no root-owned files. The runner image drops Chrome's libraries
+- On DuckDB, operator CTEs are now `AS MATERIALIZED`, as on Postgres
+  (`ConceptQL::Rdbms::DuckDB#supports_materialized?` is true). DuckDB accepts the keyword since
+  v0.8.0. `just test-duckdb` passes against libduckdb 1.5.6
+- The provenance and utilization operators (`provenance`, `one_in_two_out`, `hospitalization`,
+  `hospice`, `snf`) are now GDM-only. On omopv4_plus they fail validation as an invalid operator
+  instead of raising NoMethodError, and they are not in its metadata
+- Code format checks ("improperly formatted code") use the vocabulary_formats validator patterns.
+  CPT4, CPT4_HCPCS, HCPCS, ICD10, ICD10CM and ICD9Proc changed: CPT4 now accepts M0201, HCPCS
+  needs a leading letter, and the ICD patterns need a real dot. ATC, DRG, ICD03_Morphology,
+  ICD03_Topography, NDC, RxNorm and SNOMED are checked for the first time. ICD10PCS accepts only 7-character billable codes
+  with a real section character first (row ICD10PCS.BILLABLE), so Z561123 is now improperly
+  formatted
+- A `cpt_or_hcpcs` code gets a warning when every vocabulary rejects it, for any reason. It is
+  "improperly formatted" only when every vocabulary says so, and otherwise "unknown"
+- `first` and `last` reject the `at_least` and `within` options with the validation error
+  "unsupported option". With either option they returned every upstream row instead of one row
+  per person. The two options are no longer in the `first` and `last` metadata. A blank value
+  counts as not set. `occurrence` keeps both options
+
+### Removed
+
+- Presto support (breaking). `ConceptQL::Rdbms::Presto` and `database_type: :presto` are gone, so
+  `ConceptQL::Rdbms.generate(:presto)` raises "Unknown database_type". Its `primary_concepts`
+  emitted `DISTINCT ON`, which Presto does not support. The Rakefile's Presto scaffolding
+  (`test_script` and `driftr_script` namespaces, `convert`, `prestofy`, the Presto `cdb`) is
+  removed too. The default rake task is now `test` instead of `test_omopv4_plus`
+- The `conceptql selection_operators` command, which had raised NameError since 2016.
+- The unused query optimizer (breaking for a caller of these methods). `Query#optimized`,
+  `Operator#optimized`, `Operator#unionable?`, `Operator#dup_values` and the
+  `ConceptQL::Vocabularies::Behaviors::Sourcish` module are gone. Nothing in conceptql called
+  `optimized`, so its union merging never ran, and the generated SQL does not change.
+- The `pass_thru` operator name, which never worked. `PassThru` had no query method, so a
+  statement that used it failed to generate SQL. A `pass_thru` statement is now an invalid
+  operator. The `PassThru` class stays as the base class of union, count, sum, numeric, dedup,
+  episode, time_window and co_reported.
+- `ConceptQL::Vocabularies::Entry#is_source?`, `Entry#is_standard?` and `Entry::STANDARD_VOCABS`
+  (breaking for a caller of these). Their only use was the `Sourcish` include removed with the
+  query optimizer. Neither t_shank nor jigsaw-diagram-editor calls them.
+- The `only_columns`, `except_columns` and `query_columns` options of `dedup` (breaking). No
+  statement could use them: with any column left out, the query failed, and from JSON they did
+  nothing or broke the SQL. Like any option an operator does not read, they are now ignored, and
+  dedup groups by every column.
+
+## [0.3.0] - 2016-11-08
+
+### Added
+
+- New operators
+  - PlaceOfServiceFilter
+  - Provenance
+  - ProviderFilter
+  - CoReported
+- Support for :impala_mem_limit option
+- Initial support for dynamic columns
+- Additional columns for observation domains and drug domains
+- Support for :within, :occurrences, :at_least
+
+### Changed
+
+- Format of this [CHANGELOG](http://keepachangelog.com/en/0.3.0/)
+- Read searches Condition, Procedure, Drug, Observation
+- Rewrote OneInTwoOut to produce faster queries
+- Except operator uses different strategy under Impala
+
+### Fixed
+
+- Fixed date casting and other issues for Impala compatibility.
+- Fixed ConditionType to accept concept_ids
+
+## [0.2.0] - 2015-12-11
 
 ### Added
+
 - Optimized After/Before nodes when multi-person results are in the right stream.
 - Set end_date as coalesce(end_date, start_date) to make range if end_date missing.
 - Nodes
-    - AnyOverlap
-    - Contains
-    - Filter
-    - Ndc
-    - ObservationPeriod
-    - OneInTwoOut
-    - OverlappedBy
-    - Overlaps
-    - TrimDateEnd
-    - TrimDateStart
+  - AnyOverlap
+  - Contains
+  - Filter
+  - Ndc
+  - ObservationPeriod
+  - OneInTwoOut
+  - OverlappedBy
+  - Overlaps
+  - TrimDateEnd
+  - TrimDateStart
 - Nodes for CPRD
-    - Medcode
-    - MedcodeProcedure
-    - ObservationByEnttype
-    - Prodcode
+  - Medcode
+  - MedcodeProcedure
+  - ObservationByEnttype
+  - Prodcode
 - Nodes for SEER
-    - FromSeerVisits
-    - ToSeerVisits
+  - FromSeerVisits
+  - ToSeerVisits
 - Ability to limit results to a set of patients by setting Tree#person_ids
 - "units_source_value" and "source_value" columns in results
 - TimeWindow supports date literals
@@ -32,197 +204,134 @@ All notable changes to this project will be documented in this file.
 - Nodifier#to_metadata
 
 ### Changed
+
 - Except now allows :ignore_date option
-    - Comparison is only done on criterion_id/type
+  - Comparison is only done on criterion_id/type
 - DateRange START/END use observation_period instead of visit_occurrence
 - Operator#columns value_as_numeric => value_as_number
 - Syntax is now more "lispy"
 - Recall now uses any labeled operator as if that operator was fed to "Define" operator
 - Nodes now called Operators
 
-### Deprecated
-- Nothing.
-
 ### Removed
+
 - Let/Define operators
 
 ### Fixed
+
 - Many broken specs.
 - Union calls #from_self on incoming streams to avoid column issues
 - Concept node now works again
 - Define ensures tables are built
 
-
-## 0.1.1 - 2014-09-18
-
-### Added
-- Nothing.
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
+## [0.1.1] - 2014-09-18
 
 ### Fixed
-- Calling Query#sql no longer creates a bunch of temporary tables.
 
+- Calling Query#sql no longer creates a bunch of temporary tables.
 
-## 0.1.0 - 2014-09-04
+## [0.1.0] - 2014-09-04
 
 ### Added
+
 - Support for numeric, string, and concept_ids returned in results.
 - Many updates to the ConceptQL Specification document.
 - Added doc/implementation_notes.md to capture thoughts and bad ideas.
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
+
 - "Fake" graphs are now drawn correctly.
 - bin/conceptql doesn't bomb out drawing "fake" graphs
 
-
-## 0.0.9 - 2014-09-03
+## [0.0.9] - 2014-09-03
 
 ### Added
-- Support for MSSQL (SQL Server).
-
-### Deprecated
-- Nothing.
 
-### Removed
-- Nothing.
+- Support for MSSQL (SQL Server).
 
 ### Fixed
-- Removed debug output from Node#namify
 
+- Removed debug output from Node#namify
 
-## 0.0.8 - 2014-08-28
+## [0.0.8] - 2014-08-28
 
 ### Added
-- Support for Oracle.
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
+- Support for Oracle.
 
 ### Fixed
+
 - DateAdjuster/TimeWindow use Sequel's date_arithmetic extension to produce database agnostic date manipulation.
 - Breakage from Node#tree in GraphNodifier.
 - All tests are back to passing.
 - Changed from SHA1 to CRC32 hash since Oracle can't handle table names longer than 30 characters.
 
-
-## 0.0.6 - 2014-08-23
+## [0.0.6] - 2014-08-23
 
 ### Added
+
 - Support for Oracle
 - Tree#defined to pass type information between Define and Recall.
 - Node#sql to produce SQL for each node.
 - Graph includes row count on each edge in the diagram.
 
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
 ### Fixed
+
 - Bug in CastingNode that generate SQL returning multiple columns in a subquery.
 - Made ruby-graphviz a dependency so calling programs don't bomb out.
 - Define now passes rows on through like any other node!
 - DateAdjuster/TimeWindow use Sequel's date_arithmetic extension to produce database agnostic date manipulation.
 - All tests are back to passing.
 
-
-## 0.0.5 - 2014-08-19
-
-### Added
-- Nothing.
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
+## [0.0.5] - 2014-08-19
 
 ### Fixed
-- Bug in GraphNodifier not displaying types for `recall` nodes.
 
+- Bug in GraphNodifier not displaying types for `recall` nodes.
 
-## 0.0.4 - 2014-08-19
+## [0.0.4] - 2014-08-19
 
 ### Added
+
 - Support for 5 instead of 13 column internal representation of results.
 - `define` node, used to create "variables" in ConceptQL.
 - `recall` node, used to pull results from "variables" in ConceptQL.
 
-### Deprecated
-- Nothing.
-
 ### Removed
+
 - Support for 13 column results.
 - Dependency on a set of views to run SQL queries.
 
 ### Fixed
-- Bug where `place_of_service_code` wasn't limited to vocabulary_id 14
 
+- Bug where `place_of_service_code` wasn't limited to vocabulary_id 14
 
-## 0.0.3 - 2014-08-12
+## [0.0.3] - 2014-08-12
 
 ### Added
+
 - FakeGrapher class to make it easier to generate diagrams with experimental nodes
 - fake_graph command has returned to the `conceptql` program
 - GraphNodifier now supports:
-    - condition_type as condition_occurrence
-    - drg as procedure_occurrence
-    - vsac as misc
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
+  - condition_type as condition_occurrence
+  - drg as procedure_occurrence
+  - vsac as misc
 
 ### Fixed
-- Tree now runs #deep_symbolize_keys on incoming statements
-
 
-## 0.0.2 - 2014-07-11
-
-### Added
-- Nothing.
+- Tree now runs #deep_symbolize_keys on incoming statements
 
-### Deprecated
-- Nothing.
+## [0.0.2] - 2014-07-11
 
 ### Removed
-- Several commands from `conceptql` program
-    - fake_graph
-    - show_db_graph
-    - show_and_tell_db
-
-### Fixed
-- Nothing.
 
+- Several commands from `conceptql` program
+  - fake_graph
+  - show_db_graph
+  - show_and_tell_db
 
-## 0.0.1 - 2014-07-11
+## [0.0.1] - 2014-07-11
 
 ### Added
-- This project.
-
-### Deprecated
-- Nothing.
-
-### Removed
-- Nothing.
-
-### Fixed
-- Nothing.
 
+- This project.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2014-2026 Outcomes Insights, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -1,26 +1,32 @@
-# ConceptQL
+# ConceptQL [![Run Tests](https://github.com/outcomesinsights/conceptql/actions/workflows/run_tests.yml/badge.svg)](https://github.com/outcomesinsights/conceptql/actions/workflows/run_tests.yml)
 
 ConceptQL (pronounced concept-Q-L) is a high-level language that allows researchers to unambiguously define their research algorithms.
 
-This gem interprets the ConceptQL language and translates it into SQL queries compatible with OMOP CDM-structured data.  The ConceptQL "language" is a set of nested hashes and arrays representing search criteria and some set operations and temporal operations to glue those criteria together.
+This gem interprets the ConceptQL language and translates it into SQL queries compatible with [Generalized Data Model (GDM)](https://github.com/outcomesinsights/generalized_data_model#contexts)-structured data. The ConceptQL "language" is a set of nested arrays and hashes representing search criteria with some set operations and temporal operations to glue those criteria together.
+
+## Further Reading
+
+If you're interested in reading up on ConceptQL, a the specifications document is [available in markdown format](https://github.com/outcomesinsights/conceptql_spec).
 
 ## Motivation for ConceptQL
-Outcomes Insights intends to build a vast library of research algorithms and apply those algorithms to large databases of claims data.  Early into building the library, we realized we had to overcome two major issues:
+
+Outcomes Insights has built a vast library of research algorithms and applied those algorithms to large databases of claims data. Early into building the library, we realized we had to overcome two major issues:
 
 1. Methods sections of research papers commonly use natural language to specify the criteria used to build cohorts from a claims database.
-    - Algorithms defined in natural language are often imprecise, open to multiple interpretations, and generally difficult to reproduce.
-    - Researchers could benefit from a language that removes the ambiguity of natural language while increasing the reproducibility of their research algorithms.
+   - Algorithms defined in natural language are often imprecise, open to multiple interpretations, and generally difficult to reproduce.
+   - Researchers could benefit from a language that removes the ambiguity of natural language while increasing the reproducibility of their research algorithms.
 2. Querying against claims databases is often difficult.
-    - Hand-coding algorithms to extract cohorts from datasets is time-consuming, error-prone, and opaque.
-    - Researchers could benefit from a language that allows algorithms to be defined at a high-level and then gets translated into the appropriate queries against a database.
+   - Hand-coding algorithms to extract cohorts from datasets is time-consuming, error-prone, and opaque.
+   - Researchers could benefit from a language that allows algorithms to be defined at a high-level and then gets translated into the appropriate queries against a database.
 
 We developed ConceptQL to address these two issues.
 
-We are writing a tool that can read research algorithms defined in ConceptQL.  The tool can create a diagram for the algorithm which makes it easy to visualize and understand.  The tool can also translate the algorithm into a SQL query which runs against data structured in [OMOP's Common Data Model (CDM)](http://omop.org/CDM).  The purpose of the CDM is to standardize the format and content of observational data, so standardized applications, tools and methods can be applied to them.
+We have written a tool that can read research algorithms defined in ConceptQL. The tool can create a diagram for the algorithm which makes it easy to visualize and understand. The tool can also translate the algorithm into a SQL query which runs against data structured in the [Generalized Data Model (GDM)](https://github.com/outcomesinsights/generalized_data_model). The purpose of the GDM is to standardize the format and content of observational data, so standardized applications, tools and methods can be applied to them.
 
 For instance, using ConceptQL we can take a statement that looks like this:
-```YAML
-:icd9: '412'
+
+```JSON
+[ "icd9", "412" ]
 ```
 
 And generate a diagram that looks like this:
@@ -28,88 +34,34 @@ And generate a diagram that looks like this:
 ![](doc/diagram_0.png)
 
 And generate SQL that looks like this:
+
 ```SQL
 SELECT *
-FROM cdm_data.condition_occurrence AS co
-JOIN vocabulary.source_to_concept_map AS scm ON (c.condition_concept_id = scm.target_concept_id)
-WHERE scm.source_code IN ('412')
-AND scm.source_vocabulary_id = 2
-AND scm.source_code = co.condition_source_value
+FROM gdm_data.clinical_codes AS cc
+WHERE cc.clinical_code_concept_id IN (
+  SELECT id
+  FROM concepts
+  WHERE vocabulary_id = 'ICD9CM'
+    AND concept_code = '412'
+)
 ```
 
-As stated above, one of the goals of ConcegtQL is to make it easy to assemble fairly complex queries without having to roll up our sleeves and write raw SQL.  To accommodate this complexity, ConceptQL itself has some complexities of its own.  That said, we believe ConceptQL will help researchers define, hone, and share their research algorithms.
-
-
-## Further Reading
-
-If you're interested in reading more about ConceptQL, a rought draft of the specifications document is [available as a PDF](https://github.com/outcomesinsights/conceptql/blob/master/doc/ConceptQL%20Specification%20(alpha).pdf?raw=true) in this repository.
-
-
-## Try Before You Buy
-
-If you'd like to interact with ConceptQL a bit before deciding to dive in, head over to the [ConceptQL Sandbox](http://sandbox.cohortjigsaw.com) for an online demonstration of the language and its features.
-
+As stated above, one of the goals of ConceptQL is to make it easy to assemble fairly complex queries without having to roll up our sleeves and write raw SQL. We believe ConceptQL will help researchers define, hone, and share their research algorithms.
 
 ## Requirements
 
-ConceptQL is in an early-alpha state.  For now it is limited to working with [OMOP CDM](http://omop.org/CDM)-structured data stored in the [PostgreSQL](http://www.postgresql.org/) database.  It has been tested under [Ubuntu Linux](http://www.ubuntu.com/) and Mac OS X 10.8+.  The interpreter is written in Ruby and theoretically should be platform independent, but your mileage may vary.
-
-Specifically, ConceptQL needs:
-
-- Ruby 2.0+
-- PostgresQL 9.3+
-- Ruby's [Bundler](http://bundler.io/) gem
-- A PostgreSQL database with
-    - OMOP CDM-structured data stored in the public schema
-    - The OMOP Vocabularies stored in a schema named 'vocabulary'
-        - You may want to use [loadmop](http://github.com/outcomesinsights/loadmop) to quickly create the OMOP vocabulary schema
-
-## Installation
-
-Add this line to your application's Gemfile:
+ConceptQL works best with data stored in the [GDM](https://github.com/outcomesinsights/generalized_data_model#contexts) using [PostgreSQL](http://www.postgresql.org/) as the RDBMS engine. It has been tested under [Ubuntu Linux](http://www.ubuntu.com/). The interpreter is written in Ruby and theoretically should be platform independent, but your mileage may vary.
 
-    gem 'conceptql'
+Specifically, ConceptQL is tested against:
 
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install conceptql
-
-## Usage
-
-The easiest way to try out ConceptQL locally is to use the [conceptql-dev-box](http://github.com/outcomesinsights/conceptql-dev-box)
-
-ConceptQL comes with a [Thor](http://whatisthor.com/)-based command-line program: `conceptql`
-
-`conceptql` allows you to translate a ConceptQL statement into a SQL statement or a graph depicting the ConceptQL statement.
-
-`conceptql` will print a basic help menu for each command if you run `conceptql`
-
-See [conceptql-dev-box](http://github.com/outcomesinsights/conceptql-dev-box) for more examples on how to play with ConceptQL
-
-## Writing Your Own ConceptQL
-
-Right now, the only way to create a new ConceptQL statement is to write it out by hand in either Ruby, JSON, or YAML.
-
-If you're feeling bold, feel free to try your hand at authoring a ConceptQL statement.
-
-You might want to just take an existing statement from the statements in [test_conceptql](https://github.com/outcomesinsights/test_conceptql) and modify that.
-
-## Contributing
-
-1. Fork it ( http://github.com/outcomesinsights/conceptql/fork )
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
-
-Or, use the [conceptql-dev-box](https://github.com/outcomesinsights/conceptql-dev-box) to get yourself setup and hack on ConceptQL that way.
+- Ruby 3.2
+- PostgresQL 14
 
 ## Thanks
+
 - [Outcomes Insights, Inc.](http://outins.com)
-    - Many thanks for allowing me to release a portion of my work as Open Source Software!
-- [OMOP](http://omop.org)/[OHDSI](http://www.ohdsi.org/)
-    - Thank you for providing a great [Common Data Model](http://omop.org/CDM) and fantastic set of standardized [Vocabularies](http://omop.org/Vocabularies) upon which ConceptQL rests.
+  - Many thanks for allowing me to release a portion of my work as Open Source Software!
+- [OHDSI](http://www.ohdsi.org/)
+  - Thank you for providing the original inspiration for ConceptQL.
+- [Jeremy Evans](http://code.jeremyevans.net/)
+  - Thank you for the great contributions, ideas, and commits.

data/bin/conceptql

@@ -1,5 +1,6 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
-require_relative '../lib/conceptql/cli.rb'
+require_relative '../lib/conceptql/cli'
 
 ConceptQL::CLI.start(ARGV)

data/bin/conceptql_utils

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative '../lib/conceptql/utils_cli'
+
+ConceptQL::UtilsCLI.start(ARGV)

data/config/multiple_vocabularies.csv

@@ -0,0 +1,4 @@
+operator,vocabulary_id,domain,optional
+CPT or HCPCS,CPT4,procedure_occurrence,
+CPT or HCPCS,HCPCS,procedure_occurrence,
+CPT or HCPCS,CPT4_HCPCS,procedure_occurrence,yes