conceptql 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf5cfc7db1200e2f00e6006a54f138e6837c1d05483330c7dccb5f9b8a93a907
-  data.tar.gz: 8095c96f350258266df92fcd521d99460a9ff48af70657b6b980294a29fa6a47
+  metadata.gz: 296389d035ad781649b32662a3fafba66aa209dea6c76fe039b372463854293f
+  data.tar.gz: 9e7225d0b68c8dd72a7b6b6ade8f3ffdb67b3b1c8cd6507e53e8d84a1fa0a6bc
 SHA512:
-  metadata.gz: 3a04d687527d5d808446f3fef305805a6915679c1c079dc1edf98afdb9fa7c650525e929b89ee227fe0e9066d58c81585e8c027ff18efc5a793c30bece1454cd
-  data.tar.gz: c4302550620621d1dc241c60a0a47fe4fe31b9ee0dc143d4bd467fb52a775aff5f46eeac815a1470ef661c0103e716e518610f04d55f8ec4b4975a6c4ba9d3f4
+  metadata.gz: fb10f21a65e9d4f3dfebd0fa559d1e8c5465d8e69ecf281d62494c981d51c5c70054e8e1b04782d0a15c1ebda762aef1b36e477d1a667cce266de5d8ce053a06
+  data.tar.gz: ceb2fd1d113b3fc37af25bb613e1405ad250a7d6452c2e710020df2fc9341c4398524add0f1b0344834e74348b6f78145a91b43e73f6347edeb6c01f159e3267

data/CHANGELOG.md

@@ -4,6 +4,161 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-11-10
+
+### Added
+
+- `ConceptQL::Annotation` reads a statement that `Query#annotate` annotated, or the same
+  statement parsed back from JSON. `#each_operator` gives each operator's name, effective id
+  and annotation hash. `#errors` and `#warnings` give every problem as
+  `{ operator:, id:, message:, details: }`, and `#valid?` is true when no operator has an
+  error. Warnings never make a statement invalid. It walks the positional upstreams and the
+  `left`, `right` and `window_table` options. `Query#errors`, `Query#warnings` and
+  `Query#valid?` use it on `annotate(skip_counts: true)`, so they run no count SQL. See
+  `docs/annotation.md`.
+- The `reference_lexicon:` option of `ConceptQL::Database` gives a Database the full lexicon
+  beside the lexicon of its dataset (`Database#reference_lexicon`). When the option is not
+  given and `LEXICON_URL` is set, the `LEXICON_URL` lexicon is the reference lexicon, with
+  its own cache. Only the code checks of the vocabulary operators, `information_periods`,
+  `provenance` and `place_of_service_filter` read it. A code that the
+  dataset's lexicon does not know gets the new warning "code(s) not present in this dataset"
+  when the reference lexicon knows it, and "unknown code(s)" when it does not. Before, every
+  such code was an unknown code: on SEER-Medicare, `M84.58XP` and `W56.22`, which OHDSI
+  has, were unknown codes. Each warning lists every code. The vocabulary operators
+  are also built from the reference lexicon's vocabularies, so an operator is not an
+  "invalid operator" only because the dataset lacks its vocabulary. The SQL and the counts
+  use the dataset's lexicon only, and do not change. See `docs/annotation.md`.
+- `Query#annotate` puts an effective id first in each operator's annotation hash: the
+  operator's `id` option, or a generated `<operator>-<n>` in statement order (`icd9-1`,
+  `icd9-2`) that never repeats a given id. The statement's options do not change.
+
+### Removed
+
+- **Breaking:** `Query#scope_annotate`, the CLI command `conceptql scope_annotate`, and the
+  scope hashes behind them (`Scope#annotation`, `#add_errors`, `#add_warnings` and
+  `#add_counts`). `Query#annotate`, `ConceptQL::Annotation` and `Query#errors`,
+  `#warnings` and `#valid?` are the only way to get errors, warnings and counts. The hash of
+  `scope_annotate` was keyed by the operator's name or `id` option, so of two operators with
+  the same key, only the last one's errors, warnings and counts were kept. The CLI command
+  `conceptql annotate_statement` remains.
+- **Breaking:** the GDM lexicon (`ConceptQL::LexiconGDM`), which read the GDM vocabulary
+  tables `concepts`, `ancestors`, `mappings` and `vocabularies`. The lexicon now uses only
+  the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and
+  `vocabulary`). A real data database whose search path, and whose `LEXICON_URL` database,
+  have only the GDM tables now raises `ConceptQL::Lexicon::NoVocabularyError`, and the
+  error lists only the missing OHDSI tables. The environment variables `LEXICON_GDM_ONLY`
+  and `LEXICON_OHDSI_ONLY` are removed. The test config `slim,gdm_vocabs` is removed from
+  CI and from `just test-full`, which now runs 2 Postgres configs. No production setup
+  read the GDM tables. The gdm data model stays.
+
+### Changed
+
+- **Breaking:** `provenance` and `place_of_service_filter` give a warning, not an error, for
+  a keyword or code that the dataset's lexicon does not know, so the statement stays valid
+  and runs. The unknown keyword or code selects no record. They check it as the vocabulary
+  operators do (`ConceptQL::Database#check_codes`): "code(s) not present in this dataset"
+  when the reference lexicon knows it, "unknown code(s)" when it does not, and no warning
+  when no lexicon has the vocabulary (the 2020 rule). Each warning lists every keyword or
+  code. A `file:code` keyword of `provenance` is checked half by half, each half against its
+  own vocabularies, and a keyword with more than one `:` is unknown. Before, `provenance`
+  gave the error "unrecognized keywords" and `place_of_service_filter` the error "unknown
+  code(s)", and the plain query raised `InvalidStatementError`. So a statement could be
+  valid on one dataset and invalid on another. The checks run when the Database has a
+  lexicon but its data database is nil or a Sequel mock, and `skip_db: true` turns them off.
+  `provenance` with no keyword is still an error.
+- **Breaking:** `cpt_or_hcpcs` checks its codes as one set across CPT4, HCPCS and
+  CPT4_HCPCS. A code is known when the dataset's lexicon has it in any of them. Of the other
+  codes, one that the reference lexicon has in any of them gets "code(s) not present in this
+  dataset", and the rest get "unknown code(s)". It accepts every code (the 2020 rule) only
+  when no lexicon of the Database has a concept of any of the three. Before, an empty CPT4 or
+  HCPCS accepted every code, so on
+  SEER-Medicare, which has only CPT4_HCPCS, `cpt_or_hcpcs` never warned about a code. When
+  `etl_vocabulary_ids` lists none of the three, `cpt_or_hcpcs` gets "the data has no records
+  of this vocabulary" once and null counts. Before, it got neither.
+- **Breaking:** a vocabulary that has no concepts in the Database's lexicon no longer accepts
+  every code when the reference lexicon has concepts of it. Its codes get "code(s) not
+  present in this dataset" or "unknown code(s)". The 2020 rule (#185) now applies only when
+  no lexicon of the Database has the vocabulary: the dataset's lexicon and, when it is
+  given, the reference lexicon. `etl_vocabulary_ids` does not change the rule.
+  `ConceptQL::Database#check_codes` applies it for the vocabulary operators, `cpt_or_hcpcs`
+  and `information_periods`. So `information_periods` also asks the reference lexicon: with
+  a reference lexicon that has GDM_INFORMATION_TYPE, an information type code that it does
+  not know is "unknown code(s)", even when the dataset's lexicon lacks the vocabulary. See
+  the table in `docs/annotation.md`.
+- A vocabulary operator that gets "the data has no records of this vocabulary" gets no
+  "code(s) not present in this dataset" warning, because that warning already says that none
+  of its codes is in the data. Its codes still get "unknown code(s)", "code(s) with more than
+  one meaning" and the format warning, so a typo is not hidden. `cpt_or_hcpcs` follows the
+  same rule for its set. So `["icd10pcs", "Z561123"]` on the test fixtures gives the
+  no-records warning and "unknown code(s)".
+- **Breaking:** a vocabulary operator whose vocabulary is not in the dataset's GDM table
+  `etl_vocabulary_ids` gets the warning "the data has no records of this vocabulary", and
+  `Query#annotate` gives it null counts (`{"rows": null, "n": null}`). An operator whose
+  inputs all have null counts also has null counts. Before, such an operator counted 0 rows
+  with no warning, which said that no patient had the codes: on SEER-Medicare, `icd9cm`
+  counted 0 although the data has no ICD9CM records. A Query reads `etl_vocabulary_ids`
+  once. A dataset without the table, or with an empty one, gives no warning and the counts
+  of before. Operators that read another table than the clinical codes (`death_type`,
+  `obs_period_type`) and `cpt_or_hcpcs` are not checked. `cpt` and `hcpcs` with the
+  CPT4_HCPCS warning, which has the same text, get no second warning.
+  `ConceptQL::Diagram` keeps null counts (before, it made them 0), and
+  `ConceptQL::DiagramRenderer` draws them as `rows=NA` and `n=NA`. See `docs/annotation.md`.
+- A vocabulary operator gives the unknown-code and the reused-code warnings, and
+  `information_periods` gives its unknown-code warning, when the Database has a lexicon but
+  its data database is nil or a Sequel mock. Before, these checks ran only with a real data
+  database, although they read only the lexicon, so the Jigsaw editor
+  (`ConceptQL::Database.new(lexicon_db)`) got no unknown-code warning. `skip_db: true` still
+  turns them off. The warning that the data stores CPT4 or HCPCS codes as CPT4_HCPCS reads the
+  data, so it still needs a real data database.
+- **Breaking:** `InvalidStatementError#errors` and `Query#validation_errors` are a list of
+  every error, in the form of `ConceptQL::Annotation#errors`
+  (`{ operator:, id:, message:, details: }`). Before, they were a hash of operator name =>
+  list of `[message, *details]`, which kept the errors of only one of two operators with the
+  same name. The message of `InvalidStatementError` has one line for each error, as before.
+  `validation_errors` reads the kept result of `annotate(skip_counts: true)`, so `#sql` runs
+  no count SQL and does not annotate again.
+- **Breaking:** `Query#annotate` of a window statement keeps the `window` operator at the top
+  of its output, with the window's options and an annotation hash that holds its id
+  (`window-1`). A window table that is a ConceptQL statement is annotated in place, in the
+  `window_table` option, so each of its errors stays on the operator that has it, and
+  `Annotation#valid?` is false. Before, `annotate` dropped the window operator, and the
+  window table's errors were only in `scope_annotate`, under the key `window_table`. An
+  invalid window table still makes `#sql` raise, and the outer statement is still not
+  counted. For example, `["window", ["icd9", "412"], {"window_table": ["provider_filter", ["icd9", "412"], {"specialties": "*"}]}]` now reports the error "required option not
+  present" (`roles`) on `provider_filter-1`.
+- `union` and `co_reported` with exactly one upstream get the warning
+  `has only one upstream`. With one upstream, each passes the records of that upstream
+  through unchanged, which is almost never what the author meant. The statement stays valid,
+  and an operator with no upstream keeps the error `has no upstream`. The check is the new
+  class-level declaration `validate_at_least_two_upstreams`. Binary operators (`left` and
+  `right`) do not get the warning. Before, a one-input `union` or `co_reported` had no message.
+- **Breaking:** `read` is a vocabulary operator, built from the new `Read` row of
+  `config/vocabularies.csv` like `icd9` or `ndc`, so it gets every vocabulary check. An
+  unknown Read code gets the warning `unknown code(s)` when the lexicon has the Read
+  vocabulary. So does a code in another case than the vocabulary stores, because Read codes
+  are case-sensitive: `13g4.00` and `13G4.00` are two codes. Before, `read` was the class
+  `ConceptQL::Operators::Read`, which wrapped one vocabulary operator, and it gave no
+  vocabulary warning. The class is removed. The records and the columns of `read` do not
+  change, and it keeps the lab columns (`value_as_number`, `range_high` and the others). A
+  `union` reads `read` in its own scan, so another vocabulary operator of the same domain in
+  the union keeps its own columns. In `ConceptQL.metadata`, `read` keeps its name, preferred
+  name `READ`, long name, description, category and domain. Its `short_name` is `READ`
+  (before, null), its `aliases` is an empty list (before, null), and its argument is `codes`
+  with the text of the other vocabulary operators (before, `read_codes`).
+
+### Fixed
+
+- The warning `code(s) with more than one meaning` compares codes as the code lookup does: in
+  their own case, unless the vocabulary's vocabulary_formats row declares `(?i:...)`. Before,
+  two codes of a statement that differ only in case counted as one code with the names of
+  both.
+
+- `Query#annotate` keeps one result for each options hash, and it annotates a fresh operator
+  tree for each new options hash. Before, a second call returned the first call's result
+  whatever its options, so `annotate` after `annotate(skip_counts: true)` gave zero counts.
+  A second call on a statement with an unknown operator whose `left` had a label also added a
+  false "duplicate label" error and changed the first result.
+
 ## [0.7.0] - 2026-11-09
 
 ### Added

data/README.md

@@ -10,6 +10,10 @@ The ConceptQL specification is [docs/spec/README.md](docs/spec/README.md).
 
 `ConceptQL::DiagramRenderer` draws a statement's diagram as an SVG or a PNG. See [docs/diagram_renderer.md](docs/diagram_renderer.md).
 
+`ConceptQL::Annotation` reads the errors, warnings and counts of an annotated statement. See [docs/annotation.md](docs/annotation.md).
+
+The `reference_lexicon:` option of `ConceptQL::Database` gives a Database the full lexicon beside the lexicon of its dataset. Then a code that the dataset does not contain gets the warning "code(s) not present in this dataset", not "unknown code(s)". See [docs/annotation.md](docs/annotation.md#codes-that-the-dataset-does-not-contain).
+
 ## Motivation for ConceptQL
 
 Outcomes Insights has built a vast library of research algorithms and applied those algorithms to large databases of claims data. Early into building the library, we realized we had to overcome two major issues:

data/config/vocabularies.csv

@@ -49,6 +49,7 @@ PLCSRVC,PLCSRVC,,,,
 Procedure Type,OMOP Procedure Occurrence Type,,procedure_occurrence,,
 PROVIDER_FACILITY_TYPE,PROVIDER_FACILITY_TYPE,,,,
 Race,Race and Ethnicity Code Set (USBC),,person,,
+Read,NHS UK Read Codes Version 2 (HSCIC),READ,condition_occurrence,,
 Relationship,OMOP Relationship,,,,
 Revenue Code,UB04/CMS1450 Revenue Codes (CMS),Revenue Code,procedure_occurrence,revenue_code,
 RxNorm,RxNorm (NLM),,drug_exposure,,

data/docs/spec/README.md

@@ -274,7 +274,7 @@ ConceptQL checks each statement before it builds SQL for it. A statement with an
 
 A warning does not make a statement invalid. Examples of warnings are an unknown code, an improperly formatted code and a deprecated operator. A statement with warnings runs. When ConceptQL draws the diagram of a statement, it shows the errors and the warnings of each operator, and it does not refuse an invalid statement.
 
-ConceptQL finds the concept of each code in the vocabulary tables. It uses either the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`) or the GDM vocabulary tables (`concepts`, `ancestors`, `mappings` and `vocabularies`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no records, and a statement would return no records with no warning. Without a database, for example when ConceptQL only draws a diagram, it uses no vocabulary tables.
+ConceptQL finds the concept of each code in the vocabulary tables. It uses the OHDSI vocabulary tables (`concept`, `concept_ancestor`, `concept_relationship` and `vocabulary`). It looks for them in the search path of the data database, and then in the database that the environment variable `LEXICON_URL` names. If ConceptQL has a real data database and finds no complete set of vocabulary tables, it refuses to start. Without vocabulary tables, every code would match no records, and a statement would return no records with no warning. Without a database, for example when ConceptQL only draws a diagram, it uses no vocabulary tables. The vocabulary tables of a dataset can be trimmed to the codes that the data contains. Then ConceptQL can also read a full set of vocabulary tables, the reference lexicon, but only to tell a code that the dataset does not contain from a code that does not exist. When `LEXICON_URL` is set and the program names no other reference lexicon, the `LEXICON_URL` database is the reference lexicon.
 
 ## Selection Operators
 
@@ -376,8 +376,10 @@ Examples of how codes match:
 - `410*` matches nothing, because `*` is a wildcard only as a code alone.
 - A code that does not have the vocabulary's format gives the warning "improperly formatted code".
 - A code that the vocabulary does not contain gives the warning "unknown code(s)".
+- When ConceptQL has a full lexicon (the reference lexicon) beside the vocabulary tables of the dataset, a code that only the full lexicon contains gives the warning "code(s) not present in this dataset", not "unknown code(s)". It is a warning, so the statement stays valid on a dataset that lacks some of its codes.
 - A code that has more than one meaning in the vocabulary gives the warning "code(s) with more than one meaning".
 - Some data stores CPT-4 and HCPCS codes only in the merged vocabulary CPT4_HCPCS. On that data, `cpt` and `hcpcs` select nothing, because they read only CPT4 and HCPCS. They give the warning "the data has no records of this vocabulary", which names `cpt_or_hcpcs`. Use `cpt_or_hcpcs`, which reads all three vocabularies.
+- When the data lists the vocabularies of its records (the GDM table `etl_vocabulary_ids`) and the operator's vocabulary is not in the list, the operator gives the warning "the data has no records of this vocabulary". Its counts are null, not 0, because the data cannot show whether a patient has the codes. This warning covers all of its codes, so they get no other warning, except the warning for an improperly formatted code.
 - `rxnorm` does not find records coded with NDC, because an operator does not follow mappings between vocabularies.
 
 ### DRG Codes
@@ -1884,7 +1886,7 @@ Passes along the records whose place of service is one of the given CMS Place of
 
 - **Upstream:** one operator.
 - **Arguments:**
-  - `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a record whose place of service is any code of that vocabulary. An unknown code makes the statement invalid.
+  - `places_of_service` (codelist): One or more CMS Place of Service codes, such as 21 (inpatient hospital) or 22 (outpatient hospital). The code * matches a record whose place of service is any code of that vocabulary. An unknown code gets a warning and matches no record.
 - **Returns:** The upstream records whose place of service is one of the given codes.
 
 ______________________________________________________________________
@@ -1909,7 +1911,7 @@ Passes along the records whose provenance matches at least one of the given keyw
 
 - **Upstream:** one operator.
 - **Arguments:**
-  - `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a record came from. A code keyword, such as primary or first, says where the code was on the claim. A plain keyword matches a record when its file provenance or its code provenance matches it. A keyword also matches the provenances under it, so inpatient includes snf. A keyword of the form file:code, such as inpatient:primary, needs both the file and the code provenance to match. The keyword * matches a record that has any provenance. A statement with no keyword, or with an unknown keyword, is invalid.
+  - `provenance_types` (codelist): One or more provenance keywords, in lower case. A file keyword, such as inpatient, outpatient or carrier_claim, says which kind of file a record came from. A code keyword, such as primary or first, says where the code was on the claim. A plain keyword matches a record when its file provenance or its code provenance matches it. A keyword also matches the provenances under it, so inpatient includes snf. A keyword of the form file:code, such as inpatient:primary, needs both the file and the code provenance to match. The keyword * matches a record that has any provenance. A statement with no keyword is invalid. An unknown keyword gets a warning and matches no record.
 - **Returns:** The upstream records whose provenance matches at least one of the keywords.
 
 ______________________________________________________________________

data/docs/spec/examples/recall-operator.json

@@ -191,10 +191,13 @@
                         "children": [],
                         "counts": {
                           "condition_occurrence": {
-                            "rows": 0,
-                            "n": 0
+                            "rows": null,
+                            "n": null
                           }
-                        }
+                        },
+                        "warnings": [
+                          ["the data has no records of this vocabulary"]
+                        ]
                       }
                     ],
                     "counts": {

data/lib/conceptql/annotation.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require "json"
+
+module ConceptQL
+  # Reads a statement that Query#annotate annotated: its operators, their
+  # errors and their warnings. It accepts the Ruby result of annotate and the
+  # same result parsed back from JSON (string keys), and gives the same answer
+  # for both. It reads the statement in its JSON form, so every hash it returns
+  # has string keys. See docs/annotation.md.
+  #
+  # An operator's inputs are its positional upstreams and its left, right and
+  # window_table options. A value is an operator only when it is an annotated
+  # operator: an Array that starts with a name and holds an annotation hash. So
+  # an option whose value is a list of strings is not an operator.
+  class Annotation
+    # One operator of the statement. #id is its effective id (see
+    # Annotation.assign_ids), and #annotation is its annotation hash.
+    Operator = Data.define(:name, :id, :annotation)
+
+    # The options that hold an operator's other inputs
+    INPUT_OPTIONS = %w[left right window_table].freeze
+
+    class << self
+      # Yields each operator node of an annotated statement and its annotation
+      # hash, in statement order: an operator, then its inputs in the order of
+      # its items.
+      def each_node(node, &block)
+        return enum_for(__method__, node) unless block
+
+        yield node, annotation_hash(node)
+        node.drop(1).each do |item|
+          if item.is_a?(Hash)
+            INPUT_OPTIONS.each do |key|
+              value = lookup(item, key)
+              each_node(value, &block) if operator?(value)
+            end
+          elsif operator?(item)
+            each_node(item, &block)
+          end
+        end
+      end
+
+      # Writes the effective id of each operator into its annotation hash, as
+      # the first key, :id. It is the operator's id option when the statement
+      # gives one. Otherwise it is "<operator>-<n>", where n counts the
+      # operators of that name in statement order (icd9-1, icd9-2), and
+      # skips any id that the statement gives. The options of the statement
+      # are not changed. Query#annotate calls it.
+      def assign_ids(annotated)
+        nodes = each_node(annotated).to_a
+        used = nodes.filter_map { |node, _| given_id(node)&.to_s }.to_set
+        counters = Hash.new(0)
+        nodes.each do |node, annotation|
+          id = given_id(node) || generate_id(node.first.to_s, counters, used)
+          annotation.replace({ id: id }.merge(annotation))
+        end
+        annotated
+      end
+
+      private
+
+      def generate_id(name, counters, used)
+        loop do
+          id = "#{name}-#{counters[name] += 1}"
+          return id if used.add?(id)
+        end
+      end
+
+      def given_id(node)
+        option(node, "id")
+      end
+
+      def annotation_hash(node)
+        option(node, "annotation")
+      end
+
+      def option(node, key)
+        node.drop(1).each do |item|
+          value = lookup(item, key) if item.is_a?(Hash)
+          return value unless value.nil?
+        end
+        nil
+      end
+
+      def operator?(value)
+        value.is_a?(Array) && (value.first.is_a?(String) || value.first.is_a?(Symbol)) &&
+          annotation_hash(value).is_a?(Hash)
+      end
+
+      def lookup(hash, key)
+        hash.fetch(key) { hash[key.to_sym] }
+      end
+    end
+
+    def initialize(annotated)
+      @annotated = JSON.parse(annotated.to_json)
+    end
+
+    # Every operator, root first, in statement order. Without a block, an
+    # Enumerator.
+    def each_operator
+      return enum_for(__method__) unless block_given?
+
+      self.class.each_node(@annotated) do |node, annotation|
+        yield Operator.new(name: node.first, id: annotation["id"], annotation: annotation)
+      end
+    end
+
+    # Every error, as { operator:, id:, message:, details: }
+    def errors
+      problems("errors")
+    end
+
+    # Every warning, in the same form as #errors
+    def warnings
+      problems("warnings")
+    end
+
+    # True when no operator has an error. Warnings do not make a statement
+    # invalid.
+    def valid?
+      errors.empty?
+    end
+
+    private
+
+    def problems(kind)
+      each_operator.flat_map do |op|
+        (op.annotation[kind] || []).map do |message, *details|
+          { operator: op.name, id: op.id, message: message, details: details }
+        end
+      end
+    end
+  end
+end

data/lib/conceptql/behaviors/provenanceable.rb

@@ -27,7 +27,7 @@ module ConceptQL
     end
 
     # No keyword at all is a validation error of Provenance. It still builds
-    # the condition of "*", as an invalid pair still builds a condition.
+    # the condition of "*", as an unknown pair still builds a condition.
     def build_where_from_codes(db, codes)
       return plain_condition(db, []) if codes.empty? || codes.include?(ANY)
 
@@ -39,19 +39,31 @@ module ConceptQL
       Sequel.|(*conditions)
     end
 
-    # The keywords as written that name no provenance concept. A pair is
-    # recognized only when it has one separator, a file keyword on the left
-    # and a code keyword on the right. Case is compared exactly, as the lookup
-    # compares it: the provenance vocabularies have no vocabulary_formats row
-    # that declares case-insensitivity (seed conceptql-oji ruling "b"). So
-    # "Inpatient" names no concept, and the concept codes are in lower case.
-    # "*" is known. It is not a half of a pair.
-    def find_bad_keywords(db, codes)
-      pairs, plain = (codes - [ ANY ]).partition { |code| pair_keyword?(code) }
-      known = known_keywords(db, all_provenance_types_vocab)
-      bad = plain.reject { |code| known.include?(code.to_s) }
-      bad += pairs.reject { |pair| known_pair?(db, pair) }
-      codes & bad
+    # The keywords as written that the dataset's lexicon does not know, as
+    # [not_present, unknown] in the order written (bead conceptql-amuu).
+    # Database#check_codes sorts them with the 2020 rule, as it sorts the
+    # codes of a vocabulary operator: a keyword that the reference lexicon
+    # knows is not present in this dataset, and the rest are unknown.
+    #
+    # A plain keyword is checked against the file and the code provenance
+    # vocabularies together. A pair is checked half by half: the file keyword
+    # against the file vocabularies and the code keyword against the code
+    # vocabularies. A pair with an unknown half is unknown, and else a pair
+    # with a half that is not present is not present. A pair needs exactly
+    # one separator, or it is unknown.
+    #
+    # Case is compared exactly, as the lookup compares it: the provenance
+    # vocabularies have no vocabulary_formats row that declares
+    # case-insensitivity (seed conceptql-oji ruling "b"). So "Inpatient" names
+    # no concept, and the concept codes are in lower case. "*" is known. It is
+    # not a half of a pair.
+    def check_keywords(db, codes)
+      pairs, plain = (codes.map(&:to_s).uniq - [ ANY ]).partition { |code| pair_keyword?(code) }
+      not_present, unknown = check_codes(db, all_provenance_types_vocab, plain)
+      pair_not_present, pair_unknown = check_pairs(db, pairs)
+      [ not_present + pair_not_present, unknown + pair_unknown ].map do |bad|
+        codes.select { |code| bad.include?(code.to_s) }.uniq
+      end
     end
 
     private
@@ -60,25 +72,31 @@ module ConceptQL
       code.to_s.include?(PAIR_SEPARATOR)
     end
 
-    def known_pair?(db, pair)
-      file, code, *rest = pair.split(PAIR_SEPARATOR, -1)
-      rest.empty? &&
-        known_keywords(db, dm.file_provenance_types_vocab).include?(file) &&
-        known_keywords(db, dm.code_provenance_types_vocab).include?(code)
+    # [not_present, unknown] of the pairs.
+    def check_pairs(db, pairs)
+      halves = pairs.to_h { |pair| [ pair, pair.split(PAIR_SEPARATOR, -1) ] }
+      malformed, formed = halves.partition { |_, parts| parts.size != 2 }.map(&:to_h)
+      files_not_present, files_unknown = check_codes(db, dm.file_provenance_types_vocab, formed.values.map(&:first).uniq)
+      codes_not_present, codes_unknown = check_codes(db, dm.code_provenance_types_vocab, formed.values.map(&:last).uniq)
+      unknown = formed.select { |_, (file, code)| files_unknown.include?(file) || codes_unknown.include?(code) }.keys
+      not_present = formed.select { |_, (file, code)| files_not_present.include?(file) || codes_not_present.include?(code) }.keys
+      [ not_present - unknown, malformed.keys + unknown ]
     end
 
-    # The concept codes of the vocabularies.
-    def known_keywords(db, vocabularies)
-      dm.concept_codes(db, vocabularies)
+    # Database#check_codes, which is not asked about no codes.
+    def check_codes(db, vocabularies, codes)
+      return [ [], [] ] if codes.empty?
+
+      nodifier.cdb.check_codes(db, vocabularies, codes)
     end
 
     def all_provenance_types_vocab
       [ dm.file_provenance_types_vocab, dm.code_provenance_types_vocab ].flatten
     end
 
-    # The keywords come from the statement, whose validation reports a bad
-    # one (find_bad_keywords), so the lookup does not raise
-    # CodeCaseMismatchError.
+    # The keywords come from the statement, whose validation warns about a
+    # bad one (check_keywords), so the lookup does not raise
+    # CodeCaseMismatchError. A keyword that names no concept selects nothing.
     def prov_of(db, vocabularies, ancestors)
       dm.descendants_of(db, dm.concept_ids(db, vocabularies, ancestors, check_case: false))
     end
@@ -90,7 +108,7 @@ module ConceptQL
       )
     end
 
-    # An invalid pair (validation reports it) still builds a condition. A
+    # An unknown pair (validation warns about it) still builds a condition. A
     # half with no concept matches nothing, so the pair keeps no record.
     def pair_condition(db, pair)
       file, code = pair.split(PAIR_SEPARATOR, 2)

data/lib/conceptql/cli.rb

@@ -77,12 +77,6 @@ module ConceptQL
       pp q.annotate(skip_counts: true)
     end
 
-    desc "scope_annotate", "Reads in a statement and annotates it"
-    def scope_annotate(statement_file)
-      q = ConceptQL::Query.new(cdb(options), criteria_from_file(statement_file))
-      pp q.scope_annotate(skip_counts: true)
-    end
-
     desc "render_json STATEMENT_FILE",
          "Prints the statement(s) as a #{ConceptQL::Diagram::FORMAT} render tree (JSON) for the diagram renderer"
     long_desc <<~DESC

data/lib/conceptql/database.rb

@@ -2,6 +2,8 @@
 
 require "digest"
 require "logger"
+# Database#db_extensions names Sequel::Mock::Database before any mock exists
+require "sequel/adapters/mock"
 require_relative "lexicon"
 
 module ConceptQL
@@ -37,6 +39,7 @@ module ConceptQL
       if @lexicon && @opts[:cache_lookups]
         raise ArgumentError, "cache_lookups needs the Database to build its own Lexicon; do not pass lexicon:"
       end
+      @reference_lexicon = opts[:reference_lexicon] if opts[:reference_lexicon]
 
       @opts[:database_type] ||= (ENV["CONCEPTQL_DATABASE_TYPE"] || db_type).to_sym
       @opts[:scope_opts] = {
@@ -135,11 +138,78 @@ module ConceptQL
       @lexicon ||= Lexicon.new(self.class.lexicon_db, db, cache: @opts[:cache_lookups])
     end
 
+    # The reference lexicon: the full lexicon, beside the lexicon of the
+    # dataset (#lexicon), which can be trimmed to the codes that the data
+    # contains (bead conceptql-8oim, seed conceptql-d9w). Only the code checks
+    # (#check_codes) read it: a code that the dataset's lexicon does not know
+    # is "not present in this dataset" when the reference lexicon knows it,
+    # and an unknown code when it does not. Code-to-concept
+    # translation, the SQL and the counts use #lexicon only. The vocabulary
+    # operators are built from both, so an operator is valid whatever the
+    # dataset contains.
+    #
+    # It is the reference_lexicon: option, or else, when LEXICON_URL is set,
+    # a Lexicon of the LEXICON_URL database (Database.lexicon_db) with its own
+    # cache. It is nil when neither exists. A LEXICON_URL database without
+    # vocabulary tables raises.
+    def reference_lexicon
+      return @reference_lexicon if defined?(@reference_lexicon)
+
+      @reference_lexicon = (lexicon_url_lexicon(self.class.lexicon_db) if self.class.lexicon_db)
+    end
+
+    # The code check that every operator with codes calls (bead
+    # conceptql-dufi, seed conceptql-d9w). It returns [not_present, unknown]:
+    # of the codes that #lexicon does not know in any of the vocabularies,
+    # those that #reference_lexicon knows in any of them, and the rest. The
+    # caller gives "code(s) not present in this dataset" for the first and
+    # "unknown code(s)" for the second.
+    #
+    # The 2020 rule (refs #185): when no lexicon of the Database has a
+    # concept of any of the vocabularies, every code is accepted and both
+    # lists are empty. A lexicon that lacks a vocabulary cannot tell a real
+    # code from a typo. So with a reference lexicon, the rule applies only
+    # when both lexicons lack the vocabularies, and without one, when
+    # #lexicon lacks them. etl_vocabulary_ids does not change the rule.
+    #
+    # A code matches in its own case: a statement warns about a code in
+    # another case instead of raising CodeCaseMismatchError.
+    def check_codes(data_db, vocabulary_ids, codes)
+      vocabulary_ids = Array(vocabulary_ids).uniq
+      missing = codes - known_codes_in(lexicon, data_db, vocabulary_ids, codes)
+      return [ [], [] ] if missing.empty?
+
+      lexicons = [ lexicon, reference_lexicon ].compact
+      return [ [], [] ] if lexicons.all? { |lex| vocabulary_ids.all? { |id| lex.vocabulary_is_empty?(data_db, id) } }
+
+      not_present = reference_lexicon ? known_codes_in(reference_lexicon, data_db, vocabulary_ids, missing) : []
+      [ missing & not_present, missing - not_present ]
+    end
+
+    # The codes that a lexicon has in any of the vocabularies, with no
+    # exception for an empty vocabulary.
+    def known_codes_in(lex, data_db, vocabulary_ids, codes)
+      vocabulary_ids.flat_map do |vocabulary_id|
+        lex.known_codes(data_db, vocabulary_id, codes, check_case: false, empty_means_known: false)
+      end.uniq
+    end
+    private :known_codes_in
+
+    def lexicon_url_lexicon(lexicon_db)
+      reference = Lexicon.new(lexicon_db, nil, cache: true)
+      return reference unless reference.strategy == :no_db
+
+      raise Lexicon::NoVocabularyError, "the LEXICON_URL database has no vocabulary tables, " \
+                                        "so it cannot be the reference lexicon"
+    end
+    private :lexicon_url_lexicon
+
     # This Database's operators for a data model, as a frozen name => class
     # Hash: a vocabulary operator for every vocabulary in config/vocabularies.csv
-    # (plus the custom file) and in this Database's lexicon, with the built-in
-    # operators merged over them. A built-in wins a name clash (gender, race and
-    # ethnicity are also vocabularies). Built lazily, once per data model.
+    # (plus the custom file), in this Database's lexicon and in its reference
+    # lexicon, with the built-in operators merged over them. A built-in wins a
+    # name clash (gender, race and ethnicity are also vocabularies). Built
+    # lazily, once per data model.
     #
     # The data model is an argument because a Nodifier can be given one that
     # differs from this Database's.
@@ -148,7 +218,7 @@ module ConceptQL
       ConceptQL::DataModel.assert_supported!(data_model)
       @operators ||= {}
       @operators[data_model] ||= begin
-        vocabulary_operators = Vocabularies::DynamicVocabularies.new(lexicon).operators(data_model)
+        vocabulary_operators = Vocabularies::DynamicVocabularies.new(lexicon, reference_lexicon).operators(data_model)
         vocabulary_operators.merge(Operators.static_operators.fetch(data_model)).freeze
       end
     end

data/lib/conceptql/diagram.rb

@@ -107,10 +107,13 @@ module ConceptQL
         (annotation[:counts] || {}).keys.flatten.map(&:to_s).uniq.sort
       end
 
+      # A null count stays null: the data has no records of the operator's
+      # vocabulary, so the count is unknown, not 0 (bead conceptql-d1bd).
+      # DiagramRenderer draws it as NA.
       def counts(annotation)
         (annotation[:counts] || {})
           .sort_by { |domain, _| domain.to_s }
-          .to_h { |domain, c| [ domain.to_s, { rows: Integer(c[:rows] || 0), n: Integer(c[:n] || 0) } ] }
+          .to_h { |domain, c| [ domain.to_s, { rows: c[:rows]&.then { Integer(_1) }, n: c[:n]&.then { Integer(_1) } } ] }
       end
 
       def metadata_for(name)