tablassert 7.4.14__tar.gz → 7.5.0__tar.gz

{tablassert-7.4.14 → tablassert-7.5.0}/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 All notable changes to this project are documented in this file.
 
+## 7.5.0 - 2026-07-01
+
+### Changed
+- Publication CURIEs in `compile_subgraph()` (`lib.py`) now use the `PMCID:` namespace prefix for PubMed Central sources. A `repo: PMC` section with `publication: PMC11708054` is emitted as `PMCID:PMC11708054` (previously `PMC:PMC11708054`); non-PMC repos such as `PMID` are unaffected and continue to emit `<repo>:<publication>` (e.g., `PMID:11708054`). The `repository` edge column is unchanged and still records the raw `repo` value. Extracted via a new `publication_curie()` helper.
+
+### Added
+- New `<col> table literal value` edge column for subject, object, and qualifier nodes encoded with `method: column`. Unlike the existing `original <col>` column (which snapshots the value *after* all `fill`/`explode_by`/`regex`/`remove`/`prefix`/`suffix`/`transformations`), `<col> table literal value` captures the pristine source-cell value *before* any transformation. Emitted only for column-encoded nodes; annotations and `method: value` nodes are unaffected. Implemented via a `table_literal` flag on `Tcode.encoding()`, enabled by `Tcode.node()`.
+- Four regression tests in `test_lib.py`: `publication_curie()` for PMC and PMID namespaces, and two `Tcode` tcode-inspection tests covering presence/ordering of the table-literal column for column encodings and its absence for value encodings.
+
+### Documentation
+- Comprehensive accuracy pass across the API, configuration, and Docker documentation, reconciling every page against the current codebase. Highlights: corrected invalid examples that would not load (`syntax: TC2`; `publication` integers and missing `PMC` prefixes; a non-existent `Qualifiers` member; `reindex` placed at section level; a subject missing `method: column`), fixed wrong field types (`rows`/`row_slice`/`taxon` → `PositiveInt`, `remove` → regex patterns), corrected the QC fuzzy thresholds (`fuzz.ratio >= 20 OR partial_token_sort_ratio >= 30`), removed a non-existent `uuid:` prefix from `utils.md` return examples, fixed the `resolve_many()` parameter order and added the original-column-capture and optional QC-audit pipeline steps, corrected graph-config path resolution (CWD, not config-relative) and processing-flow ordering, documented the strict QC GPU no-fallback behavior and the `.cachassert/` working directory, and aligned `Categories` enum member names (`GENE`/`PROTEIN`) and Docker CI triggers with the source.
+
 ## 7.4.14 - 2026-06-30
 
 ### Changes

{tablassert-7.4.14 → tablassert-7.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tablassert
-Version: 7.4.14
+Version: 7.5.0
 Summary: Extract knowledge assertions from tabular data into NCATS Translator-compliant KGX NDJSON — declaratively, with entity resolution and quality control built in.
 Project-URL: Homepage, https://github.com/SkyeAv/Tablassert
 Project-URL: Source, https://github.com/SkyeAv/Tablassert

{tablassert-7.4.14 → tablassert-7.5.0}/docs/api/lib.md

@@ -2,7 +2,7 @@
 
 The `lib` module exposes `resolve_many()`, a high-level convenience function for resolving an iterable of entity strings to CURIEs without requiring manual LazyFrame construction, NLP preprocessing, or DuckDB shard management.
 
-It wraps the lower-level [`resolve()`](fullmap.md) pipeline — applying `level_one` and `level_two` normalization, opening all 10 DuckDB shard connections, executing entity resolution, and returning results as a plain Python list of row dictionaries.
+It wraps the lower-level [`resolve()`](fullmap.md) pipeline — preserving the original input text, applying `level_one` and `level_two` normalization, opening all 10 DuckDB shard connections, executing entity resolution, optionally running the QC audit (when `qc=True`), and returning results as a plain Python list of row dictionaries.
 
 ## resolve_many()
 
@@ -18,8 +18,8 @@ def resolve_many(
     taxon: Optional[str] = None,
     prioritize: Optional[list[Categories]] = None,
     avoid: Optional[list[Categories]] = None,
-    column_context: bool = True,
     qc: bool = False,
+    column_context: bool = True,
 ) -> list[dict[str, Any]]
 ```
 
@@ -52,29 +52,29 @@ Each shard contains:
 
 Optional NCBI Taxon ID for filtering results to a specific organism.
 
-Example: `"9606"` restricts matches to human-specific entities. When `None`, no taxon filtering is applied and matches from all organisms are returned.
+Example: `"9606"` restricts **gene** matches to human-specific entries; non-gene categories (e.g., diseases, chemicals) are returned regardless of taxon. When `None`, no taxon filtering is applied and matches from all organisms are returned.
 
 **`prioritize: Optional[list[Categories]]` (default: `None`)**
 
 Optional list of Biolink categories to prefer when multiple matches exist for the same input term. Categories listed here receive higher ranking scores during resolution.
 
-Example: `[Categories.Gene, Categories.Protein]` prefers gene and protein mappings over other categories like diseases or chemicals.
+Example: `[Categories.GENE, Categories.PROTEIN]` prefers gene and protein mappings over other categories like diseases or chemicals.
 
 **`avoid: Optional[list[Categories]]` (default: `None`)**
 
 Optional list of Biolink categories to exclude from results entirely. Any match belonging to an avoided category is filtered out before ranking.
 
-Example: `[Categories.Gene]` prevents gene mappings from appearing in the output, even if they would otherwise be the best match.
+Example: `[Categories.GENE]` prevents gene mappings from appearing in the output, even if they would otherwise be the best match.
 
 **`column_context: bool` (default: `True`)**
 
-Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the resolution query adds a category frequency score and prefers the category that appears most frequently across all terms in the batch. When `False`, frequency-based tie-breaking is disabled.
+Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the deduplication stage adds a category-frequency score (computed in Polars after the SQL query) and prefers the category that appears most frequently across all matched terms in the batch. When `False`, frequency-based tie-breaking is disabled.
 
 This is useful when resolving a column of related entities (e.g., all genes) — the shared context helps disambiguate terms that map to multiple categories.
 
 **`qc: bool` (default: `False`)**
 
-When `True`, runs the QC audit stage after entity resolution. The QC pipeline validates mappings through a three-stage audit: exact match, fuzzy matching via rapidfuzz, and BioBERT sentence embeddings with cosine similarity. Requires a QC runtime to be installed (`tablassert[qc]` or `tablassert[qc-cuda]`). The ONNX Runtime provider is auto-detected based on the installed package — CUDA is preferred when `onnxruntime-gpu` is available, otherwise CPU is used.
+When `True`, runs the QC audit stage after entity resolution. The QC pipeline validates mappings through a three-stage audit: exact match, fuzzy matching via rapidfuzz, and BioBERT sentence embeddings with cosine similarity. Mappings that fail all three stages are dropped from the returned list (in addition to the unresolved-entity filtering performed by `resolve()`). Requires a QC runtime to be installed (`tablassert[qc]` or `tablassert[qc-cuda]`). The ONNX Runtime provider is auto-detected based on the installed package — CUDA is preferred when `onnxruntime-gpu` is installed and `CUDAExecutionProvider` is available; if only `onnxruntime` (CPU) is installed, CPU is used. Note: if `onnxruntime-gpu` is installed but `CUDAExecutionProvider` is unavailable, QC raises rather than falling back to CPU.
 
 ### Return Value
 
@@ -101,13 +101,17 @@ Each dictionary contains the following keys (where `{col}` is the value of the `
 
 1. **Series construction** — Wraps the input iterable in a `pl.Series` with the given column name, then converts to a single-column `pl.LazyFrame`.
 
-2. **NLP normalization** — Applies `level_one()` (whitespace stripping + lowercasing) and `level_two()` (non-word character removal via `\W+`) to produce the two normalized columns required by `resolve()`.
+2. **Original column capture** — Copies the raw input column into `original {col}` via `column(lf, add("original ", col), col)` so the pre-normalization text is preserved in the output.
+
+3. **NLP normalization** — Applies `level_one()` (whitespace stripping + lowercasing) and `level_two()` (non-word character removal via `\W+`) to produce the two normalized columns required by `resolve()`.
+
+4. **DuckDB connection management** — Opens all 10 shard connections inside a `contextlib.ExitStack`, ensuring every connection is properly closed when resolution completes or if an error occurs.
 
-3. **DuckDB connection management** — Opens all 10 shard connections inside a `contextlib.ExitStack`, ensuring every connection is properly closed when resolution completes or if an error occurs.
+5. **Entity resolution** — Delegates to `fullmap.resolve()` which queries the sharded DuckDB database, ranks matches by category priority, preferred-name exactness, NLP level, and category frequency, then deduplicates to one CURIE per input string.
 
-4. **Entity resolution** — Delegates to `fullmap.resolve()` which queries the sharded DuckDB database, ranks matches by category priority, preferred-name exactness, NLP level, and category frequency, then deduplicates to one CURIE per input string.
+6. **QC audit (optional)** — When `qc=True`, runs `fullmap_audit()` on the resolved LazyFrame. Rows that fail all three audit stages are dropped from the result.
 
-5. **Collection and conversion** — Collects the lazy result into an eager `pl.DataFrame` and converts to a list of row dictionaries via `to_dicts()`.
+7. **Collection and conversion** — Collects the lazy result into an eager `pl.DataFrame` and converts to a list of row dictionaries via `to_dicts()`.
 
 ### Example Usage
 
@@ -126,7 +130,7 @@ result: list[dict[str, Any]] = resolve_many(
     entities=["TP53", "BRCA1", "EGFR", "KRAS"],
     datassert=datassert,
     taxon="9606",
-    prioritize=[Categories.Gene],
+    prioritize=[Categories.GENE],
 )
 
 # result[0] → {"original gene": "TP53", "gene": "HGNC:11998", "gene name": "TP53", ...}
@@ -147,7 +151,7 @@ result: list[dict[str, Any]] = resolve_many(
     col="disease",
     entities=["diabetes mellitus", "breast cancer", "alzheimer disease"],
     datassert=datassert,
-    avoid=[Categories.Gene, Categories.Protein],
+    avoid=[Categories.GENE, Categories.PROTEIN],
 )
 
 # result[0] → {"original disease": "diabetes mellitus", "disease": "MONDO:0005015", ...}
@@ -206,7 +210,7 @@ for row in result:
 | **Connections** | Managed internally via `ExitStack` | Must be opened externally |
 | **Output** | `list[dict[str, Any]]` | `pl.LazyFrame` |
 | **Logging** | Uses default (`log=True`) | Configurable |
-| **Context params** | Not exposed (`section_hash`, `config_file`, `tag`) | Fully configurable |
+| **Context params** | `column_context` exposed; `section_hash`, `config_file`, `tag` not exposed | Fully configurable |
 | **Use case** | Standalone batch lookups, scripting, notebooks | Internal pipeline integration |
 
 `resolve_many()` is designed for ad-hoc and programmatic use — scripts, notebooks, and one-off lookups. For pipeline integration where you need full control over logging, context metadata, and lazy evaluation, use `resolve()` directly.

{tablassert-7.4.14 → tablassert-7.5.0}/docs/api/qc.md

@@ -2,7 +2,7 @@
 
 The `qc` module validates entity resolution mappings through a multi-stage pipeline: exact matching, fuzzy matching, and BERT semantic similarity.
 
-QC runtime support is optional. Install `tablassert[qc]` for CPU inference or `tablassert[qc-cuda]` for CUDA inference on GPU 0.
+QC runtime support is optional. Install `tablassert[qc]` for CPU inference or `tablassert[qc-cuda]` for CUDA inference on GPU 0. **Strict GPU behavior:** if `onnxruntime-gpu` is installed but `CUDAExecutionProvider` is not available (e.g., a broken CUDA/cuDNN environment), `fullmap_audit()` raises a `RuntimeError` rather than silently falling back to CPU — install `tablassert[qc]` for CPU inference or fix the CUDA environment.
 
 ## fullmap_audit()
 
@@ -84,7 +84,7 @@ original == preferred_name
 
 **Performance:** O(1) string comparison
 
-Before fuzzy matching, the function also applies rule-based pass-through checks for known safe patterns (for example CHEBI/PR/UniProtKB CURIE families and selected exception prefixes).
+Before fuzzy matching, the function also applies rule-based pass-through checks: (1) the resolved CURIE matches an exempt prefix (`CHEBI`, `PR`, `UniProtKB`, `NCBIGene`, `UMLS`, `UNII`, `PUBCHEM`, `MONDO`); (2) the original text contains `:` (looks like a CURIE); (3) the preferred name matches an exception prefix (`^LOC` or `^si:`).
 
 #### Stage 2: Fuzzy Matching
 
@@ -94,11 +94,11 @@ Two fuzzy matching algorithms:
 1. **Ratio:** Overall string similarity
 2. **Partial token sort ratio:** Combined token/subsequence matching
 
-**Threshold:** 20% similarity
+**Thresholds:** `fuzz.ratio` >= 20 OR `fuzz.partial_token_sort_ratio` >= 30
 
 ```python
 fuzz.ratio(original, preferred) >= 20
-or fuzz.partial_token_sort_ratio(original, preferred) >= 20
+or fuzz.partial_token_sort_ratio(original, preferred) >= 30
 ```
 
 **Example passes:**
@@ -116,8 +116,8 @@ or fuzz.partial_token_sort_ratio(original, preferred) >= 20
 3. **Accept** if similarity >= 0.2 (20%)
 
 ```python
-embeddings = BIOBERT.encode([original, preferred])
-similarity = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
+embeddings = get_biobert(provider).encode(originals + preferreds)
+similarity = cosine_similarity(embeddings[:n], embeddings[n:]).diagonal()
 return similarity >= 0.2
 ```
 

{tablassert-7.4.14 → tablassert-7.5.0}/docs/api/utils.md

@@ -19,10 +19,8 @@ def namespace_uuid(domain: Any, *values: list[Any]) -> str
 Domain string used to create the namespace UUID.
 
 Converted to string internally. Common domains:
-- `"TABLASSERT"` - Default domain used for knowledge graph edge IDs
-- `"edges"` - Optional custom domain for edge IDs
-- `"nodes"` - For custom node IDs
-- `"tablassert"` - For application-specific IDs
+- `"TABLASSERT"` - Default domain used for KGX edge IDs (the value used by `label_edge`)
+- Other domain strings (e.g., `"edges"`, `"nodes"`) may be passed for custom namespaces, though only `"TABLASSERT"` is used internally.
 
 **`*values: list[Any]`**
 
@@ -118,7 +116,7 @@ edge_id = namespace_uuid(
   "MONDO:0005148",  # Type 2 diabetes
   "PMC11708054"  # Publication
 )
-# Returns: "uuid:a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+# Returns: "2cfea591-0f8f-33af-a7df-03da531d3359"
 ```
 
 **Benefits:**
@@ -145,7 +143,7 @@ from tablassert.utils import namespace_uuid
 
 # Generate edge ID
 edge_id = namespace_uuid("edges", "subject", "predicate", "object")
-print(edge_id)  # "uuid:12345678-1234-1234-1234-123456789abc"
+print(edge_id)  # "12345678-1234-1234-1234-123456789abc"
 ```
 
 **With qualifiers:**
@@ -183,12 +181,33 @@ Suitable for millions of ID generations per second.
 **`basespace(domain)`** - Creates namespace UUID from domain
 
 ```python
+@cache
 def basespace(domain: str) -> UUID:
   namespace = UUID("00000000-0000-0000-0000-000000000000")
   return uuid3(namespace, domain)
 ```
 
-Used internally by `namespace_uuid()`.
+Used internally by `namespace_uuid()`. Results are memoized per domain.
+
+## mkhash()
+
+Generates an `xxhash` 64-bit digest (hex string, 16 characters) for arbitrary input.
+
+```python
+def mkhash(x: Any) -> str
+```
+
+Used for compact identifier hashing (e.g., config/file identifiers in the CLI). Returns the `xxh64` hex digest of the input converted to a string.
+
+## samphash()
+
+Hashes a sampled `polars.DataFrame` for deterministic temp-file naming.
+
+```python
+def samphash(df: pl.DataFrame, n: int = 20) -> str
+```
+
+Samples up to `n` rows (default `20`) from `df`, hashes the sampled bytes via `mkhash()`, and returns the digest string.
 
 ## Next Steps
 

{tablassert-7.4.14 → tablassert-7.5.0}/docs/cli.md

@@ -26,7 +26,7 @@ Prints the installed Tablassert version to stdout and exits. This is a flag on t
 
 ## build
 
-Build A KGX Compliant Knowledge Graph From A Graph Configuration File
+Build a knowledge graph from a YAML configuration file.
 
 ### Synopsis
 
@@ -70,7 +70,7 @@ See [Graph Configuration](configuration/graph.md) for details on the YAML schema
 
 ## validate
 
-Verify The Syntax Of A Declarative Table Configuration File
+Validate section syntax from a YAML configuration file.
 
 ### Synopsis
 

{tablassert-7.4.14 → tablassert-7.5.0}/docs/configuration/advanced-example.md

@@ -23,7 +23,7 @@ template:
     local: ./DATALAKE/ALAM.XLSX
     url: https://pmc.ncbi.nlm.nih.gov/articles/instance/11708054/bin/mbio.01679-24-s0006.xlsx
     row_slice:
-      - 2  # Skip first row (header)
+      - 2  # Skip the first two rows (title + header)
       - auto  # Read to end
     sheet: all correlations
 
@@ -39,9 +39,9 @@ template:
       avoid:
         - Gene
 
-      # Remove invalid entries
+      # Strip non-value text in place (rows are not dropped)
       remove:
-        - "^NA "  # Rows starting with "NA "
+        - "^NA "  # Cells starting with "NA "
 
       # Clean taxonomic names with regex
       regex:
@@ -66,7 +66,7 @@ template:
   # Provenance: Publication and curation info
   provenance:
     repo: PMC
-    publication: 11708054
+    publication: PMC11708054
     contributors:
       - kind: curation
         name: Skye Lane Goetz
@@ -147,7 +147,7 @@ The subject field uses three regex transformations in sequence:
 ```
 `"Lactobacillus sp"` → `"Lactobacillus sp. "`
 
-> **Regex constraint:** Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Capturing groups (`(...)` / `\1`) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?<!...)`) are not allowed** and will fail validation. Express transformations as a sequence of simple anchored / character-class substitutions instead — the pipeline above is a deliberate three-step chain because no single capturing-group pattern is permitted. If the transformation can't be expressed without those features, capture the leftover context in a `miscellaneous notes` annotation rather than fighting the regex engine.
+> **Regex constraint:** Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Backreferences (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?<!...)`) are not allowed** and will fail validation. Plain groups `(...)` and non-capturing groups `(?:...)` *are* supported. Express transformations as a sequence of simple anchored / character-class substitutions where possible — the pipeline above is a deliberate three-step chain. If the transformation can't be expressed without those unsupported features, capture the leftover context in a `miscellaneous notes` annotation rather than fighting the regex engine.
 
 ### Taxonomic Filtering
 
@@ -226,6 +226,7 @@ template:
 
   statement:
     subject:
+      method: column
       encoding: A
       prioritize: [OrganismTaxon]
       avoid: [Gene]
@@ -239,18 +240,20 @@ sections:
   # Section 1: Positive correlations
   - statement:
       predicate: positively_correlated_with
-    reindex:
-      - column: B  # Correlation coefficient
-        comparison: gt
-        comparator: 0
+    source:
+      reindex:
+        - column: B  # Correlation coefficient
+          comparison: gt
+          comparator: 0
 
   # Section 2: Negative correlations
   - statement:
       predicate: negatively_correlated_with
-    reindex:
-      - column: B
-        comparison: lt
-        comparator: 0
+    source:
+      reindex:
+        - column: B
+          comparison: lt
+          comparator: 0
 ```
 
 This produces two sets of edges from one table:
@@ -296,14 +299,9 @@ template:
         - pattern: _
           replacement: ' '   # "Lactobacillus_rhamnosus" → "Lactobacillus rhamnosus"
 
-    qualifiers:
-      - qualifier: p value
-        method: column
-        encoding: E
-
   provenance:
     repo: PMC
-    publication: 12345678
+    publication: PMC12345678
     contributors:
       - kind: curation
         name: Skye Lane Goetz
@@ -324,7 +322,7 @@ template:
 
 **Both nodes from columns:** Setting `method: column` on both subject and object means both undergo entity resolution via `resolve()`. Each gets its own `prioritize` list to guide disambiguation.
 
-**`remove` vs `regex`:** `remove` filters out entire rows matching a pattern before resolution. `regex` transforms the column value in-place before resolution.
+**`remove` vs `regex`:** Both transform cell text in place before resolution (neither drops rows). `remove` strips each listed regex pattern by replacing it with an empty string; `regex` applies an ordered `pattern`→`replacement` substitution list.
 
 ---
 
@@ -364,7 +362,7 @@ template:
 
   provenance:
     repo: PMC
-    publication: 87654321
+    publication: PMC87654321
     contributors:
       - kind: curation
         name: Skye Lane Goetz

{tablassert-7.4.14 → tablassert-7.5.0}/docs/configuration/graph.md

@@ -16,7 +16,6 @@ A graph configuration file specifies:
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `syntax` | String | Configuration version (must be `"GC2"`) |
 | `name` | String | Knowledge graph name (used in output filename) |
 | `version` | String | Knowledge graph version (used in output filename) |
 | `tables` | List[Path] | Paths to table configuration YAML files |
@@ -26,6 +25,7 @@ A graph configuration file specifies:
 
 | Field | Type | Description |
 |-------|------|-------------|
+| `syntax` | String | Configuration version. Defaults to `"GC2"` (also accepts `"TC3"`); overriding is not recommended. |
 | `pubmed_db` | Path | Path to SQLite PubMed metadata database |
 | `pmc_db` | Path | Path to SQLite PMC figure captions database |
 | `log` | Boolean | Log unmatched entities and audit details during graph builds |
@@ -35,7 +35,7 @@ A graph configuration file specifies:
 
 **`syntax: "GC2"`**
 
-Configuration syntax version. Must be `"GC2"`.
+Configuration syntax version. Defaults to `"GC2"` (the type also accepts `"TC3"`); overriding the default is not recommended.
 
 **`name: string`**
 
@@ -61,7 +61,7 @@ This field only controls whether QC runs. Install `tablassert[qc]` or `tablasser
 
 **`tables: list[path]`**
 
-List of table configuration file paths. Can be absolute or relative to graph config location.
+List of table configuration file paths. Can be absolute or relative to the current working directory (paths are resolved against the process CWD, not the graph-config file location).
 
 Each table config defines:
 - Data source (Excel/CSV/TSV)
@@ -101,8 +101,7 @@ When provided, this is used when provenance specifies PMC publications.
 
 Paths can be:
 - **Absolute:** `/home/user/data/datassert`
-- **Relative to graph config:** `./tables/table1.yaml`
-- **Relative to current directory:** `../configs/table.yaml`
+- **Relative to the current working directory:** `./tables/table1.yaml` (note: paths are resolved against the process CWD, not the graph-config file location — there is no config-relative resolver)
 
 ## Minimal Example
 
@@ -139,16 +138,17 @@ pmc_db: /databases/PMCSuppCaptions.db
 When you run `tablassert build graph.yaml`:
 
 1. **Load graph configuration** - Parse YAML, validate schema
-2. **For each table in `tables`:**
-   - Load table configuration
+2. **Load table configurations** - Parse each YAML in `tables`
+3. **Extract sections** - Expand templates into per-section `Tcode` instances
+4. **Collect instructions (per section):**
    - Download source file (if URL specified)
-   - Apply transformations
-   - Resolve entities using `datassert`
-    - Validate with QC pipeline when `qc: true`
-   - Create subgraph parquet file
-3. **Aggregate subgraphs** - Merge all parquet files
-4. **Add provenance (optional)** - Query `pubmed_db` and `pmc_db` for metadata when configured
-5. **Export NDJSON** - Generate `{name}_{version}.nodes.ndjson` and `.edges.ndjson`
+   - Apply transformations and resolve entities using `datassert`
+   - Validate with the QC audit when `qc: true`
+   - Enrich with provenance: query `pubmed_db` (MeSH) and `pmc_db` (captions) when configured
+5. **Build subgraphs** - Compile each section's resolved data into a parquet file
+6. **Compile graph** - Aggregate all subgraph parquets and export `{name}_{version}.nodes.ndjson` / `.edges.ndjson`
+
+> Note: provenance enrichment (`with_mesh`/`with_captions`) and QC both run during the per-section Collect Instructions stage, *before* subgraphs are built — not as a separate post-aggregation step.
 
 ## Output Files
 
@@ -173,8 +173,8 @@ version: UNSTABLE
 tables:
   - /local_raid1/sgoetz/STORE/CONFIG/TABLASSERT/TABLE/V6/ALAMV6.yaml
 datassert: /local_raid1/sgoetz/CODE/DATASSERT/datassert
-pubmed_db: /local_raid1/sgoetz/DBSTORE/local_raid1/sgoetz/DBSTORE/PUBMED/PubMed.db
-pmc_db: /local_raid1/sgoetz/DBSTORE/local_raid1/sgoetz/DBSTORE/CAPTIONS/PMCSuppCaptions.db
+pubmed_db: /local_raid1/sgoetz/DBSTORE/PUBMED/PubMed.db
+pmc_db: /local_raid1/sgoetz/DBSTORE/CAPTIONS/PMCSuppCaptions.db
 ```
 
 This processes a single table configuration (ALAMV6.yaml) into a knowledge graph named `MULTIOMICS_KG_UNSTABLE`.

{tablassert-7.4.14 → tablassert-7.5.0}/docs/configuration/table.md

@@ -84,7 +84,7 @@ template:
   syntax: TC3
 
 sections:
-  - syntax: TC2  # Overrides (not recommended)
+  - syntax: GC2  # Valid values are TC3 (default) and GC2; overriding is not recommended
 ```
 
 ### Use Cases
@@ -100,7 +100,7 @@ template:
 ```yaml
 template:
   source: {kind: excel, local: data.xlsx, url: https://example.com/data.xlsx}
-  provenance: {repo: PMC, publication: 123, contributors: [{name: Example User, date: 27 JAN 2026}]}
+  provenance: {repo: PMC, publication: "PMC123", contributors: [{name: Example User, date: 27 JAN 2026}]}
 
 sections:
   - statement: {predicate: treats}
@@ -111,7 +111,7 @@ sections:
 ```yaml
 template:
   source: {kind: text, local: data.csv, url: https://example.com/data.csv}
-  provenance: {repo: PMID, publication: 456, contributors: [{name: Example User, date: 27 JAN 2026}]}
+  provenance: {repo: PMID, publication: "456", contributors: [{name: Example User, date: 27 JAN 2026}]}
   statement:
     subject: {method: column, encoding: A}
 
@@ -141,8 +141,8 @@ Defines the data file location and format.
 | `local` | Path | Yes | Local file path for caching |
 | `url` | URL | Yes | Download URL (HTTP/HTTPS). Validated as reachable at parse time. |
 | `sheet` | String | No | Sheet name. Defaults to `"Sheet1"`. |
-| `row_slice` | List[Int\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be an integer or `"auto"`. Mutually exclusive with `rows`. |
-| `rows` | List[Int] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
+| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
+| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
 | `reindex` | List[Reindex] | No | Conditional row filtering |
 
 **Example:**
@@ -165,8 +165,8 @@ source:
 | `local` | Path | Yes | Local file path for caching |
 | `url` | URL | Yes | Download URL. Validated as reachable at parse time. |
 | `delimiter` | String | No | Field delimiter. Defaults to `","`. |
-| `row_slice` | List[Int\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be an integer or `"auto"`. Mutually exclusive with `rows`. |
-| `rows` | List[Int] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
+| `row_slice` | List[PositiveInt\|"auto"] | No | Two-value zero-based crop bounds: `[start, stop]`. Each value may be a positive integer or `"auto"`. Mutually exclusive with `rows`. |
+| `rows` | List[PositiveInt] | No | Zero-based row indices to keep after any `row_slice` crop. Mutually exclusive with `row_slice`. |
 | `reindex` | List[Reindex] | No | Conditional filtering |
 
 **Example:**
@@ -185,11 +185,11 @@ source:
 
 Filter rows based on column values.
 
-| Field | Type | Description |
-|-------|------|-------------|
-| `column` | String | Source column letters to evaluate (`A`-`ZZZ`) |
-| `comparison` | String | Operator. Defaults to `"ne"`; allowed values are `"eq"`, `"ne"`, `"lt"`, `"le"`, `"gt"`, `"ge"`. |
-| `comparator` | String\|Int\|Float | Value to compare against. Must be a string for `"eq"`/`"ne"`, or a number for `"lt"`/`"le"`/`"gt"`/`"ge"`. |
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `column` | String | Yes | Source column letters to evaluate; constrained to `^[A-Z]{1,3}$` (`A`-`ZZZ`). |
+| `comparison` | String | No | Operator. Defaults to `"ne"`; allowed values are `"eq"`, `"ne"`, `"lt"`, `"le"`, `"gt"`, `"ge"`. |
+| `comparator` | String\|Int\|Float | Yes | Value to compare against. Must be a string for `"eq"`/`"ne"`, or a number for `"lt"`/`"le"`/`"gt"`/`"ge"`. |
 
 **Example:**
 ```yaml
@@ -232,12 +232,12 @@ Defines how to extract and resolve entities.
 |-------|------|----------|-------------|
 | `method` | String | No | `"value"` (literal) or `"column"` (source column letters). Defaults to `"value"`. |
 | `encoding` | String\|Int\|Float | Yes | Literal value or source column letters, depending on `method` |
-| `taxon` | Int | No | NCBI Taxon ID for filtering (e.g., `9606` for human) |
+| `taxon` | PositiveInt | No | NCBI Taxon ID for filtering (e.g., `9606` for human) |
 | `prioritize` | List[String] | No | Preferred Biolink categories (must be valid `Categories` enum values such as `Gene`, `Protein`) |
 | `avoid` | List[String] | No | Excluded Biolink categories (must be valid `Categories` enum values) |
 | `regex` | List[Regex] | No | Pattern replacements |
 | `fill` | String | No | Null-filling strategy: `"forward"`, `"backward"`, `"min"`, `"max"`, `"mean"`, `"zero"`, `"one"` |
-| `remove` | List[String] | No | Strings to filter out |
+| `remove` | List[Int\|Float\|String] | No | Regex patterns to remove (replaced with empty string) |
 | `prefix` | String | No | Add prefix to values |
 | `suffix` | String | No | Add suffix to values |
 | `explode_by` | String | No | Delimiter to split multi-value cells |
@@ -314,7 +314,7 @@ Prevents misclassifying organism names as genes.
 
 #### Text Transformations
 
-**`regex: list[{pattern, replacement}]`** - Pattern-based replacements
+**`regex: list[{pattern: Int|Float|String, replacement: Int|Float|String}]`** - Pattern-based replacements
 
 ```yaml
 subject:
@@ -328,18 +328,18 @@ subject:
 
 Executed in order.
 
-> **Regex dialect:** Patterns are passed directly to Polars `str.replace_all()`, which uses the Rust [`regex`](https://docs.rs/regex/) crate. Only features supported by that engine work — in particular, **capturing groups (`(...)`, `\1`) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?<!...)` are not supported** and will raise an error at parse time. Stick to character classes, anchors (`^`, `$`), quantifiers, alternation (`a|b`), and non-capturing groups (`(?:...)`) if grouping is needed. If a transformation is too complex to express, prefer chaining several simple substitutions or capturing the residual context in a `miscellaneous notes` annotation instead.
+> **Regex dialect:** Patterns are passed directly to Polars `str.replace_all()`, which uses the Rust [`regex`](https://docs.rs/regex/) crate. Only features supported by that engine work — in particular, **backreferences (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?<!...)`) are not supported** and will raise an error at parse time. Plain groups `(...)` and non-capturing groups `(?:...)` *are* supported. Stick to character classes, anchors (`^`, `$`), quantifiers, alternation (`a|b`), and grouping if needed. If a transformation is too complex to express, prefer chaining several simple substitutions or capturing the residual context in a `miscellaneous notes` annotation instead.
 
-**`remove: list[string]`** - Filter out specific strings
+**`remove: list[regex]`** - Regex patterns to remove
 
 ```yaml
 subject:
   encoding: A
   remove:
-    - "^NA "  # Remove rows starting with "NA "
+    - "^NA "  # Strip leading "NA " prefix from cell text
 ```
 
-Same regex constraints apply as the `regex` field — Polars-compatible patterns only, no capturing groups or lookarounds.
+Each entry is applied as a regex replace-with-empty-string on the cell text in place (rows are not dropped). Same regex constraints apply as the `regex` field — Polars-compatible patterns only, no backreferences or lookarounds.
 
 **`prefix` / `suffix`** - Add text
 
@@ -349,6 +349,13 @@ object:
   prefix: "CUSTOM:"  # "123" → "CUSTOM:123"
 ```
 
+**Output columns: `original <col>` vs `<col> table literal value`** - For every subject/object/qualifier node, the pipeline snapshots the cell value into two edge columns at different stages:
+
+- `<col> table literal value` - the **pristine source-cell value**, captured immediately after the column is read and *before* any `fill`, `explode_by`, `regex`, `remove`, `prefix`, `suffix`, or `transformations`. Emitted only when `method: column` (a `method: value` node has no table source).
+- `original <col>` - the **fully-transformed value**, captured *after* all of the above, i.e. the same text that is then normalized and resolved to a CURIE. Always present for subject/object/qualifier nodes.
+
+Example: with `method: column`, `encoding: A`, `remove: ["^NA "]` over a cell `"NA BRCA1"`, `subject table literal value` is `"NA BRCA1"` while `original subject` is `"BRCA1"`. Annotations never emit a table-literal column.
+
 #### Null Handling
 
 **`fill: string`** - Fill null values using a strategy
@@ -400,10 +407,10 @@ Use the `"values"` token to reference column values in transformations.
 
 Add context to edges (anatomical location, species, etc.).
 
-| Field | Type | Description |
-|-------|------|-------------|
-| `qualifier` | String | Biolink qualifier from the `Qualifiers` enum (e.g., `"species_context_qualifier"`) |
-| (inherits NodeEncoding) | | All NodeEncoding fields available |
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `qualifier` | String | Yes | Biolink qualifier from the `Qualifiers` enum (e.g., `"species_context_qualifier"`) |
+| (inherits NodeEncoding) | | | All NodeEncoding fields available |
 
 **Example:**
 ```yaml
@@ -420,7 +427,7 @@ Required metadata about data source.
 | Field | Type | Required | Description |
 |-------|------|----------|-------------|
 | `repo` | String | No | Repository. Defaults to `"PMC"`; allowed values are `"PMC"`, `"PMID"`. |
-| `publication` | String | Yes | Repository-local identifier appended to `repo:` (e.g., `"11708054"`, `"123"`). When `repo` is `"PMC"`, the value **must** start with `PMC` followed by digits (e.g., `"PMC11708054"`, `"PMC123"`). |
+| `publication` | String | Yes | Repository-local identifier emitted as a CURIE. For `repo: PMC` the value **must** start with `PMC` followed by digits (e.g., `"PMC11708054"`, `"PMC123"`) and is emitted under the `PMCID:` namespace as `PMCID:PMC...` (previously `PMC:PMC...`); for `repo: PMID` it is emitted as `PMID:<publication>` (e.g., `"11708054"` → `PMID:11708054`). The `repository` output column still records the raw `repo` value (`PMC`/`PMID`). |
 | `contributors` | List[Contributor] | Yes | Curation information |
 
 **Contributor fields:**
@@ -437,7 +444,7 @@ Required metadata about data source.
 ```yaml
 provenance:
   repo: PMC
-  publication: 11708054
+  publication: "PMC11708054"
   contributors:
     - kind: curation
       name: Skye Lane Goetz
@@ -452,10 +459,10 @@ provenance:
 
 Optional edge attributes (statistical metadata, notes, etc.).
 
-| Field | Type | Description |
-|-------|------|-------------|
-| `annotation` | String | Attribute name (e.g., `"p value"`, `"sample size"`). Underscores are automatically replaced with spaces at parse time. |
-| (inherits Encoding) | | All Encoding fields available (method, encoding, regex, etc.) |
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `annotation` | String | Yes | Attribute name (e.g., `"p value"`, `"sample size"`). Underscores are automatically replaced with spaces and the result is trimmed of leading/trailing whitespace at parse time. |
+| (inherits Encoding) | | | All Encoding fields available (method, encoding, regex, etc.) |
 
 **Example:**
 ```yaml
@@ -510,7 +517,7 @@ template:
 
   provenance:
     repo: PMID
-    publication: 12345678
+    publication: "12345678"
     contributors:
       - kind: curation
         name: Example User

{tablassert-7.4.14 → tablassert-7.5.0}/docs/docker.md

@@ -10,7 +10,7 @@ The image is based on `python:3.14-slim` with the Tablassert CLI as the entrypoi
 docker pull ghcr.io/skyeav/tablassert:latest
 ```
 
-Version-pinned tags match the git tag (e.g., `ghcr.io/skyeav/tablassert:v7.4.0`).
+Version-pinned tags match the git tag (e.g., `ghcr.io/skyeav/tablassert:v7.5.0`).
 
 ## Quick Start
 
@@ -49,7 +49,7 @@ docker run --rm \
 
 All dependencies ship in the base install, so the Docker image includes:
 
-- **Quality control** — The QC pipeline in `src/tablassert/qc.py` runs a three-stage audit: exact match, then fuzzy matching via rapidfuzz (threshold >= 20), then BioBERT sentence embeddings with cosine similarity (threshold >= 0.2). The ONNX model is cached in `.onnxassert/` (line 26).
+- **Quality control** — The QC pipeline in `src/tablassert/qc.py` runs a three-stage audit: exact match, then fuzzy matching via rapidfuzz (`fuzz.ratio` >= 20 or `partial_token_sort_ratio` >= 30), then BioBERT sentence embeddings with cosine similarity (threshold >= 0.2). The ONNX model is cached in `.onnxassert/` (line 24).
 - **Web downloads** — `src/tablassert/downloader.py` uses Playwright to download remote files with retry logic.
 - **Legacy Excel** — `modernize_xls()` in `src/tablassert/downloader.py` converts `.xls` files using pyexcel.
 
@@ -61,7 +61,8 @@ Mount these volumes to persist data across container runs:
 |---|---|---|
 | `.storassert/` | `src/tablassert/utils.py:17` — `STORE` | Intermediate Parquet storage for compiled subgraphs |
 | `.logassert/` | `src/tablassert/log.py` | Loguru log files with 100 MB rotation |
-| `.onnxassert/` | `src/tablassert/qc.py:26` — `MODEL` | Cached ONNX/BioBERT model |
+| `.onnxassert/` | `src/tablassert/qc.py:24` — `MODEL` | Cached ONNX/BioBERT model |
+| `.cachassert/` | `src/tablassert/models.py:76` — `CACHE` | diskcache store for URL-validation memoization |
 
 Example:
 
@@ -72,6 +73,7 @@ docker run --rm \
   -v ./.storassert:/app/.storassert \
   -v ./.logassert:/app/.logassert \
   -v ./.onnxassert:/app/.onnxassert \
+  -v ./.cachassert:/app/.cachassert \
   -w /app \
   ghcr.io/skyeav/tablassert:latest \
   build /data/graph-config.yaml
@@ -87,4 +89,4 @@ docker run --rm \
 
 ## CI/CD Integration
 
-Images are built by `.github/workflows/docker.yml`, which triggers on tag pushes (after autotag and PyPI publish complete). Tags match the repository version tag (e.g., `v7.2.2`).
+Images are built by `.github/workflows/docker.yml`, which triggers when the `Auto Tag Versions` workflow completes on `main` (also runnable manually via `workflow_dispatch`). Tags match the repository version tag (e.g., `v7.5.0`).

{tablassert-7.4.14 → tablassert-7.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tablassert"
-version = "7.4.14"
+version = "7.5.0"
 description = "Extract knowledge assertions from tabular data into NCATS Translator-compliant KGX NDJSON — declaratively, with entity resolution and quality control built in."
 authors = [
     { name = "Skye Lane Goetz", email = "sgoetz@isbscience.org" }

{tablassert-7.4.14 → tablassert-7.5.0}/src/tablassert/lib.py

@@ -14,7 +14,7 @@ import lazy_loader as Lazy
 from pydantic import Field, NonNegativeInt, PositiveInt
 
 from tablassert.downloader import from_url
-from tablassert.enums import Categories, EncodingMethods, Files, Tokens
+from tablassert.enums import Categories, EncodingMethods, Files, Repositories, Tokens
 from tablassert.fullmap import SHARDS, resolve
 from tablassert.log import cat
 from tablassert.models import Encoding, NodeEncoding, Section
@@ -279,11 +279,14 @@ class Tcode(Section):
     log: bool = Field(False)
     qc: bool = Field(False)
 
-    def encoding(self: Self, x: Encoding, col: str) -> list[Any]:
+    def encoding(self: Self, x: Encoding, col: str, table_literal: bool = False) -> list[Any]:
         # ? Collect Helper For Encoding Classes
         return [
             (value, (col, x.encoding)) if eq(x.method, EncodingMethods.VALUE) else None,
             (column, (col, idxname(x.encoding))) if eq(x.method, EncodingMethods.COLUMN) else None,
+            (column, (add(col, " table literal value"), col))
+            if (table_literal and eq(x.method, EncodingMethods.COLUMN))
+            else None,
             (fill, (col, x.fill)) if x.fill else None,
             (explode, (col, x.explode_by)) if x.explode_by else None,
             [(regex, (col, r.pattern, r.replacement)) for r in x.regex] if x.regex else None,
@@ -295,7 +298,7 @@ class Tcode(Section):
 
     def node(self: Self, x: NodeEncoding, col: str, conns: list[object]) -> list[Any]:
         # ? Collect Helper For NodeEncoding Classes
-        encoding: list[Any] = self.encoding(x, col)
+        encoding: list[Any] = self.encoding(x, col, table_literal=True)
         node: list[Any] = [
             (column, (add("original ", col), col)),
             (level_one, (col,)),
@@ -355,10 +358,11 @@ class Tcode(Section):
                 (value, ("section number", self.number)),
                 (value, ("status", self.status)),
                 (value, ("repository", self.provenance.repo)),
-                (value, ("publication", (self.provenance.repo + ":" + self.provenance.publication))),
+                (value, ("publication", publication_curie(self.provenance.repo, self.provenance.publication))),
                 (contributor_values, ("contributors", self.provenance.contributors)),
                 (value, ("url", str(self.source.url))),
                 (value, ("section hash", self.store.stem)),
+                (value, ("sheet name", self.source.sheet)) if eq(self.source.kind, Files.EXCEL) else None,  # pyright: ignore
                 (with_mesh, (pubmed_db, self.provenance.publication)) if pubmed_db else None,
                 (with_captions, (pmc_db, self.provenance.publication, str(self.source.url))) if pmc_db else None,
                 (sig, ()),
@@ -391,6 +395,13 @@ def normalize(
     return nodes, edges_out
 
 
+def publication_curie(repo: str, publication: str) -> str:
+    # ? Builds The Publication CURIE; PMCID Namespace For PubMed Central
+    if eq(repo, Repositories.PUBMED_CENTRAL):
+        return add("PMCID:", publication)
+    return add(repo, add(":", publication))
+
+
 def publications(
     edges: pl.LazyFrame, names: list[str] = ["id", "name", "first author", "journal", "year published"]
 ) -> tuple[pl.LazyFrame, pl.LazyFrame]:

{tablassert-7.4.14 → tablassert-7.5.0}/tests/test_lib.py

@@ -182,6 +182,59 @@ def test_tcode_collect_enables_qc_logging(fixtures_path: Path) -> None:
     assert qc_ops[1][1] == ("object", "sectionhash", "minimal_section.yaml", "passed", True)
 
 
+# ? publication_curie Uses PMCID Namespace For PubMed Central
+def test_publication_curie_pmc() -> None:
+    assert lib.publication_curie("PMC", "PMC1234567") == "PMCID:PMC1234567"
+
+
+# ? publication_curie Uses Repo Namespace For Non PMC Repositories
+def test_publication_curie_pubmed() -> None:
+    assert lib.publication_curie("PMID", "11708054") == "PMID:11708054"
+
+
+# ? Tcode Captures Table Literal Value Before Regex For Column Encoded Nodes
+def test_tcode_table_literal_value_before_regex_for_columns(fixtures_path: Path) -> None:
+    data: Any = from_yaml(fixtures_path / "minimal_section.yaml")
+    store: Path = Path("/tmp/sectionhash.parquet")
+    data["statement"]["subject"] = {
+        "method": "column",
+        "encoding": "A",
+        "regex": [{"pattern": "\\s+", "replacement": " "}],
+    }
+    data["statement"]["object"] = {"method": "column", "encoding": "B"}
+
+    tcode_model: Tcode = Tcode.model_validate(  # pyright: ignore
+        {**data, "number": 7, "config": fixtures_path / "minimal_section.yaml", "store": store}
+    )
+
+    collected: list[tuple[Any, tuple[Any]]] = tcode_model.collect([], None, None)  # pyright: ignore
+    targets: list[str] = [op[1][0] for op in collected if op[0].__name__ == "column" and len(op[1]) > 1]
+    assert "subject table literal value" in targets
+    assert "object table literal value" in targets
+
+    lit_idx: int = next(
+        i for i, op in enumerate(collected) if len(op[1]) > 0 and op[1][0] == "subject table literal value"
+    )
+    regex_idx: int = next(
+        i for i, op in enumerate(collected) if op[0].__name__ == "regex" and len(op[1]) > 0 and op[1][0] == "subject"
+    )
+    assert lit_idx < regex_idx
+
+
+# ? Tcode Omits Table Literal Value For Value Encoded Nodes
+def test_tcode_table_literal_value_absent_for_value_encoding(fixtures_path: Path) -> None:
+    data: Any = from_yaml(fixtures_path / "minimal_section.yaml")
+    store: Path = Path("/tmp/sectionhash.parquet")
+    tcode_model: Tcode = Tcode.model_validate(  # pyright: ignore
+        {**data, "number": 7, "config": fixtures_path / "minimal_section.yaml", "store": store}
+    )
+
+    collected: list[tuple[Any, tuple[Any]]] = tcode_model.collect([], None, None)  # pyright: ignore
+    targets: list[str] = [op[1][0] for op in collected if op[0].__name__ == "column" and len(op[1]) > 1]
+    assert "subject table literal value" not in targets
+    assert "object table literal value" not in targets
+
+
 # ? resolve_many Skips QC When Disabled
 def test_resolve_many_skips_qc(monkeypatch: Any, tmp_path: Path) -> None:
     calls: list[tuple[Any, ...]] = []

{tablassert-7.4.14 → tablassert-7.5.0}/uv.lock

@@ -2340,7 +2340,7 @@ wheels = [
 
 [[package]]
 name = "tablassert"
-version = "7.4.14"
+version = "7.5.0"
 source = { editable = "." }
 dependencies = [
     { name = "cyclopts" },