tablassert 7.4.14__tar.gz → 7.5.1__tar.gz

tablassert-7.5.1/AGENTS.md

@@ -0,0 +1,57 @@
+# AGENTS.md — Tablassert
+
+## Fast Start
+
+- Python package, not a monorepo. Main code lives in `src/tablassert/`; tests live in `tests/`.
+- Install with `uv sync`. Three optional extras:
+  - `--extra qc` / `--extra qc-cuda` — installs `onnxruntime` / `onnxruntime-gpu` for QC (strict runtime behavior below).
+  - `--extra rt` — runtime-compatible Polars build for CPUs missing required SIMD instructions.
+- CLI entrypoint is `tablassert.cli:APP`. Real user commands:
+  - `uv run tablassert build <graph.yaml>` — 6 pipeline stages.
+  - `uv run tablassert validate <table.yaml>` — 3 stages, syntax-only.
+
+## Source of Truth
+
+- When prose docs and code disagree, `src/tablassert/models.py` and `src/tablassert/cli.py` are authoritative. Docs drift: some pages claim shard files are `{0..11}.duckdb` or that only the `rt` extra exists; the code uses `SHARDS = 10` (so `0..9`) and exposes `qc`/`qc-cuda`/`rt` extras.
+
+## Verify Changes
+
+- Match the repo hooks before finishing: `uv run ruff check --fix .`, `uv run ruff format .`, `uv run pyright`, `uv run pytest`.
+- Full hook run: `uv run pre-commit run --all-files` (ruff, ruff-format, pyright, pytest).
+- Focused test runs:
+  - Single test: `uv run pytest tests/test_lib.py::test_name`
+  - By keyword: `uv run pytest -k "pattern"`
+  - With print output: `uv run pytest -s tests/test_lib.py`
+- Docs build: `uv run --group dev mkdocs build`.
+
+## High-Value Structure
+
+- `src/tablassert/cli.py` is the wiring layer: `build()` → `build_pipeline()`, `validate()` → `validate_pipeline()`.
+- `src/tablassert/ingests.py` loads YAML and expands table configs into section dicts.
+- `src/tablassert/lib.py` is the core pipeline:
+  - `Tcode.collect()` builds the per-section operation list.
+  - `compile_subgraph()` executes that list into parquet.
+  - `compile_graph()` aggregates subgraph parquets into KGX NDJSON.
+  - `resolve_many()` is the direct library API for batch entity resolution.
+- Entity resolution uses DuckDB shard files at `<datassert>/data/{0..9}.duckdb`, opened read-only. `datassert` is a required `Path` field on the `Graph` model (not a fixed location); `src/tablassert/fullmap.py` hardcodes `SHARDS = 10`.
+
+## Repo-Specific Gotchas
+
+- Heavy dependencies are lazy-loaded per module with `TYPE_CHECKING` + `lazy_loader`. Follow the existing pattern instead of importing heavy packages eagerly. Lazy-loaded: polars, duckdb, orjson, xxhash, polars_hash, yaml, httpx, pyexcel, onnxruntime, sentence_transformers.
+- `tests/conftest.py` autouse-mocks `httpx.head`, so model-URL validation tests never hit the network unless a test opts in.
+- Network-dependent tests are marked `@pytest.mark.network`; GPU QC tests are marked with both `network` and `gpu` in `tests/test_qc.py`.
+- QC runtime selection is strict in `src/tablassert/qc.py`: if `onnxruntime-gpu` is installed but `CUDAExecutionProvider` is unavailable, the code raises (error 06) instead of falling back to CPU. Install `tablassert[qc]` for CPU-only.
+- Downloader behavior in `src/tablassert/downloader.py` is two-path: `direct()` via `httpx` for known file URLs, `browser()` via Playwright fallback for browser-only sources. Keep payload-validation and cleanup tests intact when changing it.
+
+## Conventions That Matter Here
+
+- Start every module with `from __future__ import annotations`.
+- Annotate every variable, including locals.
+- Use `Optional[T]` / `Union[...]`, not `T | None`. Use `Path`, not `str`, for filesystem paths. Use `# pyright: ignore` to silence lazy-load false positives.
+- No docstrings on functions. Use comment markers on the line above: `# ?` description, `# !` warning, `# *` pipeline stage, `# TODO:` todo.
+- Pydantic models inherit `TablaBase` (`extra="forbid"`, `validate_assignment=True`). Required fields use `Field(...)`; optional use `Optional[T] = Field(None)`. Enums extend `str, Enum` and live in `enums.py`.
+- Per-module logger: `from tablassert.log import cat; logger = cat("MODULE")` (e.g. `cat("FULLMAP")`). The CLI uses the root `from tablassert.log import logger`. Raise `RuntimeError` for failures; `logger.warning()` for non-fatal issues.
+
+## Side Effects
+
+- The package writes working artifacts to hidden dirs in the repo root, each defined in code: `.storassert/` (`utils.STORE`), `.logassert/` (`log.LOGASSERT`), `.cachassert/` (`models.CACHE`), `.onnxassert/` (`qc.MODEL`).

{tablassert-7.4.14 → tablassert-7.5.1}/CHANGELOG.md

@@ -2,6 +2,29 @@
 
 All notable changes to this project are documented in this file.
 
+## 7.5.1 - 2026-07-01
+
+### Changed
+- Numeric annotation columns are now coerced and emitted as controlled-notation strings in NDJSON output instead of raw values. Two new pipeline steps wired into `Tcode.collect()` (`lib.py`): `clean_numeric()` lazily casts matching columns to `Float64` with `strict=False` (non-numeric entries drop to null), and `format_numeric()` renders them as strings — p-value columns (any name containing `"p value"`, case-insensitive) in scientific notation (`{:.4e}`), and `relationship strength` / `sample size` in decimal general format (`{:.4g}`, ≥4 significant figures). Non-matching columns are left untouched, and nulls are subsequently dropped by `strip_nulls()`. `math_op()` now also casts with `strict=False` so it tolerates residual junk in numeric annotation columns. `format_numeric()` formats via numpy-backed batch conversion rather than `map_elements` for throughput.
+- Removed dead `pl.Config(set_fmt_float=...)` and `pl.Config(float_precision=...)` context managers from `compile_graph()` (`lib.py`); they were no-ops for NDJSON serialization (`write_ndjson` emits raw f64 via serde shortest-repr and ignores float display options), and the `fmt`/`precision` parameters of `compile_graph()` were removed alongside them.
+- Fixed an off-by-one in the build/validate progress bar so each section loop now shows the configuration currently being processed instead of the last-completed one. `PipelineProgress.section_loop()` (`progress.py`) previously returned a single `advance(info)` callback that set the description and ticked the completed counter together, called after each item's work — so while section *K* ran the bar still displayed section *K−1*. It now returns a `(start, advance)` pair: `start(info)` updates the description to the in-flight item without incrementing, and `advance()` ticks the counter afterwards (so the counter never claims an in-flight item is complete). All five call sites in `cli.py` (TCode build, Collect, Subgraph, Graph, Validate) were updated to `start(...)` before the work and `advance()` after; the long-running Collect and Subgraph stages continue to show the full `format_section_oneline()` summary (including the `CONFIG` name) of the in-flight section.
+
+### Added
+- Fifteen regression tests in `test_lib.py` covering `numeric_columns()` detection (p-value substring, exact-name match, case-insensitivity), `clean_numeric()` (parse/coerce numeric and scientific notation, null out non-numeric junk, leave non-matching columns untouched, noop, idempotent on Float64), `format_numeric()` (scientific notation for p-value, decimal general format for relationship strength/sample size, null preservation, floating-point-noise cleaning, noop), null-stripped NDJSON rows, `compile_graph()` NDJSON emission, and `sig()` operating over a cleaned Float64 p-value column.
+- Three regression tests in `tests/test_progress.py` pinning the new two-callback contract: `start` shows the in-flight item with the counter still at zero, `advance` ticks the counter without altering the description, and a `start`/`advance` cycle keeps the description synced to the current item rather than the previous one.
+
+## 7.5.0 - 2026-07-01
+
+### Changed
+- Publication CURIEs in `compile_subgraph()` (`lib.py`) now use the `PMCID:` namespace prefix for PubMed Central sources. A `repo: PMC` section with `publication: PMC11708054` is emitted as `PMCID:PMC11708054` (previously `PMC:PMC11708054`); non-PMC repos such as `PMID` are unaffected and continue to emit `<repo>:<publication>` (e.g., `PMID:11708054`). The `repository` edge column is unchanged and still records the raw `repo` value. Extracted via a new `publication_curie()` helper.
+
+### Added
+- New `<col> table literal value` edge column for subject, object, and qualifier nodes encoded with `method: column`. Unlike the existing `original <col>` column (which snapshots the value *after* all `fill`/`explode_by`/`regex`/`remove`/`prefix`/`suffix`/`transformations`), `<col> table literal value` captures the pristine source-cell value *before* any transformation. Emitted only for column-encoded nodes; annotations and `method: value` nodes are unaffected. Implemented via a `table_literal` flag on `Tcode.encoding()`, enabled by `Tcode.node()`.
+- Four regression tests in `test_lib.py`: `publication_curie()` for PMC and PMID namespaces, and two `Tcode` tcode-inspection tests covering presence/ordering of the table-literal column for column encodings and its absence for value encodings.
+
+### Documentation
+- Comprehensive accuracy pass across the API, configuration, and Docker documentation, reconciling every page against the current codebase. Highlights: corrected invalid examples that would not load (`syntax: TC2`; `publication` integers and missing `PMC` prefixes; a non-existent `Qualifiers` member; `reindex` placed at section level; a subject missing `method: column`), fixed wrong field types (`rows`/`row_slice`/`taxon` → `PositiveInt`, `remove` → regex patterns), corrected the QC fuzzy thresholds (`fuzz.ratio >= 20 OR partial_token_sort_ratio >= 30`), removed a non-existent `uuid:` prefix from `utils.md` return examples, fixed the `resolve_many()` parameter order and added the original-column-capture and optional QC-audit pipeline steps, corrected graph-config path resolution (CWD, not config-relative) and processing-flow ordering, documented the strict QC GPU no-fallback behavior and the `.cachassert/` working directory, and aligned `Categories` enum member names (`GENE`/`PROTEIN`) and Docker CI triggers with the source.
+
 ## 7.4.14 - 2026-06-30
 
 ### Changes

{tablassert-7.4.14 → tablassert-7.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tablassert
-Version: 7.4.14
+Version: 7.5.1
 Summary: Extract knowledge assertions from tabular data into NCATS Translator-compliant KGX NDJSON — declaratively, with entity resolution and quality control built in.
 Project-URL: Homepage, https://github.com/SkyeAv/Tablassert
 Project-URL: Source, https://github.com/SkyeAv/Tablassert

{tablassert-7.4.14 → tablassert-7.5.1}/docs/api/lib.md

@@ -2,7 +2,7 @@
 
 The `lib` module exposes `resolve_many()`, a high-level convenience function for resolving an iterable of entity strings to CURIEs without requiring manual LazyFrame construction, NLP preprocessing, or DuckDB shard management.
 
-It wraps the lower-level [`resolve()`](fullmap.md) pipeline — applying `level_one` and `level_two` normalization, opening all 10 DuckDB shard connections, executing entity resolution, and returning results as a plain Python list of row dictionaries.
+It wraps the lower-level [`resolve()`](fullmap.md) pipeline — preserving the original input text, applying `level_one` and `level_two` normalization, opening all 10 DuckDB shard connections, executing entity resolution, optionally running the QC audit (when `qc=True`), and returning results as a plain Python list of row dictionaries.
 
 ## resolve_many()
 
@@ -18,8 +18,8 @@ def resolve_many(
     taxon: Optional[str] = None,
     prioritize: Optional[list[Categories]] = None,
     avoid: Optional[list[Categories]] = None,
-    column_context: bool = True,
     qc: bool = False,
+    column_context: bool = True,
 ) -> list[dict[str, Any]]
 ```
 
@@ -52,29 +52,29 @@ Each shard contains:
 
 Optional NCBI Taxon ID for filtering results to a specific organism.
 
-Example: `"9606"` restricts matches to human-specific entities. When `None`, no taxon filtering is applied and matches from all organisms are returned.
+Example: `"9606"` restricts **gene** matches to human-specific entries; non-gene categories (e.g., diseases, chemicals) are returned regardless of taxon. When `None`, no taxon filtering is applied and matches from all organisms are returned.
 
 **`prioritize: Optional[list[Categories]]` (default: `None`)**
 
 Optional list of Biolink categories to prefer when multiple matches exist for the same input term. Categories listed here receive higher ranking scores during resolution.
 
-Example: `[Categories.Gene, Categories.Protein]` prefers gene and protein mappings over other categories like diseases or chemicals.
+Example: `[Categories.GENE, Categories.PROTEIN]` prefers gene and protein mappings over other categories like diseases or chemicals.
 
 **`avoid: Optional[list[Categories]]` (default: `None`)**
 
 Optional list of Biolink categories to exclude from results entirely. Any match belonging to an avoided category is filtered out before ranking.
 
-Example: `[Categories.Gene]` prevents gene mappings from appearing in the output, even if they would otherwise be the best match.
+Example: `[Categories.GENE]` prevents gene mappings from appearing in the output, even if they would otherwise be the best match.
 
 **`column_context: bool` (default: `True`)**
 
-Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the resolution query adds a category frequency score and prefers the category that appears most frequently across all terms in the batch. When `False`, frequency-based tie-breaking is disabled.
+Controls category-frequency tie-breaking when multiple matches exist for a term. When `True`, the deduplication stage adds a category-frequency score (computed in Polars after the SQL query) and prefers the category that appears most frequently across all matched terms in the batch. When `False`, frequency-based tie-breaking is disabled.
 
 This is useful when resolving a column of related entities (e.g., all genes) — the shared context helps disambiguate terms that map to multiple categories.
 
 **`qc: bool` (default: `False`)**
 
-When `True`, runs the QC audit stage after entity resolution. The QC pipeline validates mappings through a three-stage audit: exact match, fuzzy matching via rapidfuzz, and BioBERT sentence embeddings with cosine similarity. Requires a QC runtime to be installed (`tablassert[qc]` or `tablassert[qc-cuda]`). The ONNX Runtime provider is auto-detected based on the installed package — CUDA is preferred when `onnxruntime-gpu` is available, otherwise CPU is used.
+When `True`, runs the QC audit stage after entity resolution. The QC pipeline validates mappings through a three-stage audit: exact match, fuzzy matching via rapidfuzz, and BioBERT sentence embeddings with cosine similarity. Mappings that fail all three stages are dropped from the returned list (in addition to the unresolved-entity filtering performed by `resolve()`). Requires a QC runtime to be installed (`tablassert[qc]` or `tablassert[qc-cuda]`). The ONNX Runtime provider is auto-detected based on the installed package — CUDA is preferred when `onnxruntime-gpu` is installed and `CUDAExecutionProvider` is available; if only `onnxruntime` (CPU) is installed, CPU is used. Note: if `onnxruntime-gpu` is installed but `CUDAExecutionProvider` is unavailable, QC raises rather than falling back to CPU.
 
 ### Return Value
 
@@ -101,13 +101,17 @@ Each dictionary contains the following keys (where `{col}` is the value of the `
 
 1. **Series construction** — Wraps the input iterable in a `pl.Series` with the given column name, then converts to a single-column `pl.LazyFrame`.
 
-2. **NLP normalization** — Applies `level_one()` (whitespace stripping + lowercasing) and `level_two()` (non-word character removal via `\W+`) to produce the two normalized columns required by `resolve()`.
+2. **Original column capture** — Copies the raw input column into `original {col}` via `column(lf, add("original ", col), col)` so the pre-normalization text is preserved in the output.
+
+3. **NLP normalization** — Applies `level_one()` (whitespace stripping + lowercasing) and `level_two()` (non-word character removal via `\W+`) to produce the two normalized columns required by `resolve()`.
+
+4. **DuckDB connection management** — Opens all 10 shard connections inside a `contextlib.ExitStack`, ensuring every connection is properly closed when resolution completes or if an error occurs.
 
-3. **DuckDB connection management** — Opens all 10 shard connections inside a `contextlib.ExitStack`, ensuring every connection is properly closed when resolution completes or if an error occurs.
+5. **Entity resolution** — Delegates to `fullmap.resolve()` which queries the sharded DuckDB database, ranks matches by category priority, preferred-name exactness, NLP level, and category frequency, then deduplicates to one CURIE per input string.
 
-4. **Entity resolution** — Delegates to `fullmap.resolve()` which queries the sharded DuckDB database, ranks matches by category priority, preferred-name exactness, NLP level, and category frequency, then deduplicates to one CURIE per input string.
+6. **QC audit (optional)** — When `qc=True`, runs `fullmap_audit()` on the resolved LazyFrame. Rows that fail all three audit stages are dropped from the result.
 
-5. **Collection and conversion** — Collects the lazy result into an eager `pl.DataFrame` and converts to a list of row dictionaries via `to_dicts()`.
+7. **Collection and conversion** — Collects the lazy result into an eager `pl.DataFrame` and converts to a list of row dictionaries via `to_dicts()`.
 
 ### Example Usage
 
@@ -126,7 +130,7 @@ result: list[dict[str, Any]] = resolve_many(
     entities=["TP53", "BRCA1", "EGFR", "KRAS"],
     datassert=datassert,
     taxon="9606",
-    prioritize=[Categories.Gene],
+    prioritize=[Categories.GENE],
 )
 
 # result[0] → {"original gene": "TP53", "gene": "HGNC:11998", "gene name": "TP53", ...}
@@ -147,7 +151,7 @@ result: list[dict[str, Any]] = resolve_many(
     col="disease",
     entities=["diabetes mellitus", "breast cancer", "alzheimer disease"],
     datassert=datassert,
-    avoid=[Categories.Gene, Categories.Protein],
+    avoid=[Categories.GENE, Categories.PROTEIN],
 )
 
 # result[0] → {"original disease": "diabetes mellitus", "disease": "MONDO:0005015", ...}
@@ -206,7 +210,7 @@ for row in result:
 | **Connections** | Managed internally via `ExitStack` | Must be opened externally |
 | **Output** | `list[dict[str, Any]]` | `pl.LazyFrame` |
 | **Logging** | Uses default (`log=True`) | Configurable |
-| **Context params** | Not exposed (`section_hash`, `config_file`, `tag`) | Fully configurable |
+| **Context params** | `column_context` exposed; `section_hash`, `config_file`, `tag` not exposed | Fully configurable |
 | **Use case** | Standalone batch lookups, scripting, notebooks | Internal pipeline integration |
 
 `resolve_many()` is designed for ad-hoc and programmatic use — scripts, notebooks, and one-off lookups. For pipeline integration where you need full control over logging, context metadata, and lazy evaluation, use `resolve()` directly.

{tablassert-7.4.14 → tablassert-7.5.1}/docs/api/qc.md

@@ -2,7 +2,7 @@
 
 The `qc` module validates entity resolution mappings through a multi-stage pipeline: exact matching, fuzzy matching, and BERT semantic similarity.
 
-QC runtime support is optional. Install `tablassert[qc]` for CPU inference or `tablassert[qc-cuda]` for CUDA inference on GPU 0.
+QC runtime support is optional. Install `tablassert[qc]` for CPU inference or `tablassert[qc-cuda]` for CUDA inference on GPU 0. **Strict GPU behavior:** if `onnxruntime-gpu` is installed but `CUDAExecutionProvider` is not available (e.g., a broken CUDA/cuDNN environment), `fullmap_audit()` raises a `RuntimeError` rather than silently falling back to CPU — install `tablassert[qc]` for CPU inference or fix the CUDA environment.
 
 ## fullmap_audit()
 
@@ -84,7 +84,7 @@ original == preferred_name
 
 **Performance:** O(1) string comparison
 
-Before fuzzy matching, the function also applies rule-based pass-through checks for known safe patterns (for example CHEBI/PR/UniProtKB CURIE families and selected exception prefixes).
+Before fuzzy matching, the function also applies rule-based pass-through checks: (1) the resolved CURIE matches an exempt prefix (`CHEBI`, `PR`, `UniProtKB`, `NCBIGene`, `UMLS`, `UNII`, `PUBCHEM`, `MONDO`); (2) the original text contains `:` (looks like a CURIE); (3) the preferred name matches an exception prefix (`^LOC` or `^si:`).
 
 #### Stage 2: Fuzzy Matching
 
@@ -94,11 +94,11 @@ Two fuzzy matching algorithms:
 1. **Ratio:** Overall string similarity
 2. **Partial token sort ratio:** Combined token/subsequence matching
 
-**Threshold:** 20% similarity
+**Thresholds:** `fuzz.ratio` >= 20 OR `fuzz.partial_token_sort_ratio` >= 30
 
 ```python
 fuzz.ratio(original, preferred) >= 20
-or fuzz.partial_token_sort_ratio(original, preferred) >= 20
+or fuzz.partial_token_sort_ratio(original, preferred) >= 30
 ```
 
 **Example passes:**
@@ -116,8 +116,8 @@ or fuzz.partial_token_sort_ratio(original, preferred) >= 20
 3. **Accept** if similarity >= 0.2 (20%)
 
 ```python
-embeddings = BIOBERT.encode([original, preferred])
-similarity = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
+embeddings = get_biobert(provider).encode(originals + preferreds)
+similarity = cosine_similarity(embeddings[:n], embeddings[n:]).diagonal()
 return similarity >= 0.2
 ```
 

{tablassert-7.4.14 → tablassert-7.5.1}/docs/api/utils.md

@@ -19,10 +19,8 @@ def namespace_uuid(domain: Any, *values: list[Any]) -> str
 Domain string used to create the namespace UUID.
 
 Converted to string internally. Common domains:
-- `"TABLASSERT"` - Default domain used for knowledge graph edge IDs
-- `"edges"` - Optional custom domain for edge IDs
-- `"nodes"` - For custom node IDs
-- `"tablassert"` - For application-specific IDs
+- `"TABLASSERT"` - Default domain used for KGX edge IDs (the value used by `label_edge`)
+- Other domain strings (e.g., `"edges"`, `"nodes"`) may be passed for custom namespaces, though only `"TABLASSERT"` is used internally.
 
 **`*values: list[Any]`**
 
@@ -118,7 +116,7 @@ edge_id = namespace_uuid(
   "MONDO:0005148",  # Type 2 diabetes
   "PMC11708054"  # Publication
 )
-# Returns: "uuid:a1b2c3d4-e5f6-7890-abcd-ef1234567890"
+# Returns: "2cfea591-0f8f-33af-a7df-03da531d3359"
 ```
 
 **Benefits:**
@@ -145,7 +143,7 @@ from tablassert.utils import namespace_uuid
 
 # Generate edge ID
 edge_id = namespace_uuid("edges", "subject", "predicate", "object")
-print(edge_id)  # "uuid:12345678-1234-1234-1234-123456789abc"
+print(edge_id)  # "12345678-1234-1234-1234-123456789abc"
 ```
 
 **With qualifiers:**
@@ -183,12 +181,33 @@ Suitable for millions of ID generations per second.
 **`basespace(domain)`** - Creates namespace UUID from domain
 
 ```python
+@cache
 def basespace(domain: str) -> UUID:
   namespace = UUID("00000000-0000-0000-0000-000000000000")
   return uuid3(namespace, domain)
 ```
 
-Used internally by `namespace_uuid()`.
+Used internally by `namespace_uuid()`. Results are memoized per domain.
+
+## mkhash()
+
+Generates an `xxhash` 64-bit digest (hex string, 16 characters) for arbitrary input.
+
+```python
+def mkhash(x: Any) -> str
+```
+
+Used for compact identifier hashing (e.g., config/file identifiers in the CLI). Returns the `xxh64` hex digest of the input converted to a string.
+
+## samphash()
+
+Hashes a sampled `polars.DataFrame` for deterministic temp-file naming.
+
+```python
+def samphash(df: pl.DataFrame, n: int = 20) -> str
+```
+
+Samples up to `n` rows (default `20`) from `df`, hashes the sampled bytes via `mkhash()`, and returns the digest string.
 
 ## Next Steps
 

tablassert-7.5.1/docs/changelog.md

@@ -0,0 +1,16 @@
+# Changelog
+
+The canonical release history lives in the repository root at [`CHANGELOG.md`](https://github.com/SkyeAv/Tablassert/blob/main/CHANGELOG.md).
+
+## Current Release Notes
+
+## 7.5.1 - 2026-07-01
+
+### Changed
+- Numeric annotation columns are now coerced and emitted as controlled-notation strings in NDJSON output instead of raw values. Two new pipeline steps wired into `Tcode.collect()` (`lib.py`): `clean_numeric()` lazily casts matching columns to `Float64` with `strict=False` (non-numeric entries drop to null), and `format_numeric()` renders them as strings — p-value columns (any name containing `"p value"`, case-insensitive) in scientific notation (`{:.4e}`), and `relationship strength` / `sample size` in decimal general format (`{:.4g}`, ≥4 significant figures). Non-matching columns are left untouched, and nulls are subsequently dropped by `strip_nulls()`. `math_op()` now also casts with `strict=False` so it tolerates residual junk in numeric annotation columns. `format_numeric()` formats via numpy-backed batch conversion rather than `map_elements` for throughput.
+- Removed dead `pl.Config(set_fmt_float=...)` and `pl.Config(float_precision=...)` context managers from `compile_graph()` (`lib.py`); they were no-ops for NDJSON serialization (`write_ndjson` emits raw f64 via serde shortest-repr and ignores float display options), and the `fmt`/`precision` parameters of `compile_graph()` were removed alongside them.
+- Fixed an off-by-one in the build/validate progress bar so each section loop now shows the configuration currently being processed instead of the last-completed one. `PipelineProgress.section_loop()` (`progress.py`) previously returned a single `advance(info)` callback that set the description and ticked the completed counter together, called after each item's work — so while section *K* ran the bar still displayed section *K−1*. It now returns a `(start, advance)` pair: `start(info)` updates the description to the in-flight item without incrementing, and `advance()` ticks the counter afterwards (so the counter never claims an in-flight item is complete). All five call sites in `cli.py` (TCode build, Collect, Subgraph, Graph, Validate) were updated to `start(...)` before the work and `advance()` after; the long-running Collect and Subgraph stages continue to show the full `format_section_oneline()` summary (including the `CONFIG` name) of the in-flight section.
+
+### Added
+- Fifteen regression tests in `test_lib.py` covering `numeric_columns()` detection (p-value substring, exact-name match, case-insensitivity), `clean_numeric()` (parse/coerce numeric and scientific notation, null out non-numeric junk, leave non-matching columns untouched, noop, idempotent on Float64), `format_numeric()` (scientific notation for p-value, decimal general format for relationship strength/sample size, null preservation, floating-point-noise cleaning, noop), null-stripped NDJSON rows, `compile_graph()` NDJSON emission, and `sig()` operating over a cleaned Float64 p-value column.
+- Three regression tests in `tests/test_progress.py` pinning the new two-callback contract: `start` shows the in-flight item with the counter still at zero, `advance` ticks the counter without altering the description, and a `start`/`advance` cycle keeps the description synced to the current item rather than the previous one.

{tablassert-7.4.14 → tablassert-7.5.1}/docs/cli.md

@@ -26,7 +26,7 @@ Prints the installed Tablassert version to stdout and exits. This is a flag on t
 
 ## build
 
-Build A KGX Compliant Knowledge Graph From A Graph Configuration File
+Build a knowledge graph from a YAML configuration file.
 
 ### Synopsis
 
@@ -70,7 +70,7 @@ See [Graph Configuration](configuration/graph.md) for details on the YAML schema
 
 ## validate
 
-Verify The Syntax Of A Declarative Table Configuration File
+Validate section syntax from a YAML configuration file.
 
 ### Synopsis
 

{tablassert-7.4.14 → tablassert-7.5.1}/docs/configuration/advanced-example.md

@@ -23,7 +23,7 @@ template:
     local: ./DATALAKE/ALAM.XLSX
     url: https://pmc.ncbi.nlm.nih.gov/articles/instance/11708054/bin/mbio.01679-24-s0006.xlsx
     row_slice:
-      - 2  # Skip first row (header)
+      - 2  # Skip the first two rows (title + header)
       - auto  # Read to end
     sheet: all correlations
 
@@ -39,9 +39,9 @@ template:
       avoid:
         - Gene
 
-      # Remove invalid entries
+      # Strip non-value text in place (rows are not dropped)
       remove:
-        - "^NA "  # Rows starting with "NA "
+        - "^NA "  # Cells starting with "NA "
 
       # Clean taxonomic names with regex
       regex:
@@ -66,7 +66,7 @@ template:
   # Provenance: Publication and curation info
   provenance:
     repo: PMC
-    publication: 11708054
+    publication: PMC11708054
     contributors:
       - kind: curation
         name: Skye Lane Goetz
@@ -147,7 +147,7 @@ The subject field uses three regex transformations in sequence:
 ```
 `"Lactobacillus sp"` → `"Lactobacillus sp. "`
 
-> **Regex constraint:** Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Capturing groups (`(...)` / `\1`) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?<!...)`) are not allowed** and will fail validation. Express transformations as a sequence of simple anchored / character-class substitutions instead — the pipeline above is a deliberate three-step chain because no single capturing-group pattern is permitted. If the transformation can't be expressed without those features, capture the leftover context in a `miscellaneous notes` annotation rather than fighting the regex engine.
+> **Regex constraint:** Each `pattern` is handed to Polars `str.replace_all()` (Rust `regex` crate). **Backreferences (`\1`, `\2`, …) and lookarounds (`(?=...)`, `(?<=...)`, `(?!...)`, `(?<!...)`) are not allowed** and will fail validation. Plain groups `(...)` and non-capturing groups `(?:...)` *are* supported. Express transformations as a sequence of simple anchored / character-class substitutions where possible — the pipeline above is a deliberate three-step chain. If the transformation can't be expressed without those unsupported features, capture the leftover context in a `miscellaneous notes` annotation rather than fighting the regex engine.
 
 ### Taxonomic Filtering
 
@@ -226,6 +226,7 @@ template:
 
   statement:
     subject:
+      method: column
       encoding: A
       prioritize: [OrganismTaxon]
       avoid: [Gene]
@@ -239,18 +240,20 @@ sections:
   # Section 1: Positive correlations
   - statement:
       predicate: positively_correlated_with
-    reindex:
-      - column: B  # Correlation coefficient
-        comparison: gt
-        comparator: 0
+    source:
+      reindex:
+        - column: B  # Correlation coefficient
+          comparison: gt
+          comparator: 0
 
   # Section 2: Negative correlations
   - statement:
       predicate: negatively_correlated_with
-    reindex:
-      - column: B
-        comparison: lt
-        comparator: 0
+    source:
+      reindex:
+        - column: B
+          comparison: lt
+          comparator: 0
 ```
 
 This produces two sets of edges from one table:
@@ -296,14 +299,9 @@ template:
         - pattern: _
           replacement: ' '   # "Lactobacillus_rhamnosus" → "Lactobacillus rhamnosus"
 
-    qualifiers:
-      - qualifier: p value
-        method: column
-        encoding: E
-
   provenance:
     repo: PMC
-    publication: 12345678
+    publication: PMC12345678
     contributors:
       - kind: curation
         name: Skye Lane Goetz
@@ -324,7 +322,7 @@ template:
 
 **Both nodes from columns:** Setting `method: column` on both subject and object means both undergo entity resolution via `resolve()`. Each gets its own `prioritize` list to guide disambiguation.
 
-**`remove` vs `regex`:** `remove` filters out entire rows matching a pattern before resolution. `regex` transforms the column value in-place before resolution.
+**`remove` vs `regex`:** Both transform cell text in place before resolution (neither drops rows). `remove` strips each listed regex pattern by replacing it with an empty string; `regex` applies an ordered `pattern`→`replacement` substitution list.
 
 ---
 
@@ -364,7 +362,7 @@ template:
 
   provenance:
     repo: PMC
-    publication: 87654321
+    publication: PMC87654321
     contributors:
       - kind: curation
         name: Skye Lane Goetz

{tablassert-7.4.14 → tablassert-7.5.1}/docs/configuration/graph.md

@@ -16,7 +16,6 @@ A graph configuration file specifies:
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `syntax` | String | Configuration version (must be `"GC2"`) |
 | `name` | String | Knowledge graph name (used in output filename) |
 | `version` | String | Knowledge graph version (used in output filename) |
 | `tables` | List[Path] | Paths to table configuration YAML files |
@@ -26,6 +25,7 @@ A graph configuration file specifies:
 
 | Field | Type | Description |
 |-------|------|-------------|
+| `syntax` | String | Configuration version. Defaults to `"GC2"` (also accepts `"TC3"`); overriding is not recommended. |
 | `pubmed_db` | Path | Path to SQLite PubMed metadata database |
 | `pmc_db` | Path | Path to SQLite PMC figure captions database |
 | `log` | Boolean | Log unmatched entities and audit details during graph builds |
@@ -35,7 +35,7 @@ A graph configuration file specifies:
 
 **`syntax: "GC2"`**
 
-Configuration syntax version. Must be `"GC2"`.
+Configuration syntax version. Defaults to `"GC2"` (the type also accepts `"TC3"`); overriding the default is not recommended.
 
 **`name: string`**
 
@@ -61,7 +61,7 @@ This field only controls whether QC runs. Install `tablassert[qc]` or `tablasser
 
 **`tables: list[path]`**
 
-List of table configuration file paths. Can be absolute or relative to graph config location.
+List of table configuration file paths. Can be absolute or relative to the current working directory (paths are resolved against the process CWD, not the graph-config file location).
 
 Each table config defines:
 - Data source (Excel/CSV/TSV)
@@ -101,8 +101,7 @@ When provided, this is used when provenance specifies PMC publications.
 
 Paths can be:
 - **Absolute:** `/home/user/data/datassert`
-- **Relative to graph config:** `./tables/table1.yaml`
-- **Relative to current directory:** `../configs/table.yaml`
+- **Relative to the current working directory:** `./tables/table1.yaml` (note: paths are resolved against the process CWD, not the graph-config file location — there is no config-relative resolver)
 
 ## Minimal Example
 
@@ -139,16 +138,17 @@ pmc_db: /databases/PMCSuppCaptions.db
 When you run `tablassert build graph.yaml`:
 
 1. **Load graph configuration** - Parse YAML, validate schema
-2. **For each table in `tables`:**
-   - Load table configuration
+2. **Load table configurations** - Parse each YAML in `tables`
+3. **Extract sections** - Expand templates into per-section `Tcode` instances
+4. **Collect instructions (per section):**
    - Download source file (if URL specified)
-   - Apply transformations
-   - Resolve entities using `datassert`
-    - Validate with QC pipeline when `qc: true`
-   - Create subgraph parquet file
-3. **Aggregate subgraphs** - Merge all parquet files
-4. **Add provenance (optional)** - Query `pubmed_db` and `pmc_db` for metadata when configured
-5. **Export NDJSON** - Generate `{name}_{version}.nodes.ndjson` and `.edges.ndjson`
+   - Apply transformations and resolve entities using `datassert`
+   - Validate with the QC audit when `qc: true`
+   - Enrich with provenance: query `pubmed_db` (MeSH) and `pmc_db` (captions) when configured
+5. **Build subgraphs** - Compile each section's resolved data into a parquet file
+6. **Compile graph** - Aggregate all subgraph parquets and export `{name}_{version}.nodes.ndjson` / `.edges.ndjson`
+
+> Note: provenance enrichment (`with_mesh`/`with_captions`) and QC both run during the per-section Collect Instructions stage, *before* subgraphs are built — not as a separate post-aggregation step.
 
 ## Output Files
 
@@ -173,8 +173,8 @@ version: UNSTABLE
 tables:
   - /local_raid1/sgoetz/STORE/CONFIG/TABLASSERT/TABLE/V6/ALAMV6.yaml
 datassert: /local_raid1/sgoetz/CODE/DATASSERT/datassert
-pubmed_db: /local_raid1/sgoetz/DBSTORE/local_raid1/sgoetz/DBSTORE/PUBMED/PubMed.db
-pmc_db: /local_raid1/sgoetz/DBSTORE/local_raid1/sgoetz/DBSTORE/CAPTIONS/PMCSuppCaptions.db
+pubmed_db: /local_raid1/sgoetz/DBSTORE/PUBMED/PubMed.db
+pmc_db: /local_raid1/sgoetz/DBSTORE/CAPTIONS/PMCSuppCaptions.db
 ```
 
 This processes a single table configuration (ALAMV6.yaml) into a knowledge graph named `MULTIOMICS_KG_UNSTABLE`.