tejtm 0.1.0__tar.gz

tejtm-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+dist/
+*.egg-info/
+.scratch/

tejtm-0.1.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.5
+Name: tejtm
+Version: 0.1.0
+Summary: Schema mapping with composite confidence for tabular imports
+Requires-Python: >=3.11
+Requires-Dist: rapidfuzz>=3.0
+Provides-Extra: dev
+Requires-Dist: email-validator>=2.0; extra == 'dev'
+Requires-Dist: openpyxl>=3.1; extra == 'dev'
+Requires-Dist: pydantic>=2.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: phone
+Requires-Dist: phonenumbers>=8.13; extra == 'phone'
+Provides-Extra: pydantic
+Requires-Dist: email-validator>=2.0; extra == 'pydantic'
+Requires-Dist: pydantic>=2.0; extra == 'pydantic'
+Provides-Extra: semantic
+Requires-Dist: fastembed>=0.4; extra == 'semantic'
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6; extra == 'yaml'
+Description-Content-Type: text/markdown
+
+# tejtm
+
+tejtm maps messy import columns to a target schema you define. It returns one-to-one pairs with a composite confidence and a per-scorer breakdown. It does not read files, join records, or validate rows.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+```
+
+Optional extras:
+
+```bash
+pip install -e ".[semantic,pydantic,yaml]"
+```
+
+`semantic` adds FastEmbed header similarity. `pydantic` builds a target schema from a model. `yaml` saves mapping results.
+
+## Map columns to a target schema
+
+The library takes column names plus sample values. You keep file I/O in your app.
+
+```python
+from tejtm import MappingEngine, MappingConfig, SourceColumn, SourceProfile, TargetField, TargetSchema
+
+source = SourceProfile(columns=[
+    SourceColumn("phone", samples=("+1 555-0100",)),
+    SourceColumn("contact", samples=("user@example.com",)),
+    SourceColumn("Cust Name", samples=("Ada Lovelace",)),
+])
+target = TargetSchema(fields=[
+    TargetField("phone_number", aliases=("phone",), expected_semantic_type="phone"),
+    TargetField("email", expected_semantic_type="email"),
+    TargetField("full_name", aliases=("name", "customer name")),
+])
+result = MappingEngine().propose_mappings(
+    source,
+    target,
+    config=MappingConfig(min_confidence=0.5),
+)
+for mapping in result.mappings:
+    print(mapping.source_column, "->", mapping.target_field, mapping.confidence)
+print("unmapped", result.unmapped_columns)
+print("unmatched", result.unmatched_fields)
+```
+
+You should see `phone -> phone_number`, `contact -> email`, and `Cust Name -> full_name`. Weak pairs stay out of `mappings` and land in `unmapped_columns` or `unmatched_fields`.
+
+Default `min_confidence` is `0.55`. Lower it for messy vendor headers, then auto-apply only the pairs above a threshold you pick.
+
+## Apply the mapping
+
+tejtm does not rewrite rows. Use `source_column` and `target_field` to rename keys in your own loop. For cell casting after rename, call `coerce_value` or `coerce_record`. For a Pydantic model, see [How to customize mapping](docs/customization.md).
+
+## Fixture results
+
+The six directories under `tests/fixtures/scenarios/` are the only sector samples in this repo. They are small synthetic tables, not a customer corpus. Treat a fixture as passing when every `expect_mappings` pair lands in `result.mappings` at that fixture's `min_confidence`.
+
+Measured 29 Oct 2026 in this checkout, FastEmbed absent, two identical runs. `header_embedding` abstained. Scores come from `exact_name`, `fuzzy_name`, `semantic_type`, and `value_intent` only. The limiter is fixture size (2 or 3 columns) and those lowered thresholds, not production files.
+
+| Fixture | Format | min_confidence | Mapped / expected | Lowest mapped pair |
+| --- | --- | --- | --- | --- |
+| `crm_contacts` | csv | 0.45 | 3 / 3 | `contact` -> `email` (0.549) |
+| `edge_duplicate_generic_headers` | csv | 0.35 | 2 / 2 | `ref_a` -> `order_reference` (0.601) |
+| `finance_ledger` | xlsx | 0.4 | 3 / 3 | `GL Code` -> `ledger_account` (0.752) |
+| `healthcare_appointments` | csv | 0.4 | 3 / 3 | `Clinic` -> `facility_name` (0.480) |
+| `hr_payroll` | csv | 0.4 | 3 / 3 | `Gross Pay` -> `gross_amount` (0.569) |
+| `warehouse_inventory` | csv | 0.4 | 3 / 3 | `Bin Loc` -> `storage_location` (0.807) |
+
+All 17 expected pairs mapped at those fixture thresholds. At the library default of `0.55`, 15 of 17 still map. The two that drop are `contact` -> `email` in `crm_contacts` (0.549) and `Clinic` -> `facility_name` in `healthcare_appointments` (0.480). See [About mapping limits](docs/limitations.md) for why demos sit below the default.
+
+`.venv/bin/python -m pytest -q` reported 52 passed in this checkout. That count is the test suite, not a mapping-accuracy figure.
+
+Regenerate the table:
+
+```bash
+.venv/bin/python tests/support/fixture_stats.py
+```
+
+## What to read next
+
+- [How to customize mapping](docs/customization.md) covers weights, overrides, custom scorers, exemplars, Pydantic, and YAML.
+- [About mapping limits](docs/limitations.md) covers joins, duplicate headers, optional extras, and unused config.
+- [Product spec](docs/specs/schema-mapping-spec.md) is the v1 contract.
+- [Glossary](docs/GLOSSARY.md) defines source column, target field, and value intent.
+
+Run tests with the project virtualenv:
+
+```bash
+.venv/bin/python -m pytest -q
+```

tejtm-0.1.0/README.md

@@ -0,0 +1,91 @@
+# tejtm
+
+tejtm maps messy import columns to a target schema you define. It returns one-to-one pairs with a composite confidence and a per-scorer breakdown. It does not read files, join records, or validate rows.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+```
+
+Optional extras:
+
+```bash
+pip install -e ".[semantic,pydantic,yaml]"
+```
+
+`semantic` adds FastEmbed header similarity. `pydantic` builds a target schema from a model. `yaml` saves mapping results.
+
+## Map columns to a target schema
+
+The library takes column names plus sample values. You keep file I/O in your app.
+
+```python
+from tejtm import MappingEngine, MappingConfig, SourceColumn, SourceProfile, TargetField, TargetSchema
+
+source = SourceProfile(columns=[
+    SourceColumn("phone", samples=("+1 555-0100",)),
+    SourceColumn("contact", samples=("user@example.com",)),
+    SourceColumn("Cust Name", samples=("Ada Lovelace",)),
+])
+target = TargetSchema(fields=[
+    TargetField("phone_number", aliases=("phone",), expected_semantic_type="phone"),
+    TargetField("email", expected_semantic_type="email"),
+    TargetField("full_name", aliases=("name", "customer name")),
+])
+result = MappingEngine().propose_mappings(
+    source,
+    target,
+    config=MappingConfig(min_confidence=0.5),
+)
+for mapping in result.mappings:
+    print(mapping.source_column, "->", mapping.target_field, mapping.confidence)
+print("unmapped", result.unmapped_columns)
+print("unmatched", result.unmatched_fields)
+```
+
+You should see `phone -> phone_number`, `contact -> email`, and `Cust Name -> full_name`. Weak pairs stay out of `mappings` and land in `unmapped_columns` or `unmatched_fields`.
+
+Default `min_confidence` is `0.55`. Lower it for messy vendor headers, then auto-apply only the pairs above a threshold you pick.
+
+## Apply the mapping
+
+tejtm does not rewrite rows. Use `source_column` and `target_field` to rename keys in your own loop. For cell casting after rename, call `coerce_value` or `coerce_record`. For a Pydantic model, see [How to customize mapping](docs/customization.md).
+
+## Fixture results
+
+The six directories under `tests/fixtures/scenarios/` are the only sector samples in this repo. They are small synthetic tables, not a customer corpus. Treat a fixture as passing when every `expect_mappings` pair lands in `result.mappings` at that fixture's `min_confidence`.
+
+Measured 29 Oct 2026 in this checkout, FastEmbed absent, two identical runs. `header_embedding` abstained. Scores come from `exact_name`, `fuzzy_name`, `semantic_type`, and `value_intent` only. The limiter is fixture size (2 or 3 columns) and those lowered thresholds, not production files.
+
+| Fixture | Format | min_confidence | Mapped / expected | Lowest mapped pair |
+| --- | --- | --- | --- | --- |
+| `crm_contacts` | csv | 0.45 | 3 / 3 | `contact` -> `email` (0.549) |
+| `edge_duplicate_generic_headers` | csv | 0.35 | 2 / 2 | `ref_a` -> `order_reference` (0.601) |
+| `finance_ledger` | xlsx | 0.4 | 3 / 3 | `GL Code` -> `ledger_account` (0.752) |
+| `healthcare_appointments` | csv | 0.4 | 3 / 3 | `Clinic` -> `facility_name` (0.480) |
+| `hr_payroll` | csv | 0.4 | 3 / 3 | `Gross Pay` -> `gross_amount` (0.569) |
+| `warehouse_inventory` | csv | 0.4 | 3 / 3 | `Bin Loc` -> `storage_location` (0.807) |
+
+All 17 expected pairs mapped at those fixture thresholds. At the library default of `0.55`, 15 of 17 still map. The two that drop are `contact` -> `email` in `crm_contacts` (0.549) and `Clinic` -> `facility_name` in `healthcare_appointments` (0.480). See [About mapping limits](docs/limitations.md) for why demos sit below the default.
+
+`.venv/bin/python -m pytest -q` reported 52 passed in this checkout. That count is the test suite, not a mapping-accuracy figure.
+
+Regenerate the table:
+
+```bash
+.venv/bin/python tests/support/fixture_stats.py
+```
+
+## What to read next
+
+- [How to customize mapping](docs/customization.md) covers weights, overrides, custom scorers, exemplars, Pydantic, and YAML.
+- [About mapping limits](docs/limitations.md) covers joins, duplicate headers, optional extras, and unused config.
+- [Product spec](docs/specs/schema-mapping-spec.md) is the v1 contract.
+- [Glossary](docs/GLOSSARY.md) defines source column, target field, and value intent.
+
+Run tests with the project virtualenv:
+
+```bash
+.venv/bin/python -m pytest -q
+```

tejtm-0.1.0/docs/GLOSSARY.md

@@ -0,0 +1,67 @@
+# Glossary
+
+Terms for the schema-mapping library. No implementation detail.
+
+## Source schema
+
+The column names and optional sample values from an incoming file or table. The user does not need to declare a rigid schema before import.
+
+## Target schema
+
+The canonical fields the application expects. Each field has an identifier, optional human labels, optional aliases, and optional expected semantic type.
+
+## Field
+
+One slot in the target schema. It is the unit of mapping.
+
+## Column
+
+One labeled series in the source data. It is the unit on the source side.
+
+## Mapping
+
+A proposed link from one source column to one target field, with a confidence score and an explanation of score factors.
+
+## Semantic type
+
+A coarse label for what values in a column mean. Examples: email, phone, date, freetext. Detection uses heuristics on sampled cell values.
+
+## Header similarity
+
+Similarity between source column name text and target field name or alias text. It uses string metrics and optional embedding cosine similarity.
+
+## Value intent
+
+Inference of field meaning from value patterns when headers are ambiguous. The library uses value signatures and integrator metadata, not sector-specific rules.
+
+## Value signature
+
+A compact summary of a column’s sample values: detected semantic type plus format statistics (length, digit ratio, uniqueness, and similar measures).
+
+## Exemplar samples
+
+Optional example values on a target field. They describe what correct data should look like when headers are generic or duplicated.
+
+## Scorer
+
+A pluggable component that returns a partial score in `[0, 1]` or abstains. The engine combines scorer outputs.
+
+## Score breakdown
+
+A structured record of each scorer name, weight, raw score, and short reason. The host app can show this in a review UI.
+
+## Assignment
+
+The step that picks a global set of mappings when many column–field pairs have scores. It enforces business rules such as one-to-one links.
+
+## Unmapped column
+
+A source column with no accepted mapping above the threshold.
+
+## Unmatched field
+
+A target field with no source column assigned.
+
+## Transform
+
+A separate step that merges or splits columns. It is not the same as a one-to-one mapping.

tejtm-0.1.0/docs/README.md

@@ -0,0 +1,25 @@
+# Schema mapping library writeups
+
+User-facing guides live next to this index. Start with the root [README](../README.md).
+
+- [How to customize mapping](./customization.md)
+- [About mapping limits](./limitations.md)
+
+This folder also holds research, the wayfinder map index, glossary, and the product spec.
+
+## Documents
+
+| File | Purpose |
+|------|---------|
+| [GLOSSARY.md](./GLOSSARY.md) | Domain terms |
+| [architecture-frontier.md](./architecture-frontier.md) | STE summary of the technical frontier |
+| [research-prior-art.md](./research-prior-art.md) | Surveys, matchers, Python ecosystem |
+| [research-string-heuristics.md](./research-string-heuristics.md) | Name matching metrics |
+| [research-type-detection.md](./research-type-detection.md) | Value-based semantic types |
+| [research-semantic-similarity.md](./research-semantic-similarity.md) | Embeddings and value intent |
+| [primary-sources.md](./primary-sources.md) | Bibliography URLs |
+| [specs/schema-mapping-spec.md](./specs/schema-mapping-spec.md) | Product spec (`ready-for-agent`) |
+
+## Wayfinder map
+
+Decision tickets live under [`.scratch/schema-mapping/`](../.scratch/schema-mapping/map.md). Open grilling tickets cover assignment policy and v1 scope.

tejtm-0.1.0/docs/adr/0001-value-intent-via-signatures.md

@@ -0,0 +1,23 @@
+# ADR 0001: Value intent via signatures and integrator metadata
+
+## Status
+
+Accepted
+
+## Context
+
+Homonymous headers (`name`, `id`, `date`) are common in imports. Header and type scorers alone mis-assign when two columns share a label or generic type. An early `ValueIntentScorer` hard-coded product/customer vocabulary, which conflicts with a general-purpose library.
+
+## Decision
+
+- Compute a **value signature** from column samples: detected semantic type, match confidence, and format statistics (length, digit ratio, uniqueness, whitespace ratio).
+- Compute a **field expectation** from integrator metadata only:
+  - `expected_semantic_type` maps to a generic type profile.
+  - `exemplar_samples` (optional) defines what values should look like; signature similarity is the main disambiguation tool.
+  - `description` (optional) participates only through the optional embedding scorer path, not keyword rules.
+- Sector-specific behavior belongs in user-provided exemplars, aliases, and types—not in core code.
+
+## Consequences
+
+- `TargetField` gains `exemplar_samples`.
+- Tests use scenario fixtures across industries; expectations are declared in manifest data, not in Python branches.

tejtm-0.1.0/docs/architecture-frontier.md

@@ -0,0 +1,72 @@
+# Architecture frontier
+
+ASD-STE100 strict summary. Detail lives in linked research files.
+
+## Problem
+
+Import features often reject files when headers do not match a fixed schema. Users need automatic mapping with visible confidence. The library must work for any user-defined target schema and any messy source headers.
+
+## Reference pipeline
+
+1. **Ingest profiles.** Read source column names and a sample of values per column. Read target field ids, labels, aliases, and optional expected types.
+2. **Normalize names.** Apply Unicode normalization, casefold, punctuation cleanup, and camelCase or snake_case tokenization.
+3. **Profile values.** Run semantic type detectors on samples. Store type label and confidence per column and per field expectation.
+4. **Score pairs.** For each source column and target field, run scorers. Combine with explicit weights into one score and a breakdown.
+5. **Assign globally.** Solve one-to-one assignment on the score matrix. Mark low scores as unmapped.
+6. **Return result.** Emit mappings, unmapped columns, unmatched fields, and per-pair breakdowns. The host applies renames, transforms, and validation.
+
+This mirrors COMA-style composition ([COMA PDF](https://dbs.uni-leipzig.de/files/research/publications/2002-1/pdf/COMA.pdf)) and products such as infermap and tabmodel.
+
+## Scorer set (recommended v1)
+
+| Scorer | Signal | Default weight | Notes |
+|--------|--------|----------------|-------|
+| ExactName | Normalized id or alias equality | 1.0 | Highest priority |
+| FuzzyName | Jaro-Winkler, ratio, token_sort, token_set max or blend | 0.45 | See [research-string-heuristics.md](./research-string-heuristics.md) |
+| SemanticType | Agreement between detected source type and field expectation | 0.25 | See [research-type-detection.md](./research-type-detection.md) |
+| HeaderEmbedding | Cosine similarity of column name vs field name+aliases | 0.20 | Optional extra; see [research-semantic-similarity.md](./research-semantic-similarity.md) |
+| ValueIntent | Value signature versus type profile or `exemplar_samples` | 0.30 | Runs on every pair that has a signature. Not gated on header ties. |
+
+Weights are user-tunable. A scorer that returns `None` does not enter the denominator. A score of `0.0` still does.
+
+## Semantic type catalog (built-in)
+
+Implemented by `detect_semantic_type`: `boolean`, `integer`, `number`, `freetext`, `email`, `phone`, `url`, `uuid`, `date`, `datetime`, `time`, `currency`.
+
+Not implemented: `postal_code`, `country_code`, `address_line`. Use aliases, exemplars, or a custom scorer.
+
+Each detector returns match rate on the sample and a reason string for debugging.
+
+## Value intent (homonym disambiguation)
+
+General-purpose only:
+
+1. Build a **value signature** per column (semantic type + format stats + dominant token prefix).
+2. Compare to **field expectation** from `expected_semantic_type` (generic type profile), `exemplar_samples` (integrator-provided), and optional description embedding when `[semantic]` is installed.
+3. No sector keywords in core code. See ADR `docs/adr/0001-value-intent-via-signatures.md`.
+
+## Assignment
+
+Use the Hungarian algorithm on the negated score matrix for one-to-one assignment. Enforce a minimum score cutoff. Leave source columns and target fields unpaired when below cutoff.
+
+Greedy assignment is a fallback when scipy is not installed. Document the dependency.
+
+## Lightweight default install
+
+- **Core:** Python 3.11+, `rapidfuzz`, stdlib `re` / `datetime` / `unicodedata`.
+- **Optional `[semantic]`:** `fastembed` for header embeddings.
+- **Optional `[phone]`:** `phonenumbers` for stricter phone detection.
+- **No GPU.** No PyTorch in the default wheel.
+
+## Differentiation from existing packages
+
+| Need | infermap / tabmodel | This library |
+|------|---------------------|--------------|
+| Framework | Often tied to files or Pydantic | Schema objects only; host supplies data |
+| Score transparency | Partial | Required breakdown per mapping |
+| Extensibility | Fixed scorer list | Register custom scorers and weights |
+| Value intent | Profile stats | Explicit homonym disambiguation pass |
+
+## Open decisions (grilling)
+
+See wayfinder tickets [05-global-assignment-grilling](../.scratch/schema-mapping/issues/05-global-assignment-grilling.md) and [06-library-boundaries-grilling](../.scratch/schema-mapping/issues/06-library-boundaries-grilling.md). The spec uses recommended defaults until you confirm otherwise.

tejtm-0.1.0/docs/customization.md

@@ -0,0 +1,131 @@
+# How to customize mapping
+
+Use these steps when the default engine maps the wrong column, or when you want to replay a reviewer's edits. The public call stays `MappingEngine.propose_mappings`.
+
+## Bias scorers with weights
+
+Pass scorer names to `MappingConfig.weights`. Missing names keep the defaults in `DEFAULT_WEIGHTS`.
+
+| Scorer | Default weight | Signal |
+| --- | --- | --- |
+| `exact_name` | 1.0 | Normalized header or alias equality |
+| `fuzzy_name` | 0.45 | RapidFuzz similarity on headers |
+| `semantic_type` | 0.25 | Sample type versus `expected_semantic_type` |
+| `header_embedding` | 0.20 | Optional FastEmbed cosine on headers |
+| `value_intent` | 0.30 | Value signature versus type profile or `exemplar_samples` |
+
+A scorer that returns `None` does not enter the weighted mean. A `semantic_type` mismatch returns score `0.0` with a weight, which lowers the pair.
+
+```python
+from tejtm import DEFAULT_WEIGHTS, MappingConfig, MappingEngine
+
+print(DEFAULT_WEIGHTS)
+config = MappingConfig(
+    min_confidence=0.4,
+    weights={"fuzzy_name": 0.2, "semantic_type": 1.0},
+)
+result = MappingEngine().propose_mappings(source, target, config=config)
+```
+
+## Force a pair with overrides
+
+Overrides skip inference for those columns and set confidence to `1.0`. They also reserve the target field so assignment cannot reuse it.
+
+```python
+config = MappingConfig(overrides={"Contact": "phone"}, min_confidence=0.4)
+```
+
+To replay a previous `MappingResult` on the next file, copy `source_column` to `target_field` into `overrides`. High `min_confidence` then applies only to columns the reviewer did not lock.
+
+## Disambiguate generic headers with exemplars
+
+When two columns share a vague name shape, put example values on the target field. The value-intent scorer compares sample signatures to those exemplars.
+
+```python
+TargetField(
+    "order_reference",
+    expected_semantic_type="freetext",
+    exemplar_samples=("ORD-500", "ORD-501"),
+)
+```
+
+Sector words do not belong in library code. Put them in aliases, types, or exemplars.
+
+## Register a custom scorer
+
+A scorer implements `name` and `score(column, field, context)`. Return `ScorerResult` or `None`. Pass the tuple into `MappingEngine(scorers=...)`.
+
+```python
+from tejtm.engine import MappingEngine
+from tejtm.models import ScorerResult
+from tejtm.scorers.base import ScoringContext
+
+class AlwaysPhone:
+    name = "always_phone"
+
+    def score(self, column, field, context: ScoringContext):
+        if field.id == "phone_number":
+            return ScorerResult(self.name, 0.99, 1.0, "domain hook")
+        return None
+
+engine = MappingEngine(scorers=(AlwaysPhone(),))
+```
+
+You replace the default scorer tuple when you pass `scorers`. Include the built-ins you still want.
+
+## Build the target schema from Pydantic
+
+Install `tejtm[pydantic]`. Field title becomes a label. Put aliases and types in `json_schema_extra`.
+
+```python
+from pydantic import BaseModel, EmailStr, Field
+from tejtm.pydantic_adapter import propose_and_apply
+from tejtm.models import MappingConfig
+
+class Customer(BaseModel):
+    name: str = Field(title="Full Name")
+    email: EmailStr = Field(
+        json_schema_extra={
+            "tejtm_aliases": ["Contact", "E-mail"],
+            "semantic_type": "email",
+        }
+    )
+
+result = propose_and_apply(
+    Customer,
+    [{"Contact": "a@example.com", "Full Name": "Ada"}],
+    config=MappingConfig(min_confidence=0.4),
+)
+print(result.column_rename)
+print(result.valid_rows)
+```
+
+`propose_and_apply` maps, renames keys, coerces fields that have `semantic_type`, then calls `model_validate`. Row errors land in `result.errors`.
+
+## Coerce cells after you rename
+
+```python
+from tejtm import coerce_value, coerce_record
+
+coerce_value("42", "integer")  # 42
+coerce_record({"amount": "$12.00"}, {"amount": "currency"})
+```
+
+Failed casts leave the original string. Email and URL stay strings.
+
+## Save a mapping result
+
+Install `tejtm[yaml]`. `MappingResult.to_dict()` needs no extra. YAML helpers wrap that dict.
+
+```python
+from tejtm.serialization import mapping_to_yaml, mapping_from_yaml
+
+text = mapping_to_yaml(result)
+restored = mapping_from_yaml(text)
+```
+
+This stores the proposed pairs. It does not store `MappingConfig`. Keep weights and `min_confidence` in your own config file.
+
+## Built-in semantic types
+
+`detect_semantic_type` labels a sample as one of `email`, `phone`, `url`, `uuid`, `date`, `datetime`, `time`, `boolean`, `currency`, `integer`, `number`, or `freetext`. Bare digit strings are not phones. Currency needs a `$`, `€`, or `£` sign, or a thousands separator. A number column can still match a currency field through type compatibility.

tejtm-0.1.0/docs/limitations.md

@@ -0,0 +1,53 @@
+# About mapping limits
+
+tejtm proposes column-to-field pairs. Host code owns files, row transforms, and identity. This page is the boundary, not a backlog of features to expect in the engine.
+
+## Column mapping is not record correlation
+
+The engine never joins two tables, matches people across systems, or flags duplicate rows. Spec out of scope lists duplicate detection next to file readers and web UI. If you need MDM or fuzzy entity match, run that after you rename columns.
+
+`tejtm.__all__` exports mapping types, `MappingEngine`, and coercion helpers. It does not export join or identity APIs.
+
+## One source column, one target field
+
+Assignment is global one-to-one. Two source columns cannot share a target field. Weak pairs fall below `min_confidence` and stay unmapped. Automatic many-to-one merges (split first name plus last name into `full_name`) are out of v1. Do those transforms in the host after mapping.
+
+## Duplicate headers are illegal
+
+`SourceProfile` requires unique `SourceColumn.name` values. `TargetSchema` requires unique field ids. Two Excel columns both labeled `Name` must be uniquified before you call the engine (`Name`, `Name_2`). The engine keys unmapped sets by those names, so a silent collision would drop a column.
+
+Homonyms with different names (`ref_a` versus `ref_b`, or `name` versus `product`) can still split with aliases, types, or `exemplar_samples`.
+
+## You own I/O and validation
+
+The library does not read CSV, Excel, or Parquet. Tests load fixtures through `tests/support/loaders.py` as a host pattern, not as a public module. After mapping, validate with Pydantic, Pandera, or Frictionless. `propose_and_apply` is a convenience adapter, not the core engine.
+
+## Default threshold is stricter than the demos
+
+`MappingConfig.min_confidence` defaults to `0.55`. Sector fixtures often use `0.35` to `0.45`. If you keep the default on messy vendor files, more columns stay unmapped. That is intentional. Auto-apply only what you trust.
+
+## Optional extras change scores
+
+Without `fastembed`, `HeaderEmbeddingScorer` abstains. Scenario tests then run name, type, and value-intent scorers only. Install `tejtm[semantic]` and the first FastEmbed call may download the multilingual MiniLM model. Core mapping does not open network sockets. The embedding extra can.
+
+Without `scipy`, assignment uses a greedy cost fallback instead of the Hungarian algorithm. Pairings can differ on crowded score matrices. Install SciPy if you need the Hungarian path.
+
+Without `openpyxl`, the finance XLSX scenario test cannot load. Production mapping never opens workbooks.
+
+`pyproject.toml` declares a `phone` extra that installs `phonenumbers`. Detectors do not import that package. Phone matching uses the regex in `tejtm.typing.detectors`.
+
+## What architecture docs list that code does not
+
+`docs/architecture-frontier.md` mentions `postal_code`, `country_code`, and `address_line`. `detect_semantic_type` does not implement those labels. They score as `freetext` or `integer` today. Put postal meaning in aliases and exemplars, or register a custom scorer.
+
+`MappingConfig.intent_margin` exists on the dataclass and has no runtime consumer. Setting it does nothing.
+
+Value intent always runs when it can build a signature. `intent_margin` does not gate it.
+
+## LLM scoring and ontology alignment
+
+There is no LLM scorer in the default distribution. OWL, RDF, and structural XML matching are out of scope.
+
+## Confidence is a ranking aid
+
+A high score is not a proof that the column is that field in the business sense. Show `breakdown` to reviewers. Keep a human override path. Replay those overrides on the next file.

tejtm-0.1.0/docs/primary-sources.md

@@ -0,0 +1,56 @@
+# Primary sources (bibliography)
+
+Canonical URLs and identifiers for claims in `research-semantic-similarity.md`. Prefer these over secondary summaries.
+
+## Sentence embeddings (multilingual, CPU)
+
+| ID | Resource | URL |
+|----|----------|-----|
+| [ST-PMM] | Model card: `paraphrase-multilingual-MiniLM-L12-v2` | https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 |
+| [ST-PMM-ONNX] | ONNX port (384-dim, 128 tokens) | https://huggingface.co/onnx-models/paraphrase-multilingual-MiniLM-L12-v2-onnx |
+| [SBERT-MULTI] | SBERT docs: multilingual pretrained models | https://www.sbert.net/docs/sentence_transformer/pretrained_models.html |
+| [REIMERS-KD] | Reimers & Gurevych, multilingual knowledge distillation (arXiv:2004.09813) | https://arxiv.org/abs/2004.09813 |
+| [E5-SMALL] | Model card: `intfloat/multilingual-e5-small` | https://huggingface.co/intfloat/multilingual-e5-small |
+| [E5-SMALL-ONNX] | Official ONNX weights (incl. quantized) | https://huggingface.co/intfloat/multilingual-e5-small/tree/main/onnx |
+| [E5-REPORT] | Wang et al., Multilingual E5 technical report (arXiv:2402.05672) | https://arxiv.org/abs/2402.05672 |
+
+## FastEmbed / ONNX Runtime inference
+
+| ID | Resource | URL |
+|----|----------|-----|
+| [FASTEMBED-README] | Qdrant FastEmbed README | https://github.com/qdrant/fastembed/blob/main/README.md |
+| [FASTEMBED-DOCS] | Qdrant FastEmbed documentation | https://qdrant.tech/documentation/fastembed/ |
+| [FASTEMBED-MODELS] | Supported models table (size, dim, license) | https://qdrant.github.io/fastembed/examples/Supported_Models/ |
+| [ONNXRUNTIME] | ONNX Runtime (inference engine used by FastEmbed) | https://onnxruntime.ai/ |
+
+## GloVe and short-text similarity
+
+| ID | Resource | URL |
+|----|----------|-----|
+| [GLOVE] | Pennington, Socher, Manning — GloVe (EMNLP 2014) | https://nlp.stanford.edu/pubs/glove.pdf |
+| [GLOVE-VECTORS] | Stanford GloVe download / format | https://nlp.stanford.edu/projects/glove/ |
+| [SHORT-TEXT-KENTER] | Kenter & De Rijke, short text similarity with word embeddings (CIKM 2015) | https://doi.org/10.1145/2806416.2806475 |
+
+## Column semantic typing (value-based)
+
+| ID | Resource | URL |
+|----|----------|-----|
+| [SHERLOCK] | Hulsebos et al., Sherlock (KDD 2019; arXiv:1905.10688) | https://doi.org/10.1145/3292500.3330993 · https://arxiv.org/abs/1905.10688 |
+| [SATO] | Zhang et al., Sato (PVLDB 13(11), 2020; arXiv:1911.06311) | https://doi.org/10.14778/3407790.3407793 · https://arxiv.org/abs/1911.06311 · https://www.vldb.org/pvldb/vol13/p1835-zhang.pdf |
+| [DODUO] | Suhara et al., Doduo (SIGMOD 2022; arXiv:2104.01785) | https://doi.org/10.1145/3514221.3517906 · https://arxiv.org/abs/2104.01785 |
+| [DODUO-CODE] | Doduo toolbox (Megagon Labs) | https://github.com/megagonlabs/doduo |
+| [SATO-CODE] | Sato (Megagon Labs) | https://github.com/megagonlabs/sato |
+
+## Header-only / metadata matching (related)
+
+| ID | Resource | URL |
+|----|----------|-----|
+| [CVA] | Column Vocabulary Association — dataless tables (arXiv:2409.13709) | https://arxiv.org/abs/2409.13709 |
+| [GLOSSARY-LLM] | Matching table metadata to business glossaries (OM 2023) | https://arxiv.org/abs/2309.11506 |
+
+## Corpora and type systems referenced by Sherlock/Sato/Doduo
+
+| ID | Resource | URL |
+|----|----------|-----|
+| [T2D-GOLD] | T2Dv2 gold standard (DBpedia ↔ WebTables headers) | http://webdatacommons.org/webtables/goldstandardV2.html |
+| [VIZNET] | VizNet table corpus (Hu et al., 2019) | https://arxiv.org/abs/1907.03629 |