vertiq 0.2.2__tar.gz → 0.2.3__tar.gz

{vertiq-0.2.2/vertiq.egg-info → vertiq-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vertiq
-Version: 0.2.2
+Version: 0.2.3
 Summary: Self-improving decision intelligence: a local language model that maps any operational dataset and decision models trained on real outcomes, improved through recursive self-improvement under an evaluation gate.
 Author: Biswajit Tripathy
 License: Apache-2.0
@@ -298,6 +298,20 @@ Recency baseline AUC 0.761.
 | Auto (chose boosted) | 0.812 | 0.795–0.827 | +0.043 to +0.059 |
 | scikit-learn logistic regression (reference) | 0.807 |  |  |
 
+### Real data across sectors
+
+`vertiq-report` on public data from 5 sectors, nothing tuned per sector; windows of 30, 60 and 90 days, each through the
+gate with its confidence corrected for the three, and its segment check (`benchmarks/sectors/fetch.sh`, then
+`python benchmarks/sectors/run.py`; full reports in `benchmarks/sectors/reports/`).
+
+| Sector | Data | Size | Question | Result | Patterns kept |
+|---|---|---|---|---|---|
+| Retail | UCI Online Retail II: a UK online shop's invoice lines, 2009-2010 | 28,816 records, 9,612 entities | Will a customer buy again, and how much? | Promoted: 30 days, AUC 0.881 vs 0.776; 60 days, AUC 0.888 vs 0.750; 90 days, AUC 0.890 vs 0.708 | 20 of 471 |
+| Food safety | City of Chicago food inspections since 2019 (up to 150,000) | 137,034 records, 27,264 entities | Which establishments will fail an inspection? | Promoted: 30 days, AUC 0.813 vs 0.698; 60 days, AUC 0.822 vs 0.703; 90 days, AUC 0.809 vs 0.692 | 8 of 94 |
+| Animal services | Austin Animal Center intakes (up to 200,000) | 173,812 records, 156,287 entities | Which animals will come back to the shelter? | Not promoted: best AUC 0.870 vs 0.865; the current rule stays | 20 of 266 |
+| Manufacturing | Microsoft Azure predictive maintenance sample: 100 machines, 2015 (errors, failures, machine models) | 4,680 records, 100 entities | Which machines will fail? | Promoted: 60 days, AUC 0.720 vs 0.545. Kept the rule: 30 days, 90 days | 7 of 221 |
+| Transport | NYC motor vehicle collisions since January 2025 (up to 200,000) | 121,970 records, 43,061 entities | Where will collisions happen again? | Promoted: 30 days, AUC 0.971 vs 0.894; 60 days, AUC 0.958 vs 0.859; 90 days, AUC 0.950 vs 0.840. Next value promoted: average error 0.44 vs 0.48 | 5 of 86 |
+
 ### Real exports (held out)
 
 Six public exports from different sectors, never used for training; a role is correct when it is one of the accepted answers

{vertiq-0.2.2 → vertiq-0.2.3}/README.md

@@ -268,6 +268,20 @@ Recency baseline AUC 0.761.
 | Auto (chose boosted) | 0.812 | 0.795–0.827 | +0.043 to +0.059 |
 | scikit-learn logistic regression (reference) | 0.807 |  |  |
 
+### Real data across sectors
+
+`vertiq-report` on public data from 5 sectors, nothing tuned per sector; windows of 30, 60 and 90 days, each through the
+gate with its confidence corrected for the three, and its segment check (`benchmarks/sectors/fetch.sh`, then
+`python benchmarks/sectors/run.py`; full reports in `benchmarks/sectors/reports/`).
+
+| Sector | Data | Size | Question | Result | Patterns kept |
+|---|---|---|---|---|---|
+| Retail | UCI Online Retail II: a UK online shop's invoice lines, 2009-2010 | 28,816 records, 9,612 entities | Will a customer buy again, and how much? | Promoted: 30 days, AUC 0.881 vs 0.776; 60 days, AUC 0.888 vs 0.750; 90 days, AUC 0.890 vs 0.708 | 20 of 471 |
+| Food safety | City of Chicago food inspections since 2019 (up to 150,000) | 137,034 records, 27,264 entities | Which establishments will fail an inspection? | Promoted: 30 days, AUC 0.813 vs 0.698; 60 days, AUC 0.822 vs 0.703; 90 days, AUC 0.809 vs 0.692 | 8 of 94 |
+| Animal services | Austin Animal Center intakes (up to 200,000) | 173,812 records, 156,287 entities | Which animals will come back to the shelter? | Not promoted: best AUC 0.870 vs 0.865; the current rule stays | 20 of 266 |
+| Manufacturing | Microsoft Azure predictive maintenance sample: 100 machines, 2015 (errors, failures, machine models) | 4,680 records, 100 entities | Which machines will fail? | Promoted: 60 days, AUC 0.720 vs 0.545. Kept the rule: 30 days, 90 days | 7 of 221 |
+| Transport | NYC motor vehicle collisions since January 2025 (up to 200,000) | 121,970 records, 43,061 entities | Where will collisions happen again? | Promoted: 30 days, AUC 0.971 vs 0.894; 60 days, AUC 0.958 vs 0.859; 90 days, AUC 0.950 vs 0.840. Next value promoted: average error 0.44 vs 0.48 | 5 of 86 |
+
 ### Real exports (held out)
 
 Six public exports from different sectors, never used for training; a role is correct when it is one of the accepted answers

{vertiq-0.2.2 → vertiq-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "vertiq"
-version = "0.2.2"
+version = "0.2.3"
 description = "Self-improving decision intelligence: a local language model that maps any operational dataset and decision models trained on real outcomes, improved through recursive self-improvement under an evaluation gate."
 readme = "README.md"
 requires-python = ">=3.10"

{vertiq-0.2.2 → vertiq-0.2.3}/tests/test_report.py

@@ -250,3 +250,62 @@ def test_search_inputs_read_as_words():
     from vertiq.menu import words
     assert words("share:bill[@item=Jeans]@180d") == "share of bills including Jeans over 180 days"
     assert words("since:bill[payment_mode=Cash]") == "time since the last bill with payment mode Cash"
+
+
+def _write_csv(path, rows):
+    with open(path, "w", newline="") as f:
+        w = csv.DictWriter(f, fieldnames=list(rows[0]))
+        w.writeheader()
+        w.writerows(rows)
+
+
+def _machines(tmp_path):
+    """100 machines numbered 1 to 100 (small whole numbers), an error log and a failure log, and a machine list."""
+    rnd = random.Random(9)
+    errors, failures = [], []
+    for m in range(1, 101):
+        for _ in range(rnd.randint(10, 40)):
+            errors.append({"datetime": f"2015-{rnd.randint(1, 12):02d}-{rnd.randint(1, 28):02d} 0{rnd.randint(0, 9)}:00:00",
+                           "machineID": str(m), "errorID": rnd.choice(["error1", "error2", "error3"])})
+        for _ in range(rnd.randint(2, 9)):
+            failures.append({"datetime": f"2015-{rnd.randint(1, 12):02d}-{rnd.randint(1, 28):02d} 06:00:00",
+                             "machineID": str(m), "failure": rnd.choice(["comp1", "comp2", "comp3"])})
+    _write_csv(tmp_path / "errors.csv", errors)
+    _write_csv(tmp_path / "failures.csv", failures)
+    _write_csv(tmp_path / "machines.csv", [{"machineID": str(m), "model": f"model{m % 4 + 1}", "age": str(m % 20)} for m in range(1, 101)])
+
+
+def test_several_event_files_merge_on_a_small_number_id(tmp_path):
+    _machines(tmp_path)
+    cfg = {"records": [{"file": "errors.csv", "event": "error"}, {"file": "failures.csv", "event": "failure"}],
+           "entity_table": {"file": "machines.csv"}, "target": "failure", "noun": "events", "entities": "machines"}
+    collect = {}
+    text = report.run(cfg, tmp_path, collect)
+    assert "entity 'machineID'" in text and "Merged:" in text and collect["entities"] == 100
+    assert any(p["target"] == "failure" for p in collect["predictions"])
+    # a failure code exists only on failures: no "failure is comp1 ... have type error" patterns
+    assert not any("have type" in f and "failure is" in f for f in collect["patterns"]["found"])
+
+
+def test_identifiers_are_found_by_name_and_never_taken_as_amounts():
+    from vertiq.tables import id_named, id_worded, propose
+    assert id_named("machineID") and id_named("customer_id") and id_named("Customer ID") and not id_named("paid")
+    assert id_worded("license_") and id_worded("Invoice No") and not id_worded("number_of_persons_injured")
+    rnd = random.Random(2)
+    rows = [{"date": f"2024-{rnd.randint(1, 12):02d}-{rnd.randint(1, 28):02d}", "store_id": str(rnd.randint(1, 40)),
+             "license_": str(rnd.randint(1, 2_500_000)), "amount": f"{rnd.uniform(5, 500):.2f}"} for _ in range(800)]
+    m, _ = propose(rows)
+    assert m.entity == "store_id" and m.value == "amount"
+
+
+def test_an_outcome_named_by_value_sets_the_event_type(tmp_path):
+    rnd = random.Random(4)
+    rows = [{"inspection_id": str(100000 + i), "establishment": f"Shop {rnd.randint(1, 150)}",
+             "inspection_date": f"202{rnd.randint(1, 4)}-{rnd.randint(1, 12):02d}-{rnd.randint(1, 28):02d}",
+             "results": rnd.choice(["Pass", "Pass", "Fail", "Pass w/ Conditions"])} for i in range(1500)]
+    _write_csv(tmp_path / "inspections.csv", rows)
+    collect = {}
+    text = report.run({"records": {"file": "inspections.csv"}, "target": "Fail", "noun": "inspections"}, tmp_path, collect)
+    assert "'Fail' is a value of 'results'" in text
+    assert all(p["target"] == "Fail" for p in collect["predictions"])
+    assert not any("results is" in f for f in collect["patterns"]["found"])   # the type column is not also an attribute

{vertiq-0.2.2 → vertiq-0.2.3}/vertiq/__init__.py

@@ -20,4 +20,4 @@ it likes.
   experiments  comparing message variants against each other and a control group
   verticals    per-vertical settings (windows, nouns, currency, tactics)
 """
-__version__ = "0.2.2"
+__version__ = "0.2.3"

{vertiq-0.2.2 → vertiq-0.2.3}/vertiq/patterns.py

@@ -157,10 +157,20 @@ def find(ds: Dataset, *, item_groups: dict[str, str] | None = None, noun: str =
             tests.append({"kind": "value", "feature": k[1], "n_with": a[0], "n_without": b[0],
                           "with": ma, "without": mb, "p": _mean_p_sums(a, b),
                           "big": abs(ma - mb) >= min_value_gap * abs(overall)})
+    # an attribute recorded on one event type only (a failure code exists only on failures) says nothing new
+    # about the type: comparing the two would report how the data was exported, not a pattern
+    types_of: dict[str, set] = defaultdict(set)
+    for e in events:
+        for k, v in (e.attrs or {}).items():
+            if v not in (None, "", "-"):
+                types_of[f"attr:{k}"].add(e.type)
+    one_type = {src for src, ts in types_of.items() if len(ts) == 1 and len({e.type for e in events}) > 1}
     for i, ka in enumerate(keys):
         for kb in keys[i + 1:]:
             if ka[0] == kb[0]:
                 continue                      # levels of one attribute exclude each other by construction
+            if {ka[0], kb[0]} & one_type and "type" in (ka[0], kb[0]):
+                continue
             if ka[0].startswith("item:") and kb[0].startswith("item:"):
                 continue                      # items bought together: vertiq.affinity, with its own thresholds
             # context (weekday, part of day, group) is the condition; otherwise the first feature is

{vertiq-0.2.2 → vertiq-0.2.3}/vertiq/report.py

@@ -12,6 +12,13 @@ config.json names the tables and how they map to the contract; nothing else is c
    "entity_table": {"file": "customers.csv", "trainable": "trainable"},
    "noun": "orders", "currency": "₹", "entities": "customers"}
 
+"records" can also list several event files about the same entities, each named by its "event" (default: the
+file name), with "target" naming the event to predict. "target" may also be an outcome as written in a column ("Fail" in a
+results column); that column then gives each record's type:
+
+  {"records": [{"file": "errors.csv", "event": "error"}, {"file": "failures.csv", "event": "failure"}],
+   "entity_table": {"file": "machines.csv"}, "target": "failure", "noun": "events", "entities": "machines"}
+
 Predictions try return windows of 30, 60 and 90 days ("windows": [...] or "window_days": N to change), with the
 gate's confidence corrected for the number tried, and search the inputs vertiq.menu proposes from the data
 ("tune": false to skip). A column listing each record's items in one cell is read as line items when no items
@@ -107,10 +114,44 @@ def read_table(path: str | Path) -> list[dict]:
     return [dict(zip(head, r + [""] * (len(head) - len(r)))) for r in rows[at + 1:]]
 
 
+def _build_many(config: dict, base: Path) -> tuple[object, dict, list[str]]:
+    """Several event files about the same entities (a machine's error log and its failure records, a customer's
+    orders and returns): each file is read and mapped on its own, its events named after the file (or its
+    "event"), and the files merged on the entity; an entity table joins once, at the end."""
+    from .contract import Dataset, Entity
+    ents: dict[str, Entity] = {}
+    events, groups, log = [], {}, []
+    for r in config["records"]:
+        name = r.get("event") or Path(r["file"]).stem
+        sub = {**config, "records": r, "entity_table": None, "_entity_table": config.get("entity_table"), "event": name}
+        ds, g, lg = build(sub, base)
+        log += [f"{name}: {x}" for x in lg]
+        if ds is None:
+            continue
+        for e in ds.entities:
+            old = ents.get(e.id)
+            ents[e.id] = e if old is None else Entity(e.id, old.group or e.group, {**(e.attrs or {}), **(old.attrs or {})})
+        events += ds.events
+        groups.update(g)
+    if not events:
+        return None, {}, log
+    ds = Dataset([ents[k] for k in sorted(ents)], sorted(events, key=lambda e: e.at))
+    log.append(f"Merged: {len(events):,} events of {len({e.type for e in events})} types from {len(config['records'])} files")
+    if config.get("entity_table"):
+        from .tables import attach_entities
+        et = config["entity_table"]
+        ds, more, _labels = attach_entities(ds, read_table(base / et["file"]), et.get("key"))
+        log += [f"Entities: {n}" for n in more]
+    return ds, groups, log
+
+
 def build(config: dict, base: Path = Path(".")) -> tuple[object, dict, list[str]]:
     """The dataset, item groups and loading notes from a config. Without a mapping, Vertiq proposes one from
-    the values (vertiq.tables.propose); without join columns, it finds them (vertiq.tables.find_join)."""
+    the values (vertiq.tables.propose); without join columns, it finds them (vertiq.tables.find_join).
+    "records" may also be a list of event files about the same entities (see _build_many)."""
     from dataclasses import replace
+    if isinstance(config["records"], list):
+        return _build_many(config, base)
     from .tables import find_join, find_lines, propose, records_from_lines
     rec = config["records"]
     rows = read_table(base / rec["file"])
@@ -138,12 +179,27 @@ def build(config: dict, base: Path = Path(".")) -> tuple[object, dict, list[str]
     if rec.get("mapping"):
         proposal = Mapping.parse(rec["mapping"])
     else:
-        proposal, why = propose(rows, record_key=it.get("event_key"), latest=bool(shape and shape["counts"]))
+        hint = None
+        et = config.get("entity_table") or config.get("_entity_table")
+        if et:                                        # the entity table names the entity: its key's column
+            from .tables import entity_key_hint
+            hint = entity_key_hint(rows, read_table(base / et["file"]))
+        proposal, why = propose(rows, record_key=it.get("event_key"), latest=bool(shape and shape["counts"]),
+                                entity_hint=hint)
         log += [f"Mapping proposal: {w}" for w in why]
         if proposal is None:
             return None, {}, log
     m, notes = repair(proposal, rows, exclude={it["event_key"]} if it.get("event_key") else ())
     ds, problems = apply(m, rows)
+    target = config.get("target")
+    if target and ds.events and target not in {e.type for e in ds.events}:
+        # the outcome is named as written in the data ("Fail" in a results column): that column is the event type
+        holder = next((a for a in m.attrs if any(str((e.attrs or {}).get(a, "")).strip() == target for e in ds.events)), None)
+        if holder:
+            ds = type(ds)(ds.entities, [replace(e, type=str((e.attrs or {}).get(holder) or "").strip() or "other",
+                                                attrs={k: v for k, v in (e.attrs or {}).items() if k != holder})
+                                        for e in ds.events], ds.actions)
+            log.append(f"Target: {target!r} is a value of {holder!r}, so {holder!r} gives each record's type")
     if ds.events and not any(e.type for e in ds.events):
         noun = config.get("noun", "")
         name = config.get("event") or (noun[:-1] if noun.endswith("s") else noun) or "record"
@@ -173,11 +229,16 @@ def build(config: dict, base: Path = Path(".")) -> tuple[object, dict, list[str]
     return ds, groups, log
 
 
-def run(config: dict, base: Path = Path(".")) -> str:
+def run(config: dict, base: Path = Path("."), collect: dict | None = None) -> str:
+    """The report as Markdown. `collect`, when given, is filled with the same results as data (loading notes,
+    counts, patterns, combinations, and each prediction task's gate report) for benchmarks to record."""
+    collect = collect if collect is not None else {}
     ds, groups, log = build(config, base)
+    collect["log"] = log
     noun, cur = config.get("noun", "records"), config.get("currency", "")
     if ds is None:
         return "\n".join(["# Vertiq report", "", "## Loading", ""] + [f"- {x}" for x in log] + ["", "No mapping could be proposed."]) + "\n"
+    collect.update(records=len(ds.events), entities=len(ds.entities))
     out = ["# Vertiq report", "", f"{len(ds.events)} {noun} from {len(ds.entities)} entities.", "", "## Loading", ""]
     out += [f"- {x}" for x in log]
     if not ds.events:
@@ -186,6 +247,7 @@ def run(config: dict, base: Path = Path(".")) -> str:
     days = config.get("summary_days") or max(1, span // 2)
     out += ["", "## What happened", "", narrate.summary(summary.summarise(ds, days=days), entities=config.get("entities", "entities"))]
     pat = patterns.find(ds, item_groups=groups or None, noun=noun, currency=cur)
+    collect["patterns"] = {k: pat.get(k) for k in ("status", "features", "tested")} | {"found": [f["text"] for f in pat.get("found", [])]}
     out += ["", "## Patterns", ""]
     if pat["status"] == "learning":
         out.append(f"Not enough {noun} to test patterns: {pat['records']}, {pat['need']} needed.")
@@ -197,6 +259,7 @@ def run(config: dict, base: Path = Path(".")) -> str:
     if lines:
         a = affinity.analyse(lines)
         combos = a.get("combos", [])
+        collect["combos"] = [{"items": c["items"], "bills_together": c["bills_together"], "lift": c["lift"]} for c in combos[:10]]
         out += ["", "## Items bought together", ""]
         out += ([f"- {' + '.join(c['items'])}: on {c['bills_together']} {noun} together, {c['lift']} times as often as chance" for c in combos[:10]]
                 or [f"No pair of items appears together on at least {affinity.MIN_PAIR} {noun}, the minimum for a combination."])
@@ -218,7 +281,7 @@ def run(config: dict, base: Path = Path(".")) -> str:
         # several windows are several chances to pass the gate: correct the confidence for how many are tried
         conf = 1 - (1 - harness.CONFIDENCE) / len(windows)
         if len(windows) > 1:
-            out.append(f"- Return windows tried: {', '.join(f'{w} days' for w in windows)}; each must pass the gate at "
+            out.append(f"- Windows tried: {', '.join(f'{w} days' for w in windows)}; each must pass the gate at "
                        f"{conf:.1%} confidence, corrected for trying {len(windows)}.")
         tune = config.get("tune", TUNE)
         for w in windows:
@@ -227,6 +290,7 @@ def run(config: dict, base: Path = Path(".")) -> str:
             tasks.append(_task(ds, "value", target, max(windows), harness.CONFIDENCE, tune, out))
     else:
         out.append("- No single event type to predict; set \"target\" in the config.")
+    collect["predictions"] = [t.report() for t in tasks]
     if config.get("scores"):
         out += scores(tasks, base / config["scores"], cur, config.get("entities", "entities"))
     return "\n".join(out) + "\n"

{vertiq-0.2.2 → vertiq-0.2.3}/vertiq/tables.py

@@ -24,13 +24,57 @@ def _vals(rows, c):
     return [str(r.get(c) if r.get(c) is not None else "").strip() for r in rows]
 
 
-def propose(rows: list[dict], record_key: str | None = None, latest: bool = False) -> tuple[Mapping | None, list[str]]:
+_ID_NAME = (re.compile(r"(?i)(?:^|[\s_.\-])id$"), re.compile(r"[a-z]I[dD]$"))     # customer_id, Customer ID, machineID
+
+
+_ID_WORDS = re.compile(r"(?i)(?:^|[^a-z])(?:id|ids|no|nbr|num|number|code|key|ref|reference|licen[cs]e|account|acct|"
+                       r"zip|postcode|pin|phone|mobile)(?:[^a-z]|$)|[a-z](?:Id|ID|No|Number|Code|Key)$")
+
+
+def id_worded(column: str) -> bool:
+    """Whether a column's name says it holds codes rather than amounts: "license_", "Invoice No", "zip_code",
+    "accountNumber". Such a column is never proposed as the value."""
+    if re.search(r"(?i)(?:number|num|no)[\s_.]*of(?:[^a-z]|$)", column):
+        return False                                  # "number of persons injured" is a count, not a code
+    return bool(_ID_WORDS.search(column.strip())) or id_named(column)
+
+
+def id_named(column: str) -> bool:
+    """Whether a column's name says it holds identifiers: "id", "customer_id", "Customer ID", "machineID"."""
+    return any(p.search(column.strip()) for p in _ID_NAME)
+
+
+def entity_key_hint(rows: list[dict], entity_rows: list[dict]) -> str | None:
+    """The column of `rows` whose values are the key of an entity table (a customer list, an asset register): the
+    table's first column with a distinct value on every row, matched by at least 90% of the column's values."""
+    ents = _records(entity_rows)
+    if not ents:
+        return None
+    key = next((c for c in ents[0] if len({str(r.get(c) or "").strip() for r in ents}) == len(ents)), None)
+    if key is None:
+        return None
+    keys = {str(r.get(key) or "").strip() for r in ents}
+    data = _records(rows)
+    best = None
+    for c in (data[0] if data else {}):
+        vals = {v for v in _vals(data, c) if v}
+        share = len(vals & keys) / max(1, len(vals))
+        if vals and share >= 0.9 and (best is None or share > best[0]):
+            best = (share, c)
+    return best[1] if best else None
+
+
+def propose(rows: list[dict], record_key: str | None = None, latest: bool = False,
+            entity_hint: str | None = None) -> tuple[Mapping | None, list[str]]:
     """A mapping chosen from the values: the time is the date column that is filled best, with a separate
     time-of-day column if there is one; the value is the most complete amount column; the entity is the best
     identifier that repeats (a person or asset), else the best identifier; attributes are the remaining
     columns with 2 to 12 repeated labels, and columns that identify each record (kept for joins).
     `record_key`: a column known to identify the record itself (the key line items join on); it is kept as
-    an attribute and never proposed as the entity, since it names the order, not whom it is about."""
+    an attribute and never proposed as the entity, since it names the order, not whom it is about.
+    `entity_hint`: a column known to hold the entity (its values are an entity table's key); it is the entity.
+    A column named as an identifier (id_named) that holds whole numbers counts as one with 20 or more values,
+    so machine 7 or store 12 are entities, not counts."""
     skip = structural_rows(rows)
     data = [r for i, r in enumerate(rows, start=1) if i not in skip]
     if not data:
@@ -58,7 +102,9 @@ def propose(rows: list[dict], record_key: str | None = None, latest: bool = Fals
     fmts = date_formats(_vals(data, time))
     clock = next((c for c in kind if c != time and sum(1 for v in _vals(data, c) if _TIME_OF_DAY.match(v)) >= 0.8 * len(data)), None)
     qcols = _quantities(data, kind)
-    amounts = [c for c in kind if kind[c] == "number" and fitness(data, c, "value", kind) >= 0.5 and c not in qcols
+    # an identifier (named or joined as one) is never the amount
+    amounts = [c for c in kind if kind[c] == "number" and c != entity_hint and not id_worded(c)
+               and fitness(data, c, "value", kind) >= 0.5 and c not in qcols
                and not _id_numbers(_vals(data, c))]
     value = None
     if amounts:
@@ -75,12 +121,16 @@ def propose(rows: list[dict], record_key: str | None = None, latest: bool = Fals
     for c in kind:
         if c in (time, clock, value, record_key, item_col):
             continue
-        f = fitness(data, c, "entity", kind)
-        if f < 0.2:
-            continue                                  # records without an id become one-off entities (reported)
         ph = placeholder_ids(data, c)                 # once per column, not once per value
         real = [v for v in _vals(data, c) if v not in ph]
         distinct = len(set(real)) / max(1, len(real))
+        if c == entity_hint or (id_named(c) and real and all(v.isdigit() for v in real) and len(set(real)) >= 20):
+            # named or joined as an identifier: small whole numbers here are ids, not counts
+            ids.append((True if c == entity_hint else distinct < 0.98, 2.0 if c == entity_hint else len(real) / n, c))
+            continue
+        f = fitness(data, c, "entity", kind)
+        if f < 0.2:
+            continue                                  # records without an id become one-off entities (reported)
         # an identifier is a code with letters and digits, a fixed-length whole number (an account, customer or phone
         # number), or a long number; short numbers and numbers with decimals are amounts or counts
         coded = _id_numbers(real) or sum(1 for v in real if any(ch.isdigit() for ch in v)

{vertiq-0.2.2 → vertiq-0.2.3/vertiq.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vertiq
-Version: 0.2.2
+Version: 0.2.3
 Summary: Self-improving decision intelligence: a local language model that maps any operational dataset and decision models trained on real outcomes, improved through recursive self-improvement under an evaluation gate.
 Author: Biswajit Tripathy
 License: Apache-2.0
@@ -298,6 +298,20 @@ Recency baseline AUC 0.761.
 | Auto (chose boosted) | 0.812 | 0.795–0.827 | +0.043 to +0.059 |
 | scikit-learn logistic regression (reference) | 0.807 |  |  |
 
+### Real data across sectors
+
+`vertiq-report` on public data from 5 sectors, nothing tuned per sector; windows of 30, 60 and 90 days, each through the
+gate with its confidence corrected for the three, and its segment check (`benchmarks/sectors/fetch.sh`, then
+`python benchmarks/sectors/run.py`; full reports in `benchmarks/sectors/reports/`).
+
+| Sector | Data | Size | Question | Result | Patterns kept |
+|---|---|---|---|---|---|
+| Retail | UCI Online Retail II: a UK online shop's invoice lines, 2009-2010 | 28,816 records, 9,612 entities | Will a customer buy again, and how much? | Promoted: 30 days, AUC 0.881 vs 0.776; 60 days, AUC 0.888 vs 0.750; 90 days, AUC 0.890 vs 0.708 | 20 of 471 |
+| Food safety | City of Chicago food inspections since 2019 (up to 150,000) | 137,034 records, 27,264 entities | Which establishments will fail an inspection? | Promoted: 30 days, AUC 0.813 vs 0.698; 60 days, AUC 0.822 vs 0.703; 90 days, AUC 0.809 vs 0.692 | 8 of 94 |
+| Animal services | Austin Animal Center intakes (up to 200,000) | 173,812 records, 156,287 entities | Which animals will come back to the shelter? | Not promoted: best AUC 0.870 vs 0.865; the current rule stays | 20 of 266 |
+| Manufacturing | Microsoft Azure predictive maintenance sample: 100 machines, 2015 (errors, failures, machine models) | 4,680 records, 100 entities | Which machines will fail? | Promoted: 60 days, AUC 0.720 vs 0.545. Kept the rule: 30 days, 90 days | 7 of 221 |
+| Transport | NYC motor vehicle collisions since January 2025 (up to 200,000) | 121,970 records, 43,061 entities | Where will collisions happen again? | Promoted: 30 days, AUC 0.971 vs 0.894; 60 days, AUC 0.958 vs 0.859; 90 days, AUC 0.950 vs 0.840. Next value promoted: average error 0.44 vs 0.48 | 5 of 86 |
+
 ### Real exports (held out)
 
 Six public exports from different sectors, never used for training; a role is correct when it is one of the accepted answers