mainsequence-metatable 0.1.21.dev33__tar.gz → 0.1.22.dev35__tar.gz

{mainsequence_metatable-0.1.21.dev33 → mainsequence_metatable-0.1.22.dev35}/.agents/skills/client/metatables-meta-tables/SKILL.md

@@ -64,10 +64,12 @@ are fingerprints, not table identity. See `docs/concepts/identity-and-scope.md`.
 ## SQLAlchemy contract rules
 
 Use `src/metatables/examples/tables.py` as the tested pattern. Declare an explicit
-application-prefixed physical name with `schema_table_name(app, concept)`, a
-stable logical identifier, and an intention-rich `__metatable_description__`.
-Give columns useful `info={"label": ..., "description": ...}` metadata. Explain
-row grain and downstream meaning rather than repeating types.
+application-prefixed physical name with `schema_table_name(app, concept)` and a
+stable logical identifier. Write the table description, labels and every column's
+`info={"label": ..., "description": ...}` by the rules in the
+[table discovery skill](../metatables-table-discovery/SKILL.md), which is
+authoritative for that metadata. Before authoring a new table, search for an
+existing one with that skill.
 
 Keep foreign keys and indexes in SQLAlchemy metadata. Reference explicit physical
 names using `ForeignKey`/`ForeignKeyConstraint`; do not ask users to hand-author
@@ -126,8 +128,9 @@ access; row/time limits and transaction handling remain in the API. Refer to the
 query guide for engine-specific statement/batch behavior. Application DDL belongs
 to the separate migration-provider workflow.
 
-Column search is supported. Description search, search-index refresh, and
-legacy convenience sharing/label actions are not supported API workflows.
+To find tables, use `MetaTable.search` as described in the
+[table discovery skill](../metatables-table-discovery/SKILL.md). Legacy convenience
+sharing/label actions are not supported API workflows.
 Use `get_access`, `set_access` and `revoke_access` for supported Reader/Writer
 sharing. Labels can be changed through the documented table PATCH path. Authorization is
 owned by the API catalog; see `docs/concepts/permissions-namespaces-labels.md`.

{mainsequence_metatable-0.1.21.dev33 → mainsequence_metatable-0.1.22.dev35}/.agents/skills/client/metatables-migrations/SKILL.md

@@ -31,7 +31,10 @@ Read `docs/client/define-and-migrate-tables.md`,
 `src/metatables/examples/tables.py`. Managed authoring is migration-first: define SQLAlchemy
 models, select a provider, author and apply a revision with the client, then
 finalize catalog bindings through the API. Use the
-[table skill](../metatables-meta-tables/SKILL.md) for contract design.
+[table skill](../metatables-meta-tables/SKILL.md) for contract design. The catalog
+sync in `metatables migrations upgrade` also copies changed table descriptions,
+labels and column metadata to existing tables; a metadata-only change needs no
+revision (see the [table discovery skill](../metatables-table-discovery/SKILL.md)).
 
 Check `docs/reference/capabilities.md` and `metatables migrations --help` before
 promising a command. The API's own catalog migrations are a separate history: a

mainsequence_metatable-0.1.22.dev35/.agents/skills/client/metatables-table-discovery/SKILL.md

@@ -0,0 +1,244 @@
+---
+name: metatables-table-discovery
+description: "Find existing MetaTables and time-index tables by meaning with table search, and write the table descriptions, labels and column metadata that make tables findable. Authoritative for every description, label and column `info` authored on MetaTables SQLAlchemy models in consuming applications. Excludes API/server changes, client-library implementation and table storage design."
+---
+
+# Finding tables and writing findable metadata
+
+Table search ranks the tables you can read using one document per table, built
+from the table's own metadata: identifier, description, labels, namespace, time
+index and cadence, and every column's name, type, label, description and logical
+name. What you write on a model is exactly what search reads. The decision is
+`docs/adr/api/0018-table-search.md`.
+
+This skill is authoritative for descriptions, labels and column metadata. Other
+MetaTables skills defer to it. Storage design (physical names, grain, foreign
+keys, indexes) stays with the [table skill](../metatables-meta-tables/SKILL.md);
+schema evolution with the [migrations skill](../metatables-migrations/SKILL.md).
+
+In a copied skill, resolve `docs/` paths relative to this skill's `references/`
+directory.
+
+## Part 1: Find a table
+
+Search before writing code that reads data the user describes in words, and
+before authoring a new table. Reusing a table beats creating a near-duplicate,
+which also splits future search results.
+
+```python
+from metatables import MetaTable, TimeIndexMetaTable
+
+result = TimeIndexMetaTable.search("bond prices", limit=10)
+for hit in result.hits:
+    print(hit.table.identifier, hit.score, hit.matched_columns)
+```
+
+```bash
+metatables time-index-table search "bond prices"
+metatables meta-table search "account balances" --filter namespace=ledger
+```
+
+`MetaTable.search` covers every table; `TimeIndexMetaTable.search` only
+time-index tables. Both accept the list filters (`namespace`, `labels__in`,
+`data_source_uid`, ...) to narrow a search you have already started.
+
+### Write the query
+
+- Use a short English noun phrase of domain words: what the rows are and the key
+  measure. `bond prices`, `fx spot rates`, `portfolio positions`. Not a question,
+  not SQL, not a physical table name.
+- Run two to four variants before concluding anything: a synonym (`sovereign
+  debt`), a market or vendor term (`M Bonos`, `CETES`, `Valmer`), and column words
+  (`clean price`, `yield`).
+
+### Read the results
+
+- `matched_columns` lists the columns whose name or logical name contains a query
+  word. A table that matches both in its description and in its columns is usually
+  the right one.
+- `result.semantic` is `false` while the embedding model is unavailable; ranking
+  is then by keywords only, so try the exact words likely to appear in names and
+  descriptions.
+- Before using a candidate, read its `description` and columns in the hit to
+  confirm the grain, the time index, the cadence and the units.
+- Results contain only tables you can read. When the data surely exists but
+  nothing matches, ask the user for the owner or identifier. Do not create a
+  duplicate table.
+- Use `MetaTable.column_search(q)` only for a pure column-name lookup.
+
+Do not list every table and scan it, use the list filter `q` for meaning (it
+matches identifier and physical name only), or conclude a table is absent from
+one query.
+
+## Part 2: Write findable metadata
+
+These rules apply to every `PlatformManagedMetaTable`,
+`PlatformTimeIndexMetaTable` and externally registered SQLAlchemy model.
+
+| Search reads | Write it in |
+| --- | --- |
+| Table description | `__metatable_description__` |
+| Labels | `__metatable_labels__` |
+| Identifier | `__metatable_identifier__` |
+| Namespace | `__metatable_namespace__` |
+| Column label, description, logical name | `mapped_column(..., info={"label": ..., "description": ..., "logical_name": ...})` |
+| Time index and cadence | `__time_index_name__`, `__index_names__`, `__cadence__` |
+
+Use only these attributes. Do not put metadata in SQLAlchemy `comment=` or `doc=`:
+they are fallbacks for imported tables, and `info` wins over them.
+
+### Table description
+
+Required on every table. One to three sentences, at most 500 characters, in
+English, in this order:
+
+1. **Grain.** What one row is: "One row per ISIN per valuation date."
+2. **Subject.** The plain English words a searcher would type, then the domain or
+   vendor terms: "End-of-day prices and yields of Mexican government bonds (M
+   Bonos, Udibonos and CETES)". Always include the plain English noun even when a
+   vendor term is standard: keyword search matches words, and "Bonos" does not
+   match "bond".
+3. **Source and coverage.** Who produces it and from what: "from the Valmer daily
+   price vector".
+4. **Timing,** for time-index tables: when rows arrive, in words: "published after
+   the Mexican market close".
+
+Never:
+
+- start with "Table of", "This table contains" or "Data for";
+- repeat the identifier, the physical name or the column list;
+- name another table. Say what the related thing is ("the asset each valuation
+  belongs to"). A table name in prose can reveal a table the reader may not see;
+- write placeholders (TBD, Unknown, n/a), ticket numbers, people's or customers'
+  names, credentials or URLs carrying tokens;
+- describe implementation: updater classes, hashes, Alembic.
+
+### Columns
+
+Every column gets `info` with `label` and `description`, including the primary
+key, the time index and the identity dimensions.
+
+- `label`: one to four words, sentence case, no units: `"Clean price"`.
+- `description`: one sentence giving the meaning plus whatever a reader needs to
+  use the value correctly:
+  - unit, currency or scale: "per 100 of nominal", "in MXN", "in percent: 5.25
+    means 5.25%";
+  - identifier scheme: "ISIN", "Main Sequence asset UID";
+  - allowed values of a code: "M Bono, Udibono or CETE";
+  - for times, which moment and timezone: "valuation business day at 00:00 UTC";
+  - what null means, when the column is nullable;
+  - the sign convention, where one applies.
+- Foreign-key columns describe what the referenced thing is, never the referenced
+  table's name.
+- Name new columns in full snake_case words (`clean_price`, not `cln_px`). Search
+  splits names into words, so each word becomes searchable.
+- `logical_name` only when the physical name is an abbreviation or vendor code you
+  cannot change, typically on an external table: a column `ytm` gets
+  `info={"logical_name": "yield_to_maturity", ...}`.
+
+### Labels
+
+- Two to six labels, lowercase and hyphenated: `fixed-income`.
+- Draw them from these facets: domain or asset class (`fixed-income`, `fx`,
+  `equities`, `accounting`), geography (`mexico`, `global`), data kind (`prices`,
+  `reference-data`, `positions`, `transactions`, `risk`), and source (`valmer`,
+  `banxico`) when the source is not the namespace.
+- Reuse labels you see on existing tables in search results instead of creating
+  near-duplicates (`fixed_income` next to `fixed-income`).
+- Do not repeat the namespace or the cadence as a label.
+
+### Identifier and namespace
+
+- `__metatable_identifier__` is `<namespace>.<concept>` in full words:
+  `valmer.mx_sovereign_bond_prices`. It is identity and never changes after the
+  table is published.
+- `__metatable_namespace__` is the owning application or source, not a SQL schema.
+  Its description (one or two sentences about the application or source) is
+  written once by the namespace's Writer; do not repeat it in table descriptions.
+
+### Time-index tables, in addition
+
+- The grain sentence matches `__index_names__`: `["time_index", "isin"]` → "One
+  row per ISIN per valuation date."
+- Declare `__cadence__` whenever the interval is stable.
+- The time index column's description says which moment it is (observation,
+  valuation, publication or effective date), its timezone and, for daily data,
+  the day convention.
+
+### Example
+
+```python
+from datetime import datetime
+
+from sqlalchemy import DateTime, Float, MetaData, String
+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
+
+from metatables import PlatformTimeIndexMetaTable, schema_table_name
+
+
+class Base(DeclarativeBase):
+    metadata = MetaData()
+
+
+class SovereignBondPrice(PlatformTimeIndexMetaTable, Base):
+    __tablename__ = schema_table_name("valmer", "mx_sovereign_bond_prices")
+    __metatable_namespace__ = "valmer"
+    __metatable_identifier__ = "valmer.mx_sovereign_bond_prices"
+    __metatable_description__ = (
+        "One row per ISIN per valuation date. End-of-day prices and yields of "
+        "Mexican government bonds (M Bonos, Udibonos and CETES) from the Valmer "
+        "daily price vector, published after the Mexican market close."
+    )
+    __metatable_labels__ = ["fixed-income", "mexico", "prices"]
+    __time_index_name__ = "time_index"
+    __index_names__ = ["time_index", "isin"]
+    __cadence__ = "1d"
+
+    time_index: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False,
+        info={"label": "Valuation date",
+              "description": "Business day the prices apply to, at 00:00 UTC."},
+    )
+    isin: Mapped[str] = mapped_column(
+        String(12), nullable=False,
+        info={"label": "ISIN", "description": "ISIN of the bond."},
+    )
+    instrument_type: Mapped[str] = mapped_column(
+        String(16), nullable=False,
+        info={"label": "Instrument type",
+              "description": "Bond family: M Bono, Udibono or CETE."},
+    )
+    clean_price: Mapped[float] = mapped_column(
+        Float, nullable=False,
+        info={"label": "Clean price",
+              "description": "Price excluding accrued interest, per 100 of nominal, in MXN."},
+    )
+    dirty_price: Mapped[float] = mapped_column(
+        Float, nullable=False,
+        info={"label": "Dirty price",
+              "description": "Clean price plus accrued interest, per 100 of nominal, in MXN."},
+    )
+    yield_to_maturity: Mapped[float | None] = mapped_column(
+        Float, nullable=True,
+        info={"label": "Yield to maturity",
+              "description": "Annual yield to maturity in percent; null when Valmer publishes no yield."},
+    )
+```
+
+### When metadata changes
+
+The model is the source of truth. Edit the model and run `metatables migrations
+upgrade`: its catalog sync updates the description, labels and column metadata of
+tables that already exist. A metadata-only change needs no Alembic revision.
+Re-run the registration of an external table after changing its model. Do not
+`patch` the description of a table that has a model; the next sync replaces it.
+Use `table.patch(description=..., labels=[...])` only for imported tables with no
+model.
+
+## Validation
+
+Before finishing, check every table and column you authored against Part 2:
+four-part description, two to six labels, `label` and `description` on every
+column. After `metatables migrations upgrade`, search for each new table with two
+phrases a user would type; it should appear among the first results. State
+whether you ran that search or only checked the model.

{mainsequence_metatable-0.1.21.dev33 → mainsequence_metatable-0.1.22.dev35}/.agents/skills/client/metatables-time-index-table-updates/SKILL.md

@@ -41,7 +41,9 @@ choices, first-run/backfill bounds, and whether identity must be preserved from
 the task and existing code. Clarify only unresolved choices that change the
 published contract. Keep storage description, namespace, identifier, cadence,
 columns, and FK/index metadata on the authoring model, not the updater config.
-Use intention-rich table and column descriptions; see `src/metatables/examples/tables.py`.
+Write table and column descriptions by the rules in the
+[table discovery skill](../metatables-table-discovery/SKILL.md); see
+`src/metatables/examples/tables.py`.
 
 Migrate and bind the output class before constructing the producer. Pass explicit
 `config` and `output_table`; do not create/register tables inside `update()`.

{mainsequence_metatable-0.1.21.dev33/src/mainsequence_metatable.egg-info → mainsequence_metatable-0.1.22.dev35}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mainsequence-metatable
-Version: 0.1.21.dev33
+Version: 0.1.22.dev35
 Summary: Main Sequence MetaTables Python client and standalone API
 Author-email: Main Sequence GmbH <dev@main-sequence.io>
 License-Expression: MIT
@@ -19,6 +19,7 @@ Requires-Dist: alembic<2,>=1.18
 Requires-Dist: cachetools
 Requires-Dist: cryptography<51,>=46
 Requires-Dist: fastapi<1,>=0.110
+Requires-Dist: fastembed<0.9,>=0.8.1
 Requires-Dist: httpx<1,>=0.28
 Requires-Dist: keyring<26,>=25
 Requires-Dist: numpy
@@ -29,6 +30,7 @@ Requires-Dist: pandas
 Requires-Dist: psutil
 Requires-Dist: platformdirs<5,>=4
 Requires-Dist: psycopg2-binary<3,>=2.9
+Requires-Dist: py-rust-stemmers<0.2,>=0.1.5
 Requires-Dist: pyodbc<6,>=5
 Requires-Dist: pydantic<3,>=2.11
 Requires-Dist: pyyaml

mainsequence_metatable-0.1.22.dev35/docs/adr/api/0018-search-document-pipeline.svg

@@ -0,0 +1,105 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1240" height="680" viewBox="0 0 1240 680" role="img" aria-labelledby="f17-title f17-desc">
+  <title id="f17-title">How a table's search document is built</title>
+  <desc id="f17-desc">A metadata write rebuilds the table's search document in the same catalog transaction. The API reads the table, its labels, its namespace and every column, never other tables' names, writes the document and its hash to meta_table_search_document, and derives keyword terms from it at once. In the background the document is cut into chunks that fit 512 tokens, embedded with bge-small-en-v1.5 and stored in meta_table_search_vector, one row per chunk.</desc>
+  <style>
+    .sans { font-family: -apple-system, "Segoe UI", Helvetica, Arial, sans-serif; }
+    .mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
+    .title { font-size: 21px; font-weight: 700; fill: #0f172a; }
+    .sub { font-size: 13.5px; fill: #475569; }
+    .head { font-size: 15px; font-weight: 700; fill: #0f172a; }
+    .step { font-size: 13px; font-weight: 700; fill: #0f172a; }
+    .msg { font-size: 12.5px; font-weight: 600; fill: #0f172a; }
+    .note { font-size: 12.5px; fill: #475569; }
+    .code { font-size: 12px; fill: #0f172a; }
+    .deny { fill: #b91c1c; font-weight: 700; }
+  </style>
+  <defs>
+    <marker id="f17-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse"><path d="M0,0 L10,5 L0,10 z" fill="#334155"/></marker>
+  </defs>
+
+  <rect x="0" y="0" width="1240" height="680" rx="16" fill="#ffffff"/>
+
+  <text x="40" y="48" class="sans title">How a table's search document is built</text>
+  <text x="40" y="74" class="sans sub">A metadata write rebuilds the document in its catalog transaction. Vectors follow in the background.</text>
+
+  <!-- 1. Trigger -->
+  <rect x="40" y="96" width="1160" height="48" rx="10" fill="#f8fafc" stroke="#64748b" stroke-width="1.2"/>
+  <text x="56" y="125" class="sans step">1 · Any metadata write</text>
+  <text x="240" y="125" class="sans note">register · patch · label change · import · finalize · contract update · namespace description change</text>
+  <line x1="600" y1="144" x2="600" y2="210" stroke="#334155" stroke-width="1.8" marker-end="url(#f17-arrow)"/>
+
+  <!-- Catalog rows -->
+  <text x="40" y="196" class="sans head">Read from the catalog</text>
+
+  <rect x="40" y="212" width="320" height="70" rx="10" fill="#f8fafc" stroke="#94a3b8" stroke-width="1.1"/>
+  <text x="56" y="236" class="sans step">MetaTable</text>
+  <text x="56" y="256" class="sans note">identifier, physical name, description,</text>
+  <text x="56" y="274" class="sans note">time index and cadence</text>
+
+  <rect x="40" y="294" width="320" height="52" rx="10" fill="#f8fafc" stroke="#94a3b8" stroke-width="1.1"/>
+  <text x="56" y="316" class="sans step">Labels</text>
+  <text x="56" y="336" class="sans note">label names</text>
+
+  <rect x="40" y="358" width="320" height="52" rx="10" fill="#f8fafc" stroke="#94a3b8" stroke-width="1.1"/>
+  <text x="56" y="380" class="sans step">Namespace</text>
+  <text x="56" y="400" class="sans note">name, description</text>
+
+  <rect x="40" y="422" width="320" height="70" rx="10" fill="#f8fafc" stroke="#94a3b8" stroke-width="1.1"/>
+  <text x="56" y="446" class="sans step">Columns, every one</text>
+  <text x="56" y="466" class="sans note">name, type, logical name, label,</text>
+  <text x="56" y="484" class="sans note">description</text>
+
+  <rect x="40" y="508" width="320" height="52" rx="10" fill="#ffffff" stroke="#b91c1c" stroke-width="1.1" stroke-dasharray="6 4"/>
+  <text x="56" y="530" class="sans step"><tspan class="deny">Never read</tspan></text>
+  <text x="56" y="550" class="sans note">names of other tables, which may be hidden</text>
+
+  <line x1="362" y1="330" x2="438" y2="330" stroke="#334155" stroke-width="1.8" marker-end="url(#f17-arrow)"/>
+  <text x="400" y="321" text-anchor="middle" class="sans msg">read</text>
+
+  <!-- 2. Build -->
+  <rect x="440" y="212" width="320" height="196" rx="10" fill="#eef2ff" stroke="#4f46e5" stroke-width="1.2"/>
+  <text x="456" y="236" class="sans step">2 · Build the document</text>
+  <text x="456" y="260" class="sans note">in the same catalog transaction</text>
+  <text x="456" y="282" class="sans note">sections in fixed order:</text>
+  <text x="456" y="302" class="mono code"># Table, # Namespace, # Columns</text>
+  <text x="456" y="324" class="sans note">one line per column, in column order</text>
+  <text x="456" y="346" class="sans note">empty fields are left out</text>
+  <text x="456" y="368" class="sans note">document hash: SHA-256 of the text</text>
+
+  <line x1="600" y1="408" x2="600" y2="444" stroke="#334155" stroke-width="1.8" marker-end="url(#f17-arrow)"/>
+
+  <rect x="440" y="446" width="320" height="64" rx="10" fill="#eff6ff" stroke="#1d4ed8" stroke-width="1.2"/>
+  <text x="456" y="470" class="mono step">meta_table_search_document</text>
+  <text x="456" y="492" class="sans note">document, document hash, updated time</text>
+
+  <line x1="760" y1="468" x2="838" y2="300" stroke="#334155" stroke-width="1.8" marker-end="url(#f17-arrow)"/>
+  <line x1="760" y1="488" x2="838" y2="470" stroke="#334155" stroke-width="1.8" marker-end="url(#f17-arrow)"/>
+
+  <!-- 3a. Keyword terms -->
+  <rect x="840" y="212" width="360" height="150" rx="10" fill="#ffffff" stroke="#4f46e5" stroke-width="1.2"/>
+  <text x="856" y="236" class="sans step">3a · Keyword terms</text>
+  <text x="856" y="258" class="sans note">built by the API process from the document</text>
+  <text x="856" y="280" class="sans note">split identifiers: <tspan class="mono code">clean_price → clean, price</tspan></text>
+  <text x="856" y="302" class="sans note">stem words: <tspan class="mono code">prices → price</tspan></text>
+  <text x="856" y="324" class="sans note">table fields weigh more than column fields</text>
+  <text x="856" y="346" class="sans note">searchable as soon as the write commits</text>
+
+  <!-- 3b. Chunks and vectors -->
+  <rect x="840" y="388" width="360" height="176" rx="10" fill="#ffffff" stroke="#4f46e5" stroke-width="1.2" stroke-dasharray="6 4"/>
+  <text x="856" y="412" class="sans step">3b · Chunks and vectors, in the background</text>
+  <text x="856" y="436" class="sans note">chunk 0: the <tspan class="mono code"># Table</tspan> and <tspan class="mono code"># Namespace</tspan> sections</text>
+  <text x="856" y="458" class="sans note">chunks 1…n: table header + a group of columns</text>
+  <text x="856" y="480" class="sans note">each chunk fits the model's 512 tokens</text>
+  <text x="856" y="502" class="sans note">embedded with <tspan class="mono code">bge-small-en-v1.5</tspan>, 384 dims</text>
+  <text x="856" y="524" class="sans note">redone when the document hash or model changes</text>
+
+  <line x1="1020" y1="564" x2="1020" y2="590" stroke="#334155" stroke-width="1.8" marker-end="url(#f17-arrow)"/>
+
+  <rect x="840" y="592" width="360" height="64" rx="10" fill="#eff6ff" stroke="#1d4ed8" stroke-width="1.2"/>
+  <text x="856" y="616" class="mono step">meta_table_search_vector</text>
+  <text x="856" y="638" class="sans note">one row per chunk: model, document hash, vector</text>
+
+  <!-- Footer -->
+  <text x="40" y="600" class="sans note">Search ranks the tables the caller can see with the keyword terms and the vectors.</text>
+  <text x="40" y="622" class="sans note">Until a table's vectors match its document hash, it is ranked by keywords only.</text>
+</svg>