pgkiln 0.68.0 → 0.70.0

package/docs/guide/04-pages-and-regions.md

@@ -404,6 +404,8 @@ Attributes (most of them are also in the page designer's **Report settings** for
 | `lazy` | `false` | Load the region after the page shows (see [large tables](#large-tables)) |
 | `cache` | none | Keep the rendered region: `{"scope": "user" \| "session" \| "all", "seconds": 300}` |
 | `searchable` | `true` | Show the search box (`false` = a "classic report") |
+| `fulltext` | none | (sprint 135) A `tsvector` column of the source that the search box also uses: a row is found when its text contains the search text, **or** its tsvector matches it (`websearch_to_tsquery`: stemmed words, `"phrases"`, `or`, `-word`). It may be a `hidden` column (you chose it). Without it, the only `tsvector` column the user may see is used, if there is one. Not with a search in one column |
+| `fulltext_config` | language's | (sprint 135) The text search configuration of that search (`english`, `dutch`…, the one the tsvector was made with); by default the application's primary language's, else `simple`. Only names in `pg_ts_config` are used |
 | `interactive` | `true` | Show the Actions menu (only when searchable) |
 | `sortable` | `true` | Allow sorting (`false` keeps the query's order, e.g. for trees) |
 | `mobile` | `"reflow"` | `"scroll"` keeps a horizontally scrolling table on phones |
@@ -1639,6 +1641,15 @@ Search configurations):
 | Icon | An [icon name](09-reference.md#icons), or `#COLUMN#` for a column that holds icon names |
 | Link | Where a result goes: `{"page": 3, "items": {"P3_ID": "#ID#"}}` (`#COLUMN#` is the row's value; the link gets a checksum, a modal page opens as a dialog) |
 | Maximum results | Per configuration, 1–100 (default 10); "Showing the first *n* rows." when there are more |
+| Search type | (sprint 135) `standard`: the words as below · `fulltext`: PostgreSQL full-text search (table and sql sources; see [full-text search](#full-text-search)) · (sprint 136) `vector`: by meaning, the rows whose embedding is nearest to the search text's · `hybrid`: full-text and vector together (see [vector search](#vector-search)) |
+| tsvector column or expression | `fulltext`: a `tsvector` column of the source (best: with a GIN index), or an expression such as `to_tsvector('english', title \|\| ' ' \|\| body)`. Empty: `to_tsvector` over the searchable columns on every search |
+| Text search configuration | `fulltext`: `english`, `dutch`, `german`, `simple`… (`pg_ts_config`): the one the tsvector was made with. Empty: the application's primary language's, else `simple` |
+| Did you mean | With the `pg_trgm` extension installed: when nothing is found, a link that searches for the closest words of the searched columns (any search type; reads up to 10,000 rows). Without `pg_trgm` nothing is suggested |
+| Vector provider | `vector`, `hybrid`: the [vector provider](03-builder.md#vector-providers) (static ID) that embeds the search text: the same one that embedded the rows |
+| Vector column | `vector`, `hybrid`: the source's `vector(n)` column, *n* the provider's dimensions |
+| Key column | `hybrid`: the key (e.g. `id`) that matches the full-text and the vector results |
+| Distance | `cosine` (default; most embedding models), `l2` or `inner_product`; an HNSW index helps only when it is built for the same one |
+| Minimum similarity | `vector`, `hybrid`: rows less similar than this are left out (cosine: 1 − distance; inner product: the product; l2: 1 / (1 + distance)). Empty: the nearest rows, however far |
 
 **How it searches.** Each word of the search text must be found (case-insensitively, anywhere in
 the value) in one of the searchable columns, so `clerk mil` finds MILLER the clerk. `%` and `_` are
@@ -1655,6 +1666,80 @@ select e.empno, initcap(e.ename) as name, initcap(e.job) as job, d.dname as depa
 with searchable columns `name, job, department`, title `name`, subtitle `job`, badge `department`
 and link `{"page": 3, "items": {"P3_EMPNO": "#EMPNO#"}}`.
 
+#### Full-text search
+
+A configuration of search type **fulltext** (APEX: the search type *Oracle Text*) uses PostgreSQL's full-text
+search instead of matching the words letter for letter:
+
+- **Stemming**: in the `english` configuration *running shoes* finds a row with *run* and *shoe*. The text is
+  read with `websearch_to_tsquery`, so users can write `"exact phrase"`, `shoes or boots` and `shoes -polish`.
+- **Ranking**: the best match first (`ts_rank_cd`; words close together and in a column with weight A count
+  more), then by the title.
+- **Highlights**: the title and subtitle come back with every matched word marked, the description as the
+  fragments around the matches (`ts_headline`). Everything is escaped: only ts_headline's own marks become
+  `<mark>`, the data's characters are always text.
+- **The tsvector**: name a `tsvector` column of the source. The quickest way to make one is SQL Workshop ›
+  Table Properties › **Add full-text search** ([chapter 3](03-builder.md)): a generated column over the text
+  columns, weighted, and a GIN index on it. An expression works too (a GIN index on the same expression helps);
+  with neither, the searchable columns are turned into a tsvector on every search, which reads the whole table.
+- **The text search configuration** is checked against `pg_ts_config` and passed as a query parameter; use the
+  one the tsvector was made with. Empty: the configuration of the application's primary language (`en` →
+  `english`, `nl` → `dutch`, `de` → `german`, `fr`, `es`, `it`, `pt`, `sv`, `da`, `nb`, `fi`, `ru`, `tr`, `el`,
+  `ar`…), else `simple` (no stemming; for Japanese, Chinese and Korean).
+- **Did you mean**: with `pg_trgm` installed (`create extension pg_trgm`), a configuration that asks for it
+  suggests, when nothing at all is found, the search text with each unknown word replaced by the most similar
+  word of its searched columns (as the user: row level security applies). Without `pg_trgm` nothing is
+  suggested and nothing fails.
+
+```sql
+-- what Add full-text search writes (HR example part 71, hr.policy)
+alter table hr.policy
+  add column search_vector tsvector generated always as (
+    setweight(to_tsvector('english'::regconfig, coalesce(title, '')), 'A') ||
+    setweight(to_tsvector('english'::regconfig, coalesce(body, '')), 'B')) stored;
+create index policy_search_vector_idx on hr.policy using gin (search_vector);
+```
+
+with search type `fulltext`, tsvector `search_vector`, configuration `english`, title `title` and description
+`body`. The Advisor warns when a fulltext configuration's column has no GIN index (or its tsvector is built on
+every search), reports a configuration name that doesn't exist, and a column that isn't a tsvector. A report's
+search box can use the same engine: see its `fulltext` attribute.
+
+#### Vector search
+
+A configuration of search type **vector** (APEX: the search type *Oracle Vector Search*) finds rows by meaning:
+*time off for a family move* finds the policy *Relocation leave … relocate to another city … the new house*,
+though they share no word. It needs the [pgvector](https://github.com/pgvector/pgvector) extension (see
+[chapter 3](03-builder.md#vector-providers) for installing it), a vector provider and a vector column:
+
+1. **A vector provider** (Shared Components → Vector providers, [chapter 3](03-builder.md#vector-providers)): an
+   AI service with an embedding model (OpenAI `text-embedding-3-small`, Gemini, Ollama's `nomic-embed-text` …)
+   or a SQL function returning `vector`.
+2. **A vector column** on the table, kept current: SQL Workshop › Table Properties › **Add vector column**
+   ([chapter 3](03-builder.md)) adds `embedding vector(n)`, an HNSW index and a trigger, and queues the rows for
+   their embeddings.
+3. **The configuration**: search type `vector`, the provider, the vector column, title and description columns,
+   icon and link as for any configuration.
+
+When a user searches, pgkiln embeds the search text with the provider (an AI service: one request, remembered a
+few minutes for search-as-you-type, logged under AI usage; the text goes to that provider), then asks the
+database for the nearest rows (`order by embedding <=> $1 limit n`, which the HNSW index serves) as the
+application's role, so **row level security decides what is found**. The vector is a query parameter; pgvector's
+operators are called schema-qualified. Results are not marked word by word: they need not contain the words.
+
+**Hybrid** (`hybrid`) runs the full-text search (the tsvector and text search configuration above) and the vector
+search, takes the best 50 rows of each, and ranks them together by **reciprocal rank fusion**: each row scores
+1 / (60 + its rank) in each list it is in, summed, best first. Rows found by both rise to the top; a row that only
+one of them finds still shows. Hybrid needs the key column that tells which rows are the same.
+
+Without pgvector a vector or hybrid configuration shows that the extension is missing (and the Advisor reports
+it); the other configurations answer as before. The Advisor also checks that the provider exists, that the
+column is `vector(n)` with the provider's *n*, and warns when it has no HNSW index or one for another distance.
+The HR example (part 72) adds `POLICIES_MEANING` (hybrid, with a stand-in function provider) to page 42.
+
+An AI agent can use any table or SQL configuration as a **Retrieve** tool: the model gives a text, the agent gets
+the best rows ([AI agents](03-builder.md#ai-agents)).
+
 **The region**: attributes
 
 | Key | Meaning |
@@ -1669,7 +1754,7 @@ The region's own field is a GET form (`?s<region id>=text`, so a search can be b
 without JavaScript; with JavaScript it searches while the user types and replaces only the results.
 The Advisor plans every configuration with the columns it names and reports names a region uses
 that don't exist. Search configurations are exported with the application. The HR example's page 42
-searches employees, departments and the shortcuts list.
+searches employees, departments, the HR policies (full-text, part 71; by meaning, part 72) and the shortcuts list.
 
 ### `url` (URL region)
 

package/docs/guide/09-reference.md

@@ -167,6 +167,24 @@ select meta.ai_generate('CLAUDE', 'Classify: ' || :P5_TEXT, null,
   enabled and the application is allowed); whether the provider accepts the key shows only when a
   request is made.
 
+### Embeddings in SQL
+
+(sprint 136) `meta.embed(text, provider [, app_id])` returns the embedding of a text by a
+[vector provider](03-builder.md#vector-providers) of the current application (or of `app_id`, one the
+login may see) as `real[]`; cast it for pgvector: `meta.embed(:P1_Q, 'DOC_EMBEDDINGS')::vector`. It
+works for **function providers** and calls the function as the role of the query (security invoker:
+its grants apply), checking the dimensions. A provider that uses an AI service is called by the
+pgkiln server, never from SQL: `meta.embed` raises an error that says so (use a vector search
+configuration, a Retrieve tool or a vector column's queue). `meta.vector_provider(name [, app_id])`
+returns a provider's definition (they hold no secrets).
+
+```sql
+-- the nearest documents to a text, as the application's role (row level security applies)
+select title from sales.doc
+ order by embedding <=> meta.embed('how do I return an order?', 'DOC_EMBEDDINGS')::vector
+ limit 5;
+```
+
 ### Parsing files in SQL
 
 `meta.parse_data()` reads CSV/TSV and JSON in a `bytea` (an uploaded file in `meta.temp_files`, a
@@ -488,6 +506,7 @@ navigation entries and application processes have the same `build_option` column
 | `app_ai_service` | Which applications may use which AI service, with daily limits `max_requests` and `max_tokens` (null: no limit). Not exported | no (through `meta.ai_available`) |
 | `ai_usage` | One row per AI request: `at`, `app_id`, `page_no`, `username`, `service`, `provider`, `model`, `source`, `input_tokens`, `output_tokens`, `duration_ms`, `status`, `message` (an error class, never prompt or answer text). Not exported | no |
 | `ai_tool_call` | (121) Every tool call of an [AI assistant or agent](03-builder.md#ai-agents): `at`, `app_id`, `page_no`, `region_id`, `username`, `agent`, `tool`, `type`, `arguments` (the model's, cut to 4 KB), `status` (`ok`, `error`, `refused`), `duration_ms`, `message`. Kept 30 days, not exported | no |
+| `embedding_queue` | (136) Rows whose vector column the server re-embeds: `app_id`, `provider`, `tbl` (the table's oid), `vector_column`, `key_column`, `source_columns`, `key_value`, `queued_at`, `next_at`, `attempts`, `last_error`. Filled by `meta.embedding_enqueue` triggers and Backfill; not exported | no |
 | `ai_conversation` | [AI assistant](04-pages-and-regions.md#ai_assistant-ai-assistant) conversations: one per session and region (`messages`: the provider's history, `turns`: what the page shows); deleted with the session. Not exported | no |
 | `ai_table_note` | Descriptions of tables and columns for models ([App Builder AI](03-builder.md#app-builder-ai)): `schema_name`, `table_name`, `column_name` (`''` for the table), `note`. Installation data, not exported | no |
 | `blueprint` | Saved [blueprints](03-builder.md#creating-an-application-from-a-blueprint): `name`, `spec` (the JSON), `app_id` (the application last created from it), `created_by`. Builder data, not exported | no |

package/docs/guide/12-development.md

@@ -145,6 +145,12 @@ src/
                            scramVerifier (SCRAM-SHA-256, SASLprep) and the password slot (maskSql for review and log, withVerifier to run)
   cron-schedule.ts         (sprint 101, DA-12) pg_cron schedules without a database: parseSchedule (cron fields, $, n seconds, macros),
                            nextRuns in a time zone, describeSchedule in words, the builder's presets and fields
+  vector.ts                (sprint 136, AP-1) pgvector: pgvectorStatus(), vectorSchema() (operators and the type schema-qualified),
+                           DISTANCES (cosine/l2/inner_product: operator, HNSW opclass, similarity), vectorColumnDims(); the embedding
+                           queue (meta.embedding_queue, migration 136): embeddingTick() (groups, FOR UPDATE SKIP LOCKED, batches
+                           within the provider's per-minute limit, written with devDbForApp(), retries with backoff), drainEmbeddings(),
+                           kickEmbeddings(), startEmbeddingRunner() (server.ts; EMBEDDINGS=off, EMBEDDING_INTERVAL_S), vectorTriggers()
+                           (meta.embedding_set / embedding_enqueue triggers and their arguments), backfill(), queueState()
   pgcron.ts                (sprint 101, DA-12) pg_cron for SQL Workshop → Scheduling: cronState (installed, loaded, cron.database_name,
                            access), jobs and runs, jobSql/activeSql/unscheduleSql/purgeSql (reviewed, then applySql), runNow
   secrets.ts               secrets at rest (web credentials): AES-256-GCM with PGKILN_SECRET_KEY (or a key given, for rotate-key);
@@ -176,6 +182,11 @@ src/
                            (121) an agent's effort (Claude output_config.effort, OpenAI reasoning_effort) and output limit).
                            Providers (121): anthropic, openai, gemini (OpenAI-compatible endpoint, GEMINI_BASE_URL), openai_compatible
                            (Ollama, vLLM, Azure OpenAI; base URL required, key optional): service.ts serviceKey(), baseUrlFor(), providerLabel()
+                           (sprint 136, AP-1) embeddings.ts: vector providers (meta.app.vector_providers: ai_service with an embedding
+                           model, or a function), vectorProviderProblems(), embedTexts() (the openai SDK's embeddings API, float
+                           encoding, dimensions checked, daily limits via checkDailyLimit(), meta.ai_usage source "embedding"; a
+                           function provider through meta.embed on the caller's connection), embedQuery() (search texts remembered
+                           10 minutes per process), vectorLiteral()
   debug.ts                 debug messages: DebugLog (levels, timed steps, NOTICEs of meta.debug from appTx), started in
                            loadContext, stored after the response (onResponse hook → meta.debug_save), hourly purge
   icons.ts                 icon helper (sprite in public/icons.svg)
@@ -255,7 +266,8 @@ src/
                            definitionProblems), agentFor(), prepareTools() (authorization schemes, navigate pages the user may
                            open, strict schemas), runAgentTool() (sql: read-only transaction as the app role with a time limit;
                            function: named arguments, committed; rest; javascript: queued for the browser; navigate: URL and
-                           checksum built here), logToolCall() (meta.ai_tool_call, debug level 4, 30-day purge)
+                           checksum built here), logToolCall() (meta.ai_tool_call, debug level 4, 30-day purge); (136) retrieve:
+                           retrieveTool() (QUERY only, search.ts retrieve() in a read-only transaction as the app role)
     ai-filter.ts           "Ask in your own words" on a report ("ai_filter", sprint 112: AI interactive reports): the box, the chips
                            form (Edit/Remove/Undo, undo history in the session), ask and chips routes; an AI service's structured
                            output (planSchema) or the phrase parser, then ir-plan.ts
@@ -266,6 +278,14 @@ src/
     pdf.ts                 report PDFs with report layouts (pdfkit); rows from a cursor in batches (tablePdf takes batches)
     search.ts              search region and search configurations (migration 080): words as ILIKE parameters, table/sql/list
                            sources as the app role in savepoints, marked results, configColumns/searchConfigProblems for the Advisor
+                           (sprint 135, migration 135) search type fulltext: tsvectorSql (column, expression or built from the
+                           searchable columns), websearch_to_tsquery / ts_rank_cd / ts_headline between private-use markers
+                           (headlined() escapes, only the markers become <mark>), textSearchConfig() (pg_ts_config names only,
+                           passed as $1::regconfig; tsConfigFor(language)), trigramSchema() and "did you mean" with pg_trgm
+                           (sprint 136, migration 136) search types vector (vectorRows: the text embedded by the provider, nearest
+                           rows by the schema-qualified distance operator, the vector as $1, min_similarity) and hybrid (full-text
+                           and vector rankings, 50 rows each, reciprocal rank fusion 1/(60+rank) per key_column); retrieve() for
+                           an agent's Retrieve tool
     url-region.ts          URL region (migration 078): literal origins only (urlTarget), &ITEM. URL-encoded, sandbox and height,
                            frameOrigins() for the CSP's frame-src (security.ts)
     ajax.ts                Ajax callbacks (migration 085: POST …/ajax/:name, point "ajax" processes, x01 … x10 binds) and the
@@ -353,6 +373,16 @@ src/
     object-browser.ts      (sprint 75) the Object Browser: objects by type, system objects hidden, a table in tabs, column actions as SQL to review
     table-editor.ts        (sprint 95) Table Properties: GET/POST /sql/objects/table (create, edit ?o=, ?template=), fromForm(), Preview / Apply
                            (one transaction) / Open in the worksheet
+    fulltext.ts            (sprint 135, AP-2) Table Properties › Add full-text search: GET/POST /sql/objects/fulltext?o= (text columns
+                           with weights A–D, a configuration from pg_ts_config, fulltextDdl(): a generated tsvector column and a GIN
+                           index), Preview / Apply (one transaction, recorded in the Change Log) / Open in the worksheet
+    vector-column.ts       (sprint 136, AP-1) Table Properties › Add vector column: GET/POST /sql/objects/vector?o= (a provider of the
+                           workspace's applications, source columns, distance; vectorDdl(): vector(n), HNSW index, the
+                           meta.embedding_set / embedding_enqueue trigger), Preview / Apply (Change Log; the rows queued) / worksheet;
+                           the table's vector columns with progress and Backfill (mode=backfill, all=on)
+    vector-providers.ts    (sprint 136, AP-1) Shared Components → Vector providers (/apps/:id/vector-providers: list, provider form,
+                           test, delete) and Workspace utilities → Vector providers (/vector-providers: pgvector status, Create
+                           extension for administrators, the workspace's providers and vector columns); pgvectorNote()
     data-security.ts       (sprint 119) a table's Security: GET/POST /sql/objects/security?o= (policies, roles × columns, masks), Preview / Apply
     object-actions.ts      (sprint 94) an object's ⋯ menu (objectMenu) in the worksheet and the Object Browser: Quick DDL (objectDdl), Drop with
                            dependents, Add index / trigger / policy as drafts; GET /sql/objects/action, /sql/objects/panel (Open in a panel)
@@ -495,6 +525,13 @@ test/
   changelog.test.ts        (sprint 106) DDL recognition and names, recording from Table Properties, the worksheet and Compare,
                            downloads (Flyway, Liquibase, SQL), stable numbers, the latest application export
   table-editor.test.ts     (sprint 95) the Table Properties page: preview, apply, a failing apply rolls back, template, worksheet
+  fulltext-search.test.ts  (sprint 135) full-text search configurations: stemming, rank order, highlights, websearch syntax, the
+                           configuration fallback, did you mean (pg_trgm when it can be installed), the report search on a tsvector
+                           column, the Advisor's GIN check, Add full-text search (preview, apply, change log), export/import
+  vector-search.test.ts    (sprint 136) vector providers and their checks, meta.embed, Add vector column (preview, apply, the queue
+                           with the mock's embeddings, a change queued again, Backfill, a failing batch), vector and hybrid search
+                           ("time off for a family move" → Relocation leave; RLS), the Advisor, Retrieve tools on page 49, the
+                           builder pages, export; skips what needs pgvector when it isn't installed
   object-actions.test.ts   (sprint 94) object menus: Quick DDL, Drop with dependents, Add index / trigger / policy, the panel, not-found
   data-modeler.test.ts     (sprint 98) Data Modeler: "hr core" with its FK lines, saved positions, SVG, objects, DDL, dictionary, geometry
   worksheet.test.ts        (sprint 91) the SQL Worksheet: statement at the cursor, Run Script log, plans, notices, history, navigator, Run as
@@ -556,7 +593,9 @@ test/
   find-workspace.test.ts   developers' addresses, builder mail, the workspaces mail, the same answer, limits
   ai-builder.test.ts       App Builder AI: the builder's service, SQL from a question, explain, describe tables, pages with AI
   ai-script-mock.ts        a scripted mock of Claude (SSE, tool_use with a thinking block) and OpenAI (tool calls), reply by reply
-  ai-mock.ts               a local mock of the Claude Messages API (SSE stream) and OpenAI Chat Completions
+  ai-mock.ts               a local mock of the Claude Messages API (SSE stream) and OpenAI Chat Completions; (136) OpenAI's
+                           embeddings (/embeddings, also in ai-script-mock.ts): conceptEmbedding(), eight "topics", the same
+                           as the HR example's hr.policy_embedding
   helpers.ts               a cookie-keeping test browser
   e2e/responsive.test.ts   browser tests at phone/tablet/desktop widths (Playwright)
   e2e/builder-i18n.test.ts (sprint 121) Deutsch on the sign-in page, the status bar's language menu, German screens at 390 px, CSP

package/docs/guide/15-extensions.md

@@ -63,7 +63,7 @@ that includes them.
 | **[pg_cron](https://github.com/citusdata/pg_cron)** | Run SQL on a schedule inside the database (`select cron.schedule('nightly', '0 2 * * *', 'call hr.close_month()')`) | **Automations**, `DBMS_SCHEDULER` |
 | **[pgaudit](https://github.com/pgaudit/pgaudit)** | Statement-level audit logging (reads, writes, DDL, role changes) to the server log, for compliance | Oracle Unified Auditing |
 | **PostGIS** | Geometry and geography types, spatial indexes and functions: the foundation for a map region | **Map region**, Oracle Spatial |
-| **pgvector** | Vector similarity search (embeddings) for semantic search and retrieval-augmented generation | APEX AI (RAG sources), Oracle AI Vector Search |
+| **pgvector** | Vector similarity search (embeddings) for semantic search and retrieval-augmented generation; pgkiln's vector providers, vector and hybrid search configurations and Retrieve tools use it (sprint 136, [chapter 4](04-pages-and-regions.md#vector-search)) | APEX AI (RAG sources), Oracle AI Vector Search |
 | **[plpgsql_check](https://github.com/okbob/plpgsql_check)** | Static analysis of PL/pgSQL: wrong columns, unused variables, missing `RETURN`, hidden casts. Run it on your app's functions in CI | **Advisor** (for code) |
 
 Scheduling example with pg_cron. pg_cron must be in `shared_preload_libraries` (a server setting;

package/examples/hr/README.md

@@ -42,6 +42,8 @@ Then open http://127.0.0.1:3100/a/hr and sign in as `king`, `blake`, `jones`, `a
 | `hr_49_push_notifications.sql` | Push notifications: the app may send them (users turn them on under *My account → Notifications*); a new leave request notifies the manager, a decision notifies the employee, and the notification opens the request (page 7) |
 | `hr_64_ai_agent.sql` | *HR agent* (page 46): the AI agent `HR_AGENT` (Shared Components → AI Agents) with four tools: `top_earners` (SQL, read-only as `hr_app`), `open_employee` (navigate: a link to the employee form with its checksum, only for users who may open page 3), `give_raise` (the PL/pgSQL function `hr.give_raise`, managers only) and `note` (a JavaScript tool: the custom event `hr_agent_note`). Ask *Who in Sales earns most?* Needs an AI service named `HR_ASSISTANT` (see part 41) |
 | `hr_68_component_details.sql` | *Component details* (page 47): a popup LOV over 6,000 rows that loads more as the dialog scrolls, a wait popup around a two-second process (*Recalculate*) and around an Ajax call (*Check in the background*), and an *Actions* menu whose entries run dynamic actions (event `menu_entry`) |
+| `hr_71_fulltext_search.sql` | Full-text search: `hr.policy` gets a generated `search_vector` column (title weight A, body B, english) with a GIN index, and the search configuration `POLICIES` (search type *fulltext*, *Did you mean* when `pg_trgm` is installed) joins page 42 *Search*: "works home" finds *Working from home*, the matched words highlighted |
+| `hr_72_vector_search.sql` | Search by meaning: `hr.policy` gets *Relocation leave* and a managers-only *Salary bands* with row level security (`audience`); with pgvector, the vector provider `POLICY_EMBEDDINGS` (a stand-in function, `hr.policy_embedding`, eight topics), `hr.policy.embedding vector(8)` with an HNSW index and a trigger, the hybrid search configuration `POLICIES_MEANING` on page 42 (*time off for a family move* finds *Relocation leave*), and *Policy questions* (page 49): the agent `POLICY_AGENT` with the Retrieve tool `policy_search`. Needs an AI service named `HR_ASSISTANT` for page 49 |
 
 The files run in order, each once (recorded in `public.pgkiln_seed`), so new ones can be added
 later. pgkiln's own tests use this application as their fixture (`npm test` installs it first).

package/examples/hr/hr_71_fulltext_search.sql

@@ -0,0 +1,43 @@
+-- =====================================================================
+-- HR example, part 71: a full-text search configuration (sprint 135, AP-2;
+-- migration 135; docs/guide/04-pages-and-regions.md, search region)
+--
+-- hr.policy (part 42) gets what SQL Workshop › Table Properties › Add
+-- full-text search writes: a stored generated tsvector column over the
+-- title (weight A) and the body (weight B) in the english text search
+-- configuration, and a GIN index on it.
+--
+-- The search configuration POLICIES searches it with search type fulltext:
+-- page 42 "Search" then finds "Requesting leave" for "request" and
+-- "Working from home" for "works home", the best match first and the
+-- matched words highlighted. "Did you mean" suggests the closest words when
+-- nothing is found, once the pg_trgm extension is installed (without it,
+-- nothing is suggested).
+-- =====================================================================
+
+alter table hr.policy
+  add column search_vector tsvector generated always as (
+    setweight(to_tsvector('english'::regconfig, coalesce(title, '')), 'A') ||
+    setweight(to_tsvector('english'::regconfig, coalesce(body, '')), 'B')) stored;
+create index policy_search_vector_idx on hr.policy using gin (search_vector);
+
+insert into meta.search_config (app_id, name, label, seq, source_type, source, search_type, fts_vector, fts_config, did_you_mean,
+                                title_column, description_column, icon, max_results)
+select id, 'POLICIES', 'HR policies', 25, 'table', 'hr.policy', 'fulltext', 'search_vector', 'english', true,
+       'title', 'body', 'book', 5
+  from meta.app where alias = 'hr';
+
+update meta.region r
+   set config = jsonb_set(r.config, '{configs}', (r.config->'configs') || '["POLICIES"]'::jsonb)
+         || '{"placeholder": "Search employees, departments, policies and shortcuts…"}'::jsonb
+  from meta.page p join meta.app a on a.id = p.app_id
+ where r.page_id = p.id and a.alias = 'hr' and p.page_no = 42 and r.type = 'search';
+
+insert into meta.translation (app_id, language, source, target)
+select a.id, 'nl', t.source, t.target
+  from meta.app a, (values
+  ('HR policies', 'HR-regelingen'),
+  ('Search employees, departments, policies and shortcuts…', 'Zoek medewerkers, afdelingen, regelingen en snelkoppelingen…')
+  ) as t (source, target)
+ where a.alias = 'hr'
+on conflict do nothing;

package/examples/hr/hr_72_vector_search.sql

@@ -0,0 +1,158 @@
+-- =====================================================================
+-- HR example, part 72: search by meaning (sprint 136, AP-1; migration
+-- 136; docs/guide/04-pages-and-regions.md "Vector search", 03-builder.md
+-- "Vector providers", 20-ai-agents.md)
+--
+-- hr.policy (parts 42 and 71) gets two more policies and an audience:
+-- "Salary bands" is for managers only, and row level security now shows
+-- hr_app only the policies the user may read. Every search below runs as
+-- hr_app, so an employee never finds the managers' policy.
+--
+-- With pgvector installed (the extension is created here when the server
+-- has it and this connection may; otherwise this part only adds the
+-- policies and says so):
+--
+--   - the vector provider POLICY_EMBEDDINGS (Shared Components → Vector
+--     providers) of type function: hr.policy_embedding(text), a tiny
+--     stand-in for an embedding model that maps words to eight topics
+--     (moving house, time off, home working, sickness, pay, requests,
+--     public holidays, anything else). It keeps the example working without
+--     an AI service; for real use, make the provider an AI service with an
+--     embedding model (e.g. OpenAI text-embedding-3-small, 1536 dimensions,
+--     or Ollama's nomic-embed-text, 768) and add the column again with
+--     Table Properties › Add vector column;
+--   - what Table Properties › Add vector column writes: hr.policy.embedding
+--     vector(8), an HNSW index (vector_cosine_ops) and a trigger
+--     (meta.embedding_set) that embeds a policy when its title or body
+--     changes;
+--   - the search configuration POLICIES_MEANING (search type hybrid:
+--     full-text on search_vector and the embedding, merged by reciprocal
+--     rank fusion) joins page 42 "Search": "time off for a family move"
+--     finds "Relocation leave", which uses none of those words;
+--   - page 49 "Policy questions": an AI assistant region with the agent
+--     POLICY_AGENT, whose Retrieve tool policy_search hands the model the
+--     policies that match a question (as hr_app). Like parts 41, 42 and 64
+--     it needs an AI service named HR_ASSISTANT.
+-- =====================================================================
+
+alter table hr.policy add column audience text not null default 'all' check (audience in ('all', 'managers'));
+
+insert into hr.policy (title, body, audience) values
+  ('Relocation leave', 'Employees who relocate to another city for the company get three paid days to settle into the new house, and help with the removal costs.', 'all'),
+  ('Salary bands', 'Managers review the salary bands every spring; a raise above the band needs the director''s approval.', 'managers');
+
+alter table hr.policy enable row level security;
+create policy policy_visible on hr.policy for select
+  using (audience = 'all' or meta.has_role('manager') or meta.has_role('admin'));
+
+-- a stand-in for an embedding model: each word counts for one of eight topics (unknown words a little for the
+-- last), and the vector is normalised, so the cosine of two texts says how much they share topics
+create function hr.policy_embedding(p_text text) returns real[]
+language sql immutable parallel safe set search_path = pg_catalog as $$
+  with w as (select lower(m[1]) as w from regexp_matches(coalesce(p_text, ''), '([[:alpha:]]+)', 'g') m),
+  d as (select case
+                 when w ~ '^(relocat|move|moving|removal|house|settl|city|family)' then 1
+                 when w ~ '^(leave|time|off|holiday|vacation|days|absence)$' then 2
+                 when w ~ '^(home|remote|working|work)$' then 3
+                 when w ~ '^(sick|ill|doctor)' then 4
+                 when w ~ '^(salar|pay|band|raise|wage|scale)' then 5
+                 when w ~ '^(request|approv|manager)' then 6
+                 when w ~ '^(public|closed|office)' then 7
+                 else 8 end as dim,
+               case when w ~ '^(relocat|move|moving|removal|house|settl|city|family|leave|time|off|holiday|vacation|days|absence|home|remote|working|work|sick|ill|doctor|salar|pay|band|raise|wage|scale|request|approv|manager|public|closed|office)'
+                    then 1.0 else 0.05 end as weight
+          from w),
+  v as (select i, coalesce((select sum(weight) from d where d.dim = i), 0) as x from generate_series(1, 8) i),
+  n as (select sqrt(sum(x * x)) as n from v)
+  select array_agg((case when n.n = 0 then (case when i = 8 then 1 else 0 end) else x / n.n end)::real order by i) from v, n
+$$;
+grant execute on function hr.policy_embedding(text) to hr_app;
+
+update meta.app set
+  vector_providers = vector_providers || $j$[
+    {"name": "POLICY_EMBEDDINGS", "type": "function", "function": "hr.policy_embedding", "dimensions": 8,
+     "description": "A stand-in for an embedding model (eight topics); use an AI service for real texts"}
+  ]$j$::jsonb,
+  ai_tools = ai_tools || $j$[
+    {"name": "policy_search", "type": "retrieve", "search_config": "POLICIES_MEANING", "max_rows": 3,
+     "description": "The HR policies that match a question, best first (title and text). Use it for every question about rules, leave, pay or working arrangements."}
+  ]$j$::jsonb,
+  ai_agents = ai_agents || $j$[
+    {"name": "POLICY_AGENT", "description": "Answers questions about the HR policies the user may read",
+     "service": "HR_ASSISTANT", "effort": "low", "tools": ["policy_search"], "max_steps": 3, "max_tokens": 1500,
+     "system": "You answer questions of &APP_USER. about the company's HR policies. Look the policies up with policy_search and answer only from what it returns; say so when no policy covers the question. Name the policy you used. Be brief."}
+  ]$j$::jsonb
+ where alias = 'hr';
+
+-- page 49: questions about the policies, answered through the Retrieve tool
+insert into meta.page (app_id, page_no, name, title, parent_page)
+select id, 49, 'Policy questions', 'Policy questions', 1 from meta.app where alias = 'hr';
+
+insert into meta.region (page_id, seq, title, type, columns, template, source, condition, config)
+select p.id, r.seq, r.title, r.type, r.cols, 'standard', r.source, r.cond, r.config::jsonb
+  from meta.page p join meta.app a on a.id = p.app_id, (values
+  (10, 'Not set up yet', 'static', 12,
+   '<p class="alert alert-info">Policy questions need the pgvector extension in the database and an AI service named <b>HR_ASSISTANT</b> (App Builder → Workspace utilities → AI services, allowed for the HR application).</p>',
+   'not (meta.ai_available(''HR_ASSISTANT'') and exists (select from pg_catalog.pg_extension where extname = ''vector''))', '{}'),
+  (20, 'Ask about the policies', 'ai_assistant', 8, null,
+   'meta.ai_available(''HR_ASSISTANT'') and exists (select from pg_catalog.pg_extension where extname = ''vector'')',
+   $j${"agent": "POLICY_AGENT",
+       "welcome": "Hello! Ask me about the HR policies, e.g. how much time off you get when you move for the company.",
+       "placeholder": "e.g. Do I get time off for a family move?",
+       "max_turns": 20,
+       "error_message": "The policy assistant could not answer just now. Please try again later."}$j$)
+  ) as r (seq, title, type, cols, source, cond, config)
+ where a.alias = 'hr' and p.page_no = 49;
+
+insert into meta.nav_entry (app_id, seq, label, icon, target_page)
+select id, 30, 'Policy questions', 'book', 49 from meta.app where alias = 'hr';
+
+-- with pgvector: the extension (when this connection may create it), the vector column, the search configuration
+-- and the search page
+do $$
+declare
+  v_schema text;
+begin
+  if not exists (select from pg_extension where extname = 'vector') then
+    if exists (select from pg_available_extensions where name = 'vector') and (select rolsuper from pg_roles where rolname = current_user) then
+      create extension vector;
+    else
+      raise notice 'HR part 72: pgvector is not installed: hr.policy gets no vector column (install pgvector, then add one with Table Properties › Add vector column).';
+      return;
+    end if;
+  end if;
+  select n.nspname into v_schema from pg_extension e join pg_namespace n on n.oid = e.extnamespace where e.extname = 'vector';
+  execute format('alter table hr.policy add column embedding %I.vector(8)', v_schema);
+  execute format('create index policy_embedding_idx on hr.policy using hnsw (embedding %I.vector_cosine_ops)', v_schema);
+  execute format($t$create trigger embedding_embed before insert or update of title, body on hr.policy
+                    for each row execute function meta.embedding_set(%L, 'POLICY_EMBEDDINGS', 'embedding', 'id', 'title', 'body')$t$,
+                 (select id from meta.app where alias = 'hr'));
+  -- the policies there are: embedded now (the trigger does it for new and changed ones)
+  execute format($u$update hr.policy set embedding = translate(meta.embed(meta.embedding_text(to_jsonb(policy), '{title,body}'), 'POLICY_EMBEDDINGS', %s)::text, '{}', '[]')::text::%I.vector$u$,
+                 (select id from meta.app where alias = 'hr'), v_schema);
+  insert into meta.search_config (app_id, name, label, seq, source_type, source, search_type, fts_vector, fts_config,
+                                  vector_provider, vector_column, key_column, distance, min_similarity,
+                                  title_column, description_column, icon, max_results)
+  select id, 'POLICIES_MEANING', 'Policies by meaning', 26, 'table', 'hr.policy', 'hybrid', 'search_vector', 'english',
+         'POLICY_EMBEDDINGS', 'embedding', 'id', 'cosine', 0.6,
+         'title', 'body', 'book', 3
+    from meta.app where alias = 'hr';
+  update meta.region r
+     set config = jsonb_set(r.config, '{configs}', (r.config->'configs') || '["POLICIES_MEANING"]'::jsonb)
+    from meta.page p join meta.app a on a.id = p.app_id
+   where r.page_id = p.id and a.alias = 'hr' and p.page_no = 42 and r.type = 'search';
+end $$;
+
+insert into meta.translation (app_id, language, source, target)
+select a.id, 'nl', t.source, t.target
+  from meta.app a, (values
+  ('Policies by meaning', 'Regelingen op betekenis'),
+  ('Policy questions', 'Vragen over regelingen'),
+  ('Ask about the policies', 'Vraag naar de regelingen'),
+  ('Not set up yet', 'Nog niet ingesteld'),
+  ('Hello! Ask me about the HR policies, e.g. how much time off you get when you move for the company.',
+   'Hallo! Vraag me naar de HR-regelingen, bijv. hoeveel vrij je krijgt als je voor het bedrijf verhuist.'),
+  ('e.g. Do I get time off for a family move?', 'bijv. Krijg ik vrij voor een verhuizing met het gezin?')
+  ) as t (source, target)
+ where a.alias = 'hr'
+on conflict do nothing;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pgkiln",
-  "version": "0.68.0",
+  "version": "0.70.0",
   "description": "Open source low-code, SQL-driven application builder for PostgreSQL, inspired by Oracle APEX",
   "keywords": [
     "postgresql",

package/public/builder.css

@@ -1597,3 +1597,6 @@ pre.wrap-pre { white-space: pre-wrap; overflow-wrap: anywhere; }
   --font: "Inter", "Segoe UI Variable Text", "Segoe UI", system-ui, -apple-system, "Apple SD Gothic Neo", "Malgun Gothic", "Noto Sans KR", "Noto Sans CJK KR", "Source Han Sans KR", sans-serif; }
 :is(.ide-lang-form, .signin-langs) [lang] { font-family: var(--font); }
 :is(.ide, .ide-login):lang(ko) { word-break: keep-all; overflow-wrap: break-word; }
+
+/* (sprint 136) a button inside an alert (e.g. Add vector column without pgvector) wraps on a phone instead of overflowing */
+.alert .btn { white-space: normal; overflow-wrap: anywhere; max-width: 100%; }