pgkiln 0.51.0 → 0.53.0

package/README.md

@@ -123,7 +123,7 @@ developing pgkiln itself.
 
 ## What's in the box
 
-For a full comparison with Oracle APEX 26.1, including what's missing, see
+For a full comparison with Oracle APEX 26.2, including what's missing, see
 **[docs/apex-feature-parity.md](docs/apex-feature-parity.md)**.
 
 **Runtime (Universal Theme-style UI, responsive on phone, tablet and desktop)**

package/SECURITY.md

@@ -99,6 +99,15 @@ first version (MVP). Every finding below has a regression test in
 | SQL Worksheet transactions (sprint 107) | A worksheet's open transaction is held in server memory under the caller's builder session id plus the tab key and checked against the workspace: every lookup starts from the caller's own session, so another developer or another sign-in never reaches it. Its connection is a plain client built from the workspace's developer login (or the *Run as* application's), never returned to a pool, and closed at the end, so `SET ROLE` reaches only roles that login has and nothing leaks into the builder's pools; *Run as* can't change while it is open. Rolled back on idle timeout, sign-out, session expiry, workspace switch, tab close and server stop, with `idle_in_transaction_session_timeout` as a backstop; at most `WORKSHEET_TXN_MAX_PER_DEVELOPER` per developer and `WORKSHEET_TXN_MAX` per server, each one database connection outside the pools. CSRF and a developer sign-in on every endpoint. |
 | Release check (sprint 109) | Off by default (`PGKILN_RELEASE_CHECK`, or Instance settings for administrators). When on, the server sends one GET to GitHub's releases API at most once a day per process, with a 5-second timeout and a 1 MB limit, and sends nothing about the installation; only a semver version and an https link are kept from the answer, and they are escaped on the Workspace home. Pages never wait for it. |
 | Data Reporter datasets and reporting applications (sprint 110) | Datasets are managed by the workspace's developers with CSRF on every POST; their schema must be one the workspace's SQL Workshop connection may use (never pgkiln's own or the system's), and tables, columns, joins and filters are checked against the catalog. The runtime builds a dataset's SELECT with quoted identifiers, literal values and whitelisted operators, and runs reports as the application's role. A reporting application's role gets SELECT only on the datasets' tables and views, so RLS applies; viewers never reach the report editor (report parameters in the URL are ignored), and only `admin_authz` users unpublish or delete others' reports. Dataset files are checked like a saved dataset before import. |
+| AI Agents and tools (sprint 111) | SQL tools are read-only (functions that change data and data-modifying `WITH` fail) and run as the application's role, so row level security applies; the model can't set `:APP_USER`. A Navigate tool links only to pages the user may open, with item values of that page and a checksum pgkiln builds. An agent's authorization scheme refuses other users before any AI call. Tool arguments and model output reach the page escaped. AI service keys stay on the server: not in builder pages, the tool call log, an export or the runtime role's reach. The builder's agent and tool forms need a developer and the CSRF token. |
+| AI interactive reports (sprint 112) | "Ask in your own words" (the phrase parser and the AI answer) only sets the report's own settings over its shown columns, never a hidden one; operators, functions, colours, chart types and computed-column expressions come from whitelists and are checked; values are query parameters or escaped literals. Chips are edited and removed through a CSRF-checked form whose fields are checked again; a report without `ai_filter`, a region of another page and signed-out users are refused. |
+| Team Development (sprint 113) | Issues, labels, milestones, templates and saved views belong to the developer's current workspace only, and can be neither read nor changed from another; assignees and @mentions are the workspace's developers, links go only to its applications. Every POST needs the session's CSRF token; titles, bodies, comments and names are escaped, Markdown is made safe, label colours come from a palette. Application users and signed-out visitors can't reach it. |
+| Workflow 26.2 (sprint 114) | Migrating running instances needs a developer of the application and the CSRF token; unmapped steps and other applications' workflows are refused. A task ends only with an outcome of its definition. Task event SQL runs as the application's role (after the commit, or in the request for *fail* events). Draft versions can't be activated in the builder, with SQL or into the active steps. The diagram export is for developers only, served as a sandboxed attachment with escaped labels. |
+| Component details (sprint 115) | The popup LOV search needs the CSRF token, caps the page size and page number and binds the search term. A wait popup's message is page data shown as text. A `menu_entry` dynamic action with an authorization the user lacks is neither sent to the page nor run (403); menu entries the user may not use are left out, labels are escaped, and the database refuses crafted action names. |
+| App Builder list details (sprint 116) | `meta.app_change` (Updated by) is written only for a builder POST that passed the developer checks (sign-in, CSRF token, workspace, locks) and succeeded; posts that change nothing record nothing. Developer names and lock owners are escaped in the report, the cards and the lock label; another workspace's applications are not listed. The table is closed to the runtime role and not exported. |
+| Pillar navigation appearances (sprint 117) | The appearance and the header strip colours are changed only by a developer with the CSRF token; only keys of a fixed list and `#rrggbb` colours are saved, and a hand-edited theme can't put markup or CSS into the page (a `<body>` class and the nonce'd `<style>`). A strip colour from an item is used only when the value is `#rrggbb`. |
+| Mail queue, workspace mail and network access (sprint 118) | Administration → Mail queue: administrators see every workspace, a workspace's administrators only their current workspace's applications, other developers get 403; Retry and Purge (one mail or all failed) check the mail's workspace again on the server and need the CSRF token; bodies and attachments are never shown. A workspace's own mail service is managed by its administrators only; its secret is encrypted like a web credential's, write-only, never shown or exported (not in the workspace export); it has no API base URL or token URL, and its SMTP host must pass the installation's allow-list and the workspace's network access list and resolve to a public address (unless `PGKILN_REST_PRIVATE_HOSTS` lists it), resolved once and connected to by that address (no DNS rebinding). A workspace's network access list narrows `PGKILN_REST_ALLOWED_HOSTS` on the server for every outgoing web call of its applications (`meta.web_request`, REST data sources incl. write-back and the response cache, invoke API, token endpoints, object storage, the builder's Test, REST source catalogs), on every redirect hop; it can't widen the installation's list or the private-address checks. Refusals go to the debug log and `meta.network_refusal` (7 days). Also fixed: *REST source catalogs → read from a URL* fetched the URL before checking the developer's session (any visitor could make the server call an allowed host); it is now read only after the developer and CSRF checks. |
+| Data security per table (sprint 119) | SQL Workshop → a table → Security writes `GRANT` / `REVOKE` and a masking view, so it needs a signed-in developer and the CSRF token, and works only on tables of the workspace's schemas (never pgkiln's own or the system's: 404). Table, column, view and role names reach SQL quoted; view names and application roles that could break out of their quotes are refused. The masking role is `nologin`, a member of the reader roles, so row level security still applies inside the view, and the generated SQL refuses to run when that role would bypass RLS. MCP `run_query` with an application runs as that application's database role (read-only) and refuses unknown applications. |
 
 ## Findings of the MVP review and their fixes
 
@@ -218,6 +227,6 @@ A second full review, mainly of what end users and API callers can reach. Regres
 5. Give each app its own `db_role` with the minimum grants; add RLS where rows are per user or per team. REST modules and AutoREST refuse to run without one on a privileged connection (403); run the Advisor after upgrading.
 6. Restrict network access to `/builder` (reverse proxy or firewall) if developers are a small group.
 7. With a REST API: set a long random `API_JWT_SECRET` (the same in PostgREST), change the `pgkiln_authenticator` password, configure `db-pre-request = meta.api_check`, expose only the `api` schema, and run PostgREST behind HTTPS.
-8. With REST data sources: list only the hosts you need in `PGKILN_REST_ALLOWED_HOSTS` (avoid `*`), set a long random `PGKILN_SECRET_KEY` (32+ characters) kept outside the database and its backups, and use `PGKILN_REST_PRIVATE_HOSTS` only for internal services you mean to reach.
+8. With REST data sources: list only the hosts you need in `PGKILN_REST_ALLOWED_HOSTS` (avoid `*`), set a long random `PGKILN_SECRET_KEY` (32+ characters) kept outside the database and its backups, and use `PGKILN_REST_PRIVATE_HOSTS` only for internal services you mean to reach. Narrow it further per workspace under Administration → Network access (sprint 118).
 9. With header authentication: list only your proxy's addresses in `PGKILN_AUTH_HEADER_PROXIES`, make sure users can't reach pgkiln directly, and have the proxy remove any incoming copy of the header.
 10. With database-account authentication: list exact roles or use a dedicated membership role, and limit what those roles may connect to in `pg_hba.conf` (they can reach the database with their password outside pgkiln too).

package/db/migrations/121_ai_agents.sql

@@ -0,0 +1,94 @@
+-- =====================================================================
+-- 121: AI agents and AI tools (sprint 111, WS-4; APEX 26.1 / 26.2: Shared
+--      Components → AI Agents, AI Tools), more AI providers, and a log of
+--      every tool call
+--
+-- meta.app.ai_tools: the application's AI tools, a JSON list. A tool is
+--   what an agent may do, each one an approved, constrained action:
+--     {"name": "top_earners", "type": "sql", "description": "…",
+--      "sql": "select … where dname ilike :DNAME", "parameters": {"DNAME": {"type": "string"}},
+--      "max_rows": 20, "timeout_s": 10, "authz": "MANAGER"}
+--   Types: sql (a query, run as the application's role through RLS in a
+--   read-only transaction with a time limit), function (a PL/pgSQL
+--   function called with named arguments, as the application's role; it
+--   may change data), rest (a REST data source of the application),
+--   javascript (a function registered with pgkiln.actions.register and/or
+--   the dynamic actions of the custom event with the tool's action name,
+--   run in the user's browser after the answer) and navigate (the model
+--   picks one of the tool's pages and item values; pgkiln checks the user
+--   may open the page and builds the URL with its checksum).
+-- meta.app.ai_agents: the application's AI agents, a JSON list:
+--     {"name": "HR_AGENT", "description": "…", "service": "HR_ASSISTANT",
+--      "system": "…", "effort": "low" | "medium" | "high" | null,
+--      "tools": ["top_earners", "open_employee"], "max_steps": 5,
+--      "max_tokens": 2000, "authz": "MANAGER"}
+--   An AI assistant region names an agent ("agent" in its config) instead
+--   of a service, a system prompt and tools of its own.
+-- Both are definitions of the application: export_app copies every column
+-- of meta.app, so they travel with it (no redefinition of export_app /
+-- import_app). An export made before 121 has neither key: the trigger
+-- below turns the NULL from jsonb_populate_record into an empty list.
+-- The builder checks both lists (src/runtime/ai-agents.ts); the runtime
+-- checks them again before it uses them.
+--
+-- meta.ai_service.provider: also "gemini" (Google Gemini through its
+--   OpenAI-compatible endpoint) and "openai_compatible" (any endpoint that
+--   speaks OpenAI's Chat Completions: Ollama, vLLM, Azure OpenAI, a
+--   gateway). An OpenAI-compatible service needs a base URL; its key may
+--   be empty (a local Ollama needs none).
+--
+-- meta.ai_tool_call: one row per tool call of an assistant or agent (app,
+--   page, user, agent, tool, type, the arguments the model gave, status,
+--   duration, a short message). The arguments may hold user data: only
+--   the owner reads the table (Activity → AI usage); kept 30 days.
+--   Installation data, like meta.ai_usage: not exported.
+-- =====================================================================
+
+alter table meta.app add column ai_tools jsonb not null default '[]'
+  check (jsonb_typeof(ai_tools) = 'array');
+alter table meta.app add column ai_agents jsonb not null default '[]'
+  check (jsonb_typeof(ai_agents) = 'array');
+comment on column meta.app.ai_tools is 'AI tools (121): what the application''s AI agents may do (sql, function, rest, javascript, navigate)';
+comment on column meta.app.ai_agents is 'AI agents (121): AI service, system prompt, reasoning effort, tools, limits and authorization of each agent';
+
+create function meta.app_ai_default() returns trigger
+language plpgsql set search_path = pg_catalog as $$
+begin
+  new.ai_tools := coalesce(new.ai_tools, '[]');
+  new.ai_agents := coalesce(new.ai_agents, '[]');
+  return new;
+end $$;
+create trigger app_ai_default before insert or update on meta.app
+  for each row execute function meta.app_ai_default();
+
+alter table meta.ai_service drop constraint ai_service_provider_check;
+alter table meta.ai_service add constraint ai_service_provider_check
+  check (provider in ('anthropic', 'openai', 'gemini', 'openai_compatible'));
+alter table meta.ai_service add constraint ai_service_compatible_url
+  check (provider <> 'openai_compatible' or base_url is not null);
+-- conversations keep the provider's format (Gemini and OpenAI-compatible: OpenAI's)
+alter table meta.ai_conversation drop constraint ai_conversation_provider_check;
+alter table meta.ai_conversation add constraint ai_conversation_provider_check
+  check (provider in ('anthropic', 'openai', 'gemini', 'openai_compatible'));
+
+create table meta.ai_tool_call (
+  id          bigserial primary key,
+  at          timestamptz not null default now(),
+  app_id      int not null references meta.app on delete cascade,
+  page_no     int,
+  region_id   int,
+  username    text,
+  -- null: a tool of the region itself (no agent)
+  agent       text,
+  tool        text not null,
+  type        text not null,
+  -- the model's arguments (checked), cut to a few kilobytes
+  arguments   jsonb not null default '{}',
+  status      text not null check (status in ('ok', 'error', 'refused')),
+  duration_ms int not null default 0,
+  message     text
+);
+create index on meta.ai_tool_call (app_id, id desc);
+create index on meta.ai_tool_call (at);
+revoke all on meta.ai_tool_call from public;
+comment on table meta.ai_tool_call is 'Tool calls of AI assistants and agents (121), with their arguments; kept 30 days';

package/db/migrations/122_team_development.sql

@@ -0,0 +1,186 @@
+-- =====================================================================
+-- 122: Team Development (APEX: Team Development in the workspace)
+--
+-- An issue tracker per workspace for the team that builds its
+-- applications: issues (title, Markdown body, open or closed) with labels,
+-- a milestone, assignees (the workspace's developers), a link to an
+-- application and one of its pages, comments with @mentions of developers
+-- (shown in the builder, never mailed) and a history of changes. Labels
+-- belong to label groups (e.g. Type: Bug, Feature, Task; Priority: High,
+-- Medium, Low); templates prefill new issues (kind 'issue') or comments
+-- (kind 'response', canned answers). Developers save their own filtered
+-- views of the list.
+--
+-- An issue may come from a user's feedback (meta.feedback, Activity →
+-- Feedback) or a developer comment (meta.dev_comment): linked back.
+--
+-- Workspace data, like the SQL Workshop's scripts: written by the builder
+-- as the owner, never read by the runtime; not part of an application's
+-- export (kept by pgkiln import --replace) but part of the workspace export
+-- (Workspace utilities → Export and import, part "team").
+-- =====================================================================
+
+create table meta.team_label_group (
+  id           bigint generated always as identity primary key,
+  workspace_id int  not null references meta.workspace on delete cascade,
+  name         text not null check (name ~ '^\S(.{0,38}\S)?$'),
+  description  text check (length(description) <= 500),
+  seq          int  not null default 10
+);
+create unique index team_label_group_name on meta.team_label_group (workspace_id, lower(name));
+
+create table meta.team_label (
+  id           bigint generated always as identity primary key,
+  workspace_id int  not null references meta.workspace on delete cascade,
+  group_id     bigint references meta.team_label_group on delete set null,
+  name         text not null check (name ~ '^\S(.{0,38}\S)?$'),
+  -- a colour of a fixed palette (builder.css draws them); never free CSS
+  color        text not null default 'gray' check (color in ('red', 'orange', 'yellow', 'green', 'teal', 'blue', 'purple', 'gray')),
+  description  text check (length(description) <= 500),
+  seq          int  not null default 10
+);
+create unique index team_label_name on meta.team_label (workspace_id, lower(name));
+
+create table meta.team_milestone (
+  id           bigint generated always as identity primary key,
+  workspace_id int  not null references meta.workspace on delete cascade,
+  name         text not null check (name ~ '^\S(.{0,38}\S)?$'),
+  description  text check (length(description) <= 500),
+  due_date     date,
+  status       text not null default 'open' check (status in ('open', 'closed')),
+  created_at   timestamptz not null default now()
+);
+create unique index team_milestone_name on meta.team_milestone (workspace_id, lower(name));
+
+create table meta.team_template (
+  id           bigint generated always as identity primary key,
+  workspace_id int  not null references meta.workspace on delete cascade,
+  kind         text not null check (kind in ('issue', 'response')),
+  name         text not null check (name ~ '^\S(.{0,58}\S)?$'),
+  -- issue templates: the new issue's title and labels (by name)
+  title        text check (length(title) <= 200),
+  body         text not null default '' check (length(body) <= 20000),
+  labels       text[] not null default '{}' check (cardinality(labels) <= 20),
+  seq          int  not null default 10
+);
+create unique index team_template_name on meta.team_template (workspace_id, kind, lower(name));
+
+create table meta.team_issue (
+  id             bigint generated always as identity primary key,
+  workspace_id   int  not null references meta.workspace on delete cascade,
+  title          text not null check (length(btrim(title)) between 1 and 200),
+  body           text not null default '' check (length(body) <= 20000),
+  status         text not null default 'open' check (status in ('open', 'closed')),
+  milestone_id   bigint references meta.team_milestone on delete set null,
+  -- the application (of the same workspace) and page the issue is about
+  app_id         int references meta.app on delete set null,
+  page_no        int check (page_no >= 0),
+  -- where it came from: a user's feedback or a developer comment
+  feedback_id    bigint references meta.feedback on delete set null,
+  dev_comment_id int references meta.dev_comment on delete set null,
+  created_by     text not null,
+  created_at     timestamptz not null default now(),
+  updated_at     timestamptz not null default now(),
+  closed_by      text,
+  closed_at      timestamptz
+);
+create index team_issue_ws on meta.team_issue (workspace_id, status, updated_at desc);
+create index team_issue_page on meta.team_issue (app_id, page_no) where status = 'open';
+create index team_issue_feedback on meta.team_issue (feedback_id) where feedback_id is not null;
+create index team_issue_comment_src on meta.team_issue (dev_comment_id) where dev_comment_id is not null;
+
+create table meta.team_issue_label (
+  issue_id bigint not null references meta.team_issue on delete cascade,
+  label_id bigint not null references meta.team_label on delete cascade,
+  primary key (issue_id, label_id)
+);
+create index on meta.team_issue_label (label_id);
+
+create table meta.team_issue_assignee (
+  issue_id bigint not null references meta.team_issue on delete cascade,
+  username text   not null references meta.developer (username) on delete cascade on update cascade,
+  primary key (issue_id, username)
+);
+create index on meta.team_issue_assignee (username);
+
+create table meta.team_issue_comment (
+  id         bigint generated always as identity primary key,
+  issue_id   bigint not null references meta.team_issue on delete cascade,
+  author     text not null,
+  body       text not null check (length(btrim(body)) between 1 and 20000),
+  created_at timestamptz not null default now()
+);
+create index on meta.team_issue_comment (issue_id, created_at);
+
+-- the history of an issue: who changed what, when
+create table meta.team_issue_event (
+  id       bigint generated always as identity primary key,
+  issue_id bigint not null references meta.team_issue on delete cascade,
+  at       timestamptz not null default now(),
+  username text,
+  action   text not null check (action ~ '^[a-z_]{1,30}$'),
+  detail   text check (length(detail) <= 1000)
+);
+create index on meta.team_issue_event (issue_id, at);
+
+-- @mentions of the workspace's developers in an issue's body or a comment; shown in the builder until seen
+create table meta.team_mention (
+  id         bigint generated always as identity primary key,
+  issue_id   bigint not null references meta.team_issue on delete cascade,
+  comment_id bigint references meta.team_issue_comment on delete cascade,
+  username   text   not null references meta.developer (username) on delete cascade on update cascade,
+  by_user    text,
+  created_at timestamptz not null default now(),
+  seen_at    timestamptz
+);
+create index on meta.team_mention (username) where seen_at is null;
+
+-- a developer's saved views of the issue list (the filters as a query string)
+create table meta.team_view (
+  id           bigint generated always as identity primary key,
+  workspace_id int  not null references meta.workspace on delete cascade,
+  username     text not null references meta.developer (username) on delete cascade on update cascade,
+  name         text not null check (name ~ '^\S(.{0,38}\S)?$'),
+  query        text not null default '' check (length(query) <= 1000),
+  created_at   timestamptz not null default now()
+);
+create unique index team_view_name on meta.team_view (workspace_id, username, lower(name));
+
+-- the label's group, the issue's milestone and its labels belong to the issue's workspace
+create function meta.team_same_workspace() returns trigger
+language plpgsql set search_path = pg_catalog as $$
+begin
+  -- (nested: a record field is only read in the branch of its own table)
+  if tg_table_name = 'team_label' then
+    if new.group_id is not null
+       and (select g.workspace_id from meta.team_label_group g where g.id = new.group_id) is distinct from new.workspace_id then
+      raise exception 'the label group belongs to another workspace';
+    end if;
+  elsif tg_table_name = 'team_issue' then
+    if new.milestone_id is not null
+       and (select m.workspace_id from meta.team_milestone m where m.id = new.milestone_id) is distinct from new.workspace_id then
+      raise exception 'the milestone belongs to another workspace';
+    end if;
+  elsif tg_table_name = 'team_issue_label' then
+    if (select l.workspace_id from meta.team_label l where l.id = new.label_id)
+       is distinct from (select i.workspace_id from meta.team_issue i where i.id = new.issue_id) then
+      raise exception 'the label belongs to another workspace';
+    end if;
+  end if;
+  return new;
+end $$;
+create trigger team_same_workspace before insert or update on meta.team_label for each row execute function meta.team_same_workspace();
+create trigger team_same_workspace before insert or update on meta.team_issue for each row execute function meta.team_same_workspace();
+create trigger team_same_workspace before insert or update on meta.team_issue_label for each row execute function meta.team_same_workspace();
+
+revoke all on meta.team_label_group, meta.team_label, meta.team_milestone, meta.team_template, meta.team_issue, meta.team_issue_label,
+  meta.team_issue_assignee, meta.team_issue_comment, meta.team_issue_event, meta.team_mention, meta.team_view from public;
+
+comment on table meta.team_issue is 'Team Development (122): issues of a workspace; workspace data (the workspace export), not exported with an application';
+comment on table meta.team_label is 'Team Development (122): labels of a workspace''s issues, in label groups, with a colour of a fixed palette';
+comment on table meta.team_label_group is 'Team Development (122): label groups (e.g. Type, Priority)';
+comment on table meta.team_milestone is 'Team Development (122): milestones of a workspace';
+comment on table meta.team_template is 'Team Development (122): issue templates and canned responses';
+comment on table meta.team_issue_event is 'Team Development (122): the history of an issue';
+comment on table meta.team_mention is 'Team Development (122): @mentions of developers, shown in the builder until seen';
+comment on table meta.team_view is 'Team Development (122): saved views of the issue list, per developer';