pgkiln 0.31.0 → 0.36.0

package/README.md

@@ -17,7 +17,7 @@ them.
 > pgkiln is an independent open source project and is not affiliated with Oracle. Oracle and APEX are trademarks of Oracle.
 
 <p align="center">
-  <img src="docs/images/builder-page-designer.png" alt="The page designer: component tree, layout and property editor" width="100%">
+  <img src="https://raw.githubusercontent.com/pgkiln/pgkiln/main/docs/images/builder-page-designer.png" alt="The page designer: component tree, layout and property editor" width="100%">
 </p>
 
 ## Screenshots
@@ -37,7 +37,7 @@ screenshots are made by `npm run screenshots`.
 | **App Builder** home | **SQL Workshop**: object browser with RLS and grants |
 
 <p align="center">
-  <img src="docs/images/app-phone.png" alt="The dashboard on a phone" width="260"><br>
+  <img src="https://raw.githubusercontent.com/pgkiln/pgkiln/main/docs/images/app-phone.png" alt="The dashboard on a phone" width="260"><br>
   <em>Responsive on phones and tablets, and installable as a Progressive Web App</em>
 </p>
 

package/SECURITY.md

@@ -76,6 +76,16 @@ first version (MVP). Every finding below has a regression test in
 | Plug-ins (0.31.0) | Only developers (session, CSRF, workspace guard, app locks) import, remove and install plug-ins; `meta.import_plugin` is not executable by the runtime role, which can only read `meta.plugin`. Plug-in files are checked before anything is stored: type, names, static file names and types, base64 contents, attributes, a region's template against the template allow-list, a process function as a `schema.function` identifier (checked again by the database). Install SQL never runs on import: a developer reads and runs it as the application's role, in one transaction. Pages get only a plug-in's name and attribute values (escaped JSON in `data-plugin-attrs`, or in the page's JSON), and only for a plug-in of the matching type; its JavaScript comes from same-origin static files. A plug-in's code runs with the application's rights: install only what you trust. |
 | Zips and parsing in SQL (0.31.0) | `meta.unpacked_file`/`meta.unpacked_entry` are closed to applications; the functions that read them are security definer and only return entries of content the caller already holds (looked up by its SHA-256). The server unpacks only zip/xlsx it receives, at most 2,000 files and 200 MB (declared sizes checked before inflating, actual sizes after), kept 24 hours. Zip entry names are relative paths (no `..`, no leading `/`). XML with a DTD or entity declarations is refused; a row selector is an element name or path, never XPath. |
 | Object storage (0.31.0) | Bucket requests go through the web client: allow-list, private-address checks at connect time, no secrets on cross-origin redirects; the credential (type `aws_sigv4`, secret encrypted) must be valid for the bucket URL and is used only for object storage. Keys are generated by the server (`<prefix><uuid>/<cleaned name>`); prefixes can't contain `..`. Downloads keep the checksum, page access and RLS checks (the row is read as the app's role before the object is fetched) and the sandboxing headers; users never see the bucket or credential. |
+| Lost update detection (sprint 50) | `meta.row_version()` runs with the caller's rights (the app's role: grants and RLS apply); the key is a typed parameter and column names are quoted identifiers. A posted row version that is not an MD5 hex string is ignored; changing it only weakens the user's own protection. |
+| URL region (sprint 50) | The origin must be written out in the source (https, http only for localhost, or a path on this server; no user:password, no substitution before the path); `&ITEM.` values are URL-encoded and the result is checked to have the same origin. The CSP gets `frame-src` for exactly the origins of the page's visible URL regions; the frame is sandboxed (`allow-top-navigation` is never offered) with `referrerpolicy=strict-origin-when-cross-origin`. |
+| Search region (sprint 50) | The search text is a query parameter (`ILIKE … ESCAPE`, wildcards escaped, at most 8 words of 200 characters); configuration column names are checked and quoted, table sources resolved with `regclass`; queries run as the app's role in savepoints. Only the application's own configurations count; results link with checksums; list sources show only entries the user may see. The region endpoint serves search regions only after page access and the region's checks. |
+| Developer toolbar (sprint 50) | Shown only when the cookie `pgkiln_devbar` (path `/a`, HttpOnly, SameSite=Lax; a random token whose SHA-256 is on the builder session, separate from the builder's token) belongs to a live builder session of a developer who may open the application (administrator or workspace member); otherwise its routes are 404. Its POSTs need the app session's CSRF token. The viewer never shows password items. Debug per session is set only by the toolbar (`__DEBUG_LEVEL` can't be posted or put in a URL), and `meta.debug_save` stores a session's log only for a known level of a session of that application. |
+| E-mail (sprint 50) | The mail tables and the internal functions are closed to application and runtime roles. `meta.send_mail` and friends act for `meta.app_id()` only when the caller's role (`current_setting('role')`, unchanged by SECURITY DEFINER) is that application's database role, so application code can't send in another application's name or after `RESET ROLE`. Addresses are plain addresses (no CR/LF, no display names), the subject one line, at most 50 recipients, bodies 1 MB, attachments 20 × 10 MB per mail and only for a mail queued in the same transaction; From must be one of the service's allowed senders; a daily limit per application. Template placeholders are filled in one pass and HTML-escaped unless `!RAW`. Service secrets are AES-256-GCM encrypted with `PGKILN_SECRET_KEY`, write-only, never exported, logged or put in error messages; only administrators manage services. nodemailer runs with file and URL access off. |
+| Checked imports (sprint 51) | The Advisor runs in the import's transaction inside a savepoint that is rolled back (its `SET LOCAL ROLE` and settings with it): SQL is only planned (`EXPLAIN` without ANALYZE), PL/pgSQL blocks compiled into a `pg_temp` function, nothing is executed. A refused import rolls back entirely, its backup included. |
+| Backups (sprint 51) | `meta.app_backup` and `meta.backup_app` are closed to the runtime and application roles; the builder serves a backup only under its own application's URL (developer session, CSRF for restore and delete, the application's lock and Run only respected for a restore). A backup is an export: it holds no secrets. |
+| Availability and Run only (sprint 51) | Checked on every page request, file download, REST module call and, through `meta.api_check`, every PostgREST request; "developers only" needs the developer toolbar's token of a live builder session (never an API token); restricted compares user names case-insensitively. Run only is enforced on the server for every builder POST of the application (423 for JSON), except its availability, backups, locks, comments and feedback answers. |
+| Feedback (sprint 51) | Needs the application's setting, the page's access rules and the session's CSRF token; at most 10 per session and hour, 4000 characters, escaped everywhere; users see only their own feedback; the runtime role may only insert and read `meta.feedback`; answers need a developer session and are matched to the application in the URL. |
+| Page behaviour (sprint 52) | Ajax callbacks need the CSRF token and page access, run only processes with point *ajax* of this page or application (SQL only) with their authorization, condition and build option, as the app's role; posted values set only items the user may change, `x01`–`x10` are binds `:X01`–`:X10` and can't replace the built-in binds; at most 1000 rows come back. The download action runs only the download process its own (visible) dynamic action names. The error handling function runs as the app's role in its own transaction with a 2 s limit; its failure keeps the original message. *Stay signed in* needs that application's signed-in session and its CSRF token. |
 | Session state protection | Item values in URLs, and the row keys of interactive grids, carry an HMAC checksum bound to app, page and user. Hidden, display, read-only and unauthorized items cannot be set by a form post. |
 | Request integrity | A CSRF token on every POST (pages, AJAX, login, logout). Buttons are re-validated on submit. |
 | Output | Auto-escaping HTML templates, strict CSP (no inline scripts, no inline styles: the page's one `<style>` carries a per-response nonce), `X-Frame-Options`, `nosniff`, `no-store`. Images may also come from the map tile server (`MAP_TILE_URL`, an http(s) origin only); map data travels as JSON in a `<script type="application/json">` element with `<` escaped. A map area in a report URL (`r<id>_bb`) is parsed into four range-checked numbers before it reaches SQL; anything else is ignored. The browser fetches tiles from that server directly, so it sees users' IP addresses, map areas and the site's origin (sent as `Referer`, which OpenStreetMap requires; never the page's path or query): self-host tiles where that matters. |

package/db/migrations/076_form_lost_update.sql

@@ -0,0 +1,50 @@
+-- =====================================================================
+-- 076: lost update detection for forms (APEX: "Prevent Lost Updates" and
+-- "Lock Row" of the Form - Automatic Row Processing (DML) process)
+--
+-- When a form region fetches its row, the page also gets a checksum of the
+-- row's form columns (meta.row_version). An update or delete through the
+-- form_dml process locks the row (FOR UPDATE), computes the checksum again
+-- and refuses the save when it differs: someone else changed the row since
+-- the page was shown.
+--
+-- The checksum is computed under fixed settings (UTC, ISO dates, all float
+-- digits), so it does not depend on the user's time zone or the session's
+-- formats. The function runs with the caller's rights: the application's
+-- role, its grants and its row level security policies.
+-- =====================================================================
+
+create function meta.row_version(p_table regclass, p_pk_column text, p_pk text, p_columns text[], p_lock boolean default false)
+returns text
+language plpgsql
+set timezone = 'UTC'
+set datestyle = 'ISO, YMD'
+set intervalstyle = 'iso_8601'
+set extra_float_digits = 3
+set search_path = pg_catalog
+as $$
+declare
+  v_cols text;
+  v_type text;
+  v_out  text;
+begin
+  -- the key's own type, so the lookup uses the primary key's index
+  select format_type(a.atttypid, a.atttypmod) into v_type
+    from pg_attribute a where a.attrelid = p_table and a.attname = p_pk_column and a.attnum > 0 and not a.attisdropped;
+  if v_type is null then
+    raise exception 'column % does not exist in %', p_pk_column, p_table;
+  end if;
+  -- each column as text in the row, in the given order
+  select string_agg(format('%I::text', c), ', ' order by o) into v_cols
+    from unnest(p_columns) with ordinality as x(c, o);
+  execute format('select md5(row(%s)::text) from %s where %I = $1::%s%s',
+                 coalesce(v_cols, format('%I::text', p_pk_column)), p_table, p_pk_column, v_type,
+                 case when p_lock then ' for update' else '' end)
+    into v_out using p_pk;
+  return v_out;
+end
+$$;
+grant execute on function meta.row_version(regclass, text, text, text[], boolean) to public;
+
+comment on function meta.row_version(regclass, text, text, text[], boolean) is
+  'A checksum of a row''s columns (lost update detection of forms, 076); NULL when the row is not found or not visible';

package/db/migrations/077_warn_unsaved.sql

@@ -0,0 +1,41 @@
+-- =====================================================================
+-- 077: warn on unsaved changes (APEX: the page attribute "Warn on Unsaved
+-- Changes" and the item attribute "Warn on Unsaved Changes: Ignore")
+--
+-- With meta.page.warn_unsaved the browser asks before the user leaves the
+-- page (a link, a redirect button, closing or reloading the tab) while an
+-- item of the page form has a changed value. Submitting the page doesn't
+-- ask. An item is left out with {"warn_unsaved": false} in its attributes.
+-- Off for existing pages; the create page wizards switch it on for forms.
+-- =====================================================================
+
+alter table meta.page add column warn_unsaved boolean not null default false;
+
+-- an export made before 077 has no warn_unsaved: NULL becomes false
+create function meta.page_warn_unsaved_default() returns trigger
+language plpgsql set search_path = pg_catalog as $$
+begin
+  new.warn_unsaved := coalesce(new.warn_unsaved, false);
+  return new;
+end $$;
+create trigger page_warn_unsaved before insert or update on meta.page
+  for each row execute function meta.page_warn_unsaved_default();
+
+comment on column meta.page.warn_unsaved is 'Ask before leaving the page while items have unsaved changes (077)';
+
+-- The create page wizards switch it on for the form pages they make (APEX's default
+-- for new pages): meta.wizard_form (047) is wrapped.
+alter function meta.wizard_form(meta.app, regclass, int, text, int, boolean, text[]) rename to wizard_form_047;
+create function meta.wizard_form(
+  p_app meta.app, p_table regclass, p_page int, p_label text, p_return int, p_modal boolean, p_columns text[] default null
+) returns int
+language plpgsql as $$
+declare
+  v_page int := meta.wizard_form_047(p_app, p_table, p_page, p_label, p_return, p_modal, p_columns);
+begin
+  update meta.page set warn_unsaved = true where id = v_page;
+  return v_page;
+end
+$$;
+-- like the original (047): for the builder only, not for applications
+revoke all on function meta.wizard_form(meta.app, regclass, int, text, int, boolean, text[]) from public;

package/db/migrations/078_url_region.sql

@@ -0,0 +1,12 @@
+-- =====================================================================
+-- 078: the URL region (APEX: URL region): another site's page in the
+-- region, in a sandboxed iframe. The source is the URL; its origin is
+-- literal (an item can't choose the host), and the page's
+-- Content-Security-Policy frames exactly the origins of its URL regions
+-- (src/runtime/url-region.ts).
+-- =====================================================================
+
+alter table meta.region drop constraint region_type_check;
+alter table meta.region add constraint region_type_check check (type in (
+  'report', 'form', 'chart', 'cards', 'static', 'grid', 'calendar', 'dynamic', 'facets', 'tasks', 'workflows', 'map', 'tree',
+  'template_component', 'smart_filters', 'display_selector', 'list', 'data_reporter', 'ai_assistant', 'plugin', 'url'));

package/db/migrations/079_developer_toolbar.sql

@@ -0,0 +1,53 @@
+-- =====================================================================
+-- 079: the developer toolbar (APEX: Runtime Developer Toolbar) and the
+-- session state viewer
+--
+-- A builder session hands the browser a second, random token in the cookie
+-- pgkiln_devbar (path /a, HttpOnly). Pages of an application show the
+-- developer toolbar when that token belongs to a live builder session of a
+-- developer who may open the application (src/runtime/devbar.ts). The token
+-- only shows the toolbar: it is not the builder session's token.
+-- =====================================================================
+
+alter table meta.session add column devbar_hash text;
+create unique index session_devbar_hash on meta.session (devbar_hash) where devbar_hash is not null;
+
+comment on column meta.session.devbar_hash is 'Builder sessions: sha256 of the developer toolbar token (079)';
+
+-- Debug messages of a session the toolbar put in debug (state __DEBUG_LEVEL, set only by
+-- POST /a/<alias>/<page>/devbar/debug for a developer) are stored even when the
+-- application is not in debug. Same as 051 otherwise.
+create or replace function meta.debug_save(p_app_id int, p_page_no int, p_username text, p_session_id uuid, p_method text, p_path text,
+                                p_status int, p_level int, p_started_at timestamptz, p_elapsed_ms numeric, p_entries jsonb)
+returns bigint
+language plpgsql security definer set search_path = meta, pg_catalog as $$
+declare
+  v_id bigint;
+  v_level int;
+begin
+  -- only for an application that is in debug now, or (079) a session the developer toolbar put in debug
+  select debug_level into v_level from meta.app where id = p_app_id;
+  if coalesce(v_level, 0) = 0 then
+    select (s.state->>'__DEBUG_LEVEL')::int into v_level
+      from meta.session s
+     where s.id = p_session_id and s.app_id = p_app_id and s.state->>'__DEBUG_LEVEL' in ('1', '2', '4', '6', '9');
+  end if;
+  if coalesce(v_level, 0) = 0 then
+    return null;
+  end if;
+  insert into meta.debug_view (app_id, page_no, username, session_id, method, path, status, level, started_at, elapsed_ms, entries)
+  values (p_app_id, p_page_no, left(p_username, 200), p_session_id, left(p_method, 10), left(p_path, 500), p_status,
+          least(greatest(coalesce(p_level, v_level), 1), 9), coalesce(p_started_at, now()), p_elapsed_ms,
+          least(coalesce(jsonb_array_length(p_entries), 0), 2000))
+  returning id into v_id;
+  insert into meta.debug_message (view_id, seq, elapsed_ms, duration_ms, level, component, message)
+  select v_id, e.ord::int, coalesce((e.v->>'ms')::numeric(10, 1), 0), (e.v->>'dur')::numeric(10, 1),
+         least(greatest(coalesce((e.v->>'level')::int, 4), 1), 9), left(e.v->>'component', 100), left(coalesce(e.v->>'text', ''), 4000)
+    from jsonb_array_elements(coalesce(p_entries, '[]')) with ordinality as e(v, ord)
+   where e.ord <= 2000;
+  if random() < 0.02 then
+    perform meta.debug_purge();
+  end if;
+  return v_id;
+end
+$$;

package/db/migrations/080_search.sql

@@ -0,0 +1,326 @@
+-- =====================================================================
+-- 080: search configurations and the search region (APEX 22.2: Search
+-- Configurations in Shared Components, the Search region)
+--
+-- A search configuration says what one source contributes to a search: a
+-- table or view, a SELECT, or a list (Shared Components → Lists); the
+-- columns searched; and which columns make each result's title, subtitle,
+-- description, badge and icon, and where a result links to. A region of
+-- type "search" searches the configurations it names with one search
+-- field (or a page item) and shows the results grouped per configuration
+-- (src/runtime/search.ts). The searches run as the application's role, the
+-- search text is a query parameter.
+--
+-- export_app / import_app are redefined from 069 to carry search_configs.
+-- =====================================================================
+
+create table meta.search_config (
+  id                 serial primary key,
+  app_id             int not null references meta.app (id) on delete cascade,
+  name               text not null check (name ~ '^[A-Z][A-Z0-9_]{0,59}$'),
+  label              text not null check (length(label) between 1 and 200),
+  seq                int not null default 10,
+  source_type        text not null default 'table' check (source_type in ('table', 'sql', 'list')),
+  -- a table or view (schema.name), a SELECT, or the name of a list
+  source             text not null,
+  -- the columns the search text is looked for in (empty: title, subtitle and description)
+  searchable_columns text[] not null default '{}',
+  title_column       text,
+  subtitle_column    text,
+  description_column text,
+  badge_column       text,
+  -- an icon name, or #COLUMN# for a column that holds icon names
+  icon               text,
+  -- where a result links to: {"page": 3, "items": {"P3_ID": "#ID#"}}
+  link               jsonb,
+  max_results        int not null default 10 check (max_results between 1 and 100),
+  build_option       text,
+  unique (app_id, name)
+);
+grant select on meta.search_config to pgkiln_runtime;
+
+-- an export made before 080 has none; one made with fewer columns gets the defaults
+create function meta.search_config_defaults() returns trigger
+language plpgsql set search_path = pg_catalog as $$
+begin
+  new.seq := coalesce(new.seq, 10);
+  new.source_type := coalesce(new.source_type, 'table');
+  new.searchable_columns := coalesce(new.searchable_columns, '{}');
+  new.max_results := coalesce(new.max_results, 10);
+  return new;
+end $$;
+create trigger search_config_defaults before insert or update on meta.search_config
+  for each row execute function meta.search_config_defaults();
+
+comment on table meta.search_config is 'Search configurations (080): what a search region searches, exported with the application';
+
+alter table meta.region drop constraint region_type_check;
+alter table meta.region add constraint region_type_check check (type in (
+  'report', 'form', 'chart', 'cards', 'static', 'grid', 'calendar', 'dynamic', 'facets', 'tasks', 'workflows', 'map', 'tree',
+  'template_component', 'smart_filters', 'display_selector', 'list', 'data_reporter', 'ai_assistant', 'plugin', 'url', 'search'));
+
+-- ---------------------------------------------------------------- export / import
+-- Same as 069, plus search_configs.
+create or replace function meta.export_app(p_alias text) returns jsonb
+language sql stable set search_path = meta, pg_catalog as $$
+  select jsonb_build_object(
+    'format', 'pgkiln/2',
+    'app', to_jsonb(a) - 'id' - 'created_at' - 'updated_at',
+    'authz_schemes', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.authz_scheme x where x.app_id = a.id), '[]'),
+    'app_items', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.app_item x where x.app_id = a.id), '[]'),
+    'app_processes', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.seq, x.id) from meta.app_process x where x.app_id = a.id), '[]'),
+    'lovs', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.lov x where x.app_id = a.id), '[]'),
+    'group_roles', coalesce((select jsonb_agg(to_jsonb(x) - 'app_id' order by x.group_name, x.role) from meta.app_group_role x where x.app_id = a.id), '[]'),
+    'text_messages', coalesce((select jsonb_agg(to_jsonb(x) - 'app_id' order by x.name, x.language) from meta.text_message x where x.app_id = a.id), '[]'),
+    'translations', coalesce((select jsonb_agg(to_jsonb(x) - 'app_id' order by x.language, x.source) from meta.translation x where x.app_id = a.id), '[]'),
+    'report_layouts', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' - 'logo' || jsonb_build_object('logo', encode(x.logo, 'base64')) order by x.name)
+                                  from meta.report_layout x where x.app_id = a.id), '[]'),
+    'automations', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' - 'next_run_at' - 'last_run_at' - 'last_status' - 'code' order by x.name)
+                               from meta.automation x where x.app_id = a.id), '[]'),
+    -- (044) the actions of the automations, by automation name
+    'automation_actions', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.automation_name collate "C", x.seq, x.name collate "C")
+                                      from meta.automation_action x where x.app_id = a.id), '[]'),
+    'document_templates', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.document_template x where x.app_id = a.id), '[]'),
+    'task_definitions', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.task_definition x where x.app_id = a.id), '[]'),
+    'workflow_definitions', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.workflow_definition x where x.app_id = a.id), '[]'),
+    'rest_modules', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.rest_module x where x.app_id = a.id), '[]'),
+    'template_components', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.static_id) from meta.template_component x where x.app_id = a.id), '[]'),
+    'build_options', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.build_option x where x.app_id = a.id), '[]'),
+    -- (030) secrets never leave the installation: they are entered again after an import
+    -- (050) nor do the password and refresh token of the OAuth2 flows, or the token state
+    'web_credentials', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' - 'secret_enc' - 'password_enc' - 'refresh_token_enc' - 'token_refreshed_at' order by x.name)
+                                   from meta.web_credential x where x.app_id = a.id), '[]'),
+    -- (050) the synchronisation's state (next and last run) belongs to the installation
+    'rest_sources', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' - 'sync_next_at' - 'sync_last_at' - 'sync_last_status' order by x.name)
+                                from meta.rest_source x where x.app_id = a.id), '[]'),
+    -- (041) data load definitions
+    'data_load_definitions', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.data_load_def x where x.app_id = a.id), '[]'),
+    -- (042) lists; entries keep their ids so parents can be remapped on import
+    'lists', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.list x where x.app_id = a.id), '[]'),
+    'list_entries', coalesce((select jsonb_agg(to_jsonb(x) - 'app_id' order by x.list_name, x.parent_id nulls first, x.seq, x.id) from meta.list_entry x where x.app_id = a.id), '[]'),
+    -- (042) supporting objects: install, upgrade and deinstall scripts (never run on import)
+    'supporting_scripts', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.supporting_script x where x.app_id = a.id), '[]'),
+    -- (069) plug-ins (their files are static files, their templates template components)
+    'plugins', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.plugin x where x.app_id = a.id), '[]'),
+    -- (080) search configurations
+    'search_configs', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' order by x.name) from meta.search_config x where x.app_id = a.id), '[]'),
+    -- (068) static application files (JavaScript, CSS, images, fonts) as base64
+    'static_files', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'app_id' - 'content' - 'updated_at' || jsonb_build_object('content', translate(encode(x.content, 'base64'), E'\n', '')) order by x.name)
+                                from meta.static_file x where x.app_id = a.id), '[]'),
+    -- nav entries and regions keep their ids, so parents and references can be remapped on import
+    'nav', coalesce((select jsonb_agg(to_jsonb(x) - 'app_id' order by x.parent_id nulls first, x.seq, x.id) from meta.nav_entry x where x.app_id = a.id), '[]'),
+    'pages', coalesce((
+      select jsonb_agg(to_jsonb(p) - 'id' - 'app_id' || jsonb_build_object(
+        'regions', coalesce((select jsonb_agg(to_jsonb(r) - 'page_id' order by r.seq, r.id) from meta.region r where r.page_id = p.id), '[]'),
+        'items', coalesce((select jsonb_agg(to_jsonb(i) - 'id' - 'page_id' order by i.seq, i.id) from meta.item i where i.page_id = p.id), '[]'),
+        'buttons', coalesce((select jsonb_agg(to_jsonb(b) - 'id' - 'page_id' order by b.seq, b.id) from meta.button b where b.page_id = p.id), '[]'),
+        'dynamic_actions', coalesce((select jsonb_agg(to_jsonb(d) - 'id' - 'page_id' order by d.seq, d.id) from meta.dynamic_action d where d.page_id = p.id), '[]'),
+        'validations', coalesce((select jsonb_agg(to_jsonb(v) - 'id' - 'page_id' order by v.seq, v.id) from meta.validation v where v.page_id = p.id), '[]'),
+        'processes', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'page_id' order by x.seq, x.id) from meta.process x where x.page_id = p.id), '[]'),
+        'computations', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'page_id' order by x.seq, x.id) from meta.computation x where x.page_id = p.id), '[]'),
+        'branches', coalesce((select jsonb_agg(to_jsonb(x) - 'id' - 'page_id' order by x.seq, x.id) from meta.branch x where x.page_id = p.id), '[]')
+      ) order by p.page_no)
+      from meta.page p where p.app_id = a.id), '[]'))
+  from meta.app a
+  where a.alias = p_alias
+$$;
+
+create or replace function meta.import_app(p_doc jsonb, p_alias text default null) returns int
+language plpgsql set search_path = meta, pg_catalog as $$
+declare
+  v_app_id  int;
+  v_page_id int;
+  v_page    jsonb;
+  v_e       jsonb;
+  v_rmap    jsonb;
+  v_nmap    jsonb := '{}';
+  v_lmap    jsonb := '{}';
+  v_new_id  int;
+begin
+  if p_doc->>'format' is distinct from 'pgkiln/2' then
+    raise exception 'unsupported export format %', coalesce(p_doc->>'format', '(none)');
+  end if;
+
+  insert into meta.app
+  select (jsonb_populate_record(null::meta.app, p_doc->'app' || jsonb_build_object(
+            'id', nextval('meta.app_id_seq'),
+            'alias', coalesce(p_alias, p_doc->'app'->>'alias'),
+            'created_at', now(), 'updated_at', now()))).*
+  returning id into v_app_id;
+
+  insert into meta.authz_scheme
+  select (jsonb_populate_record(null::meta.authz_scheme, e || jsonb_build_object('id', nextval('meta.authz_scheme_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'authz_schemes', '[]')) e;
+  insert into meta.app_item
+  select (jsonb_populate_record(null::meta.app_item, e || jsonb_build_object('id', nextval('meta.app_item_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'app_items', '[]')) e;
+  insert into meta.app_process
+  select (jsonb_populate_record(null::meta.app_process, e || jsonb_build_object('id', nextval('meta.app_process_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'app_processes', '[]')) e;
+  insert into meta.lov
+  select (jsonb_populate_record(null::meta.lov, e || jsonb_build_object('id', nextval('meta.lov_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'lovs', '[]')) e;
+  insert into meta.app_group_role
+  select (jsonb_populate_record(null::meta.app_group_role, e || jsonb_build_object('app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'group_roles', '[]')) e;
+  insert into meta.text_message
+  select (jsonb_populate_record(null::meta.text_message, e || jsonb_build_object('app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'text_messages', '[]')) e;
+  insert into meta.translation
+  select (jsonb_populate_record(null::meta.translation, e || jsonb_build_object('app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'translations', '[]')) e;
+  insert into meta.report_layout
+  select (jsonb_populate_record(null::meta.report_layout, (e - 'logo') || jsonb_build_object(
+            'id', nextval('meta.report_layout_id_seq'), 'app_id', v_app_id,
+            'logo', null))).*
+    from jsonb_array_elements(coalesce(p_doc->'report_layouts', '[]')) e;
+  update meta.report_layout l
+     set logo = decode(e->>'logo', 'base64')
+    from jsonb_array_elements(coalesce(p_doc->'report_layouts', '[]')) e
+   where l.app_id = v_app_id and l.name = e->>'name' and e->>'logo' is not null;
+  -- imported automations start switched off: the copy must not run the original's jobs unasked.
+  -- (044) a file with actions: the automations' code is ignored; an older file: code becomes the single action
+  insert into meta.automation
+  select (jsonb_populate_record(null::meta.automation, '{"error_handling": "stop"}'::jsonb
+            || jsonb_strip_nulls(case when p_doc ? 'automation_actions' then e - 'code' else e end)
+            || jsonb_build_object('id', nextval('meta.automation_id_seq'), 'app_id', v_app_id, 'enabled', false))).*
+    from jsonb_array_elements(coalesce(p_doc->'automations', '[]')) e;
+  insert into meta.automation_action
+  select (jsonb_populate_record(null::meta.automation_action, '{"seq": 10}'::jsonb || jsonb_strip_nulls(e)
+            || jsonb_build_object('id', nextval('meta.automation_action_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'automation_actions', '[]')) e;
+  insert into meta.document_template
+  select (jsonb_populate_record(null::meta.document_template, e || jsonb_build_object('id', nextval('meta.document_template_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'document_templates', '[]')) e;
+  insert into meta.task_definition
+  select (jsonb_populate_record(null::meta.task_definition, e || jsonb_build_object('id', nextval('meta.task_definition_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'task_definitions', '[]')) e;
+  insert into meta.workflow_definition
+  select (jsonb_populate_record(null::meta.workflow_definition, e || jsonb_build_object('id', nextval('meta.workflow_definition_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'workflow_definitions', '[]')) e;
+  insert into meta.rest_module
+  select (jsonb_populate_record(null::meta.rest_module, e || jsonb_build_object('id', nextval('meta.rest_module_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'rest_modules', '[]')) e;
+  -- (028) template components: regions and report columns refer to them by static id
+  insert into meta.template_component
+  select (jsonb_populate_record(null::meta.template_component, e || jsonb_build_object('id', nextval('meta.template_component_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'template_components', '[]')) e;
+  -- (029) build options (components name them by name)
+  insert into meta.build_option
+  select (jsonb_populate_record(null::meta.build_option, e || jsonb_build_object('id', nextval('meta.build_option_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'build_options', '[]')) e;
+  -- (030) web credentials arrive without a secret, whatever the document holds
+  -- (050) nor a password or refresh token
+  insert into meta.web_credential
+  select (jsonb_populate_record(null::meta.web_credential, '{"type": "basic", "valid_for": [], "grant_type": "client_credentials"}'::jsonb
+            || jsonb_strip_nulls(e - 'secret_enc' - 'password_enc' - 'refresh_token_enc' - 'token_refreshed_at')
+            || jsonb_build_object('id', nextval('meta.web_credential_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'web_credentials', '[]')) e;
+  -- (050) write-back operations and synchronisation; an imported synchronisation starts switched off
+  insert into meta.rest_source
+  select (jsonb_populate_record(null::meta.rest_source, '{"method": "GET", "headers": {}, "params": [], "columns": [], "cache_seconds": 0, "timeout_s": 10, "max_rows": 1000,
+                                                          "operations": {}, "key_columns": [], "sync_mode": "merge", "sync_delete": false, "sync_time_zone": "UTC"}'::jsonb
+            || jsonb_strip_nulls(e - 'sync_next_at' - 'sync_last_at' - 'sync_last_status')
+            || jsonb_build_object('id', nextval('meta.rest_source_id_seq'), 'app_id', v_app_id, 'sync_enabled', false))).*
+    from jsonb_array_elements(coalesce(p_doc->'rest_sources', '[]')) e;
+
+  -- (041) data load definitions (the data_load process names them)
+  insert into meta.data_load_def
+  select (jsonb_populate_record(null::meta.data_load_def, '{"format": "auto", "headers": true, "mode": "append", "skip_errors": false, "columns": []}'::jsonb
+            || jsonb_strip_nulls(e) || jsonb_build_object('id', nextval('meta.data_load_def_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'data_load_definitions', '[]')) e;
+
+  -- (042) lists and their entries, supporting scripts. Entries are inserted
+  -- without a parent first and the parents set afterwards, so the order of
+  -- the entries in the document doesn't matter (a moved entry may have a
+  -- higher id than its children).
+  insert into meta.list
+  select (jsonb_populate_record(null::meta.list, '{"type": "static"}'::jsonb || jsonb_strip_nulls(e) || jsonb_build_object('id', nextval('meta.list_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'lists', '[]')) e;
+  for v_e in select * from jsonb_array_elements(coalesce(p_doc->'list_entries', '[]')) loop
+    insert into meta.list_entry
+    select (jsonb_populate_record(null::meta.list_entry, '{"target_items": {}}'::jsonb || jsonb_strip_nulls(v_e) || jsonb_build_object(
+              'id', nextval('meta.list_entry_id_seq'), 'app_id', v_app_id, 'parent_id', null))).*
+    returning id into v_new_id;
+    if v_e->>'id' is not null then
+      v_lmap := v_lmap || jsonb_build_object(v_e->>'id', v_new_id);
+    end if;
+  end loop;
+  update meta.list_entry x
+     set parent_id = (v_lmap->>(e->>'parent_id'))::int
+    from jsonb_array_elements(coalesce(p_doc->'list_entries', '[]')) e
+   where e->>'parent_id' is not null and v_lmap ? (e->>'parent_id') and v_lmap ? (e->>'id')
+     and x.id = (v_lmap->>(e->>'id'))::int;
+  insert into meta.supporting_script
+  select (jsonb_populate_record(null::meta.supporting_script, '{"kind": "install", "seq": 10, "script": ""}'::jsonb || jsonb_strip_nulls(e)
+            || jsonb_build_object('id', nextval('meta.supporting_script_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'supporting_scripts', '[]')) e;
+
+  -- (069) plug-ins
+  insert into meta.plugin
+  select (jsonb_populate_record(null::meta.plugin, e || jsonb_build_object('id', nextval('meta.plugin_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'plugins', '[]')) e;
+
+  -- (080) search configurations (search regions name them)
+  insert into meta.search_config
+  select (jsonb_populate_record(null::meta.search_config, '{"seq": 10, "source_type": "table", "searchable_columns": [], "max_results": 10}'::jsonb
+            || jsonb_strip_nulls(e) || jsonb_build_object('id', nextval('meta.search_config_id_seq'), 'app_id', v_app_id))).*
+    from jsonb_array_elements(coalesce(p_doc->'search_configs', '[]')) e;
+
+  -- (068) static application files
+  insert into meta.static_file (app_id, name, mime, content, updated_at)
+  select v_app_id, e->>'name', e->>'mime', decode(coalesce(e->>'content', ''), 'base64'), now()
+    from jsonb_array_elements(coalesce(p_doc->'static_files', '[]')) e;
+
+  for v_e in select * from jsonb_array_elements(coalesce(p_doc->'nav', '[]')) loop
+    insert into meta.nav_entry
+    select (jsonb_populate_record(null::meta.nav_entry, v_e || jsonb_build_object(
+              'id', nextval('meta.nav_entry_id_seq'), 'app_id', v_app_id,
+              'parent_id', v_nmap->>(v_e->>'parent_id')))).*
+    returning id into v_new_id;
+    v_nmap := v_nmap || jsonb_build_object(v_e->>'id', v_new_id);
+  end loop;
+
+  for v_page in select * from jsonb_array_elements(coalesce(p_doc->'pages', '[]')) loop
+    insert into meta.page
+    select (jsonb_populate_record(null::meta.page, v_page || jsonb_build_object('id', nextval('meta.page_id_seq'), 'app_id', v_app_id))).*
+    returning id into v_page_id;
+
+    v_rmap := '{}';
+    for v_e in select * from jsonb_array_elements(coalesce(v_page->'regions', '[]')) loop
+      insert into meta.region
+      select (jsonb_populate_record(null::meta.region, v_e || jsonb_build_object('id', nextval('meta.region_id_seq'), 'page_id', v_page_id))).*
+      returning id into v_new_id;
+      v_rmap := v_rmap || jsonb_build_object(v_e->>'id', v_new_id);
+    end loop;
+    -- facet and map regions point at their report region by id
+    update meta.region r
+       set config = jsonb_set(r.config, '{report}', to_jsonb((v_rmap->>(r.config->>'report'))::int))
+     where r.page_id = v_page_id and r.type in ('facets', 'smart_filters', 'map') and v_rmap ? (r.config->>'report');
+
+    insert into meta.item
+    select (jsonb_populate_record(null::meta.item, e || jsonb_build_object('id', nextval('meta.item_id_seq'), 'page_id', v_page_id, 'region_id', v_rmap->>(e->>'region_id')))).*
+      from jsonb_array_elements(coalesce(v_page->'items', '[]')) e;
+    insert into meta.button
+    select (jsonb_populate_record(null::meta.button, e || jsonb_build_object('id', nextval('meta.button_id_seq'), 'page_id', v_page_id, 'region_id', v_rmap->>(e->>'region_id')))).*
+      from jsonb_array_elements(coalesce(v_page->'buttons', '[]')) e;
+    insert into meta.dynamic_action
+    select (jsonb_populate_record(null::meta.dynamic_action, e || jsonb_build_object('id', nextval('meta.dynamic_action_id_seq'), 'page_id', v_page_id, 'affected_region_id', v_rmap->>(e->>'affected_region_id')))).*
+      from jsonb_array_elements(coalesce(v_page->'dynamic_actions', '[]')) e;
+    insert into meta.validation
+    select (jsonb_populate_record(null::meta.validation, e || jsonb_build_object('id', nextval('meta.validation_id_seq'), 'page_id', v_page_id))).*
+      from jsonb_array_elements(coalesce(v_page->'validations', '[]')) e;
+    insert into meta.process
+    select (jsonb_populate_record(null::meta.process, e || jsonb_build_object('id', nextval('meta.process_id_seq'), 'page_id', v_page_id, 'region_id', v_rmap->>(e->>'region_id')))).*
+      from jsonb_array_elements(coalesce(v_page->'processes', '[]')) e;
+    insert into meta.computation
+    select (jsonb_populate_record(null::meta.computation, e || jsonb_build_object('id', nextval('meta.computation_id_seq'), 'page_id', v_page_id))).*
+      from jsonb_array_elements(coalesce(v_page->'computations', '[]')) e;
+    insert into meta.branch
+    select (jsonb_populate_record(null::meta.branch, e || jsonb_build_object('id', nextval('meta.branch_id_seq'), 'page_id', v_page_id))).*
+      from jsonb_array_elements(coalesce(v_page->'branches', '[]')) e;
+  end loop;
+
+  return v_app_id;
+end
+$$;