pgkiln 0.47.0 → 0.49.0

package/docs/guide/03-builder.md

@@ -102,7 +102,41 @@ Links), added on the workspace's page by administrators and the workspace's admi
 `https://` URL, an optional description and an order. While the workspace is current an **Extensions** icon in the
 rail opens them, each in a new tab (`rel="noopener noreferrer"`).
 
-### Export and import
+#### Scheduling with pg_cron
+
+(sprint 101; Database Actions: Scheduling) **SQL Workshop → Scheduling** (administrators only) manages the jobs of
+[pg_cron](https://github.com/citusdata/pg_cron), the PostgreSQL extension that runs SQL on a schedule inside the
+database server. pgkiln needs pg_cron 1.4 or later, created in pgkiln's database; the page says what is missing when
+it isn't (the package, `shared_preload_libraries = 'pg_cron'`, `cron.database_name`, `create extension pg_cron`).
+Without pg_cron it lists **pgkiln's own scheduled work** read-only: every application's automations (Shared Components
+→ Automations, [chapter 6](06-processing.md)) with their schedule, next and last run, and the latest background process
+jobs. With pg_cron that list is behind a toggle under the jobs.
+
+- **Jobs** (`cron.job`): name, the command, the schedule with its meaning in words (*At 02:00, every day*), the **next
+  run**, the last run's status and time (and how many of the last ten failed), database and the role it runs as;
+  a search field. Times are in pg_cron's time zone (`cron.timezone`, GMT unless set). pg_cron shows each role only its
+  own jobs: when pgkiln's owner connection is neither a superuser nor `BYPASSRLS`, the page says so.
+- **Create job** / **Edit**: name (unique per role), database, *Run as* (a role that can sign in; only a superuser
+  schedules for another role), *Active*, the command (SQL, run outside a transaction block as pg_cron runs it, so
+  `vacuum` works) and the **schedule**: five cron fields, `n seconds` (1 to 59) or `@hourly` … `@yearly`, `@reboot`.
+  A **schedule builder** (one field each for minute, hour, day of month, month and day of week, `$` for the last day
+  of the month) and **common schedules** fill it in; the **preview** shows it in words with the **next five runs**,
+  as you type (without JavaScript: the *Use* and *Preview* buttons). pgkiln explains schedules as pg_cron reads them:
+  lists, ranges, steps, month and day names, 7 for Sunday, and a day matching either the day of month or the day of
+  week when both are restricted.
+- **Review SQL** shows the statements (`cron.schedule_in_database`, `cron.alter_job` with only what changed, an
+  `update cron.job` for a new name; the command dollar-quoted); **Apply** runs them in one transaction and only if they
+  are still what was reviewed. **Disable / Enable** (`cron.alter_job(…, active := …)`) applies at once and shows the
+  statement; **Unschedule…** (`cron.unschedule`) after a review. Every change goes to the activity log (event `cron_job`).
+- **Run now**: pg_cron has no run-now, so pgkiln runs the job's stored command once itself: a new connection to the
+  job's database with the owner connection's login, `SET ROLE` to the job's role (the owner must be allowed to), no
+  transaction block around it, a 10-minute statement timeout. The page shows the result, the duration and notices.
+  The run is not in pg_cron's history; the activity log has it (event `cron_run_now`).
+- **Run history** (`cron.job_run_details`, newest first, 200 at most): status, start, duration and message, filtered
+  by job, status or a text in the message or command. **Purge…** deletes runs older than 7, 30 or 90 days, or all
+  (of every job or one), after showing the `delete` and how many runs it removes: pg_cron keeps every run until then.
+
+## Export and import
 
 Workspace utilities → **Export and import** (`/builder/workspace-export`; APEX: Workspace Utilities → Export), for
 administrators and the workspace's administrators: the current workspace's components as one JSON file (format
@@ -183,7 +217,7 @@ Every builder page has the same frame, modelled on an IDE:
   pages of an application); at the bottom the **Administration** menu (Workspace utilities; for administrators of the
   current workspace *Manage service*: [workspace message and environment banner](#workspace-message-and-environment-banner),
   [workspace utilization](#workspace-utilization) and the workspace's page; Manage users and groups, Developers,
-  Monitor activity, Change my password), Help, and your account menu (the circle with your initials) with the
+  Monitor activity, [Database users](#database-users) for administrators, Change my password), Help, and your account menu (the circle with your initials) with the
   **builder theme** (*Dark*, the default; *Light*; or *System*, which follows the operating system),
   Change password and Sign out. The theme is remembered on this device, also after signing out;
   it only changes the builder, not your applications.
@@ -199,6 +233,13 @@ Every builder page has the same frame, modelled on an IDE:
 
 On phones the rail gets narrower and the breadcrumb shows only the current page.
 
+The builder's look comes from design tokens at the top of `public/builder.css` (sprint 105): CSS custom properties
+for colours (a warm neutral palette with the kiln's ember for primary buttons, links, focus rings and selections),
+surfaces and their shadows, corner radii, the type scale and transitions, in a light and a dark set. The rail, the
+toolbar and the status bar stay dark in both themes. Transitions are switched off when the operating system asks
+for reduced motion. The builder uses the system's fonts (Inter when it is installed); it loads nothing from
+other sites.
+
 (sprint 86) An address under `/builder` that leads nowhere (a deleted application or page, a typo, an
 application of a workspace you are not a member of) shows a **Page not found** page in the builder's
 theme with a link to the App Builder home; a server error shows **Something went wrong** (the error
@@ -1254,7 +1295,18 @@ session's **schema** (APEX: the default schema): it goes first in the search pat
 you sign out. Tasks link to Quick SQL, blueprints, table descriptions, sample data and Generate DDL.
 
 (sprint 75) The tools are tabs: Home, SQL Worksheet, Object Browser, SQL Scripts and Utilities; **More** opens the
-others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compare, AI), and the open one is a tab too.
+others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compare, Data Modeler, AI), and the open one is a tab too.
+
+(sprint 96; Database Actions: the Launchpad) The home page starts with **Pinned & recently visited**: the pages you
+pinned (a tool, a database object from the Object Browser, a saved worksheet from its file menu; at most 24, per
+workspace) and the SQL Workshop pages you opened last in this browser. Below, the tools are in three tabs,
+**Development**, **Administration** and **Monitoring**, each with a line about what it does and a pin; the **More**
+menu uses the same headings.
+
+**Search** (**Ctrl/Cmd + K** on any builder page, or the field in the header) finds tools, applications, pages
+(by name or number) and database objects (tables, views, materialized views, functions and procedures) by name;
+the arrow keys and Enter open one. Applications and pages come from your workspace only; objects from the schemas
+the SQL Workshop's connection may use. Without JavaScript the header's field opens a page of results.
 
 - **SQL Worksheet** (`/builder/sql/worksheet`, sprint 91; Database Actions: SQL; was SQL Commands, whose address
   `/builder/sql/commands` opens the same page): one screen with a **navigator** on the left, the **editor** with a
@@ -1271,6 +1323,8 @@ others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compar
     function and the trigger) and **Add policy** (row level security on, a policy to fill in); **Drop…** writes the
     DROP statement with a comment that lists what depends on the object (views, foreign keys…). Nothing runs: you
     review the statement and run it.
+    (sprint 100) Tables and views also have **REST › Enable / Disable**: it opens the object's page in **RESTful
+    Services** when it is published, or the Enable form for it (below).
   - (sprint 95) **Table Properties** (a table's **Edit…**, **Use as template**, the Object Browser's *Edit table* and
     its *Table (form)* tile): a form with **Columns** (name, type, primary key, not null, default, identity,
     comment; move them up or down for a new table, remove them), **Unique keys**, **Foreign keys** (the table it refers
@@ -1299,6 +1353,12 @@ others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compar
     width, and the step's conditions (filter, index and join conditions, sort keys). Autotrace adds the actual time,
     rows, loops and buffers, the time spent in the step itself, marks the **slowest step**, and flags estimates that
     are 10 times too high or too low (run `ANALYZE` on the table). **Text plan** shows it like psql.
+  - (sprint 97) **Chart** (**Ctrl + F11**, or *Chart* above the Query Result) draws the last query with pgkiln's
+    own charts: the first text or date column gives the labels, the number columns the values; change the type (bar,
+    column, stacked bar, line, area, donut, pie, scatter), the label column or the values and it draws again. Name it
+    and **Save chart**: SQL Workshop → **Charts** lists the saved charts (shared by the workspace) and draws the ones
+    on its dashboard; a chart's page changes its settings, opens its query in the worksheet or deletes it. A saved
+    chart's SELECT runs read-only with a 10 second timeout and at most 500 rows.
   - **Notices** lists the `RAISE NOTICE`, `INFO` and `WARNING` messages of the last run with their time.
     **SQL History** (sprint 92) lists your last 200 statements in this workspace with how they ended, their rows and
     time; running the same statement again counts on its row (*3×*). Search it, show only the ones that failed or
@@ -1383,6 +1443,27 @@ others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compar
   The state is in the URL (tables in the order chosen, joins, functions and the tables' places), so
   a query can be bookmarked; everything except dragging works without JavaScript. Only names from
   the catalog and the fixed functions are used, and condition values are string literals.
+- **Data Modeler** (SQL Workshop → More → Data Modeler, `/builder/sql/modeler`): named relational
+  **diagrams** of a schema's tables and views, kept per workspace and shared by its developers. Create
+  one with a name, a schema and, optionally, the objects (`orders, customers`, `ord*`, or `*` for all);
+  **Add objects to diagram** searches the schema (part of a name, a pattern with `*`, or `*`) and adds
+  the ticked ones. Each object is a box with its columns, their types, `P` (primary key), `F` (foreign
+  key), `U` (unique) and `*` for not null; a foreign key between two shown tables is a line from the
+  referencing table (a crow's foot: many) to the referenced one (a bar: one), dashed with a circle
+  when the foreign key may be empty. A box's name opens **Table Properties** (a view: the Object
+  Browser). The toolbar: **Save** (Ctrl+S), **Undo** / **Redo** (Ctrl+Z, Ctrl+Y), **Zoom out** / **Zoom
+  in** (or Ctrl+wheel), **Fit**, **Auto layout** (referenced tables to the left of the tables that
+  reference them; undoable), a **Mini-map** (click or drag in it to move around), **SVG** and **PNG**
+  exports of the diagram as shown, **Print**, **Generate DDL** (the CREATE statements of the shown
+  objects with their keys, indexes, triggers, row level security, policies, comments and grants: to
+  download or open in the SQL Worksheet) and **Data dictionary** (each object's columns with type,
+  nullability, default, keys and comments, its constraints and what references it; to print, save as
+  PDF from the print dialog, or download as an HTML file). Drag a box to move it, or focus it and use
+  the arrow keys (Shift for bigger steps). The diagram keeps only the objects' names and positions
+  (`meta.sql_diagram`): columns and keys are read from the catalog each time, so it never goes stale;
+  an object dropped from the schema is reported and left out on the next save. Without JavaScript the
+  diagram shows at its saved positions, and adding, removing, auto layout, Export SVG, DDL and the
+  dictionary all work. Diagrams are installation data, not exported with applications.
 - **Load Data**: load a CSV, TSV, Excel, JSON or XML file into a new table (with inferred column
   types) or an existing one (append, merge by primary key, or replace), with a per-row error
   report, or with a saved **data load definition**; a mapping can be saved as one
@@ -1414,8 +1495,37 @@ others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compar
   replaces views and functions, drops and creates constraints, indexes, triggers, policies and
   grants that differ, and lists for review what it doesn't change: drops of what only the target
   has (commented out), columns whose identity or generation differs, and types, domains and
-  sequences that differ. The same from the command line (`pgkiln ddl`, `pgkiln compare`,
+  sequences that differ. (sprint 106) *Apply and record* runs the script on the target in one transaction and records
+  it in the Change Log (below). The same from the command line (`pgkiln ddl`, `pgkiln compare`,
   [chapter 18](18-cli.md)) and for AI agents (`generate_ddl`, `compare_schemas`).
+- **RESTful Services** (`/builder/sql/rest`, sprint 100; Database Actions: REST › AutoREST): publish a table or
+  view as REST endpoints. Choose the object (of a schema the SQL Workshop may use), the **application** it is
+  published through, a URL name, the **methods** (GET: rows and one row; POST; PUT; DELETE; single rows need a primary
+  key), **who may call it** (a bearer token of the application, or explicitly *public*), optional application
+  **roles** and the page size. The application decides the rest: the statements run as its database role (never the
+  object's owner, a superuser or a role that bypasses row level security), with its tokens and OAuth clients and its
+  accounts' roles. The object's page checks the role's grants per method, row level security and (for a view)
+  `security_invoker`, lists the endpoints with their URLs, and shows cURL and `fetch` examples and the OpenAPI link.
+  Details in [chapter 13](13-rest-api.md#autorest-tables-and-views-from-the-sql-workshop).
+  (sprint 104; Database Actions: REST › Modules) The **Modules** tab edits the REST modules of the workspace's
+  applications as a tree: a module (its base path and its privilege: roles every handler needs), its templates (URIs
+  with `:parameters`) and their handlers (method, source type: collection query, single row, SQL, PL/pgSQL or media,
+  who may call it, roles, documented parameters). Each handler has a **Try it** panel: the request goes to the real
+  endpoint without a token or as an account of the application, and the response is shown in the page. Details in
+  [chapter 13](13-rest-api.md#modules-templates-and-handlers-in-the-sql-workshop).
+- **Change Log** (`/builder/sql/changes`, sprint 106; Database Actions: Liquibase): DDL that succeeds in the **SQL
+  Worksheet**, **Table Properties** (*Apply*) and **Schema Comparison** (*Apply and record*: the change script run on
+  the target in one transaction) is recorded as a *change set*: the schema it changes, its SQL, who ran it, when,
+  where from and a number per schema that is never given again (deleting a change set keeps the others' numbers).
+  Statements without a schema-qualified name count for the worksheet's schema. Only schema DDL is kept (CREATE,
+  ALTER, DROP, COMMENT, GRANT and REVOKE on objects): no queries or data changes, and nothing about roles, databases
+  or servers, so passwords never end up in the log. The page lists a schema's change sets with their SQL; rename one
+  (its file name) or delete it from the log (the database is not changed). It marks which change sets were made
+  before the latest **Export** of an application of the workspace (*in it*) and which after (*still to ship*).
+  *Download migration* writes all of them, those not yet written to a file, or those after the latest export, as
+  **Flyway** versioned migrations (`V0001__add_bonus.sql`; several in a zip), a **Liquibase** formatted SQL changelog
+  (`--liquibase formatted sql`, `--changeset author:0001-add_bonus`) or one plain SQL script; each change set also
+  has its own Flyway file. Change sets belong to the workspace and are not part of an application export.
 - **Utilities** (`/builder/sql/utilities`; APEX: SQL Workshop → Utilities): tiles for Load and Unload Data, Sample
   Data, the Query Builder, DDL and Compare, and three pages of their own. Each runs in a read-only transaction with a
   15-second statement timeout, on the SQL Workshop's connection (an isolated workspace's developers see only the
@@ -1426,12 +1536,162 @@ others (Quick SQL, Query Builder, Load and Unload Data, Sample Data, DDL, Compar
     to the Object Browser. pgkiln's own `meta` schema and the system's are left out.
   - **About Database**: the PostgreSQL version, database size, uptime, a few settings (encoding, time zone,
     connections, memory, row security), the installed extensions and the schemas with their size.
-  - **Database Monitor** (administrators only): the client sessions of this database from `pg_stat_activity`, active
-    first, with their state, wait event, running time, the sessions that block them and the query (as far as the
-    owner may read other roles' statistics). Read only: it cancels nothing.
+  - **SQL Monitor** (administrators only): see [SQL Monitor](#sql-monitor) below (it replaced the Database Monitor;
+    the old address `/builder/sql/utilities/monitor` redirects).
+  - **Database Users** (administrators only; Administration menu → *Database users*, `/builder/db-users`; sprint 99,
+    Database Actions: Database Users): see [Database users](#database-users).
+  - **Schema Export / Import** (administrators only; `/builder/sql/data-pump`; sprint 103, Database Actions: Data
+    Pump): see [Schema export and import](#schema-export-and-import).
+  - **Scheduling** (administrators only; `/builder/sql/scheduling`, on the Launchpad under *Administration*, the
+    Utilities page and the search; sprint 101, Database Actions: Scheduling): [pg_cron](#scheduling-with-pg_cron) jobs.
 
 Because the SQL Workshop runs as the owner, restrict who gets a developer account.
 
+### SQL Monitor
+
+(sprint 102; Database Actions: Real Time SQL Monitor and Performance Hub) **SQL Workshop → SQL Monitor**
+(`/builder/sql/monitor`, administrators only; on the Launchpad under *Monitoring*, in the Utilities and in the search,
+Ctrl+K) shows what the database is doing, from PostgreSQL's statistics views:
+
+- **Sessions**: tiles with the sessions (of `max_connections`), the active ones, those idle in a transaction, the
+  blocked ones and the longest running statement; then the client sessions of this database (`pg_stat_activity`;
+  idle sessions on request, *Show idle sessions*) with user, application and client address, state (idle in a
+  transaction with its time), wait event, running time, the sessions that **block** it (the row is highlighted) and
+  the query. A query with a `query_id` links to its statement page. **Cancel** (`pg_cancel_backend`: stop the running
+  statement) and **Terminate** (`pg_terminate_backend`: roll back and disconnect) ask for a confirmation first; only
+  client sessions of this database can be signalled, and the owner role needs the right to (a superuser, the
+  session's own role, or `pg_signal_backend` for non-superusers' sessions).
+- **Wait events**: what the active sessions of the server wait for now, counted per type and event (*CPU: running*
+  when they don't wait).
+- **In progress** (only when something runs): VACUUM, ANALYZE, CREATE INDEX, CLUSTER and COPY from the
+  `pg_stat_progress_*` views, with the phase and a progress bar.
+- The tiles and tables **refresh in place** every 5, 15 or 60 seconds, with a **Pause** button; the choice is
+  remembered in the browser. Without JavaScript, *Refresh* reloads the page.
+- **Top SQL**: the statements of this database from `pg_stat_statements`, every role: calls, total and mean time
+  (with bars), rows, shared blocks hit and read; sort by any column (click the heading again to reverse), search the
+  text, **Reset statistics** (the whole server, after a confirmation). Literals show as `$1`, `$2`….
+- **A statement**: its text, its totals (calls, total and mean time, rows, blocks, temporary blocks) and a row per
+  role with standard deviation, minimum and maximum; **Explain** shows its plan as a tree like the worksheet's. It is
+  `EXPLAIN` only, never `EXPLAIN ANALYZE` (the statement is someone else's): a text with parameters gets a generic
+  plan (`EXPLAIN (GENERIC_PLAN)`, PostgreSQL 16 or later), in a read-only transaction that is rolled back, with
+  a 5-second timeout and one statement only. Choose the schema its unqualified names live in. **Open in worksheet**
+  puts the text in the SQL Worksheet.
+
+Top SQL needs the `pg_stat_statements` extension, which the server must load (`shared_preload_libraries`); pgkiln
+creates the extension when it can (migration 016). Without it the page says how to turn it on. The per-application
+**Top SQL** on an application's Activity page shows the same statistics for its database role only.
+
+### Database users
+
+(sprint 99; Database Actions: Database Users) **Administration → Database users** (administrators only; also on the
+SQL Workshop's Launchpad under *Administration*) lists the PostgreSQL roles of the server (`pg_roles`; the predefined
+`pg_` roles are left out): whether they can sign in, superuser / create DB / create role / bypass RLS / replication,
+the connection limit, **valid until** (with *in N days* or *expired*), the roles they are a member of and what they own
+(schemas, objects in this database, objects elsewhere). PostgreSQL doesn't record when a role was created.
+The tiles on top filter the list: roles that can sign in, **passwords that expire within 30 days**, roles with
+**BYPASSRLS**, roles **without sign-in that own objects**, and pgkiln's own roles; a search field filters by name.
+
+**Create database user** (or a role's page) is a form: name, *Can sign in* (LOGIN), the password (*No password* /
+*Set a password*; for an existing role *Keep*, *Set a new password* or *Remove*), **valid until** (a date; empty: no
+expiry), the connection limit, the roles it is a **member of** (group roles and the predefined roles; not pgkiln's own,
+superusers, login roles or the roles that reach the server's files), its **default schema** (the search path in this
+database: `ALTER ROLE … IN DATABASE … SET search_path`), a comment, and per schema **Usage** and **Create**,
+**Future tables** (default privileges, *read* or *read and write*, on what pgkiln's owner connection creates there) and
+**Existing tables** (a one-off grant or revoke on the tables there now). pgkiln's own schemas aren't offered.
+
+**Review SQL** shows the statements (only what differs from what the catalog says now), with the password as
+`'********'`; the password itself is typed on that page (twice) and **Apply** runs the statements in one
+transaction: all or none, and the error with the statement that failed. Apply runs only what the review showed:
+when the form or the role changed in between, the review comes again. The masked statements go to the activity log
+(event `db_role`).
+
+**Drop role…** lists what the role owns and warns about objects in other databases; it reassigns them to another role
+(`REASSIGN OWNED`) or, with a tick, drops them (`DROP OWNED`), revokes its privileges in this database and drops the
+role, again after showing the SQL.
+
+pgkiln's **own roles** are marked *pgkiln* and are read only here: the owner and runtime connections (`DATABASE_URL`,
+`RUNTIME_DATABASE_URL`), every `pgkiln…` role (`pgkiln_runtime`, `pgkiln_app_code`, `pgkiln_authenticator`,
+`pgkiln_anon`, isolated workspaces' logins) and the bootstrap superuser. Roles **in use** by an application (parsing
+schema, REST API role) or an isolated workspace can be changed but not dropped. New names can't start with `pg_` or
+`pgkiln`. The owner connection needs `CREATEROLE` (and, on PostgreSQL 16 and later, ADMIN on roles it didn't create)
+for these changes; PostgreSQL's own error says so otherwise.
+
+### Schema export and import
+
+(sprint 103; Database Actions: Data Pump) **SQL Workshop → Schema Export / Import** (administrators only; on the
+Launchpad under *Administration* and on the Utilities page) dumps a schema to a file with **pg_dump** and loads a
+file into a schema with **pg_restore**, as background jobs. The list shows every job: export or import, the schema,
+its status (*running*, *succeeded*, *failed*, *cancelled*, or *uploaded* for a file waiting for its options), the
+file's size, when it started, how long it took and who ran it. A job's page has its options, pg_dump's or the
+import's **log**, and **Download**, **Import…**, **Cancel** (while it runs) and **Delete** (the job and its file).
+
+**Export a schema**: pick a schema (not pgkiln's own `meta`, PostgreSQL's or an extension's), then
+
+- the **format**: *custom* (compressed, `pg_dump -Fc`; what Import reads) or *plain SQL* (for `psql`; not imported
+  here; owners are left out);
+- the **content**: definitions and data, definitions only (`--schema-only`) or data only (`--data-only`), with or
+  without **grants**;
+- the **tables, views and sequences**: the whole schema, only the checked ones (`--table`: the file then has no
+  CREATE SCHEMA and no functions or types), or all but the checked ones (`--exclude-table`).
+
+pg_dump runs with an argument list (no shell) on the owner connection (`DATABASE_URL`); the password goes in
+`PGPASSWORD`, never on the command line. The file is kept in the dump directory for `PGKILN_DUMP_KEEP_DAYS` days
+(default 7); then the scheduler removes it and the job stays in the list.
+
+**Import**: upload a custom or tar archive of **one schema** (`pg_dump -Fc -n <schema>`, up to `PGKILN_DUMP_MAX_MB`,
+default 2048 MB; the file is streamed to the dump directory), or pick a kept export. The next page shows what the
+archive holds (`pg_restore --list`): the schema, the tables, the object types and how many of each, the server and
+pg_dump versions. Then choose
+
+- the **schema** to import into: the dump's own name or another one, **new or existing**. Another name renames the
+  dump's schema in the same transaction (*create schema as*);
+- the **content** (definitions and data, definitions only, data only: data goes into the tables that are there, in
+  foreign key order, parents first);
+- **Drop the dump's objects in the schema first** (`pg_restore --clean --if-exists`, to replace what an earlier import
+  made) and **Grants**.
+
+pg_restore writes the script (`pg_restore -f -`, without a connection of its own) and pgkiln runs it on one
+connection in **one transaction**, so an error rolls everything back:
+
+```sql
+begin;
+alter schema hr rename to pgkiln_dump_42;   -- importing hr as hr_copy while hr exists: the original steps aside
+alter schema hr_copy rename to hr;          -- only when hr_copy exists: the dump's names now mean it
+-- the dump's statements and COPY data (CREATE SCHEMA left out when the schema exists)
+alter schema hr rename to hr_copy;
+alter schema pgkiln_dump_42 rename to hr;
+commit;
+```
+
+Other sessions keep seeing the original schema until the commit. Views, column defaults, triggers, policies and
+SQL-standard function bodies refer to objects, so they follow the rename; **names inside PL/pgSQL bodies** and other
+text (dynamic SQL, a function's `SET search_path`) still say the dump's schema after a rename. Objects are owned by
+pgkiln's owner role (`--no-owner`); security labels, tablespaces, publications and subscriptions are left out.
+
+Every statement is checked before it runs: the settings pg_dump writes, CREATE / ALTER / COMMENT / GRANT / DROP of
+objects **in the dump's schema**, `COPY … FROM stdin` into its tables and `setval` of its sequences. Anything else is
+refused and rolls the import back: objects in other schemas (`meta` among them), roles and role memberships, `SET
+ROLE`, `OWNER TO`, `SET SCHEMA`, `DO`, `COPY` from or to files and programs, extensions, and psql meta-commands (other
+than pg_dump's `\restrict`). An archive with several schemas, extensions or large objects is refused at the preview.
+A dump is still code: its functions and triggers run as pgkiln's owner role. **Import only dumps you trust.**
+
+**The client programs.** pg_dump can't dump a server newer than itself, so pgkiln looks for pg_dump and pg_restore of
+the server's major version or newer, the newest it finds: `PGKILN_PG_BIN` (a directory), else
+`/usr/lib/postgresql/<n>/bin` (Debian, Ubuntu), `/usr/pgsql-<n>/bin` (RHEL), Homebrew's `postgresql@<n>`, else `PATH`.
+Without them the page says what to install (`postgresql-client-17` for a PostgreSQL 17 server). The Docker image
+includes PostgreSQL 17's client programs and keeps the files in `/app/dumps` (a volume in `deploy/compose.yaml`).
+
+| Variable | Default | |
+|---|---|---|
+| `PGKILN_PG_BIN` | (searched) | the directory of pg_dump and pg_restore |
+| `PGKILN_DUMP_DIR` | `<temp>/pgkiln-dumps` | where the files are kept; one server's directory: with several pgkiln servers, a job's file is on the server that ran it |
+| `PGKILN_DUMP_KEEP_DAYS` | 7 | days a file is kept |
+| `PGKILN_DUMP_MAX_MB` | 2048 | the largest upload |
+
+A job runs in the server that started it: when that server stops, the scheduler marks its running jobs *failed
+(interrupted)* within two minutes. Exports, uploads, imports and downloads go to the activity log (`schema_export`,
+`schema_upload`, `schema_import`, `schema_download`).
+
 ## Export and import
 
 **Export** downloads the application as JSON: pages, all components, shared components and

package/docs/guide/08-security.md

@@ -511,6 +511,35 @@ refused ([chapter 16](16-files.md#data-loading)). The SQL Workshop (the SQL Work
 Quick SQL, Load Data, Unload Data, Sample Data) runs as the owner instead, so every developer can change any table: keep
 developer accounts to people you trust with the database.
 
+The REST endpoints pgkiln serves (an application's REST modules and the tables and views published with **SQL
+Workshop → RESTful Services**, AutoREST) run as the app role too, never as the owner: these grants, column grants
+and the row level security below decide what an API caller may read and change. AutoREST refuses a role that owns
+the published object or bypasses row level security, and an object answers only to a bearer token of its application
+unless the developer chose *public* for it ([chapter 13](13-rest-api.md#autorest-tables-and-views-from-the-sql-workshop)).
+(sprint 104) REST module handlers refuse to run as a superuser, a role that bypasses row level security or a member
+of pgkiln's owner (403). Their parameters are bound: SQL handlers get escaped literals, PL/pgSQL handlers read
+transaction-local settings through `meta.rest_param()`; a request can't set the server's binds (`APP_USER`, …) or
+real settings. Media handlers are served with a sandboxing Content-Security-Policy and `nosniff`. *Try it* in the
+SQL Workshop calls an endpoint as an account of the application, as a developer issuing a token could
+([chapter 13](13-rest-api.md#modules-templates-and-handlers-in-the-sql-workshop)).
+
+(sprint 106) The SQL Workshop's **Change Log** keeps the schema DDL developers run from the worksheet, Table
+Properties and Schema Comparison, per workspace (only the workspace's developers see it). Statements about roles,
+databases, servers, user mappings and subscriptions are never recorded, so passwords and connection strings stay
+out of it; it holds what the DDL says otherwise (defaults, comments, function bodies).
+
+### Database users and their passwords
+
+(sprint 99) Administrators create and change PostgreSQL roles in the builder (**Administration → Database users**,
+[chapter 3](03-builder.md#database-users)). Passwords never reach PostgreSQL or a page in plain text: pgkiln hashes
+a new password itself into a **SCRAM-SHA-256 verifier** (random salt, 4096 iterations, SASLprep, like psql's
+`\password`) and the statement carries only that verifier, as a literal built with pg's `escapeLiteral`. The review
+and the activity log show `PASSWORD '********'`; the password is typed on the review page, posted once with Apply and
+not stored, echoed or logged. A server with `log_statement = ddl` logs the verifier, not the password. Every name is
+quoted with `escapeIdentifier`, every value is checked (dates, numbers, schema names, grantable roles) and quoted.
+The page is for administrators only (others get 403), every change needs the CSRF token, and pgkiln's own roles can't
+be changed, dropped or granted from it, so the page can't take pgkiln's own connections away.
+
 ## Session state protection
 
 - **URL items**: on pages with protection *Arguments must have checksum* (the default), item

package/docs/guide/12-development.md

@@ -69,6 +69,9 @@ src/
   sqlscript.ts             SQL scripts: splitScript() (statements, line numbers and offsets, psql commands), runScript() (stop/continue, transaction, savepoints)
   sqlworksheet.ts          (sprint 91) the worksheet's runs: pickStatements() (at the cursor, a selection, all), runStatements() (a log per
                            statement, psql command tags, notices, error line and column), explainStatement() (EXPLAIN / rolled-back ANALYZE)
+  datamodel.ts             (sprint 98) Data Modeler diagrams: readModel() (catalog), boxSize(), fkPath() (crow's feet), autoLayout(), diagramSvg()
+  changelog.ts             (sprint 106, migration 118) the change log: isSchemaDdl(), ddlTarget() (schema and object), changeName()
+                           (add_bonus, create_index_x), recordChanges() (grouped per schema), Flyway / Liquibase / plain SQL writers
   tabledef.ts              (sprint 95) a table as a definition: readTable(), checkTable(), tableSql() (CREATE, or ALTER from the catalog's version)
   sqlformat.ts             (sprint 93) formatSql(): the worksheet's Format (clauses per line, keyword case; strings and comments kept)
   sqlplan.ts               (sprint 92) EXPLAIN (FORMAT JSON) as a tree: parsePlan() (self time, slowest step, misestimates), planText()
@@ -86,6 +89,19 @@ src/
   mvt.ts                   Mapbox Vector Tile encoding for map layers served as tiles
   yamltext.ts              the text style of the directory export: a strict YAML subset, written and read
   webclient.ts             outgoing HTTP to web services: allow-list, address checks at connect time (SSRF), redirects, limits
+  datapump.ts              (sprint 103, DA-16) Schema Export / Import jobs (meta.dump_job): findTools (pg_dump/pg_restore ≥ the
+                           server's major), startExport (pg_dump, argument array, PGPASSWORD), readToc/parseToc (pg_restore --list),
+                           startImport (pg_restore -f - run in one transaction between schema renames; data-only in FK order),
+                           cancel, delete, dumpTick (interrupted jobs, expired files; called by the scheduler)
+  dumpscript.ts            (sprint 103) the import's script runner: ScriptLexer (psql-style split, COPY data, \restrict),
+                           checkStatement (only the dump's schema; no roles, DO, COPY to/from files…), COPY FROM STDIN as a pg submittable
+  dbroles.ts               (sprint 99, DA-13) Database Users' SQL: roleSql (CREATE/ALTER ROLE, memberships, search path, schema
+                           and default privileges: only what differs from the catalog), dropSql, checkRole/checkPassword,
+                           scramVerifier (SCRAM-SHA-256, SASLprep) and the password slot (maskSql for review and log, withVerifier to run)
+  cron-schedule.ts         (sprint 101, DA-12) pg_cron schedules without a database: parseSchedule (cron fields, $, n seconds, macros),
+                           nextRuns in a time zone, describeSchedule in words, the builder's presets and fields
+  pgcron.ts                (sprint 101, DA-12) pg_cron for SQL Workshop → Scheduling: cronState (installed, loaded, cron.database_name,
+                           access), jobs and runs, jobSql/activeSql/unscheduleSql/purgeSql (reviewed, then applySql), runNow
   secrets.ts               secrets at rest (web credentials): AES-256-GCM with PGKILN_SECRET_KEY
   websources.ts            web credentials (OAuth2 client credentials/password/refresh token grants, token cache, stored refresh
                            tokens) and REST data sources: requests, JSON paths, typed rows, response cache, write-back operations
@@ -148,7 +164,13 @@ src/
     maps.ts                map region (data for Leaflet: layers with a query each, markers, clusters or heat, PostGIS geometry as GeoJSON, report filter by area or distance; list fallback, head assets)
     spatial.ts             spatial filtering on the server: map area and distance parsing, PostGIS detection and SQL (ST_Intersects, ST_DWithin), lat/lng fallback (bounding box, haversine)
     pwa.ts                 Progressive Web App: manifest, service worker route, icons (PNG encoder), offline page
-    rest.ts                REST modules: handler checks, matching, bearer tokens, execution (collections stream from a cursor), OpenAPI
+    rest.ts                REST modules: handler and template checks, matching, bearer tokens, module privilege, the constrained-role
+                           check, execution (collections stream from a cursor), OpenAPI (templates, documented parameters, roles)
+    rest-handlers.ts       (sprint 104, migration 117) PL/pgSQL handlers (binds as settings, DO block, meta.rest_reply) and media
+                           handlers (content type and file name cleaned, sandboxed response)
+    autorest.ts            (sprint 100, migration 113) AutoREST: /a/:alias/autorest/<object>/ for the tables and views SQL Workshop →
+                           RESTful Services published (meta.rest_object): caller() of rest.ts, SET ROLE to the app role, owner/BYPASSRLS
+                           check, columns from the catalog (column grants), filters/q/order/paging, POST/PUT/DELETE by primary key, OpenAPI
     rest-sources.ts        REST data sources in apps: regions and LOVs as SQL over "rest", the invoke_api process (items; the call is websources.ts invoke()),
                            write-back of forms (fetch, form_dml) and grids (grid_dml) through the source's operations
     tree.ts                tree region
@@ -221,6 +243,11 @@ src/
     errors.ts              (sprint 86) the builder's HTML 404 and 500 pages for browsers (wantsHtml, a not-found handler and an onSend hook)
     arrange.ts             page designer layout changes: move, column span, create from the gallery, undo / redo
     sql.ts                 SQL Workshop: the home and the session's schema
+    launchpad.ts           (sprint 96) the SQL Workshop home's Launchpad (pins in meta.builder_pin, 109; tool groups), POST /builder/pins,
+                           GET /builder/palette (the builder's search: JSON for Ctrl+K, a results page without JavaScript)
+    sql-charts.ts          (sprint 97) charts of a query: chartSpec(), drawChart() (runtime/charts.ts, a PageCss in a nonce'd <style>), the
+                           worksheet's Chart tab, SQL Workshop → Charts (meta.sql_chart, 110; read-only runs)
+    sql-tools.ts           (sprint 96) SQL_TOOLS: every SQL Workshop tool with its group, tab and description (home, More menu, search)
     worksheet.ts           (sprint 91) the SQL Worksheet (Database Actions' SQL; /sql/commands too): navigator (pickSchema, objects with
                            columns, objectStatements), toolbar, output tabs, history (meta.sql_history, 108), plan trees, Run as an app's role, (sprint 93) Save / Rename / Download as SQL Scripts, bind and substitution values, POST /sql/worksheet/format; GET /sql/worksheet/navigator
     object-browser.ts      (sprint 75) the Object Browser: objects by type, system objects hidden, a table in tabs, column actions as SQL to review
@@ -228,10 +255,20 @@ src/
                            (one transaction) / Open in the worksheet
     object-actions.ts      (sprint 94) an object's ⋯ menu (objectMenu) in the worksheet and the Object Browser: Quick DDL (objectDdl), Drop with
                            dependents, Add index / trigger / policy as drafts; GET /sql/objects/action, /sql/objects/panel (Open in a panel)
+    data-modeler.ts        (sprint 98) SQL Workshop → Data Modeler: /sql/modeler (diagrams, meta.sql_diagram, 111), /:id (SVG), /:id/objects, /layout,
+                           /rename, /delete, /svg (export), /ddl (diagramDdl), /dictionary (data dictionary, ?download=1)
     scripts.ts             SQL Workshop → SQL Scripts: editor, upload/download, run, results per statement, run history
     quicksql.ts            SQL Workshop → Quick SQL page (preview, save as script, run)
     querybuilder.ts        SQL Workshop → Query Builder: catalog, joins by foreign key or drawn, functions/group by, buildQuery() from the URL; the canvas is in builder.js
     users.ts               user directory and identity providers
+    datapump.ts            (sprint 103, DA-16) SQL Workshop → Schema Export / Import (/builder/sql/data-pump, administrators only):
+                           jobs, export form, upload (streamed to PGKILN_DUMP_DIR), archive preview and import options, job log, download
+    db-users.ts            (sprint 99, DA-13) Administration → Database Users (/builder/db-users, administrators only): PostgreSQL
+                           roles with highlights, create/edit (review the masked SQL, then Apply in one transaction), drop with
+                           reassigning; pgkiln's own roles read-only, roles of applications and workspaces not dropped
+    scheduling.ts          (sprint 101, DA-12) SQL Workshop → Scheduling (/builder/sql/scheduling, administrators only): pg_cron jobs
+                           with next runs, create/edit with a schedule builder and preview (review the SQL, then Apply), disable,
+                           run now, unschedule, run history and purge; without pg_cron the install note and pgkiln's automations
     api.ts                 per-app REST API page (API role, tokens)
     globalization.ts       translations, XLIFF/CSV, text messages
     dataload.ts            SQL Workshop → Load Data (with definitions, save a mapping as one); data load definition spec (Shared Components)
@@ -243,11 +280,18 @@ src/
     region-settings.ts     page designer: settings forms for grid, chart (gauge, drill-down), cards, calendar (views, create, drag and drop), facets, smart filters, display selector, list
     search.ts              app search, "where used" (appEntries, search, whereUsed, usedInPanel)
     advisor.ts             Advisor: EXPLAIN every SQL fragment, reference checks, plpgsql_check
-    ddl-tools.ts           SQL Workshop → DDL and Compare (src/ddl.ts)
+    ddl-tools.ts           SQL Workshop → DDL and Compare (src/ddl.ts); (sprint 106) the Apply and record form
+    changelog.ts           (sprint 106) SQL Workshop → Change Log (/builder/sql/changes): change sets per schema, latest app export,
+                           downloads (POST, mark exported_at), rename, delete; /sql/compare/apply (one transaction, recorded)
     sql-utilities.ts       SQL Workshop → Utilities (sprint 62): the tools' tiles, OBJECT_REPORTS (fixed catalogue SQL over the
-                           connection's usable schemas), About Database, Database Monitor (administrators); read only, with a timeout
+                           connection's usable schemas), About Database; read only, with a timeout; /sql/utilities/monitor redirects to the SQL Monitor
     login-page.ts          Application settings → Login page and Forgot password; Create a login page (page 9999)
-    top-sql.ts             Top SQL per app role from pg_stat_statements
+    top-sql.ts             Top SQL per app role from pg_stat_statements; (sprint 102) statementsState / statementsNote (installed and loaded?)
+    sql-monitor.ts         (sprint 102, DA-14) SQL Workshop → SQL Monitor (administrators): /sql/monitor (sessions, wait events, progress;
+                           ?fragment=1 the live part), POST /sql/monitor/session (cancel / terminate), /sql/monitor/top (topStatements:
+                           pg_stat_statements sortable, bars), /reset, /statements/:queryid (totals by role, explainForMonitor: EXPLAIN only,
+                           one statement by the extended protocol, read only, generic plan for $1), /statements/:queryid/worksheet
+    plan-view.ts           (sprint 92; shared since 102) planView(): a plan (src/sqlplan.ts) as a tree, for the worksheet and the SQL Monitor
     diagnostics.ts         Activity → Debug messages (level, list, one request's entries, purge); Workspace utilities →
                            Installation (version, install/upgrade runs, applied and missing migrations; administrators)
     ldap.ts                Users → LDAP directories
@@ -259,6 +303,12 @@ src/
     form-designer.ts       page designer: Form Designer settings (designers' scheme; sources: key, actions, offered columns, labels, lists of values)
     workingcopies.ts       Working copies: list and create, compare with differences, merge or refresh with conflict choices, delete
     rest.ts                REST module endpoints list and curl example (Shared Components)
+    autorest.ts            (sprint 100) SQL Workshop → RESTful Services (/builder/sql/rest): publish a table or view through an
+                           application of the workspace (checks: schema the workshop may use, app role not the owner), its page
+                           (grants per method, RLS, security_invoker, endpoints, cURL/fetch, OpenAPI link), settings, disable
+    rest-modules.ts        (sprint 104) SQL Workshop → RESTful Services → Modules (/builder/sql/rest/modules): modules of the
+                           workspace's applications, the module › template › handler tree, handler forms, Try it (inject with a
+                           short-lived token), Run only / lock checks
     workflows.ts           workflow versions, diagram and instances (Shared Components)
     process-jobs.ts        page designer: the Jobs tab of a background chain process
     template-spec.ts       template component property form (Shared Components)
@@ -290,11 +340,15 @@ public/
   app.css                  theme (light/dark, responsive; --font, --font-size, --radius for style variants; template option classes to-*)
   app.js                   client runtime: dialogs (dialog_closed actions), popup LOVs, dynamic actions (focus, classes, messages), grids (add/duplicate rows, master-detail refresh, move/resize columns, copy/paste of cell ranges), menus, lazy regions, maps (Leaflet layers, marker clusters, heat layer, layer legend, area/distance filter) (no inline JS)
   code-editor.js, .css     builder code editor: enhances <textarea data-code>, highlighting, suggestions (no dependencies)
+  palette.js               (sprint 96) the builder's search dialog (Ctrl+K), recently visited SQL Workshop pages, the Launchpad's tabs
+  sql-monitor.js           (sprint 102) the SQL Monitor: reloads the sessions' live part every 5 / 15 / 60 s, Pause (localStorage pgkiln.sqlMonitor)
+  data-modeler.js, .css    (sprint 98) the Data Modeler: drag boxes, undo/redo, save, zoom/fit, auto layout, mini-map, SVG/PNG export, print
   worksheet.js, .css       (sprint 91) the SQL Worksheet: run keys with the cursor, output tabs, navigator swap and search, drag and drop, splitters;
                            (92) grid sort, value dialog, history filter; (93) worksheet tabs in localStorage, Files, binds dialog, format, find, case
-  builder.css              builder only: IDE look (dark chrome, icon rail, panes), builder light/dark tokens
+  builder.css              builder only: IDE look (dark chrome, icon rail, panes); design tokens at the top (colour,
+                           elevation, radius, type, motion; light and dark sets, sprint 105), then the shared controls
   builder.js               builder only: tabs, component tree, property filter, drag and drop on the layout
-  builder-icons.svg        builder only: icons of the rail, toolbar and designer (b-*)
+  builder-icons.svg        builder only: icons of the rail, toolbar and designer (b-*); b-kiln is the brand mark (brandMark() in ui.ts)
 test/
   binds.test.ts            unit tests
   security.test.ts         security regression tests (in-process, against the database)
@@ -311,9 +365,17 @@ test/
   unload.test.ts           Unload Data: CSV/JSON/XLSX/XML output, read back with Load Data, read-only and one-statement checks, streaming
   app-from-file.test.ts    Create → From a file: proposed names and types, app + table + rows + pages, row errors, login, validation
   workshop.test.ts         SQL scripts, Quick SQL pages, query builder, data load definitions (Load Data, the process, export)
+  cron-schedule.test.ts    (sprint 101) pg_cron schedules: parsing and errors, next runs across time zones and daylight saving, words
+  scheduling.test.ts       (sprint 101) SQL Workshop → Scheduling: the SQL, the no-pg_cron page, create/edit/disable/run now/unschedule,
+                           history and purge; against pg_cron when the test database has it, else test/pgcron-mock.ts (a stand-in)
+  sql-charts.test.ts       (sprint 97) the worksheet's Chart tab, saving a chart, the Charts dashboard, a chart's page, read-only runs
+  launchpad.test.ts        (sprint 96) Launchpad groups, pins (builder pages only, at most 24), the search (JSON and page)
   tabledef.test.ts         (sprint 95) table definitions round trip: CREATE, read, ALTER, read again; mistakes
+  changelog.test.ts        (sprint 106) DDL recognition and names, recording from Table Properties, the worksheet and Compare,
+                           downloads (Flyway, Liquibase, SQL), stable numbers, the latest application export
   table-editor.test.ts     (sprint 95) the Table Properties page: preview, apply, a failing apply rolls back, template, worksheet
   object-actions.test.ts   (sprint 94) object menus: Quick DDL, Drop with dependents, Add index / trigger / policy, the panel, not-found
+  data-modeler.test.ts     (sprint 98) Data Modeler: "hr core" with its FK lines, saved positions, SVG, objects, DDL, dictionary, geometry
   worksheet.test.ts        (sprint 91) the SQL Worksheet: statement at the cursor, Run Script log, plans, notices, history, navigator, Run as
   quicksql.test.ts         Quick SQL parser and DDL generator
   xml.test.ts              XML reader: rows, attributes, paths, refused DTDs and entities, limits
@@ -327,6 +389,7 @@ test/
   charts.test.ts           chart markup per kind (geometry as classes), gauges, Gantt time axis and dependencies, pyramid, polar, range, box plot, stock, drill-down links
   calendar.test.ts         calendar views, create links, moving events (pure and over HTTP)
   rest-sources.test.ts     REST data sources, web credentials, SSRF checks, invoke_api (mock service + HR page 23)
+  rest-modules.test.ts     (sprint 104) modules, templates, PL/pgSQL and media handlers, OpenAPI, the builder's tree and Try it
   workflow-invoke.test.ts  workflow invoke_api steps: the call between transactions, faults, retry, lease, Advisor, export (mock service)
   large-tables.test.ts     row ranges, max_rows, row limits, lazy regions, region caching, streamed downloads (HR page 25)
   grid.test.ts             interactive grid: aggregates, layouts per user, saved grid reports, master-detail, row actions (HR page 27)
@@ -360,6 +423,7 @@ test/
   helpers.ts               a cookie-keeping test browser
   e2e/responsive.test.ts   browser tests at phone/tablet/desktop widths (Playwright)
   e2e/code-editor.test.ts  the code editor in a browser: highlighting, keys, suggestions, touch, screen readers
+  e2e/data-modeler.test.ts (sprint 98) the Data Modeler in a browser: drag, save, reload, undo/redo, zoom, mini-map, exports, phone
   e2e/worksheet.test.ts    (sprint 91) the SQL Worksheet in a browser: run keys, tabs, navigator, drag and drop, splitters, phone, no JavaScript
   e2e/items.test.ts        sprint 26 item types in a browser: editors, tags, stars, dates, reveal; without JavaScript
   e2e/designer.test.ts     page designer: panes per width, drag and drop, keyboard, Arrange buttons, builder theme