pgkiln 0.48.0 → 0.50.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. package/SECURITY.md +3 -2
  2. package/db/migrations/116_dump_jobs.sql +41 -0
  3. package/db/migrations/117_rest_handlers.sql +63 -0
  4. package/db/migrations/118_change_log.sql +54 -0
  5. package/docs/apex-feature-parity.md +9 -9
  6. package/docs/guide/01-installation.md +7 -0
  7. package/docs/guide/03-builder.md +227 -10
  8. package/docs/guide/08-security.md +11 -0
  9. package/docs/guide/12-development.md +64 -13
  10. package/docs/guide/13-rest-api.md +59 -1
  11. package/docs/guide/15-extensions.md +1 -1
  12. package/examples/hr/hr_62_rest_module.sql +39 -0
  13. package/package.json +1 -1
  14. package/public/app.css +1 -1
  15. package/public/builder-icons.svg +2 -0
  16. package/public/builder.css +478 -173
  17. package/public/builder.js +48 -0
  18. package/public/code-editor.css +32 -79
  19. package/public/code-editor.js +58 -3
  20. package/public/data-modeler.css +17 -14
  21. package/public/sql-monitor.js +73 -0
  22. package/public/worksheet.css +14 -7
  23. package/public/worksheet.js +119 -3
  24. package/src/app.ts +6 -0
  25. package/src/automations.ts +6 -0
  26. package/src/binds.ts +2 -1
  27. package/src/builder/advisor.ts +24 -2
  28. package/src/builder/autorest.ts +3 -2
  29. package/src/builder/changelog.ts +240 -0
  30. package/src/builder/components.ts +1 -1
  31. package/src/builder/data-modeler.ts +1 -1
  32. package/src/builder/datapump.ts +364 -0
  33. package/src/builder/db-users.ts +2 -2
  34. package/src/builder/ddl-tools.ts +13 -4
  35. package/src/builder/find-workspace.ts +2 -2
  36. package/src/builder/object-browser.ts +4 -4
  37. package/src/builder/plan-view.ts +25 -0
  38. package/src/builder/querybuilder.ts +1 -1
  39. package/src/builder/quicksql.ts +1 -1
  40. package/src/builder/rest-modules.ts +629 -0
  41. package/src/builder/rest.ts +1 -0
  42. package/src/builder/routes.ts +4 -2
  43. package/src/builder/scheduling.ts +495 -0
  44. package/src/builder/scripts.ts +1 -1
  45. package/src/builder/sql-charts.ts +1 -1
  46. package/src/builder/sql-monitor.ts +397 -0
  47. package/src/builder/sql-tools.ts +6 -2
  48. package/src/builder/sql-utilities.ts +7 -39
  49. package/src/builder/sql.ts +6 -0
  50. package/src/builder/table-editor.ts +6 -3
  51. package/src/builder/top-sql.ts +24 -6
  52. package/src/builder/ui.ts +6 -4
  53. package/src/builder/worksheet.ts +173 -41
  54. package/src/changelog.ts +109 -0
  55. package/src/cron-schedule.ts +282 -0
  56. package/src/datapump.ts +546 -0
  57. package/src/dumpscript.ts +407 -0
  58. package/src/pgcron.ts +225 -0
  59. package/src/runtime/rest-handlers.ts +75 -0
  60. package/src/runtime/rest.ts +116 -22
  61. package/src/session.ts +5 -0
  62. package/src/sqlworksheet.ts +9 -6
  63. package/src/worksheet-txn.ts +270 -0
@@ -102,7 +102,41 @@ Links), added on the workspace's page by administrators and the workspace's admi
102
102
  `https://` URL, an optional description and an order. While the workspace is current an **Extensions** icon in the
103
103
  rail opens them, each in a new tab (`rel="noopener noreferrer"`).
104
104
 
105
- ### Export and import
105
+ #### Scheduling with pg_cron
106
+
107
+ (sprint 101; Database Actions: Scheduling) **SQL Workshop → Scheduling** (administrators only) manages the jobs of
108
+ [pg_cron](https://github.com/citusdata/pg_cron), the PostgreSQL extension that runs SQL on a schedule inside the
109
+ database server. pgkiln needs pg_cron 1.4 or later, created in pgkiln's database; the page says what is missing when
110
+ it isn't (the package, `shared_preload_libraries = 'pg_cron'`, `cron.database_name`, `create extension pg_cron`).
111
+ Without pg_cron it lists **pgkiln's own scheduled work** read-only: every application's automations (Shared Components
112
+ → Automations, [chapter 6](06-processing.md)) with their schedule, next and last run, and the latest background process
113
+ jobs. With pg_cron that list is behind a toggle under the jobs.
114
+
115
+ - **Jobs** (`cron.job`): name, the command, the schedule with its meaning in words (*At 02:00, every day*), the **next
116
+ run**, the last run's status and time (and how many of the last ten failed), database and the role it runs as;
117
+ a search field. Times are in pg_cron's time zone (`cron.timezone`, GMT unless set). pg_cron shows each role only its
118
+ own jobs: when pgkiln's owner connection is neither a superuser nor `BYPASSRLS`, the page says so.
119
+ - **Create job** / **Edit**: name (unique per role), database, *Run as* (a role that can sign in; only a superuser
120
+ schedules for another role), *Active*, the command (SQL, run outside a transaction block as pg_cron runs it, so
121
+ `vacuum` works) and the **schedule**: five cron fields, `n seconds` (1 to 59) or `@hourly` … `@yearly`, `@reboot`.
122
+ A **schedule builder** (one field each for minute, hour, day of month, month and day of week, `$` for the last day
123
+ of the month) and **common schedules** fill it in; the **preview** shows it in words with the **next five runs**,
124
+ as you type (without JavaScript: the *Use* and *Preview* buttons). pgkiln explains schedules as pg_cron reads them:
125
+ lists, ranges, steps, month and day names, 7 for Sunday, and a day matching either the day of month or the day of
126
+ week when both are restricted.
127
+ - **Review SQL** shows the statements (`cron.schedule_in_database`, `cron.alter_job` with only what changed, an
128
+ `update cron.job` for a new name; the command dollar-quoted); **Apply** runs them in one transaction and only if they
129
+ are still what was reviewed. **Disable / Enable** (`cron.alter_job(…, active := …)`) applies at once and shows the
130
+ statement; **Unschedule…** (`cron.unschedule`) after a review. Every change goes to the activity log (event `cron_job`).
131
+ - **Run now**: pg_cron has no run-now, so pgkiln runs the job's stored command once itself: a new connection to the
132
+ job's database with the owner connection's login, `SET ROLE` to the job's role (the owner must be allowed to), no
133
+ transaction block around it, a 10-minute statement timeout. The page shows the result, the duration and notices.
134
+ The run is not in pg_cron's history; the activity log has it (event `cron_run_now`).
135
+ - **Run history** (`cron.job_run_details`, newest first, 200 at most): status, start, duration and message, filtered
136
+ by job, status or a text in the message or command. **Purge…** deletes runs older than 7, 30 or 90 days, or all
137
+ (of every job or one), after showing the `delete` and how many runs it removes: pg_cron keeps every run until then.
138
+
139
+ ## Export and import
106
140
 
107
141
  Workspace utilities → **Export and import** (`/builder/workspace-export`; APEX: Workspace Utilities → Export), for
108
142
  administrators and the workspace's administrators: the current workspace's components as one JSON file (format
@@ -199,6 +233,15 @@ Every builder page has the same frame, modelled on an IDE:
199
233
 
200
234
  On phones the rail gets narrower and the breadcrumb shows only the current page.
201
235
 
236
+ The builder's look comes from design tokens at the top of `public/builder.css` (sprint 105): CSS custom properties
237
+ for colours (a warm neutral palette with the kiln's ember for primary buttons, links, focus rings and selections),
238
+ surfaces and their shadows, corner radii, the type scale and transitions, in a light and a dark set; the code editor's syntax colours (`--ce-*`: keywords, strings, numbers, comments, binds,
239
+ quoted names, operators, the matching bracket, the current line, the selection and the line numbers) are tokens there
240
+ too (sprint 108). The rail, the
241
+ toolbar and the status bar stay dark in both themes. Transitions are switched off when the operating system asks
242
+ for reduced motion. The builder uses the system's fonts (Inter when it is installed); it loads nothing from
243
+ other sites.
244
+
202
245
  (sprint 86) An address under `/builder` that leads nowhere (a deleted application or page, a typo, an
203
246
  application of a workspace you are not a member of) shows a **Page not found** page in the builder's
204
247
  theme with a link to the App Builder home; a server error shows **Something went wrong** (the error
@@ -663,7 +706,13 @@ handlers, the SQL Workshop and the import box) opens in a small code editor:
663
706
  binds, `&ITEM.` substitutions, `$$` bodies), JSON and HTML (tags, attributes, `&ITEM.` and
664
707
  `{{tags}}`). The language follows the component type: a `static` region's source is HTML, a
665
708
  `regex` validation is plain text, an `execute_sql` dynamic action is PL/pgSQL.
666
- - **Line numbers**, the line and column of the cursor, and the **matching bracket** marked.
709
+ - **Line numbers**, the line and column of the cursor, the **current line** (a band and its number in bold
710
+ while the field has the focus) and the **matching bracket** marked.
711
+ - **Colours from the builder's theme** (sprint 108): keywords take the kiln's ember, types, functions, strings,
712
+ numbers, binds and quoted names cool hues, comments a muted grey, each at least 4.5:1 (WCAG AA) on the editor's
713
+ background in the light and the dark theme. Read-only SQL (the Object Browser's DDL and source tabs, the Data
714
+ Modeler's DDL, generated SQL in Quick SQL, Query Builder, Scheduling, Database Users and the table editor, the SQL
715
+ Monitor's text, a saved chart's query, the Change Log, a script's results) is highlighted the same way.
667
716
  - **Tab / Shift+Tab** indent and outdent (the selected lines, too); **Enter** keeps the
668
717
  indentation and indents after `(`, `begin`, `loop`, `then`; brackets and quotes are closed for
669
718
  you and typing the closer steps over it. Press **Escape, then Tab** to move on to the next field.
@@ -1340,11 +1389,41 @@ the SQL Workshop's connection may use. Without JavaScript the header's field ope
1340
1389
  (**Ctrl + F**, **Ctrl + H**) with a regular expression or exact case, **Go to line** (**Ctrl + G**), and
1341
1390
  **Ctrl + /** comments the selected lines out or in. The **?** menu lists every key; **F1** opens this help. The
1342
1391
  upload button opens Load Data for the navigator's schema.
1343
- - Each statement commits on its own. Statements run as the **owner connection** (an isolated workspace's
1392
+ - Each statement commits on its own (**Autocommit**, like Database Actions). Statements run as the **owner connection** (an isolated workspace's
1344
1393
  developer login for its developers), on a connection of their own that is closed afterwards, so `SET ROLE` or
1345
1394
  `SET search_path` doesn't leak. **Run as** picks an application of your workspace instead: the statements then
1346
1395
  run as its database role, with `pgkiln.app_id` set, so its grants and row level security policies apply. Use
1347
- it to test policies; it is no security boundary, since the SQL could `RESET ROLE`.
1396
+ it to test policies; it is no security boundary, since the SQL could `RESET ROLE`. A `BEGIN` typed in
1397
+ Autocommit lasts for the run only: `begin; …; commit;` in one Run Script works, but a transaction still open at
1398
+ the end of the run is rolled back, with a warning under *Notices*.
1399
+ - (sprint 107) **Transaction mode** (more than Database Actions, which always autocommits): the toolbar's
1400
+ **Commit** menu switches a worksheet tab between *Autocommit* and *Transaction*. In Transaction mode the tab's
1401
+ statements (Run Statement and Run Script) stay in one transaction on a connection of its own until you press
1402
+ **Commit** or **Rollback**: a SELECT in the same tab sees the changes, other sessions (and your other tabs) don't
1403
+ until you commit. A badge shows *Transaction open*, the number of statements and its age; after a reload it shows
1404
+ the server's real state, and leaving the page with an open transaction asks first.
1405
+ - `COMMIT` or `ROLLBACK` typed in the editor end the transaction too (the next run starts a new one); a `BEGIN`
1406
+ typed inside it only warns. Statements PostgreSQL refuses inside a transaction (`VACUUM`, `CREATE INDEX
1407
+ CONCURRENTLY`, `ALTER SYSTEM`, `CREATE DATABASE`, …) are refused before they run, so they can't abort it.
1408
+ - A statement that fails leaves the transaction **aborted** (*Transaction failed*): every later statement fails
1409
+ until **Rollback**; Commit then rolls back as well, and says so.
1410
+ - **Explain Plan** and **Autotrace** run inside the open transaction behind a savepoint: the plan sees its changes,
1411
+ Autotrace's own changes are undone and the transaction stays as it was.
1412
+ - **Run as**: the transaction keeps the role it started with. Changing *Run as* while it is open is refused:
1413
+ commit or roll back first. `SET ROLE` typed in the editor reaches only the roles of the connection's login, as
1414
+ in Autocommit.
1415
+ - DDL reaches the **Change Log** when the transaction commits, never when it is rolled back. SQL History records
1416
+ each statement as it runs.
1417
+ - The transaction is **rolled back** when it is left without a run for `WORKSHEET_TXN_IDLE_SECONDS` (default
1418
+ 300, five minutes; the badge and the next run say so), when you close its worksheet tab or switch it to
1419
+ Autocommit (after asking), sign out, switch workspaces, and when the server stops. The database ends the
1420
+ connection itself a minute later (`idle_in_transaction_session_timeout`) should the server not.
1421
+ - Each open transaction holds a database connection outside the pools: a developer has at most
1422
+ `WORKSHEET_TXN_MAX_PER_DEVELOPER` (default 3) and the server `WORKSHEET_TXN_MAX` (default 20); past that a run in
1423
+ Transaction mode is refused. A transaction belongs to the builder session, workspace and worksheet tab that
1424
+ started it: another developer, or another sign-in of yours, never reaches it. It lives in the server's memory,
1425
+ so with several pgkiln servers behind a load balancer the browser must stay on one (sticky sessions).
1426
+ - Without JavaScript the worksheet runs in Autocommit only.
1348
1427
  - With script, the panes fill the window. Drag the splitters (or focus one and use the arrow keys) to resize
1349
1428
  them, hide the navigator, or use **Fullscreen** to see only the editor (**Escape** leaves). Sizes are
1350
1429
  remembered in the browser. Without script, the buttons post the form and the panes stack; Run Statement then
@@ -1412,8 +1491,9 @@ the SQL Workshop's connection may use. Without JavaScript the header's field ope
1412
1491
  when the foreign key may be empty. A box's name opens **Table Properties** (a view: the Object
1413
1492
  Browser). The toolbar: **Save** (Ctrl+S), **Undo** / **Redo** (Ctrl+Z, Ctrl+Y), **Zoom out** / **Zoom
1414
1493
  in** (or Ctrl+wheel), **Fit**, **Auto layout** (referenced tables to the left of the tables that
1415
- reference them; undoable), a **Mini-map** (click or drag in it to move around), **SVG** and **PNG**
1416
- exports of the diagram as shown, **Print**, **Generate DDL** (the CREATE statements of the shown
1494
+ reference them; undoable), a **Mini-map** (click or drag in it to move around; the button shows when it is on),
1495
+ **SVG** and **PNG** exports of the diagram as shown (in light colours, whichever theme the builder uses; on screen the
1496
+ boxes, lines and mini-map follow the light or dark theme), **Print**, **Generate DDL** (the CREATE statements of the shown
1417
1497
  objects with their keys, indexes, triggers, row level security, policies, comments and grants: to
1418
1498
  download or open in the SQL Worksheet) and **Data dictionary** (each object's columns with type,
1419
1499
  nullability, default, keys and comments, its constraints and what references it; to print, save as
@@ -1454,7 +1534,8 @@ the SQL Workshop's connection may use. Without JavaScript the header's field ope
1454
1534
  replaces views and functions, drops and creates constraints, indexes, triggers, policies and
1455
1535
  grants that differ, and lists for review what it doesn't change: drops of what only the target
1456
1536
  has (commented out), columns whose identity or generation differs, and types, domains and
1457
- sequences that differ. The same from the command line (`pgkiln ddl`, `pgkiln compare`,
1537
+ sequences that differ. (sprint 106) *Apply and record* runs the script on the target in one transaction and records
1538
+ it in the Change Log (below). The same from the command line (`pgkiln ddl`, `pgkiln compare`,
1458
1539
  [chapter 18](18-cli.md)) and for AI agents (`generate_ddl`, `compare_schemas`).
1459
1540
  - **RESTful Services** (`/builder/sql/rest`, sprint 100; Database Actions: REST › AutoREST): publish a table or
1460
1541
  view as REST endpoints. Choose the object (of a schema the SQL Workshop may use), the **application** it is
@@ -1465,6 +1546,29 @@ the SQL Workshop's connection may use. Without JavaScript the header's field ope
1465
1546
  accounts' roles. The object's page checks the role's grants per method, row level security and (for a view)
1466
1547
  `security_invoker`, lists the endpoints with their URLs, and shows cURL and `fetch` examples and the OpenAPI link.
1467
1548
  Details in [chapter 13](13-rest-api.md#autorest-tables-and-views-from-the-sql-workshop).
1549
+ (sprint 104; Database Actions: REST › Modules) The **Modules** tab edits the REST modules of the workspace's
1550
+ applications as a tree: a module (its base path and its privilege: roles every handler needs), its templates (URIs
1551
+ with `:parameters`) and their handlers (method, source type: collection query, single row, SQL, PL/pgSQL or media,
1552
+ who may call it, roles, documented parameters). Each handler has a **Try it** panel: the request goes to the real
1553
+ endpoint without a token or as an account of the application, and the response is shown in the page. Details in
1554
+ [chapter 13](13-rest-api.md#modules-templates-and-handlers-in-the-sql-workshop).
1555
+ - **Change Log** (`/builder/sql/changes`, sprint 106; Database Actions: Liquibase): DDL that succeeds in the **SQL
1556
+ Worksheet**, **Table Properties** (*Apply*) and **Schema Comparison** (*Apply and record*: the change script run on
1557
+ the target in one transaction) is recorded as a *change set*: the schema it changes, its SQL, who ran it, when,
1558
+ where from and a number per schema that is never given again (deleting a change set keeps the others' numbers).
1559
+ Statements without a schema-qualified name count for the worksheet's schema. Only schema DDL is kept (CREATE,
1560
+ ALTER, DROP, COMMENT, GRANT and REVOKE on objects): no queries or data changes, and nothing about roles, databases
1561
+ or servers, so passwords never end up in the log. The page lists a schema's change sets with their SQL; rename one
1562
+ (its file name) or delete it from the log (the database is not changed). It marks which change sets were made
1563
+ before the latest **Export** of an application of the workspace (*in it*) and which after (*still to ship*).
1564
+ *Download migration* writes all of them, those not yet written to a file, or those after the latest export, as
1565
+ **Flyway** versioned migrations (`V0001__add_bonus.sql`; several in a zip), a **Liquibase** formatted SQL changelog
1566
+ (`--liquibase formatted sql`, `--changeset author:0001-add_bonus`) or one plain SQL script; each change set also
1567
+ has its own Flyway file. Change sets belong to the workspace and are not part of an application export.
1568
+ - **JavaScript (MLE)**: not planned. Database Actions has a JavaScript tool for Oracle's Multilingual Engine;
1569
+ PostgreSQL runs JavaScript in the database only through the [plv8](15-extensions.md) extension. Where plv8 is
1570
+ installed, write `create function … language plv8` in the SQL Worksheet like any other function: the navigator,
1571
+ the Object Browser and Quick DDL show it. pgkiln adds no JavaScript editor of its own.
1468
1572
  - **Utilities** (`/builder/sql/utilities`; APEX: SQL Workshop → Utilities): tiles for Load and Unload Data, Sample
1469
1573
  Data, the Query Builder, DDL and Compare, and three pages of their own. Each runs in a read-only transaction with a
1470
1574
  15-second statement timeout, on the SQL Workshop's connection (an isolated workspace's developers see only the
@@ -1475,14 +1579,51 @@ the SQL Workshop's connection may use. Without JavaScript the header's field ope
1475
1579
  to the Object Browser. pgkiln's own `meta` schema and the system's are left out.
1476
1580
  - **About Database**: the PostgreSQL version, database size, uptime, a few settings (encoding, time zone,
1477
1581
  connections, memory, row security), the installed extensions and the schemas with their size.
1478
- - **Database Monitor** (administrators only): the client sessions of this database from `pg_stat_activity`, active
1479
- first, with their state, wait event, running time, the sessions that block them and the query (as far as the
1480
- owner may read other roles' statistics). Read only: it cancels nothing.
1582
+ - **SQL Monitor** (administrators only): see [SQL Monitor](#sql-monitor) below (it replaced the Database Monitor;
1583
+ the old address `/builder/sql/utilities/monitor` redirects).
1481
1584
  - **Database Users** (administrators only; Administration menu → *Database users*, `/builder/db-users`; sprint 99,
1482
1585
  Database Actions: Database Users): see [Database users](#database-users).
1586
+ - **Schema Export / Import** (administrators only; `/builder/sql/data-pump`; sprint 103, Database Actions: Data
1587
+ Pump): see [Schema export and import](#schema-export-and-import).
1588
+ - **Scheduling** (administrators only; `/builder/sql/scheduling`, on the Launchpad under *Administration*, the
1589
+ Utilities page and the search; sprint 101, Database Actions: Scheduling): [pg_cron](#scheduling-with-pg_cron) jobs.
1483
1590
 
1484
1591
  Because the SQL Workshop runs as the owner, restrict who gets a developer account.
1485
1592
 
1593
+ ### SQL Monitor
1594
+
1595
+ (sprint 102; Database Actions: Real Time SQL Monitor and Performance Hub) **SQL Workshop → SQL Monitor**
1596
+ (`/builder/sql/monitor`, administrators only; on the Launchpad under *Monitoring*, in the Utilities and in the search,
1597
+ Ctrl+K) shows what the database is doing, from PostgreSQL's statistics views:
1598
+
1599
+ - **Sessions**: tiles with the sessions (of `max_connections`), the active ones, those idle in a transaction, the
1600
+ blocked ones and the longest running statement; then the client sessions of this database (`pg_stat_activity`;
1601
+ idle sessions on request, *Show idle sessions*) with user, application and client address, state (idle in a
1602
+ transaction with its time), wait event, running time, the sessions that **block** it (the row is highlighted) and
1603
+ the query. A query with a `query_id` links to its statement page. **Cancel** (`pg_cancel_backend`: stop the running
1604
+ statement) and **Terminate** (`pg_terminate_backend`: roll back and disconnect) ask for a confirmation first; only
1605
+ client sessions of this database can be signalled, and the owner role needs the right to (a superuser, the
1606
+ session's own role, or `pg_signal_backend` for non-superusers' sessions).
1607
+ - **Wait events**: what the active sessions of the server wait for now, counted per type and event (*CPU: running*
1608
+ when they don't wait).
1609
+ - **In progress** (only when something runs): VACUUM, ANALYZE, CREATE INDEX, CLUSTER and COPY from the
1610
+ `pg_stat_progress_*` views, with the phase and a progress bar.
1611
+ - The tiles and tables **refresh in place** every 5, 15 or 60 seconds, with a **Pause** button; the choice is
1612
+ remembered in the browser. Without JavaScript, *Refresh* reloads the page.
1613
+ - **Top SQL**: the statements of this database from `pg_stat_statements`, every role: calls, total and mean time
1614
+ (with bars), rows, shared blocks hit and read; sort by any column (click the heading again to reverse), search the
1615
+ text, **Reset statistics** (the whole server, after a confirmation). Literals show as `$1`, `$2`….
1616
+ - **A statement**: its text, its totals (calls, total and mean time, rows, blocks, temporary blocks) and a row per
1617
+ role with standard deviation, minimum and maximum; **Explain** shows its plan as a tree like the worksheet's. It is
1618
+ `EXPLAIN` only, never `EXPLAIN ANALYZE` (the statement is someone else's): a text with parameters gets a generic
1619
+ plan (`EXPLAIN (GENERIC_PLAN)`, PostgreSQL 16 or later), in a read-only transaction that is rolled back, with
1620
+ a 5-second timeout and one statement only. Choose the schema its unqualified names live in. **Open in worksheet**
1621
+ puts the text in the SQL Worksheet.
1622
+
1623
+ Top SQL needs the `pg_stat_statements` extension, which the server must load (`shared_preload_libraries`); pgkiln
1624
+ creates the extension when it can (migration 016). Without it the page says how to turn it on. The per-application
1625
+ **Top SQL** on an application's Activity page shows the same statistics for its database role only.
1626
+
1486
1627
  ### Database users
1487
1628
 
1488
1629
  (sprint 99; Database Actions: Database Users) **Administration → Database users** (administrators only; also on the
@@ -1518,6 +1659,82 @@ schema, REST API role) or an isolated workspace can be changed but not dropped.
1518
1659
  `pgkiln`. The owner connection needs `CREATEROLE` (and, on PostgreSQL 16 and later, ADMIN on roles it didn't create)
1519
1660
  for these changes; PostgreSQL's own error says so otherwise.
1520
1661
 
1662
+ ### Schema export and import
1663
+
1664
+ (sprint 103; Database Actions: Data Pump) **SQL Workshop → Schema Export / Import** (administrators only; on the
1665
+ Launchpad under *Administration* and on the Utilities page) dumps a schema to a file with **pg_dump** and loads a
1666
+ file into a schema with **pg_restore**, as background jobs. The list shows every job: export or import, the schema,
1667
+ its status (*running*, *succeeded*, *failed*, *cancelled*, or *uploaded* for a file waiting for its options), the
1668
+ file's size, when it started, how long it took and who ran it. A job's page has its options, pg_dump's or the
1669
+ import's **log**, and **Download**, **Import…**, **Cancel** (while it runs) and **Delete** (the job and its file).
1670
+
1671
+ **Export a schema**: pick a schema (not pgkiln's own `meta`, PostgreSQL's or an extension's), then
1672
+
1673
+ - the **format**: *custom* (compressed, `pg_dump -Fc`; what Import reads) or *plain SQL* (for `psql`; not imported
1674
+ here; owners are left out);
1675
+ - the **content**: definitions and data, definitions only (`--schema-only`) or data only (`--data-only`), with or
1676
+ without **grants**;
1677
+ - the **tables, views and sequences**: the whole schema, only the checked ones (`--table`: the file then has no
1678
+ CREATE SCHEMA and no functions or types), or all but the checked ones (`--exclude-table`).
1679
+
1680
+ pg_dump runs with an argument list (no shell) on the owner connection (`DATABASE_URL`); the password goes in
1681
+ `PGPASSWORD`, never on the command line. The file is kept in the dump directory for `PGKILN_DUMP_KEEP_DAYS` days
1682
+ (default 7); then the scheduler removes it and the job stays in the list.
1683
+
1684
+ **Import**: upload a custom or tar archive of **one schema** (`pg_dump -Fc -n <schema>`, up to `PGKILN_DUMP_MAX_MB`,
1685
+ default 2048 MB; the file is streamed to the dump directory), or pick a kept export. The next page shows what the
1686
+ archive holds (`pg_restore --list`): the schema, the tables, the object types and how many of each, the server and
1687
+ pg_dump versions. Then choose
1688
+
1689
+ - the **schema** to import into: the dump's own name or another one, **new or existing**. Another name renames the
1690
+ dump's schema in the same transaction (*create schema as*);
1691
+ - the **content** (definitions and data, definitions only, data only: data goes into the tables that are there, in
1692
+ foreign key order, parents first);
1693
+ - **Drop the dump's objects in the schema first** (`pg_restore --clean --if-exists`, to replace what an earlier import
1694
+ made) and **Grants**.
1695
+
1696
+ pg_restore writes the script (`pg_restore -f -`, without a connection of its own) and pgkiln runs it on one
1697
+ connection in **one transaction**, so an error rolls everything back:
1698
+
1699
+ ```sql
1700
+ begin;
1701
+ alter schema hr rename to pgkiln_dump_42; -- importing hr as hr_copy while hr exists: the original steps aside
1702
+ alter schema hr_copy rename to hr; -- only when hr_copy exists: the dump's names now mean it
1703
+ -- the dump's statements and COPY data (CREATE SCHEMA left out when the schema exists)
1704
+ alter schema hr rename to hr_copy;
1705
+ alter schema pgkiln_dump_42 rename to hr;
1706
+ commit;
1707
+ ```
1708
+
1709
+ Other sessions keep seeing the original schema until the commit. Views, column defaults, triggers, policies and
1710
+ SQL-standard function bodies refer to objects, so they follow the rename; **names inside PL/pgSQL bodies** and other
1711
+ text (dynamic SQL, a function's `SET search_path`) still say the dump's schema after a rename. Objects are owned by
1712
+ pgkiln's owner role (`--no-owner`); security labels, tablespaces, publications and subscriptions are left out.
1713
+
1714
+ Every statement is checked before it runs: the settings pg_dump writes, CREATE / ALTER / COMMENT / GRANT / DROP of
1715
+ objects **in the dump's schema**, `COPY … FROM stdin` into its tables and `setval` of its sequences. Anything else is
1716
+ refused and rolls the import back: objects in other schemas (`meta` among them), roles and role memberships, `SET
1717
+ ROLE`, `OWNER TO`, `SET SCHEMA`, `DO`, `COPY` from or to files and programs, extensions, and psql meta-commands (other
1718
+ than pg_dump's `\restrict`). An archive with several schemas, extensions or large objects is refused at the preview.
1719
+ A dump is still code: its functions and triggers run as pgkiln's owner role. **Import only dumps you trust.**
1720
+
1721
+ **The client programs.** pg_dump can't dump a server newer than itself, so pgkiln looks for pg_dump and pg_restore of
1722
+ the server's major version or newer, the newest it finds: `PGKILN_PG_BIN` (a directory), else
1723
+ `/usr/lib/postgresql/<n>/bin` (Debian, Ubuntu), `/usr/pgsql-<n>/bin` (RHEL), Homebrew's `postgresql@<n>`, else `PATH`.
1724
+ Without them the page says what to install (`postgresql-client-17` for a PostgreSQL 17 server). The Docker image
1725
+ includes PostgreSQL 17's client programs and keeps the files in `/app/dumps` (a volume in `deploy/compose.yaml`).
1726
+
1727
+ | Variable | Default | |
1728
+ |---|---|---|
1729
+ | `PGKILN_PG_BIN` | (searched) | the directory of pg_dump and pg_restore |
1730
+ | `PGKILN_DUMP_DIR` | `<temp>/pgkiln-dumps` | where the files are kept; one server's directory: with several pgkiln servers, a job's file is on the server that ran it |
1731
+ | `PGKILN_DUMP_KEEP_DAYS` | 7 | days a file is kept |
1732
+ | `PGKILN_DUMP_MAX_MB` | 2048 | the largest upload |
1733
+
1734
+ A job runs in the server that started it: when that server stops, the scheduler marks its running jobs *failed
1735
+ (interrupted)* within two minutes. Exports, uploads, imports and downloads go to the activity log (`schema_export`,
1736
+ `schema_upload`, `schema_import`, `schema_download`).
1737
+
1521
1738
  ## Export and import
1522
1739
 
1523
1740
  **Export** downloads the application as JSON: pages, all components, shared components and
@@ -516,6 +516,17 @@ Workshop → RESTful Services**, AutoREST) run as the app role too, never as the
516
516
  and the row level security below decide what an API caller may read and change. AutoREST refuses a role that owns
517
517
  the published object or bypasses row level security, and an object answers only to a bearer token of its application
518
518
  unless the developer chose *public* for it ([chapter 13](13-rest-api.md#autorest-tables-and-views-from-the-sql-workshop)).
519
+ (sprint 104) REST module handlers refuse to run as a superuser, a role that bypasses row level security or a member
520
+ of pgkiln's owner (403). Their parameters are bound: SQL handlers get escaped literals, PL/pgSQL handlers read
521
+ transaction-local settings through `meta.rest_param()`; a request can't set the server's binds (`APP_USER`, …) or
522
+ real settings. Media handlers are served with a sandboxing Content-Security-Policy and `nosniff`. *Try it* in the
523
+ SQL Workshop calls an endpoint as an account of the application, as a developer issuing a token could
524
+ ([chapter 13](13-rest-api.md#modules-templates-and-handlers-in-the-sql-workshop)).
525
+
526
+ (sprint 106) The SQL Workshop's **Change Log** keeps the schema DDL developers run from the worksheet, Table
527
+ Properties and Schema Comparison, per workspace (only the workspace's developers see it). Statements about roles,
528
+ databases, servers, user mappings and subscriptions are never recorded, so passwords and connection strings stay
529
+ out of it; it holds what the DDL says otherwise (defaults, comments, function bodies).
519
530
 
520
531
  ### Database users and their passwords
521
532
 
@@ -68,8 +68,15 @@ src/
68
68
  xml.ts safe XML reader (no DTDs or entities, limits) and xmlTable(): rows from a repeating element (unit tested)
69
69
  sqlscript.ts SQL scripts: splitScript() (statements, line numbers and offsets, psql commands), runScript() (stop/continue, transaction, savepoints)
70
70
  sqlworksheet.ts (sprint 91) the worksheet's runs: pickStatements() (at the cursor, a selection, all), runStatements() (a log per
71
- statement, psql command tags, notices, error line and column), explainStatement() (EXPLAIN / rolled-back ANALYZE)
71
+ statement, psql command tags, notices, error line and column), explainStatement() (EXPLAIN / rolled-back ANALYZE;
72
+ a savepoint inside a worksheet transaction)
73
+ worksheet-txn.ts (sprint 107) the worksheet's transaction mode: an in-memory registry of pinned pg.Clients per (builder session,
74
+ worksheet tab), beginTxn() / endTxn() (caps per developer and server), accountRun() (statement count, DDL kept for
75
+ the Change Log until COMMIT), outsideTxnOnly(), sweepTxns() (idle timeout, ended sessions); onSessionEnd and
76
+ onClosePools roll back; WORKSHEET_TXN_IDLE_SECONDS / _MAX_PER_DEVELOPER / _MAX
72
77
  datamodel.ts (sprint 98) Data Modeler diagrams: readModel() (catalog), boxSize(), fkPath() (crow's feet), autoLayout(), diagramSvg()
78
+ changelog.ts (sprint 106, migration 118) the change log: isSchemaDdl(), ddlTarget() (schema and object), changeName()
79
+ (add_bonus, create_index_x), recordChanges() (grouped per schema), Flyway / Liquibase / plain SQL writers
73
80
  tabledef.ts (sprint 95) a table as a definition: readTable(), checkTable(), tableSql() (CREATE, or ALTER from the catalog's version)
74
81
  sqlformat.ts (sprint 93) formatSql(): the worksheet's Format (clauses per line, keyword case; strings and comments kept)
75
82
  sqlplan.ts (sprint 92) EXPLAIN (FORMAT JSON) as a tree: parsePlan() (self time, slowest step, misestimates), planText()
@@ -87,9 +94,19 @@ src/
87
94
  mvt.ts Mapbox Vector Tile encoding for map layers served as tiles
88
95
  yamltext.ts the text style of the directory export: a strict YAML subset, written and read
89
96
  webclient.ts outgoing HTTP to web services: allow-list, address checks at connect time (SSRF), redirects, limits
97
+ datapump.ts (sprint 103, DA-16) Schema Export / Import jobs (meta.dump_job): findTools (pg_dump/pg_restore ≥ the
98
+ server's major), startExport (pg_dump, argument array, PGPASSWORD), readToc/parseToc (pg_restore --list),
99
+ startImport (pg_restore -f - run in one transaction between schema renames; data-only in FK order),
100
+ cancel, delete, dumpTick (interrupted jobs, expired files; called by the scheduler)
101
+ dumpscript.ts (sprint 103) the import's script runner: ScriptLexer (psql-style split, COPY data, \restrict),
102
+ checkStatement (only the dump's schema; no roles, DO, COPY to/from files…), COPY FROM STDIN as a pg submittable
90
103
  dbroles.ts (sprint 99, DA-13) Database Users' SQL: roleSql (CREATE/ALTER ROLE, memberships, search path, schema
91
104
  and default privileges: only what differs from the catalog), dropSql, checkRole/checkPassword,
92
105
  scramVerifier (SCRAM-SHA-256, SASLprep) and the password slot (maskSql for review and log, withVerifier to run)
106
+ cron-schedule.ts (sprint 101, DA-12) pg_cron schedules without a database: parseSchedule (cron fields, $, n seconds, macros),
107
+ nextRuns in a time zone, describeSchedule in words, the builder's presets and fields
108
+ pgcron.ts (sprint 101, DA-12) pg_cron for SQL Workshop → Scheduling: cronState (installed, loaded, cron.database_name,
109
+ access), jobs and runs, jobSql/activeSql/unscheduleSql/purgeSql (reviewed, then applySql), runNow
93
110
  secrets.ts secrets at rest (web credentials): AES-256-GCM with PGKILN_SECRET_KEY
94
111
  websources.ts web credentials (OAuth2 client credentials/password/refresh token grants, token cache, stored refresh
95
112
  tokens) and REST data sources: requests, JSON paths, typed rows, response cache, write-back operations
@@ -152,7 +169,10 @@ src/
152
169
  maps.ts map region (data for Leaflet: layers with a query each, markers, clusters or heat, PostGIS geometry as GeoJSON, report filter by area or distance; list fallback, head assets)
153
170
  spatial.ts spatial filtering on the server: map area and distance parsing, PostGIS detection and SQL (ST_Intersects, ST_DWithin), lat/lng fallback (bounding box, haversine)
154
171
  pwa.ts Progressive Web App: manifest, service worker route, icons (PNG encoder), offline page
155
- rest.ts REST modules: handler checks, matching, bearer tokens, execution (collections stream from a cursor), OpenAPI
172
+ rest.ts REST modules: handler and template checks, matching, bearer tokens, module privilege, the constrained-role
173
+ check, execution (collections stream from a cursor), OpenAPI (templates, documented parameters, roles)
174
+ rest-handlers.ts (sprint 104, migration 117) PL/pgSQL handlers (binds as settings, DO block, meta.rest_reply) and media
175
+ handlers (content type and file name cleaned, sandboxed response)
156
176
  autorest.ts (sprint 100, migration 113) AutoREST: /a/:alias/autorest/<object>/ for the tables and views SQL Workshop →
157
177
  RESTful Services published (meta.rest_object): caller() of rest.ts, SET ROLE to the app role, owner/BYPASSRLS
158
178
  check, columns from the catalog (column grants), filters/q/order/paging, POST/PUT/DELETE by primary key, OpenAPI
@@ -234,7 +254,8 @@ src/
234
254
  worksheet's Chart tab, SQL Workshop → Charts (meta.sql_chart, 110; read-only runs)
235
255
  sql-tools.ts (sprint 96) SQL_TOOLS: every SQL Workshop tool with its group, tab and description (home, More menu, search)
236
256
  worksheet.ts (sprint 91) the SQL Worksheet (Database Actions' SQL; /sql/commands too): navigator (pickSchema, objects with
237
- columns, objectStatements), toolbar, output tabs, history (meta.sql_history, 108), plan trees, Run as an app's role, (sprint 93) Save / Rename / Download as SQL Scripts, bind and substitution values, POST /sql/worksheet/format; GET /sql/worksheet/navigator
257
+ columns, objectStatements), toolbar, output tabs, history (meta.sql_history, 108), plan trees, Run as an app's role, (sprint 93) Save / Rename / Download as SQL Scripts, bind and substitution values, POST /sql/worksheet/format; GET /sql/worksheet/navigator;
258
+ (sprint 107) transaction mode: the txn field, Commit / Rollback (mode=commit|rollback), txnBadge(), GET/POST /sql/worksheet/txn (JSON)
238
259
  object-browser.ts (sprint 75) the Object Browser: objects by type, system objects hidden, a table in tabs, column actions as SQL to review
239
260
  table-editor.ts (sprint 95) Table Properties: GET/POST /sql/objects/table (create, edit ?o=, ?template=), fromForm(), Preview / Apply
240
261
  (one transaction) / Open in the worksheet
@@ -246,9 +267,14 @@ src/
246
267
  quicksql.ts SQL Workshop → Quick SQL page (preview, save as script, run)
247
268
  querybuilder.ts SQL Workshop → Query Builder: catalog, joins by foreign key or drawn, functions/group by, buildQuery() from the URL; the canvas is in builder.js
248
269
  users.ts user directory and identity providers
270
+ datapump.ts (sprint 103, DA-16) SQL Workshop → Schema Export / Import (/builder/sql/data-pump, administrators only):
271
+ jobs, export form, upload (streamed to PGKILN_DUMP_DIR), archive preview and import options, job log, download
249
272
  db-users.ts (sprint 99, DA-13) Administration → Database Users (/builder/db-users, administrators only): PostgreSQL
250
273
  roles with highlights, create/edit (review the masked SQL, then Apply in one transaction), drop with
251
274
  reassigning; pgkiln's own roles read-only, roles of applications and workspaces not dropped
275
+ scheduling.ts (sprint 101, DA-12) SQL Workshop → Scheduling (/builder/sql/scheduling, administrators only): pg_cron jobs
276
+ with next runs, create/edit with a schedule builder and preview (review the SQL, then Apply), disable,
277
+ run now, unschedule, run history and purge; without pg_cron the install note and pgkiln's automations
252
278
  api.ts per-app REST API page (API role, tokens)
253
279
  globalization.ts translations, XLIFF/CSV, text messages
254
280
  dataload.ts SQL Workshop → Load Data (with definitions, save a mapping as one); data load definition spec (Shared Components)
@@ -260,11 +286,18 @@ src/
260
286
  region-settings.ts page designer: settings forms for grid, chart (gauge, drill-down), cards, calendar (views, create, drag and drop), facets, smart filters, display selector, list
261
287
  search.ts app search, "where used" (appEntries, search, whereUsed, usedInPanel)
262
288
  advisor.ts Advisor: EXPLAIN every SQL fragment, reference checks, plpgsql_check
263
- ddl-tools.ts SQL Workshop → DDL and Compare (src/ddl.ts)
289
+ ddl-tools.ts SQL Workshop → DDL and Compare (src/ddl.ts); (sprint 106) the Apply and record form
290
+ changelog.ts (sprint 106) SQL Workshop → Change Log (/builder/sql/changes): change sets per schema, latest app export,
291
+ downloads (POST, mark exported_at), rename, delete; /sql/compare/apply (one transaction, recorded)
264
292
  sql-utilities.ts SQL Workshop → Utilities (sprint 62): the tools' tiles, OBJECT_REPORTS (fixed catalogue SQL over the
265
- connection's usable schemas), About Database, Database Monitor (administrators); read only, with a timeout
293
+ connection's usable schemas), About Database; read only, with a timeout; /sql/utilities/monitor redirects to the SQL Monitor
266
294
  login-page.ts Application settings → Login page and Forgot password; Create a login page (page 9999)
267
- top-sql.ts Top SQL per app role from pg_stat_statements
295
+ top-sql.ts Top SQL per app role from pg_stat_statements; (sprint 102) statementsState / statementsNote (installed and loaded?)
296
+ sql-monitor.ts (sprint 102, DA-14) SQL Workshop → SQL Monitor (administrators): /sql/monitor (sessions, wait events, progress;
297
+ ?fragment=1 the live part), POST /sql/monitor/session (cancel / terminate), /sql/monitor/top (topStatements:
298
+ pg_stat_statements sortable, bars), /reset, /statements/:queryid (totals by role, explainForMonitor: EXPLAIN only,
299
+ one statement by the extended protocol, read only, generic plan for $1), /statements/:queryid/worksheet
300
+ plan-view.ts (sprint 92; shared since 102) planView(): a plan (src/sqlplan.ts) as a tree, for the worksheet and the SQL Monitor
268
301
  diagnostics.ts Activity → Debug messages (level, list, one request's entries, purge); Workspace utilities →
269
302
  Installation (version, install/upgrade runs, applied and missing migrations; administrators)
270
303
  ldap.ts Users → LDAP directories
@@ -279,6 +312,9 @@ src/
279
312
  autorest.ts (sprint 100) SQL Workshop → RESTful Services (/builder/sql/rest): publish a table or view through an
280
313
  application of the workspace (checks: schema the workshop may use, app role not the owner), its page
281
314
  (grants per method, RLS, security_invoker, endpoints, cURL/fetch, OpenAPI link), settings, disable
315
+ rest-modules.ts (sprint 104) SQL Workshop → RESTful Services → Modules (/builder/sql/rest/modules): modules of the
316
+ workspace's applications, the module › template › handler tree, handler forms, Try it (inject with a
317
+ short-lived token), Run only / lock checks
282
318
  workflows.ts workflow versions, diagram and instances (Shared Components)
283
319
  process-jobs.ts page designer: the Jobs tab of a background chain process
284
320
  template-spec.ts template component property form (Shared Components)
@@ -309,14 +345,20 @@ src/
309
345
  public/
310
346
  app.css theme (light/dark, responsive; --font, --font-size, --radius for style variants; template option classes to-*)
311
347
  app.js client runtime: dialogs (dialog_closed actions), popup LOVs, dynamic actions (focus, classes, messages), grids (add/duplicate rows, master-detail refresh, move/resize columns, copy/paste of cell ranges), menus, lazy regions, maps (Leaflet layers, marker clusters, heat layer, layer legend, area/distance filter) (no inline JS)
312
- code-editor.js, .css builder code editor: enhances <textarea data-code>, highlighting, suggestions (no dependencies)
348
+ code-editor.js, .css builder code editor: enhances <textarea data-code>, highlighting, suggestions (no dependencies);
349
+ (sprint 108) the current line, and read-only <pre data-code-view="sql|plpgsql|json|html"> highlighted;
350
+ its colours are the --ce-* tokens in builder.css
313
351
  palette.js (sprint 96) the builder's search dialog (Ctrl+K), recently visited SQL Workshop pages, the Launchpad's tabs
352
+ sql-monitor.js (sprint 102) the SQL Monitor: reloads the sessions' live part every 5 / 15 / 60 s, Pause (localStorage pgkiln.sqlMonitor)
314
353
  data-modeler.js, .css (sprint 98) the Data Modeler: drag boxes, undo/redo, save, zoom/fit, auto layout, mini-map, SVG/PNG export, print
315
354
  worksheet.js, .css (sprint 91) the SQL Worksheet: run keys with the cursor, output tabs, navigator swap and search, drag and drop, splitters;
316
- (92) grid sort, value dialog, history filter; (93) worksheet tabs in localStorage, Files, binds dialog, format, find, case
317
- builder.css builder only: IDE look (dark chrome, icon rail, panes), builder light/dark tokens
355
+ (92) grid sort, value dialog, history filter; (93) worksheet tabs in localStorage, Files, binds dialog, format, find, case;
356
+ (107) Autocommit / Transaction per tab, the badge from the server, beforeunload, rollback on closing a tab
357
+ builder.css builder only: IDE look (dark chrome, icon rail, panes); design tokens at the top (colour,
358
+ elevation, radius, type, motion; light and dark sets, sprint 105; code editor --ce-*, sprint 108),
359
+ then the shared controls
318
360
  builder.js builder only: tabs, component tree, property filter, drag and drop on the layout
319
- builder-icons.svg builder only: icons of the rail, toolbar and designer (b-*)
361
+ builder-icons.svg builder only: icons of the rail, toolbar and designer (b-*); b-kiln is the brand mark (brandMark() in ui.ts)
320
362
  test/
321
363
  binds.test.ts unit tests
322
364
  security.test.ts security regression tests (in-process, against the database)
@@ -333,13 +375,20 @@ test/
333
375
  unload.test.ts Unload Data: CSV/JSON/XLSX/XML output, read back with Load Data, read-only and one-statement checks, streaming
334
376
  app-from-file.test.ts Create → From a file: proposed names and types, app + table + rows + pages, row errors, login, validation
335
377
  workshop.test.ts SQL scripts, Quick SQL pages, query builder, data load definitions (Load Data, the process, export)
378
+ cron-schedule.test.ts (sprint 101) pg_cron schedules: parsing and errors, next runs across time zones and daylight saving, words
379
+ scheduling.test.ts (sprint 101) SQL Workshop → Scheduling: the SQL, the no-pg_cron page, create/edit/disable/run now/unschedule,
380
+ history and purge; against pg_cron when the test database has it, else test/pgcron-mock.ts (a stand-in)
336
381
  sql-charts.test.ts (sprint 97) the worksheet's Chart tab, saving a chart, the Charts dashboard, a chart's page, read-only runs
337
382
  launchpad.test.ts (sprint 96) Launchpad groups, pins (builder pages only, at most 24), the search (JSON and page)
338
383
  tabledef.test.ts (sprint 95) table definitions round trip: CREATE, read, ALTER, read again; mistakes
384
+ changelog.test.ts (sprint 106) DDL recognition and names, recording from Table Properties, the worksheet and Compare,
385
+ downloads (Flyway, Liquibase, SQL), stable numbers, the latest application export
339
386
  table-editor.test.ts (sprint 95) the Table Properties page: preview, apply, a failing apply rolls back, template, worksheet
340
387
  object-actions.test.ts (sprint 94) object menus: Quick DDL, Drop with dependents, Add index / trigger / policy, the panel, not-found
341
388
  data-modeler.test.ts (sprint 98) Data Modeler: "hr core" with its FK lines, saved positions, SVG, objects, DDL, dictionary, geometry
342
389
  worksheet.test.ts (sprint 91) the SQL Worksheet: statement at the cursor, Run Script log, plans, notices, history, navigator, Run as
390
+ worksheet-txn.test.ts (sprint 107) transaction mode: the tab's transaction, Commit / Rollback, typed COMMIT, aborted, idle timeout, refusals,
391
+ Explain in it, Change Log at COMMIT, the JSON endpoint, sign-out
343
392
  quicksql.test.ts Quick SQL parser and DDL generator
344
393
  xml.test.ts XML reader: rows, attributes, paths, refused DTDs and entities, limits
345
394
  printing.test.ts report PDFs
@@ -352,6 +401,7 @@ test/
352
401
  charts.test.ts chart markup per kind (geometry as classes), gauges, Gantt time axis and dependencies, pyramid, polar, range, box plot, stock, drill-down links
353
402
  calendar.test.ts calendar views, create links, moving events (pure and over HTTP)
354
403
  rest-sources.test.ts REST data sources, web credentials, SSRF checks, invoke_api (mock service + HR page 23)
404
+ rest-modules.test.ts (sprint 104) modules, templates, PL/pgSQL and media handlers, OpenAPI, the builder's tree and Try it
355
405
  workflow-invoke.test.ts workflow invoke_api steps: the call between transactions, faults, retry, lease, Advisor, export (mock service)
356
406
  large-tables.test.ts row ranges, max_rows, row limits, lazy regions, region caching, streamed downloads (HR page 25)
357
407
  grid.test.ts interactive grid: aggregates, layouts per user, saved grid reports, master-detail, row actions (HR page 27)
@@ -384,9 +434,10 @@ test/
384
434
  ai-mock.ts a local mock of the Claude Messages API (SSE stream) and OpenAI Chat Completions
385
435
  helpers.ts a cookie-keeping test browser
386
436
  e2e/responsive.test.ts browser tests at phone/tablet/desktop widths (Playwright)
387
- e2e/code-editor.test.ts the code editor in a browser: highlighting, keys, suggestions, touch, screen readers
437
+ e2e/code-editor.test.ts the code editor in a browser: highlighting, keys, suggestions, touch, screen readers, syntax colours at AA in both themes (sprint 108)
388
438
  e2e/data-modeler.test.ts (sprint 98) the Data Modeler in a browser: drag, save, reload, undo/redo, zoom, mini-map, exports, phone
389
- e2e/worksheet.test.ts (sprint 91) the SQL Worksheet in a browser: run keys, tabs, navigator, drag and drop, splitters, phone, no JavaScript
439
+ e2e/worksheet.test.ts (sprint 91) the SQL Worksheet in a browser: run keys, tabs, navigator, drag and drop, splitters, phone, no JavaScript;
440
+ (107) transaction mode: badge, reload, Rollback, closing a tab, Autocommit switch, beforeunload
390
441
  e2e/items.test.ts sprint 26 item types in a browser: editors, tags, stars, dates, reveal; without JavaScript
391
442
  e2e/designer.test.ts page designer: panes per width, drag and drop, keyboard, Arrange buttons, builder theme
392
443
  e2e/calendar.test.ts calendar drag and drop and create on click, view switching, chart drill-down
@@ -437,7 +488,7 @@ CI (`.github/workflows/ci.yml`) runs these jobs against PostgreSQL 17:
437
488
  - **test**: typecheck and `npm test` on a fresh database;
438
489
  - **e2e**: the browser tests, uploading the screenshots as an artifact; `test/e2e/accessibility.test.ts` runs
439
490
  axe-core (WCAG 2.1 A and AA rules) on every page of the HR example (light, dark, Iris) and the builder's main pages
440
- and allows no violation;
491
+ (in both builder themes, with the SQL Workshop and the wizards, since sprint 108) and allows no violation;
441
492
  - **docker**: builds the image and starts the compose stack as a user would.
442
493
 
443
494
  - **upgrade**: installs each earlier release (`v0.31.0` …) with its HR example (`git archive <tag> | tar -x`,
@@ -31,11 +31,12 @@ answer 404.
31
31
  |---|---|
32
32
  | `method` | `GET`, `POST`, `PUT`, `PATCH` or `DELETE` |
33
33
  | `path` | Segments with `:parameters`, e.g. `orders/:id/lines` |
34
- | `type` | `collection`: a SELECT, returned page by page as `{items, offset, limit, has_more}` (`?limit=` up to 500, `?offset=`, `page_size` for the default); the page streams from a cursor as a chunked JSON array (an error before the first rows is still a 4xx/5xx, a failure after them ends the response short) · `item`: one row as an object, 404 when there is none · `sql`: statements; the first row of the last one is the response (201 for POST, 204 without a row; `status` overrides) |
34
+ | `type` | `collection`: a SELECT, returned page by page as `{items, offset, limit, has_more}` (`?limit=` up to 500, `?offset=`, `page_size` for the default); the page streams from a cursor as a chunked JSON array (an error before the first rows is still a 4xx/5xx, a failure after them ends the response short) · `item`: one row as an object, 404 when there is none · `sql`: statements; the first row of the last one is the response (201 for POST, 204 without a row; `status` overrides) · `plpgsql` (sprint 104): a PL/pgSQL block that answers with `meta.rest_reply(body, status)` ([below](#modules-templates-and-handlers-in-the-sql-workshop)) · `media`: a query whose row has `content_type`, `content` and optionally `filename`, sent as that content |
35
35
  | `source` | The SQL. Binds: path parameters, query parameters and the fields of a JSON (or form) body, upper case (`:ID`, `:CUSTOMER`); `:BODY` is the whole JSON body. `:APP_USER` (the caller), `:APP_ID` and `:APP_ALIAS` are set by the server: a parameter of that name (or `APP_SESSION`, `APP_PAGE_ID`, `REQUEST`, `APP_LANGUAGE`) is ignored, and can't be a path parameter |
36
36
  | `roles` | The caller needs one of these roles (as `meta.has_role()` sees them) |
37
37
  | `auth` | `token` (default) or `public` (no token; the SQL runs with `meta.app_user()` = `nobody`) |
38
38
  | `description` | For the OpenAPI description |
39
+ | `params` | (sprint 104) Documented parameters: `[{"name": "start_date", "in": "body", "type": "string", "required": true, "description": "…"}]` (`in`: `query` or `body`; types `string`, `integer`, `number`, `boolean`). They go into the OpenAPI document; a required one that is missing or empty answers 400 |
39
40
 
40
41
  **Calling them.** Send `Authorization: Bearer <token>` with a token from **App → REST API**: a token
41
42
  issued for an account, or one from `POST /oauth/token` for an OAuth client ([below](#tokens)). An
@@ -63,6 +64,63 @@ curl -X POST http://127.0.0.1:3100/a/hr/rest/v1/leave -H "Authorization: Bearer
63
64
  -H "Content-Type: application/json" -d '{"start_date": "2027-05-03", "end_date": "2027-05-04", "reason": "dentist"}'
64
65
  ```
65
66
 
67
+ ## Modules, templates and handlers in the SQL Workshop
68
+
69
+ **SQL Workshop → RESTful Services → Modules** (sprint 104; Oracle Database Actions / ORDS: REST › Modules) edits
70
+ the same REST modules the ORDS way, for every application of the workspace:
71
+
72
+ - a **module** is a base path, `/a/<alias>/rest/<name>/`, with a title, a description, *Enabled* and its
73
+ **privilege**: application roles (as `meta.has_role()` sees them) of which a caller needs one for *every*
74
+ handler of the module, public ones included;
75
+ - a **template** is a URI below it with `:parameters`, e.g. `emp/:id`, with a description for the OpenAPI document
76
+ (a template can exist before its first handler; renaming it moves its handlers);
77
+ - a **handler** answers one method of a template: its source type (*collection query*, *single row*, *SQL*,
78
+ *PL/pgSQL*, *media*), the source, who may call it (a token, or explicitly *public*), further roles, the page size
79
+ or status, and documented parameters, one per line: `start_date body string required First day`.
80
+
81
+ The tree on the left of each page shows module › templates › handlers. A Run only or locked application refuses
82
+ changes. The module's page checks the role the handlers run as and links the module's **OpenAPI** document.
83
+
84
+ **PL/pgSQL handlers** run their block as a `DO` statement as the application's database role. A `DO` block takes
85
+ no parameters, so pgkiln sets the request's binds as transaction-local settings (with query parameters: values
86
+ never become SQL text) and turns `:NAME` in the block into `meta.rest_param('NAME')` (text, NULL when absent):
87
+
88
+ ```sql
89
+ declare
90
+ v_id int;
91
+ begin
92
+ v_id := sales.create_order(:CUSTOMER, :TOTAL::numeric);
93
+ perform meta.rest_reply(jsonb_build_object('id', v_id), 201);
94
+ end
95
+ ```
96
+
97
+ `meta.rest_reply(body jsonb, status int)` sets the response (status 200–599, also 4xx for your own errors; the
98
+ transaction is still committed); without it the handler answers 204. A `raise exception` answers 400 with its
99
+ message and rolls back. Statements without `begin … end` are wrapped in one.
100
+
101
+ **Media handlers** run a query whose first row has `content_type`, `content` (bytea or text) and optionally
102
+ `filename`; the response is that content, sent with `X-Content-Type-Options: nosniff` and a sandboxing
103
+ Content-Security-Policy, so HTML or SVG from a table can't run script in the application's origin. No row: 404.
104
+
105
+ **Try it.** A handler's page sends a request to the real endpoint, with path parameters, a query string and a JSON
106
+ body, either without a token (to see the 401) or *as* an account of the application with a short-lived token (the
107
+ same as issuing one on App → REST API). A PUT, POST or DELETE changes data. The response's status and body are
108
+ shown in the page; it works without JavaScript.
109
+
110
+ **Example:** the HR example's module `hr` (`examples/hr/hr_62_rest_module.sql`): `GET emp/:id` (single row),
111
+ `GET emp/:id/photo` (media), `POST leave` (PL/pgSQL with body parameters) and `GET payroll/:deptno` (roles manager or
112
+ admin):
113
+
114
+ ```bash
115
+ curl http://127.0.0.1:3100/a/hr/rest/hr/emp/7839 -H "Authorization: Bearer $TOKEN"
116
+ curl -X POST http://127.0.0.1:3100/a/hr/rest/hr/leave -H "Authorization: Bearer $TOKEN" \
117
+ -H "Content-Type: application/json" -d '{"start_date": "2027-05-03", "end_date": "2027-05-04", "reason": "dentist"}'
118
+ # {"id": 12, "status": "requested"}
119
+ ```
120
+
121
+ Handlers never run as a superuser, a role that bypasses row level security or pgkiln's owner: such a request
122
+ answers 403 (set `RUNTIME_DATABASE_URL` and give the application its own database role).
123
+
66
124
  ## AutoREST: tables and views from the SQL Workshop
67
125
 
68
126
  **SQL Workshop → RESTful Services** (`/builder/sql/rest`; Database Actions: a table's *REST › Enable*, ORDS
@@ -19,7 +19,7 @@ setup`) and on practically every managed service. Install with `create extension
19
19
  | **unaccent** | Accent-insensitive search (`café` finds `cafe`), essential in multilingual apps | Text search options |
20
20
  | **citext** | Case-insensitive text columns (e-mail addresses, codes) without `lower()` everywhere | NLS_COMP=LINGUISTIC |
21
21
  | **btree_gist** | Exclusion constraints on ranges, e.g. **no overlapping bookings/leave** enforced by the database, race-free | Custom PL/SQL checks |
22
- | **pg_stat_statements** | Which SQL is slow or called most, per role, so per app (`db_role`) | Monitor Activity → Top SQL |
22
+ | **pg_stat_statements** | Which SQL is slow or called most, per role, so per app (`db_role`) | Monitor Activity → Top SQL; SQL Workshop → SQL Monitor (Top SQL, a statement's totals) |
23
23
  | **tablefunc** | `crosstab()` for pivot tables | Interactive report *Pivot* |
24
24
  | **ltree** | Hierarchies (org charts, categories, menus) with fast subtree queries | Tree region, `CONNECT BY` |
25
25
  | **postgres_fdw**, **file_fdw** | Query another PostgreSQL database, or a CSV file, as if it were a table | Remote servers / database links |