@hublo/sentinel 1.5.0-alpha.8 → 1.5.0

package/docs/sql-verification-study.md

@@ -0,0 +1,401 @@
+# Verifying our SQL: the study behind the role
+
+Research record for the SQL guardrails work (HUB-45374), written 2026-10-07. It is **not a
+specification of an implementation**, it is the record that decides what to implement: what was
+measured, what was decided, and the reason attached to each decision, so that the next person can
+disagree with a reason rather than with a preference.
+
+Every figure was measured on the monorepo on `origin/main` on 2026-10-07, with Prisma 6.19.3 and
+`libpg-query` 16.7.3. **Every figure can be re-run, once the harness is rebuilt**: it is kept as an
+instrument in pull request #89, which is deliberately not merged. A closed pull request's diff stays
+readable, and its `experiments/sql-chain/README.md` lists the prerequisites and the commands, with
+two gaps found since and recorded in section 4: the generated client is built locally rather than
+committed, and one prerequisite is missing from the list.
+
+---
+
+## 0. The question
+
+Catch a dangerous SQL query before it reaches production, the way lint catches a code defect. At
+the moment the code is written, not after a deploy, and therefore without a database holding data.
+
+---
+
+## 1. Two populations that do not add up
+
+The repository's SQL lives in two worlds that never overlap, counted in different units. A
+percentage computed in one does not compare to a percentage computed in the other.
+
+| world                 | what is counted                                  | unit   | total |
+| --------------------- | ------------------------------------------------ | ------ | ----: |
+| SQL inside TypeScript | places where code queries the database           | a call | 7 181 |
+| `.sql` files          | files on disk: migrations, and the few read live | a file | 1 045 |
+
+Verified: **no ORM call sits in a `.sql` file.** A third place belongs to neither: **125 objects**
+installed in the database by migrations, which run without any code naming them.
+
+---
+
+## 2. Four regimes of visibility
+
+Where SQL is stored says little. What matters for a check is whether its text can be read without
+running the code.
+
+| regime                           |          volume | what a static tool sees |
+| -------------------------------- | --------------: | ----------------------- |
+| raw SQL with a constant skeleton |       221 files | the whole text          |
+| raw SQL composed at run time     |        48 files | fragments only          |
+| SQL generated by the ORM         | **6 909 calls** | **nothing**             |
+| SQL living in the database       | **125 objects** | only old migrations     |
+
+Extracting a tagged template is exact, not approximate: taking the literal parts and numbering the
+holes `$1`, `$2` reproduces what Prisma would send, because the interpolations become bound
+parameters. `Prisma.raw` is the exception, since it injects text rather than a parameter.
+
+**The files are not where one expects.** 1 045 `.sql` files, of which **1 036 are migrations** and
+**9 are read at run time** (three of those under `docs/`). What the migrations installed still runs
+every day: **63 triggers and 62 functions** by distinct name, plus 418 indexes and 320 unique
+indexes. **No views**: an earlier count said six, and the pattern was matching a table named
+`user_grid_view` and its indexes.
+
+**The hard case.** `network`'s query builder: **1 986 lines of TypeScript, 26 filter strategies**,
+each deciding independently whether it applies. The set of applied strategies is an arbitrary
+subset, so the skeleton space is a lattice, not a list. This disqualifies "freeze the generated SQL
+into versioned `.sql` files": a lattice cannot be enumerated, only sampled.
+
+---
+
+## 3. Stock, flow, and per pull request
+
+A write-time check never reads the whole repository. It reads what changed. Three questions, three
+answers, and only the third says whether a tool would earn its place.
+
+|                                           |    ORM |        raw |
+| ----------------------------------------- | -----: | ---------: |
+| the stock, what the code contains         | 96,2 % |      3,8 % |
+| the flow, lines added over 90 days        | 91,5 % |      8,5 % |
+| per PR, among those touching the database | 75,6 % | **24,4 %** |
+
+Measured over **2 886 real pull requests** (first-parent commits, 90 days): 2 481 add no query at
+all, 306 add ORM calls only, 25 add raw SQL only, 74 add both. **405 touch the database (14,0 %),
+and 99 of those (24,4 %) add raw SQL.**
+
+> ⚠️ Method correction worth recording: an earlier pass counted merge commits. Only **79 of the
+> last 400** first-parent commits are merges, the rest are squashes, and the merges are mostly
+> back-merges with huge diffs. Counting them measured the wrong population.
+
+---
+
+## 4. The chain, measured end to end
+
+A first benchmark measured only compilation, on eleven queries **transcribed by hand**. That is not
+a benchmark of a tool, it is a benchmark of a path a human had already cleared. The two missing
+links were written and everything re-run from files on disk.
+
+| link                  |     time | note                                  |
+| --------------------- | -------: | ------------------------------------- |
+| 1, read and translate | 14,80 ms | paid once per **file**, not per query |
+| 2, compile            |  3,77 ms | 25 statements produced                |
+| 3, parse and check    |  1,12 ms | 0 parse failure                       |
+
+**The link that had to be written costs 3,9 times the one that already existed.** On the ten
+queries that pass, the SQL is **identical byte for byte** to the hand transcription. The eleventh
+is refused: a ternary in its `where`, and a placeholder cannot stand in for an unknown predicate.
+
+### Real coverage
+
+Over the whole `network` microservice, production code only:
+
+|                                        |   sites |       translated |         compile |
+| -------------------------------------- | ------: | ---------------: | --------------: |
+| reads (`find*`, `count`, `aggregate`)  |     242 |     215 (88,8 %) | **215 (100 %)** |
+| writes (`create`, `update`, `delete*`) |     231 |     161 (69,7 %) |             160 |
+| **total**                              | **473** | **376 (79,5 %)** |         **375** |
+
+**79,5 %, against the 90,5 % a regex estimate predicted.** And a fact no estimate would have
+produced: **no production Prisma call is fully literal. Zero out of 473.** Everything the chain
+handles is an approximation. Refusals fall 65 % on write positions.
+
+### Cost per pull request
+
+Ten real merged pull requests replayed, files read at the merge commit, external timing:
+
+|                  | whole touched file |  diff only |
+| ---------------- | -----------------: | ---------: |
+| queries examined |              2 778 |        114 |
+| **median time**  |         **0,73 s** | **0,71 s** |
+| worst case       |             1,20 s |     0,75 s |
+| compile failures |         **32,8 %** |      6,9 % |
+
+**The trade-off is not paid in seconds, it is paid in noise.** The wide mode costs the same time
+for 24 times more queries, and returns five times more failures because touched files carry specs
+and test factories. Both modes have their case: a PR making a column nullable can break a query the
+diff never touched; a 361-file PR that changes no query line makes the wide mode check 1 153
+queries for nothing. **Not decided here.**
+
+**A fixed cost is paid once per process, before a single query is examined, and it is module
+loading rather than the interpreter.** Bare Node starts in **33 ms** here, median of 9 runs, while
+a single `typescript` import costs **122 ms**, and a reading of this document measured 127 ms
+independently. The SQL parser and the generated client load on top of that. If the compiling client
+must be generated, add 2,5 s.
+
+> The 122 ms is the `typescript` the monorepo root actually **resolves**, and that is not the
+> version its `devDependencies` names: `dependencies` aliases the name onto
+> `@typescript/typescript6` 6.0.2, which is what the symlink points at, while `devDependencies`
+> still asks for 5.9.3. Here it changes nothing, and that was worth measuring rather than
+> assuming: 5.9.3 loads in 129 ms against 122, ranges overlapping, so the two are indistinguishable
+> at this precision. The declaration is a trap for a reader re-measuring, not a correction to make.
+
+> ⚠️ **An earlier revision of this section printed "390 ms of the total is Node starting up, 71 %
+> of the clock", and both halves were wrong.** The attribution is off by a factor of twelve, as
+> above. And neither figure can come from the run that produced the medians in the table: that
+> script reports no startup cost at all and starts its own clock after its imports, so the
+> overhead is outside every number it prints. 390 ms over the 0,73 s median would be 53 %, not
+> 71 %; the digits 71 match the diff-only median in the table rather than any ratio. The total
+> overhead is therefore withdrawn rather than corrected. Re-measuring it means **rebuilding** the
+> harness and not merely re-running it: the scripts are in #89, but the generated client and the
+> compiler the translator loads are built locally and committed nowhere.
+
+> ⚠️ And one prerequisite of #89 is missing from its own README, found while tracing the figure
+> above: `translate.mjs` resolves `./ts-lib/lib/typescript.js`, a compiler copy sitting beside the
+> scripts, which the four listed prerequisites never mention. A reader following the README stops
+> on a missing module, and the version that produced link 1 is unrecorded. What that costs is
+> bounded and worth naming: the translation timing and the 79,5 % coverage both pass through that
+> compiler, so both are reproducible in method and not to the digit.
+
+### Build cost
+
+| component              | lines of code |
+| ---------------------- | ------------: |
+| the translator, link 1 |       **463** |
+| the schema parser      |            73 |
+| the SQL parse and rule |        **43** |
+
+**The rule costs 43 lines, the translator eleven times more.** Six defects were found in it **by
+measurement, not by reading**. On the eleven starting queries it looked perfect; the 473 call sites
+of `network` revealed the rest.
+
+---
+
+## 5. The result that flips the priority
+
+|                            | ORM SQL                        | raw SQL                             |
+| -------------------------- | ------------------------------ | ----------------------------------- |
+| volume                     | 6 909 calls                    | 272 calls in 129 files              |
+| extracting the query       | **463 lines**, 79,5 % coverage | the text is **already in the file** |
+| the 19 May pattern appears | **never observed**, 0 of 2 319 | **yes**, that is where it happened  |
+| presence per PR            | 75,6 % of database PRs         | **24,4 %**                          |
+
+**The difficult road was built toward the half where the incident cannot be seen.** Raw SQL is both
+the easiest to extract and the only place the pattern currently lives.
+
+> ⚠️ **What that zero does not say.** It is an observation on our query shapes, not a property of
+> Prisma. Checked in the compiler binary: `UNION` and `UNION ALL` are in its string table, next to
+> `EXISTS` and `NOT EXISTS`. **A code path exists.** Which query shape triggers it was not
+> investigated. A rule there would be silent today, not useless tomorrow.
+
+---
+
+## 6. Why raw SQL is the sensitive place
+
+Three supports hold, of three different kinds. A fourth was tested and does not hold.
+
+| support                                                            | kind                 |
+| ------------------------------------------------------------------ | -------------------- |
+| the documented incident came from it, both file versions recovered | verified             |
+| it loses the schema check the ORM gives for free                   | structural           |
+| composed, it has no text before execution: 48 files                | measured             |
+| but it does not attract more fixes: 35,8 % against 33,0 %          | **tested, negative** |
+
+> ⚠️ Those last two figures count a **wider** population than the rest of this record: they take
+> as hand-written SQL the `$queryRaw` and `$executeRaw` calls, their `Unsafe` variants, **and also
+> `Prisma.sql` and `Prisma.raw`**, which are fragments. That is why the share moves from 24,4 % to
+> about a third, and the two numbers must not be read against the ones in section 3. The
+> fix-versus-feature comparison itself uses the same definition on both sides, so it stands.
+
+So the defensible claim is not that hand-written SQL breaks more often. It is that it is **the only
+place this class of defect can live, and the only place review has nothing to look at.**
+
+---
+
+## 7. A second family, found on the way
+
+The 19 May incident was about performance. Security is a different family and it lives elsewhere.
+Of **25 calls to the `Unsafe` variants**, 13 insert a value into the SQL text. **Twelve are test
+utilities. One is production code**, in the authentication library, where permission names are
+placed between quotes in the query text with no escaping. The path from the HTTP body down to the
+query carries **no runtime validation**: the DTO accepts any string, the use case checks nothing,
+and the entity's type is erased at run time.
+
+**Triage: real defect, not an emergency.** The endpoint is behind `SuperAdminAuthGuard`. That
+removes the urgency, not the defect: an injection reaches the whole database on that connection and
+turns an account compromise into a database compromise. The fix is small: validate against the
+permission enumeration, and bind parameters instead of inserting text.
+
+> ✅ **Fixed, and the speed is part of the finding.** `4b6c272`, HUB-45606, pull request #22178,
+> five days after the reading that found it: the `$executeRawUnsafe` became a tagged `$executeRaw`
+> over `jsonb_to_recordset`, the payload bound as a single parameter. The case is kept here with
+> its resolution rather than deleted, because it is the evidence for the paragraph below: a defect
+> of this family was found by reading 25 call sites, and closed in a week. Raised in review by
+> Amrou Ouni, whose remark also exposed that the local checkout this record was last re-read
+> against was **15 days behind** `main`.</br>
+> ⚠️ A record that quotes the state of the code has a shelf life. Every figure here is dated
+> 2026-10-07, and none of them is re-verified by this note.
+
+This family is **far cheaper to detect** than performance: a value inserted into an `Unsafe` call is
+visible in the TypeScript, with no compilation, no SQL parsing and no database. The perimeter is
+finite: 25 sites.
+
+---
+
+## 8. What is still out of reach
+
+| limit                                                                                            | measured or not  |
+| ------------------------------------------------------------------------------------------------ | ---------------- |
+| no database, so no `EXPLAIN`: the chain checks shape, never cost                                 | by construction  |
+| one schema of the **33** that carry a generator: 54,3 % of candidate PRs have no compilable site | measured         |
+| the cost of generating the 31 clients the chain does not build: no duration is claimed, see 11   | not measured     |
+| raw SQL is invisible to the chain: 272 calls in 129 files                                        | measured         |
+| false positives of the detector need a real TypeScript type checker                              | partially        |
+| equivalence with production SQL proven on 10 queries, not on 375                                 | measured on 10   |
+| the rule's false negatives: nobody measured what it lets through                                 | **not measured** |
+| everything ran single-process on a laptop: nothing in CI, nothing parallel                       | not measured     |
+| the fixed per-process overhead: the harness must be rebuilt before it can be timed               | **withdrawn**    |
+
+---
+
+## 9. The nine tools examined
+
+Three questions per tool, not to be conflated: **what it checks**, **the input it takes**, and
+**what it requires to run**. A tool can check the right thing and remain unusable because it cannot
+be fed.
+
+| tool              | checks                                                     | input                | database                                        | state                                           |
+| ----------------- | ---------------------------------------------------------- | -------------------- | ----------------------------------------------- | ----------------------------------------------- |
+| **squawk**        | 54 rules, **100 % migrations**. No query rule              | `.sql` files         | none                                            | npm 2.67.0, 2026-10-04, Rust, Apache-2.0        |
+| **libpg-query**   | nothing. It turns SQL into a tree: PostgreSQL's own parser | SQL                  | none                                            | active, **WASM only**, one version per PG major |
+| **plpgsql_check** | **the body of PL/pgSQL functions**, triggers included      | functions in the DB  | **inside the server**                           | 782 stars, C, MIT, 2026-10-05                   |
+| **SafeQL**        | a query against the schema: columns, types                 | **source code**      | **yes, always**, shadow DB even with migrations | active, 2026-10-01, MIT                         |
+| **sqlcheck**      | 26 generic query anti-patterns                             | SQL                  | none                                            | **dormant** since 2024-02, C++                  |
+| **SQLFluff**      | 73 rules of **style and structure**                        | SQL text             | none                                            | active, Python, MIT                             |
+| **pglinter**      | 21 global health rules with thresholds                     | system catalogs      | **inside the server**                           | 170 stars, Rust, 2026-07                        |
+| **pgTAP**         | **assertions**. It analyses no body                        | hand-written asserts | **inside the server**                           | 1 167 stars, active                             |
+| **Bytebase**      | a rule engine with real textual anti-patterns              | its own workflow     | varies, `EXPLAIN` for some                      | 14 537 stars, Go, MIT except enterprise         |
+
+### Proof by execution
+
+Two queries written with a defect no index fixes, passed to the only two tools that run without a
+database:
+
+| tool           | verdict on those two queries                                    |
+| -------------- | --------------------------------------------------------------- |
+| squawk 2.67.0  | **0 finding.** It parses them and no rule applies               |
+| SQLFluff 4.1.0 | 10 findings, **all style or structure**. Silent on both defects |
+
+**So the answer to the study's central question is no, and it is now demonstrated rather than
+assumed: no tool judges a query's cost outside a database.**
+
+### What sqlcheck's catalogue teaches, even though we drop the tool
+
+Four of our six post-mortem patterns have an equivalent there; **pagination by `OFFSET` and
+repeated computation have none**. In the other direction, five of its rules are worth taking:
+_String Concatenation_ (exactly the security family above), _SELECT \*_, _ORDER BY RAND_,
+_Eliminate Unnecessary DISTINCT_, _Implicit Column Usage_.
+
+### The third regime
+
+`plpgsql_check` really analyses PL/pgSQL function bodies, triggers included. **77 of our 78 function
+declarations are `LANGUAGE plpgsql`**, so it would cover essentially all of them. But it is a server
+extension: it does not run on a pull request, it runs against a database where migrations are
+already applied. And there is **no view to cover**, the earlier count was a false positive.
+
+> Only squawk, SQLFluff and libpg-query were actually executed. pglinter, plpgsql_check and pgTAP
+> require a server with their extension, which was not set up.
+
+---
+
+## 10. What we take, what we write, what we leave
+
+**No overlap.** Each tool covers an area the others do not see, and two areas are covered by
+nobody. Those are what sentinel writes.
+
+| what                                        | category    | do we take it       | where it runs                             |
+| ------------------------------------------- | ----------- | ------------------- | ----------------------------------------- |
+| **squawk**                                  | third-party | **yes, as is**      | in sentinel, on the diff's migrations     |
+| **plpgsql_check**                           | third-party | **yes, as is**      | **outside sentinel**, against a database  |
+| **libpg-query**                             | library     | yes, as the engine  | inside our rules                          |
+| the rule on `Unsafe` calls                  | our rules   | **to write**        | in sentinel, on the diff's TypeScript     |
+| raw SQL extraction and its rules            | our rules   | **to write**        | in sentinel, on the diff's TypeScript     |
+| the full ORM chain                          | our rules   | **to write, later** | in sentinel, on the diff's TypeScript     |
+| SafeQL, sqlcheck, SQLFluff, pglinter, pgTAP | third-party | **no**              | sqlcheck's catalogue still inspires rules |
+
+Two near-misses, which are complements rather than duplicates. **Injection appears twice, in two
+places**: `plpgsql_check` sees it inside a stored function, our TypeScript rule sees it in
+application code, and neither sees the other. And **we could rewrite squawk with libpg-query**,
+which is the only real duplication possible and precisely what not to do.
+
+**plpgsql_check does not go into sentinel.** It is an extension living in the server, analysing
+functions already installed. Adopting it means deciding against which database and how often it
+runs, which is an infrastructure question. It belongs with `pg_stat_statements`.
+
+### The order, cheapest first
+
+1. **squawk on migrations.** The only gain that requires writing nothing.
+2. **The rule on `Unsafe` calls.** No compilation, no SQL parsing, no database. 25 sites.
+3. **Raw SQL with libpg-query.** Where the 19 May pattern lives, and the text is already in the file.
+4. **The ORM chain.** 463 lines for 79,5 % coverage, and silent today. Last.
+
+Outside sentinel, in parallel: **plpgsql_check**, to scope with infrastructure.
+
+---
+
+## 11. How it fits into sentinel
+
+The role inherits everything the existing roles settled: a target declared in `package.json`,
+adoption through `--init` with declared settings surviving a re-init, a pure adapter with the engine
+owning the IO, banned patterns in the `nest` preset, permissive while adoption runs, and a column on
+the migration status page.
+
+**One thing is new.** Every current role only reads files from the repository. The ORM part needs a
+**built artifact**: a Prisma client generated in compiler mode, one per schema.
+
+### In CI
+
+1. the changed files of the pull request
+2. does any of them contain a query: a text scan, and **86 % of pull requests stop here**
+3. which schema, resolved from the module's import
+4. that schema's compiling client, from cache
+5. compile the changed queries
+6. check the SQL, and attach each finding to the source line that produced it
+
+The intermediate files are those compiling clients, **one per schema carrying a `generator`, of
+which this repository has 33 of 52 `.prisma` files**: 20 under `libs/nest/db-*`, 12 in
+microservices, and one Backstage template that carries a `generator` but no `model` and an
+unresolved `${{ values.dbName }}` output, so it is not a client at all. ⚠️ It matched the count
+because its own path contains `apps/nest/microservices/`. That leaves **32 real clients**, of which
+the chain already builds one, so 31 would have to be generated. An earlier draft said 22 and no
+count supports it.
+
+⛔ The total those 31 would cost is **not** stated here, and the earlier "about 80 seconds" is
+withdrawn rather than adjusted to 78. It rested on a flat 2,5 s per client, which is the one figure
+in section 4 carrying no basis while its neighbours carry theirs, and which the section 7 table
+already marks `not measured`. A flat average cannot carry it either way: these schemas run from
+**218 models and 4 828 lines** (`main-db`) down to **1 model and 23 lines**, a factor of 210.
+Caching the generation is argued below on the clients' number and staleness, not on a duration. They are not committed:
+generated code carrying megabytes of WebAssembly, stale as soon as the schema moves. They are
+cached, with the `.prisma` file as the declared input.
+
+> ⚠️ The existing scripts already generate the **runtime** clients into
+> `node_modules/@prisma/<name>-database`, and those run in production with the Rust engine. The
+> compiling client must not overwrite them; it needs its own output.
+
+---
+
+## Sources
+
+- The harness and every script: pull request #89, `experiments/sql-chain/`
+- `pganalyze/libpg_query`: <https://github.com/pganalyze/libpg_query>
+- squawk: <https://squawkhq.com/> · plpgsql_check: <https://github.com/okbob/plpgsql_check>
+- SafeQL: <https://safeql.dev/> · SQLFluff: <https://www.sqlfluff.com/>
+- Prisma without the Rust engine:
+  <https://www.prisma.io/docs/orm/v6/prisma-client/setup-and-configuration/no-rust-engine>