upkeep-rails 0.2.5-aarch64-linux-gnu

data/docs/spikes/SQLGLOT_VS_AREL_FINDINGS.md

@@ -0,0 +1,136 @@
+# SQLGlot-first vs Arel, and missing semantic bindings
+
+## Outcome
+
+A SQL-first decoder is viable enough to pursue. It can use one policy for both
+Active Record-generated SQL and handwritten SQL, while lowering both into a
+small generic dependency graph. The spike handles the cases where the current
+Arel decoder succeeds and cases it intentionally rejects: raw predicates, raw
+joins, correlated `EXISTS`, and CTEs.
+
+This is not yet a drop-in replacement. The current Ruby gem exposes only the
+parser-shaped JSON AST. The underlying Rust library has useful semantic APIs,
+but v0.10.12 also has correctness gaps that Upkeep must test and either fix
+upstream or normalize locally.
+
+## Comparison
+
+| Concern | Current Arel decoder | SQLGlot-first spike |
+| --- | --- | --- |
+| Hash/Arel predicates | Structural, typed nodes | Parsed from final SQL |
+| Raw predicate/order | Rejected as opaque | Analyzed |
+| Raw join/source | Rejected as opaque | Analyzed |
+| Correlated subquery | Becomes opaque when handwritten | Inner/outer columns and equality edge found |
+| CTE | Depends on Arel shape | Physical CTE sources found; CTE name suppressed |
+| Simple predicate values | Preserves Ruby values/binds | Recovers SQL literal values |
+| Bind type fidelity | Strong | Lost after `to_sql` unless passed separately |
+| Parser dependency | Rails/Arel internals | Native SQLGlot dependency |
+| Decoder policy | Structured vs opaque branches | One SQL policy with explicit confidence/warnings |
+
+The generic result is deliberately not a catalog of SQL features:
+
+```text
+physical source ── referenced column
+        │
+        ├── equality edge ── other source.column
+        └── constant predicate (optional DNF group)
+
+query shape: limit / offset / distinct / group / having
+```
+
+SQL constructs belong in the decoder. `DependencyPlan` should only consume the
+graph above plus confidence/coverage. That keeps `Linked` source kinds generic.
+
+## Ruby bindings worth adding
+
+The Ruby `sqlglot` gem 0.1.1 currently binds parse, generate, transpile, and
+version. The standalone binding proves these existing `sql-glot-rust` APIs are
+directly useful:
+
+1. `MappingSchema` + `qualify_columns`
+   - resolves unqualified columns using Active Record's schema;
+   - expands wildcards;
+   - makes alias handling less dependent on a Ruby AST walker.
+2. `build_scope`
+   - exposes sources per query scope;
+   - separates child subqueries and CTE scopes;
+   - exposes all columns, including WHERE/JOIN columns;
+   - marks external columns and correlated subqueries.
+3. `lineage`
+   - useful for selected/output values and derived expressions;
+   - **not** sufficient for invalidation because filter-only and join-only
+     sources are intentionally absent.
+
+One JSON-returning `analyze` FFI entry point is preferable to many fine-grained
+calls: parse, qualify, build scopes, and compute requested output lineages in
+Rust, then cross the FFI boundary once.
+
+## Challenging observations
+
+- Correlated `EXISTS` works: the inner scope is correlated, includes
+  `efforts.milestone_id` and `efforts.status`, and reports
+  `milestones.id` as an external column.
+- Output lineage for `SELECT milestones.id ... EXISTS (...)` contains
+  `milestones` but not `efforts`. The scope tree contains both. Upkeep should
+  derive invalidation inputs from scopes/predicates, not output lineage.
+- Schema qualification resolves unqualified columns and expands selected
+  columns.
+- CTE qualification works, and the CTE child scope points at its physical
+  table. However, v0.10.12 currently represents the root reference to the CTE
+  as `Source::Table("active_cards")`, not `Source::Scope`. This needs an
+  upstream fix or a narrowly tested normalizer.
+- Parse failures cross the proposed FFI boundary as structured JSON errors,
+  rather than null pointers with lost diagnostics.
+
+The earlier Pulse #1802 corpus remains relevant: parsing covers operator
+expressions, functions, nested queries, aliases, and raw SQL more uniformly
+than class-by-class Arel handling. It does not remove the need for semantic
+tests around scope resolution and Rails-emitted dialect details.
+
+## Performance
+
+Warm macOS arm64 measurements over 5,000 analyses:
+
+| Analyzer | Mean per analysis |
+| --- | ---: |
+| Arel, structured query | 18.3–18.6 µs |
+| SQLGlot, same structured query | 47.7 µs |
+| SQLGlot, raw join + predicate + order | 89.9–90.3 µs |
+
+The SQL-first path is about 2.6× slower on the comparable structured query, but
+still below 0.1 ms in this corpus. Schema metadata was cached; without that
+cache, reflection dominated at roughly 0.6 ms.
+
+This analysis should be cached by normalized query shape and schema version. It
+should not run once per database change or once per dependent render during
+fan-out. The fan-out cost is then graph matching, not SQL parsing.
+
+The standalone release dylib is 3.1 MB on this machine. The complete local Cargo
+release directory is larger (88 MB) but is build output, not shipped runtime.
+
+## Recommendation
+
+Proceed with a production-shaped SQL-first prototype behind the existing
+`ActiveRecordQuery.analyze` contract:
+
+1. Add the semantic `analyze` binding to the Ruby gem (preferably upstream).
+2. Feed it schema data from Active Record's schema cache.
+3. Lower its result into one generic dependency graph.
+4. Keep Arel temporarily as a parity oracle and as the source of bind values/type
+   metadata, not as a second invalidation policy.
+5. Run the full query corpus against both decoders and record false-negative,
+   conservative, and unsupported results.
+6. Remove the Arel decoder only after SQL-first parity and performance targets
+   are met.
+
+The key architectural distinction is “two decoders during migration,” not “two
+policies forever.”
+
+## Source references
+
+- [`sql-glot-ruby` v0.1.1 native bindings](https://github.com/AccountAim/sql-glot-ruby/blob/v0.1.1/lib/sqlglot/native.rb)
+- [`sql-glot-rust` v0.10.12 scope analysis](https://github.com/protegrity/sql-glot-rust/blob/v0.10.12/src/optimizer/scope_analysis.rs)
+- [`sql-glot-rust` v0.10.12 column qualification](https://github.com/protegrity/sql-glot-rust/blob/v0.10.12/src/optimizer/qualify_columns.rs)
+- [`sql-glot-rust` v0.10.12 lineage](https://github.com/protegrity/sql-glot-rust/blob/v0.10.12/src/optimizer/lineage.rs)
+- [Active Record 8.1.3 query methods](https://github.com/rails/rails/blob/v8.1.3/activerecord/lib/active_record/relation/query_methods.rb)
+- [Arel 8.1.3 bound SQL literals](https://github.com/rails/rails/blob/v8.1.3/activerecord/lib/arel/nodes/bound_sql_literal.rb)

data/docs/spikes/query-source-analysis-comparison/FINDINGS.md

@@ -0,0 +1,169 @@
+# Opaque-query source analysis comparison
+
+Date: 2026-07-24
+
+## Decision
+
+Use the Ruby `sqlglot` gem as Upkeep's first automatic fallback when Arel
+contains opaque SQL. Keep an Upkeep-owned, scope-aware AST walker and extract
+tables only.
+
+Do not add user-selectable query policies. The runtime should follow one
+resolution chain:
+
+1. Use Arel for precise column and predicate coverage.
+2. Use SQLGlot for conservative named-table coverage.
+3. Resolve views through adapter schema metadata where practical.
+4. Refuse the boundary when a source remains hidden or unresolvable.
+
+Database-native inspection can improve individual adapters later, but it should
+not be the primary cross-adapter mechanism.
+
+## Corpus
+
+The named-source corpus has 12 queries across PostgreSQL, MySQL, and SQLite:
+
+- raw predicates and order expressions;
+- raw and schema-qualified joins;
+- correlated `EXISTS` and `IN` subqueries;
+- derived tables;
+- ordinary, shadowing, and recursive CTEs;
+- PostgreSQL full-text search; and
+- MySQL JSON expressions.
+
+A separate four-case physical-source corpus tests sources hidden behind views
+and SQL functions.
+
+The correctness rule is strict: an approach fails if it omits any physical
+table that could affect the result. Extra tables are conservative but still
+recorded because they increase fanout.
+
+## Results
+
+Benchmarks are local single-process measurements on Apple Silicon. They compare
+orders of magnitude, not production latency.
+
+| Approach | Named-source correctness | Local cost | Portability | Finding |
+| --- | ---: | ---: | --- | --- |
+| Ruby SQLGlot 0.1.1 | 12/12 | ~67 μs/parse | PostgreSQL, MySQL, SQLite; CRuby native gem | Best overall trade-off |
+| `sqlparser-rs` 0.62.0 | 12/12 | ~21 μs/parse | Multi-dialect Rust library | Fastest, but Upkeep would own a Ruby binding and native release matrix |
+| Python SQLGlot 27.20.0 | 12/12 | ~501 μs/parse | Multi-dialect, separate Python runtime | Correct but operationally inappropriate for a Rails gem |
+| `pg_query` 6.2.2 | 4/4 PostgreSQL | ~104 μs/parse | PostgreSQL and CRuby only | Mature and exact for PostgreSQL, but does not solve adapter portability |
+| PostgreSQL 17.9 `EXPLAIN (FORMAT JSON)` | 4/4 PostgreSQL | ~82 μs/explain | PostgreSQL connection required | Correct baseline; expands views and stable SQL functions |
+| SQLite 3.53 `EXPLAIN QUERY PLAN` text | 5/6 SQLite | ~12 μs/explain | SQLite only | Unsafe: reports aliases instead of physical tables |
+| SQLite authorizer during prepare | 6/6 SQLite | ~15 μs/prepare | SQLite only | Correct baseline and exposed by the existing Ruby `sqlite3` gem |
+
+MySQL `EXPLAIN FORMAT=JSON` was not executed because no local MySQL server was
+available. Adding it would still leave Upkeep with three adapter-specific
+implementations and three different result formats.
+
+## Important failures
+
+### Convenience metadata is not a correctness API
+
+The Ruby SQLGlot wrapper's `Sqlglot::Query#tables` missed tables inside
+correlated subqueries, `IN` subqueries, and derived tables. Its underlying AST
+contained all sources. The scope-aware recursive extractor in the original
+SQLGlot spike recovered all 12 cases.
+
+### Syntax parsers cannot expand database objects
+
+All four parsers return the named view or table function, not the physical
+tables behind it:
+
+| Query source | Parser result | Physical dependency |
+| --- | --- | --- |
+| SQLite `published_posts` view | `published_posts` | `posts` |
+| PostgreSQL `active_accounts` view | `active_accounts` | `accounts` |
+| PostgreSQL `account_events(42)` | no table, or the function name | `audit.events` |
+
+This is expected: a parser has SQL text but not the connected database schema.
+
+### Query plans are useful but not complete
+
+PostgreSQL JSON `EXPLAIN` expanded:
+
+- the `active_accounts` view to `accounts`; and
+- an inlineable `STABLE` SQL function to `audit.events`.
+
+It did not expose the table read by an otherwise identical `VOLATILE` SQL
+function. The plan contained a function scan with no underlying relation.
+Therefore `EXPLAIN` cannot be treated as a universal proof of physical sources.
+
+SQLite's textual plan reported `physical_posts`, a query alias, instead of the
+real `posts` table. Its authorizer callback reported the physical table
+correctly, but installing and restoring a connection-global authorizer safely
+inside Rails requires concurrency and reentrancy work.
+
+## Dependency and packaging comparison
+
+- Ruby SQLGlot ships precompiled glibc Linux and macOS gems for x86-64 and
+  ARM64. Other platforms require Cargo and Git. The current arm64 macOS gem
+  includes a roughly 2.4 MB dylib.
+- `pg_query` compiled a roughly 3.4 MB native extension locally and embeds the
+  real PostgreSQL parser.
+- `sqlparser-rs` produced a roughly 7.1 MB standalone release executable in the
+  spike. A production choice would require a maintained Ruby extension or FFI
+  boundary.
+- Python SQLGlot adds an entire second language runtime to a Rails process or
+  requires an out-of-process service.
+
+The Ruby SQLGlot wrapper is young. Upkeep should pin its supported version and
+keep contract tests for the AST node shapes it consumes.
+
+## Proposed production contract
+
+The fallback analyzer should return one of:
+
+```ruby
+Resolved.new(tables: ["posts", "users"], coverage: :tables)
+Unresolved.new(reason: "table-valued function account_events")
+```
+
+It must never silently convert an empty or partially understood source set into
+the model's primary table. Specifically:
+
+- normalize `public.accounts` and `main.posts` to the table names used by
+  Active Record change events;
+- exclude CTE and derived-table aliases with query-scope awareness;
+- validate extracted names against tables and views in the connected schema;
+- expand views through adapter metadata or mark them unresolved;
+- detect table-valued functions and other source nodes that do not resolve to
+  physical tables;
+- mark every SQLGlot-derived collection non-appendable; and
+- always replay the whole render site for table-coverage dependencies.
+
+This leaves refusal as a correctness outcome, not a configurable policy:
+ordinary raw SQL works automatically, while genuinely hidden database behavior
+is rejected rather than guessed.
+
+## Reproduction
+
+Each directory is intentionally isolated from production dependencies:
+
+```sh
+# Ruby SQLGlot
+cd docs/spikes/query-source-analysis-comparison/ruby-sqlglot
+mise exec -- ruby -S bundle install
+mise exec -- ruby -rbundler/setup run.rb
+
+# pg_query
+cd docs/spikes/query-source-analysis-comparison/pg-query
+mise exec -- ruby -S bundle install
+mise exec -- ruby -rbundler/setup run.rb
+
+# Python SQLGlot
+python3 -m venv /tmp/upkeep-python-sqlglot
+/tmp/upkeep-python-sqlglot/bin/pip install -r python-sqlglot/requirements.txt
+/tmp/upkeep-python-sqlglot/bin/python python-sqlglot/run.py
+
+# sqlparser-rs
+cd docs/spikes/query-source-analysis-comparison/sqlparser-rs
+mise x rust@stable -- cargo run --release
+
+# SQLite native approaches
+python3 docs/spikes/query-source-analysis-comparison/explain/sqlite.py
+```
+
+The PostgreSQL runner expects a clean database through `DATABASE_URL`; it
+creates schemas, tables, a view, and two functions in that database.

data/docs/spikes/sqlglot-first-active-record/README.md

@@ -0,0 +1,26 @@
+# SQLGlot-first Active Record analysis spike
+
+This spike asks whether `Relation#to_sql` plus database schema metadata can replace
+the Arel decoder. The analyzer never calls `Relation#arel` and produces the same
+small semantic contract Upkeep needs:
+
+- physical tables and referenced columns;
+- simple constant predicates in DNF groups;
+- column-equality edges (joins and correlations);
+- limit and appendability shape.
+
+Run:
+
+```sh
+mise x ruby@3.4.7 -- ruby sqlglot_active_record_query_test.rb
+mise x ruby@3.4.7 -- ruby benchmark.rb
+```
+
+The important comparison is not “can SQLGlot understand every SQL node?” It is
+whether both structured Active Record and raw SQL lower into this generic contract.
+The tests include raw predicates, raw joins, a correlated `EXISTS`, a CTE inside a
+derived source, and collection-shape modifiers.
+
+The analyzer caches schema metadata by connection for the benchmark. A real
+integration should key that cache by Active Record's schema version and clear it
+when Rails clears its schema cache.

data/docs/spikes/sqlglot-query-analysis/FINDINGS.md

@@ -0,0 +1,110 @@
+# SQLGlot query-analysis spike
+
+Date: 2026-07-24
+
+## Question
+
+Can Upkeep use the `sqlglot` Ruby gem to recover table dependencies from
+Active Record relations whose Arel nodes contain opaque SQL strings?
+
+## Verdict
+
+Yes, with one important constraint: use SQLGlot's parsed AST and a small
+Upkeep-owned recursive table extractor. Do not use
+`Sqlglot::Query#tables` as the correctness boundary.
+
+The parser accepted every representative PostgreSQL, MySQL, and SQLite query
+in the original corpus. Scope-aware recursive extraction from `Table` AST
+nodes recovered every expected source table. The follow-up corpus derived from
+Pulse #1802 found one important PostgreSQL extension gap: pg_trgm's custom `<%`
+operator does not parse.
+
+## Evidence
+
+The corpus covers:
+
+- raw predicates and order expressions;
+- raw joins;
+- correlated `EXISTS` subqueries;
+- `IN` subqueries;
+- derived tables in `FROM` and `JOIN`;
+- CTEs;
+- CTEs that shadow a physical table;
+- recursive CTEs;
+- PostgreSQL full-text search;
+- PostgreSQL schema-qualified sources;
+- MySQL JSON expressions; and
+- SQLite quoting and syntax.
+
+Results:
+
+| Path | Result |
+| --- | --- |
+| SQLGlot parsing | 12/12 query cases parsed |
+| `Sqlglot::Query#tables` | 4 failures in the original 10-case corpus |
+| Scope-aware AST `Table` extraction | 12/12 table sets correct |
+| Invalid SQL | Raised `Sqlglot::ParseError` |
+| Local parse/extract benchmark | about 60 μs/query across 10,000 iterations |
+
+The [Pulse #1802 follow-up](PULSE_1802_FINDINGS.md) adds ten parseable,
+production-derived cases covering JSONB operators, nested boolean ranges,
+self-correlated and aggregate subqueries, repeated `EXISTS`, aggregate
+`CASE`, joins, projections, and pg_search expressions. It also preserves the
+realistic pg_search `<%` failure as an expected regression test.
+
+The wrapper helper missed physical tables inside correlated subqueries, `IN`
+subqueries, and derived tables. The underlying AST contained those tables in
+regular nested `Table` nodes, so a generic recursive walk recovered them.
+
+Run the spike:
+
+```sh
+cd docs/spikes/sqlglot-query-analysis
+mise exec -- ruby -S bundle install
+mise exec -- ruby -rbundler/setup sqlglot_query_analysis_test.rb
+```
+
+## Proposed Upkeep boundary
+
+Keep the existing Arel analyzer as the precise path. When it encounters an
+opaque SQL predicate, order, join, or source:
+
+1. Parse the relation's final SQL using the dialect selected from the Active
+   Record adapter.
+2. Recursively collect physical `Table` nodes.
+3. Remove CTE names from the collected set.
+4. Register table-coverage dependencies.
+5. Disable append, prepend, remove, and member-replace proofs.
+6. Re-run and update the entire render site when any collected table changes.
+
+Only table extraction should ship initially. SQLGlot exposes columns, but
+unqualified columns, aliases, projections, and derived outputs require semantic
+resolution. They are unnecessary for a correct conservative fallback.
+
+## Risks
+
+- The Ruby wrapper is new (`0.1.1`). Upkeep needs contract tests for the AST
+  node shapes it consumes.
+- Precompiled gems currently cover glibc Linux and macOS on x86-64 and ARM64.
+  Other platforms build the Rust library from source and require Cargo and Git.
+- A SQL parser sees named SQL sources, not tables hidden behind database views,
+  stored functions, or extensions. Those queries still need a database/schema
+  resolver or refusal.
+- PostgreSQL permits extension-defined operators. SQLGlot 0.1.1 rejects
+  pg_trgm's `<%` word-similarity operator even though it accepts the surrounding
+  pg_search tsearch and ranking expressions.
+- An extracted schema-qualified name must be normalized to the table names used
+  by Upkeep change events.
+- Parser success is not proof that every named source corresponds to an
+  observed Active Record table. Upkeep must validate extracted names against
+  the connected schema before registering them.
+
+## Recommendation
+
+Proceed with SQLGlot as an experimental fallback analyzer behind internal code,
+not a user-selectable policy. Keep refusal only as the terminal outcome when
+parsing fails or source names cannot be reconciled with Upkeep's observed
+schema.
+
+Before merging production integration, expand the corpus with SQL captured from
+the benchmark applications and run it on PostgreSQL, MySQL, and SQLite CI.

data/docs/spikes/sqlglot-query-analysis/PULSE_1802_FINDINGS.md

@@ -0,0 +1,95 @@
+# Pulse #1802 SQLGlot edge-case findings
+
+Date: 2026-07-24
+
+Source: [fetchly/Pulse#1802](https://github.com/fetchly/Pulse/pull/1802)
+
+## Result
+
+SQLGlot can eliminate most of the SQL-to-Arel and SQL-to-Ruby rewrites in this
+PR if Upkeep accepts conservative table-level dependencies and replays the
+whole render site.
+
+The Ruby SQLGlot 0.1.1 parser plus the scope-aware extractor handled all ten
+parseable Pulse-derived cases. One realistic `pg_search` query remains
+unparseable because SQLGlot does not recognize pg_trgm's custom `<%`
+word-similarity operator.
+
+The same minimal `<%` query also fails with Python SQLGlot 27.20.0. PostgreSQL's
+`@@` full-text-search operator, casts, `to_tsquery`, `ts_rank`, and the
+`word_similarity(...)` function all parse; `<%` is the isolated failure.
+
+Run the corpus:
+
+```sh
+cd docs/spikes/sqlglot-query-analysis
+mise exec -- ruby -rbundler/setup pulse_1802_edge_cases_test.rb
+```
+
+Observed result:
+
+```text
+13 runs, 15 assertions, 0 failures, 0 errors
+```
+
+The expected `<%` parse failure is asserted as part of that result.
+
+## PR coverage
+
+| PR query shape | Parse / table extraction | Tables recovered | What this means for Pulse |
+| --- | --- | --- | --- |
+| `Assignment.current`, `overlapping`, and `current_and_upcoming` nested date predicates | Full | `assignments` | The string predicates did not need Arel rewrites for conservative reactivity. |
+| `Effort::Filterable.not_overdue` and `not_design_efforts_types` disjunctions | Full | `efforts` | Ordinary `NULL`, comparison, `AND`, and `OR` predicates are routine. |
+| `Project::Staffable#requirements_change_summary` overlapping-range predicate | Full | `project_team_requirements` | This could stay in SQL instead of loading and filtering every requirement in Ruby. |
+| `Effort::Filterable.by_blockers` self-correlated `EXISTS` with an alias | Full | `efforts` | The extractor deduplicates the outer and aliased inner physical table. |
+| Audit JSONB containment `@>` plus `::jsonb` | Full | `audits` | The helper did not need an `Arel::Nodes::InfixOperation` for table dependency recovery. |
+| Audit JSON extraction `-> ... IS NOT NULL` | Full | `audits` | PostgreSQL JSON operators do not block the table-level fallback. |
+| `Effort::Filterable.over_budget` correlated `COALESCE(SUM(...), 0)` scalar subquery | Full | `efforts`, `time_logs` | Both the result table and aggregate input table are recovered. |
+| `Milestone.status_case_sql` with `CASE`, repeated correlated `EXISTS` / `NOT EXISTS`, and `IN` | Full with custom AST walk | `milestones`, `efforts` | The original SQL can be tracked conservatively. `Sqlglot::Query#tables` alone misses `efforts`. |
+| `active_requirements_count` aggregate `SUM(CASE ...)` over a join | Full | `project_team_requirements`, `positions` | It could remain a database aggregate; a change to either table triggers full replay. |
+| Two-column `pluck` across a left join | Full | `assignments`, `teams` | Raw projection strings are irrelevant to table discovery. The Arel projection rewrite is not required by this fallback. |
+| `pg_search` tsearch-only query and trigram ranking function | Full | `projects` | SQLGlot handles the functions, casts, rank expression, and `@@`. |
+| `pg_search` word-similarity predicate using `<%` | **Parse failure** | None | Upkeep must refuse, use a PostgreSQL-specific secondary parser, or add a narrowly tested normalization for custom operators. |
+
+## What “covered” means
+
+Coverage here is deliberately conservative. SQLGlot proves which named tables
+can change the result; it does not prove which individual changes alter
+membership, ordering, aggregates, or projected values.
+
+For every SQLGlot-derived dependency, Upkeep should therefore:
+
+- subscribe to every recovered table;
+- rerun the complete render site on any change to one of those tables; and
+- disable append, prepend, remove, and member-replace optimizations.
+
+That is enough to preserve the original SQL for the range filters, JSONB
+predicates, correlated subqueries, status `CASE`, aggregate, and `pluck` cases
+above. It trades precision for correctness without introducing a second
+user-maintained policy.
+
+## What SQLGlot does not address in the PR
+
+- The `<%` custom PostgreSQL operator prevents parsing the complete
+  `pg_search` relation. Materializing that relation remains necessary unless
+  Upkeep adds another safe fallback.
+- SQLGlot cannot discover physical tables hidden behind views or stored
+  functions from SQL text alone.
+- The milestone row-target and lazy-frame subscription fixes concern render
+  capture and subscription lifetime, not query analysis.
+- Controller runtime installation and cross-controller invalidation are
+  runtime concerns, not parser concerns.
+- Parsing an aggregate or `pluck` does not make its result incrementally
+  patchable. It only makes full replay safe.
+
+## Recommendation
+
+Use SQLGlot automatically after the precise Arel analyzer fails. For this PR,
+that would cover every query rewrite except the full pg_search predicate. Keep
+the scope-aware AST extractor: the convenience `Query#tables` API misses the
+inner `efforts` and `time_logs` sources in the two hardest cases.
+
+Treat a SQLGlot parse failure as unresolved. If pg_search is important enough
+to justify broader coverage, the next focused spike should compare
+PostgreSQL's real parser (`pg_query`) against a generic custom-operator
+normalization; the latter must not silently alter source structure.

data/docs/spikes/sqlglot-semantic-bindings/README.md

@@ -0,0 +1,29 @@
+# Missing SQLGlot semantic bindings spike
+
+The `sqlglot` Ruby gem 0.1.1 binds only parse, generate, transpile, and version.
+Its underlying `protegrity/sql-glot-rust` v0.10.12 already contains semantic
+APIs that are useful to Upkeep:
+
+- `MappingSchema` and `qualify_columns`;
+- `build_scope`, including physical/scoped sources, columns, external columns,
+  and correlation;
+- output-column `lineage`.
+
+This spike adds one JSON-over-FFI call around those APIs and a small Ruby wrapper.
+It does not fork or modify the upstream dependency.
+
+Build and run:
+
+```sh
+cargo build --release
+mise x ruby@3.4.7 -- ruby binding_test.rb
+```
+
+The output-lineage test is intentional: lineage follows a selected value back to
+its source, while Upkeep also needs filter/join/order dependencies. Scope analysis
+contains those columns, so `scope`, not output lineage alone, is the useful base
+for a dependency decoder.
+
+Current v0.10.12 caveat: a CTE's child scope is present and correct, but the
+outer source with the CTE's name is still emitted as a table source. The test
+locks this behavior down so an upstream fix is visible.