grain 0.0.1 → 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 5a98d985852245285e65d2e58f8a2ae02e2c998f58d49ee70ef5f99f3b13e3d9
4
- data.tar.gz: 7ee5d68d74314f57bbcd0315e4cd9c9d0b2b79d2b2ffd8e3a376d5e09c8c537a
3
+ metadata.gz: 5cad390077ec6feb274d0c3331f77a00c1f0766cf2edcff03460b9f3f1fb85e6
4
+ data.tar.gz: 73d1b7338f8e0ea57e9448ae4f246b17eeae2525cd45854205ed2634a4d631c9
5
5
  SHA512:
6
- metadata.gz: 48aa997aee1584516efc868aac66685a54d6c6b1caf89a70a10d7e42259023eac2e1c4575dc86fc2e95e42378295bb512081028ea70725897f888ef3bd8774f2
7
- data.tar.gz: 2d78ba7b8411d9bf0146c8bb6c458958e6f12c7c89c61397e34d0047fe312ac8a3562d0c88b7959b99d569b050385329daf8b38c99e314e15a4da96266f7b8c1
6
+ metadata.gz: 96f65b733a5d74dd8d7bea62d6f89f3dc7250a2364352e877f4ac95fbf0be5c4612adf65f6ea9a0c13e1285afe66f361d497963667e0f2cb1638cf2a980fdfb3
7
+ data.tar.gz: 5b0be5e034a803da62a573991ba4dd058b94e7e890b7785095536e59e26954836345d10c11641e2fb7c12a4a0c7047949f44e0781c5d9ac35f7f36243369c798
data/CHANGELOG.md CHANGED
@@ -1,5 +1,84 @@
1
1
  ## [Unreleased]
2
2
 
3
+ ## [0.0.2] - 2026-08-21
4
+
5
+ Everything here came out of using Grain in two real applications: a football
6
+ pool (golbet) and a multi-tenant church management API (ekklesia).
7
+
8
+ ### Breaking
9
+
10
+ - **Reads require their tenant.** `Rollup.by(...)` without a `for(tenant: ...)`
11
+ now raises `Grain::MissingTenantError` instead of aggregating every tenant into
12
+ one number. The read it replaces did not return a wrong figure, it returned
13
+ another tenant's, and nothing about the result gave that away. A `nil` tenant is
14
+ refused for the same reason: no cell can hold one, so the read came back a clean
15
+ zero. Spanning every tenant is asked for by name with `Rollup.across_tenants`,
16
+ which also makes those reads greppable.
17
+ - **Joined tables are aliased `g_<path>` instead of `j0`, `j1`.** Named by the
18
+ whole association path, because two routes can end at the same table and one
19
+ alias for both is ambiguous. A measure expression written against a generated
20
+ alias has to be updated.
21
+ - **Tables a measure reads through now log every update**, which changes the
22
+ triggers a rollup installs. Regenerate the table migration for every rollup that
23
+ uses `through:` — see Upgrading.
24
+
25
+ ### Added
26
+
27
+ - **`through:` on a measure**, so an expression can read columns from tables the
28
+ fact joins to: `sum: "CASE WHEN g_match.status = 'finished' THEN 1 ELSE 0 END",
29
+ through: :match`. Those tables are joined into the recompute and watched by the
30
+ triggers, because a column they own can change what a measure computes without
31
+ any fact row moving.
32
+ - **`Grain::Installer.install!` and `rake grain:triggers`**, which re-attach the
33
+ trigger function and every trigger. Needed after any `schema.rb` load: the
34
+ schema format cannot represent functions or triggers, so loading it creates the
35
+ tables and silently drops everything that keeps them correct. That is how test
36
+ databases are built and what `db:reset` does.
37
+ - **`Grain::DrainJob`** and `config.queue`, so keeping rollups fresh is a
38
+ scheduled ActiveJob entry rather than a cron line invoking rake. Overlapping
39
+ runs are safe: claiming uses `FOR UPDATE SKIP LOCKED`.
40
+
41
+ ### Fixed
42
+
43
+ - **A change to a table only a measure read never reached the rollup.** The
44
+ trigger fired and the log took the row, but the registry did not route that
45
+ table to any rollup, so the worker dropped it and the aggregate drifted in
46
+ silence — the one failure mode Grain exists to prevent.
47
+ - **Rollup discovery `require`d files Zeitwerk manages**, which either
48
+ double-defines the class or fails outright. It constantizes them now, and no
49
+ longer blows up when `app/rollups` does not exist.
50
+ - **The railtie pushed `app/rollups` into `autoload_paths`** from an initializer,
51
+ by which point that array is frozen. Rails already autoloads and eager loads
52
+ everything under `app/`, so the hook was both broken and unnecessary.
53
+ - `for(dimension: [])` raised a Postgres syntax error from `IN ()`. An empty list
54
+ is what `current_user.churches.ids` hands over when there are none, and it now
55
+ matches nothing instead of failing.
56
+ - `for(dimension: [id, nil])` silently dropped the null coordinate: `IN` treats a
57
+ null as an unknown that equals nothing, so the cells with no value for that
58
+ dimension — the ones a dashboard labels "uncategorised" — fell out of the answer
59
+ with no sign they had been left out. A list containing `nil` now matches them,
60
+ the same as passing `nil` on its own already did.
61
+ - A `time` dimension resolved from a **nullable** column is now reported as
62
+ nullable, so the rollup takes the surrogate key path instead of declaring the
63
+ bucket `NOT NULL` inside the primary key. Before this, the first fact row with
64
+ no timestamp failed to insert — at write time, on the application's own table,
65
+ with nothing pointing at Grain.
66
+ - The install generator's closing instructions named the wrong generator for
67
+ step 3.
68
+
69
+ ### Upgrading from 0.0.1
70
+
71
+ 1. Any read without a tenant now raises. Add `for(tenant: ...)`, or
72
+ `across_tenants` where crossing them is the point.
73
+ 2. Rename generated join aliases in measure expressions: `j0` becomes
74
+ `g_<association path>`.
75
+ 3. Regenerate and run the table migration for every rollup
76
+ (`bin/rails generate grain:table <Rollup>`), so the triggers cover the tables
77
+ your measures read through.
78
+ 4. Call `Grain::Installer.install!` wherever your test suite loads the schema.
79
+ Without it the triggers do not exist in the test database, the rollups never
80
+ update, and the suite passes anyway.
81
+
3
82
  ## [0.0.1] - 2026-08-19
4
83
 
5
84
  First published release. Feature complete for a first pass and tested end to end
data/CLAUDE.md ADDED
@@ -0,0 +1,145 @@
1
+ # Grain — contexto del proyecto
2
+
3
+ Gema Ruby: **agregados pre-calculados y mantenidos incrementalmente dentro del Postgres de la
4
+ propia aplicación**, para que los dashboards de Rails respondan en milisegundos. Publicada como
5
+ `grain` 0.0.1 en RubyGems. El repo remoto (`github.com/grainrb/grain`) todavía no existe.
6
+
7
+ Documentos hermanos: `../proyecto-grain.md` (plan de negocio, modelo open core, criterios para
8
+ matar el proyecto), `../ideas-negocio.md`, `../ekklesia/CLAUDE.md` (integración en curso).
9
+
10
+ ## Estado
11
+
12
+ - **208 tests, 0 fallas. Rubocop limpio.** `bundle exec rake` corre ambos.
13
+ - Alcance de la v1 **completo**: definición, esquema, generadores, triggers, worker, `verify`,
14
+ backfill, API de lectura, `DrainJob`.
15
+ - **Probada en una app real** (`../golbet`, porras de fútbol): coincide con la implementación de
16
+ referencia en Ruby y lee entre 6x y 62x más rápido.
17
+ - **Segunda app, completa** (`../ekklesia`, multi-tenant de verdad con `acts_as_tenant`): dos
18
+ rollups, `verify` limpio, 26 specs que los comparan contra los endpoints que reemplazan, el
19
+ worker programado con Solid Queue (probado en Linux: el drenado ocurre solo) y los tres endpoints
20
+ de `stats` ya leyendo el rollup. Salieron tres cambios de ahí: las lecturas exigen tenant
21
+ (incompatible), la dimensión de tiempo nulable, y los filtros de lista con `nil` o vacíos.
22
+ Detalle en `../ekklesia/CLAUDE.md`.
23
+ - **Hay cambios sin commitear** en grain y en golbet (DrainJob, Installer, README). Los commits
24
+ los hace el usuario; yo redacto los mensajes y no ejecuto `git commit`.
25
+ - Versión en `lib/grain/version.rb` = 0.0.1. Pendiente publicar 0.0.2 con los nueve arreglos que
26
+ salieron del dogfooding.
27
+
28
+ ## Cómo trabajar aquí
29
+
30
+ ```bash
31
+ docker run -d --name grain-pg -e POSTGRES_PASSWORD=grain \
32
+ -e POSTGRES_DB=grain_test -p 5433:5432 postgres:18
33
+ bundle exec rake # tests + rubocop
34
+ ```
35
+
36
+ Los tests de integración se saltan solos si no hay Postgres. `GRAIN_TEST_DATABASE_URL` lo
37
+ reapunta. Los tests corren los generadores de verdad, cargan los archivos que escriben y los
38
+ aplican a la base viva: **afirmar sobre strings generados es confianza falsa**, y ya pasó tres
39
+ veces que un string perfecto era semánticamente incorrecto.
40
+
41
+ ## La API
42
+
43
+ ```ruby
44
+ class OrderRevenueRollup < Grain::Rollup
45
+ fact LineItem, where: { order: { state: "paid" } }
46
+
47
+ tenant :store_id, via: { order: :store_id } # obligatorio
48
+ time :ordered_on, via: { order: :placed_on }, grain: :day # opcional
49
+ dimension :product_id, via: :product_id # columna local
50
+ dimension :category_id, via: { product: :category_id } # un salto
51
+ dimension :currency, via: { order: :currency }, immutable: true
52
+
53
+ measure :line_count, count: true
54
+ measure :revenue_cents, sum: "quantity * unit_price_cents", type: :bigint
55
+ measure :paid_lines, sum: "CASE WHEN g_order.state = 'paid' THEN 1 ELSE 0 END",
56
+ type: :bigint, through: :order
57
+ ratio :average_unit_price, of: :revenue_cents, over: :line_count
58
+ end
59
+ ```
60
+
61
+ Generadores: `grain:install` (una vez), `grain:rollup NAME` (esqueleto), `grain:table NAME`
62
+ (migración de tabla + triggers; **regenerar cada vez que cambia la definición**).
63
+ Tareas: `grain:drain`, `grain:verify` (sale con código != 0), `grain:backfill ROLLUP=`,
64
+ `grain:triggers`.
65
+ API: `Rollup.for(...).between(...).by(...)`, `.verify(repair:)`, `.backfill(from:, pause:)`,
66
+ `Grain::Worker.drain`, `Grain::DrainJob`, `Grain::Installer.install!`.
67
+
68
+ ## Decisiones tomadas — no relitigar
69
+
70
+ 1. **Recomputar una celda es la primitiva; los deltas son la optimización, y la v1 no los tiene.**
71
+ El worker trabaja por lotes, así que 1000 inserts en 10 celdas son 10 recomputaciones.
72
+ 2. **La regla que gobierna todo**: recomputar una celda que no hacía falta es inofensivo; no
73
+ recomputar una que sí, es el único bug imperdonable. Ante la duda, recomputa.
74
+ 3. **Recomputar es DELETE + INSERT, no upsert.** Una celda puede quedar legítimamente vacía y un
75
+ upsert dejaría los números viejos para siempre.
76
+ 4. **Solo cadenas `belongs_to`, máximo 3 saltos.** Es aritmética: por `belongs_to` cada fila de
77
+ hecho cae en exactamente una celda. Cruzar un `has_many` duplicaría cada conteo.
78
+ 5. **`tenant` obligatorio, `time` opcional.** Sin `time` el rollup es un counter cache verificable.
79
+ 6. **`sum`/`min`/`max` exigen `type:` explícito.** Adivinar redondearía la plata de alguien.
80
+ 7. **Un trigger por tabla fuente, nunca por rollup**, y la lista de columnas es la **unión** entre
81
+ todos los rollups que la vigilan.
82
+ 8. **Las tablas de hechos y las que leen las medidas registran todos los updates** (SQL arbitrario,
83
+ no se sabe qué columnas lo alimentan). Las demás se reducen con precisión.
84
+ 9. **Alias `f` para el hecho, `g_<camino>` para los joins.** Por el camino completo, no el último
85
+ salto: dos rutas pueden terminar en la misma tabla.
86
+ 10. **Los ratios se guardan en dos partes y se dividen al leer.** Nunca pre-divididos.
87
+ 11. **Grain emite archivos de migración**, no crea tablas en runtime, y la migración es una
88
+ fotografía (el SQL va literal, no leído de la gema).
89
+ 12. **Las lecturas exigen el tenant; cruzarlo se pide por su nombre** (`across_tenants`). Una
90
+ lectura sin tenant no devuelve un número equivocado, devuelve el de otro inquilino, y no hay
91
+ default seguro que Grain pueda elegir: no sabe de quién es el dato que le toca a quien
92
+ pregunta. Un tenant `nil` se rechaza igual, porque ninguna celda puede tener uno y la lectura
93
+ volvería como un cero limpio. La escotilla existe porque el caso legítimo existe (un panel de
94
+ administración), y así queda greppable.
95
+ 13. **Los agregados se castean al tipo declarado.** `SUM` sobre `bigint` da `numeric`; sin el cast
96
+ hay falsos positivos permanentes en `verify` y `BigDecimal` en las lecturas.
97
+
98
+ ## Patrones de bug que ya morderon — no reintroducir
99
+
100
+ - **`schema.rb` no representa funciones ni triggers.** Cargar el esquema (así se construyen las
101
+ bases de test, y así funciona `db:reset`) crea las tablas y borra todo lo que las mantiene
102
+ correctas. Sin señal alguna. Se arregla con `Grain::Installer.install!`.
103
+ - **Los tests que leen un rollup tienen que drenar.** Incluido uno que muta datos después de que
104
+ su propio setup ya drenó.
105
+ - **ActiveRecord reporta `bigint` como `:integer` con `limit: 8`.** Tomarlo literal da llaves de
106
+ 4 bytes contra fuentes de 8.
107
+ - **`create_table primary_key: [...]` junto con `id: false`** no crea llave primaria alguna, en
108
+ silencio.
109
+ - **`Array({a: :b})`** convierte el hash en pares.
110
+ - **Enseñarle a los triggers a vigilar una tabla sin enseñárselo al `Registry`**: el trigger
111
+ dispara, el log se llena, el worker la ignora.
112
+ - **`FULL OUTER JOIN` en Postgres no acepta `IS NOT DISTINCT FROM`.** `verify` usa `UNION ALL` +
113
+ `GROUP BY`, que además ya trata los nulos como iguales.
114
+ - **Afirmar sobre números sin afirmar el tipo.** `assert_equal 1400, BigDecimal(1400)` pasa.
115
+ - **`[nil].any?` es `false`, y `[false].any?` también.** Preguntar por verdadez si una lista de
116
+ valores trae algo tira justo los valores que hay que tratar. Va con `empty?`. Recién mordió al
117
+ arreglar los filtros de lista: el `IS NULL` no se agregaba nunca y el `nil` de la lista
118
+ desaparecía en silencio, que es el bug que se estaba arreglando.
119
+ - **`IN` trata al nulo como desconocido, no como coordenada.** `for(dim: [id, nil])` dejaba fuera
120
+ las celdas sin valor —las que un panel muestra como "sin categoría"— sin señal. Y `IN ()` no es
121
+ SQL válido: una lista vacía es `FALSE`, no un error de sintaxis.
122
+ - **Tratar una dimensión de tiempo como exenta de la nulabilidad de su fuente.** Guarda un bucket,
123
+ no el timestamp, pero el bucket de un nulo es nulo. Declararlo `NOT NULL` dentro de la llave
124
+ primaria hacía fallar el primer insert **en la tabla de hechos de la app**, lejos de Grain.
125
+
126
+ ## Limitaciones vigentes (están en el README)
127
+
128
+ Solo Postgres (15+ si alguna dimensión es nulable). Solo medidas aditivas — sin conteos distintos
129
+ ni percentiles (necesitan HyperLogLog / t-digest, y son **Grain Pro**). Solo grano diario. Un
130
+ hecho por rollup, sin rollups sobre rollups (Pro). Sin deltas. `pause:` es espera fija, no
131
+ throttling adaptativo por lag de replicación. Un rollup con modelo roto se salta con advertencia.
132
+
133
+ ## Lo que sigue
134
+
135
+ 1. Actualizar el bloque de estado del README (ya está probada en una app real) y publicar 0.0.2.
136
+ Ojo: 0.0.2 ya lleva **un cambio incompatible** (el tenant obligatorio en las lecturas). Está en
137
+ el CHANGELOG bajo `Unreleased`. golbet no se rompe — su única lectura ya pasaba el tenant, y su
138
+ suite corre verde contra la gema parcheada.
139
+ 2. La descripción publicada en RubyGems dice "maintains it with deltas", que **es falso** y
140
+ contradice al propio README. Los metadatos no se editan; se corrige en 0.0.2.
141
+ 3. El artículo fundacional: *"Cómo matamos nuestras vistas materializadas"*, con los números de
142
+ golbet. Según `../proyecto-grain.md` es todo el mercadeo del primer año.
143
+ 4. **Las 5-10 conversaciones con equipos Rails que tengan dashboards pesados.** Sigue sin hacerse,
144
+ y es el criterio de muerte escrito en frío: si nadie reconoce el problema como serio, no hay
145
+ negocio. El código va muy adelante de la validación.
data/README.md CHANGED
@@ -31,11 +31,14 @@ OrderRevenueRollup.for(store: current_store)
31
31
 
32
32
  > ### Status
33
33
  >
34
- > Version 0.0.1. Feature complete for a first pass and tested end to end against
35
- > a live PostgreSQL, including the generators, the triggers, the worker,
36
- > verification, backfilling and reads. **Not yet used in a real application**, so
37
- > the API may still change. Treat it as something to read and argue with rather
38
- > than something to put in front of customers this week.
34
+ > Version 0.0.2, and **in use in two real applications**: a football pool, where
35
+ > it agrees with the Ruby implementation it replaced and reads between 6x and 62x
36
+ > faster, and a multi-tenant church management API, where its dashboard endpoints
37
+ > now read rollups. Everything in 0.0.2 came out of that including one breaking
38
+ > change, so read the CHANGELOG before upgrading.
39
+ >
40
+ > Still 0.x. The API can still change, and the limitations below are real rather
41
+ > than theoretical.
39
42
 
40
43
  ## The problem
41
44
 
@@ -112,7 +115,19 @@ what makes it true about the past.
112
115
  $ bin/rails grain:drain
113
116
  ```
114
117
 
115
- Run that on a schedule, or call `Grain::Worker.drain` from a job of your own.
118
+ Or schedule `Grain::DrainJob`, which does the same thing from ActiveJob:
119
+
120
+ ```yaml
121
+ # config/recurring.yml, with Solid Queue
122
+ production:
123
+ grain_drain:
124
+ class: Grain::DrainJob
125
+ schedule: every minute
126
+ ```
127
+
128
+ Overlapping runs are safe and need no guard: claiming uses `FOR UPDATE SKIP
129
+ LOCKED`, so two drains at once split the work rather than repeat it. A slow run
130
+ caught by the next tick is not a problem.
116
131
 
117
132
  **5. Read it.**
118
133
 
@@ -145,6 +160,8 @@ workspace, an organisation.
145
160
  tenant :store_id, via: { order: :store_id }
146
161
  ```
147
162
 
163
+ Reads refuse to run without it. See [Reading](#reading).
164
+
148
165
  ### `time` — optional
149
166
 
150
167
  The bucket rows fall into. Only `grain: :day` in this release.
@@ -160,6 +177,11 @@ Timestamps are resolved to a calendar day in an explicit zone (`config.time_zone
160
177
  UTC by default). Left to the database session, the same row would land in
161
178
  different buckets for different callers.
162
179
 
180
+ A nullable source column is allowed, and the rows with no timestamp collect in a
181
+ null bucket — the bucket of a null is null, so the rollup takes the surrogate key
182
+ path like any other nullable dimension. If those rows have no business being in
183
+ the aggregate at all, keep them out with the fact's `where:` instead.
184
+
163
185
  ### `dimension`
164
186
 
165
187
  ```ruby
@@ -188,8 +210,31 @@ measure :revenue_cents, sum: "quantity * unit_price_cents", type: :bigint
188
210
  measure :largest_line, max: "quantity * unit_price_cents", type: :bigint
189
211
  ```
190
212
 
191
- `count`, `sum`, `min` and `max`. Expressions are your own SQL over the fact table,
192
- which is aliased `f` if you need to qualify a column.
213
+ `count`, `sum`, `min` and `max`. Expressions are your own SQL. The fact table is
214
+ aliased `f`.
215
+
216
+ An expression can read columns from a related table by declaring the associations
217
+ it needs with `through:`. Joined tables are aliased `g_` plus the path that
218
+ reaches them, so the expression stays readable:
219
+
220
+ ```ruby
221
+ measure :paid_lines,
222
+ sum: "CASE WHEN g_order.state = 'paid' THEN 1 ELSE 0 END",
223
+ type: :bigint,
224
+ through: :order
225
+
226
+ measure :local_cents,
227
+ sum: "CASE WHEN g_order_store.currency = 'COP' THEN f.quantity * f.unit_price_cents ELSE 0 END",
228
+ type: :bigint,
229
+ through: { order: :store }
230
+ ```
231
+
232
+ Grain watches every table a measure reads, so changing one of those columns
233
+ rebuilds the affected cells even though no fact row moved. Those tables log
234
+ **every** update rather than a narrowed set, for the same reason the fact table
235
+ does: the expression is arbitrary SQL and there is no telling which of its
236
+ columns feed the measure. That is a real cost — weigh it before pointing a
237
+ measure at a table that is written constantly.
193
238
 
194
239
  `sum`, `min` and `max` require an explicit `type:`. `count` does not, since
195
240
  counting rows always yields an integer. The others aggregate arbitrary SQL whose
@@ -274,8 +319,14 @@ mine.by(:product_id).to_h # keyed by group
274
319
  mine.sql # the statement, for when you want to look
275
320
  ```
276
321
 
322
+ - **The tenant is required.** A read that does not name one raises
323
+ `Grain::MissingTenantError` rather than totalling every tenant at once. That
324
+ read does not return a wrong number, it returns somebody else's, and nothing
325
+ about the result looks unusual — so it fails instead of warning. Grain has no
326
+ safe default here: it cannot know whose data the caller is entitled to.
277
327
  - **`for`** filters any dimension. Values may be ids, ActiveRecord objects,
278
- arrays, or `nil` (which matches a null coordinate).
328
+ arrays, or `nil` (which matches a null coordinate). A `nil` *tenant* is refused
329
+ too: no cell can have one, so the read would come back a clean, wrong zero.
279
330
  - **`between`** takes two dates or a range.
280
331
  - **`by`** groups. Coarsen the time bucket with `by(ordered_on: :month)` —
281
332
  `:day`, `:week`, `:month`, `:quarter`, `:year`.
@@ -288,6 +339,16 @@ mine.sql # the statement, for when you want to
288
339
  - Narrowing returns a new query, so a base query can be handed around and reused.
289
340
  - Results come back typed — `Date` and `Integer`, not the driver's strings.
290
341
 
342
+ Spanning every tenant is a real thing to want — a platform-wide total, an admin
343
+ dashboard — and it is asked for by name:
344
+
345
+ ```ruby
346
+ OrderRevenueRollup.across_tenants.by(:product_id).revenue_cents
347
+ ```
348
+
349
+ Grepping for `across_tenants` then lists every read that crosses the boundary,
350
+ which is not something you can do with an omission.
351
+
291
352
  ## Backfilling
292
353
 
293
354
  ```ruby
@@ -368,6 +429,54 @@ silence. Precise where it can be, conservative where it cannot.
368
429
  The column list is the union across every rollup that watches a table, so adding
369
430
  a rollup never narrows a trigger another one depends on.
370
431
 
432
+ ## Triggers and schema loading
433
+
434
+ **`schema.rb` cannot represent a function or a trigger.** Rails' schema dumper
435
+ knows tables, columns and indexes, and nothing else, so loading a schema creates
436
+ every table and silently drops everything that keeps them correct. That is how
437
+ test databases are built by default, and what `db:reset` and restoring a dump both
438
+ do — leaving a rollup that never updates and a test suite that passes anyway.
439
+
440
+ Re-attach them after any schema load:
441
+
442
+ ```console
443
+ $ bin/rails grain:triggers
444
+ ```
445
+
446
+ ```ruby
447
+ Grain::Installer.install! # safe to call any number of times
448
+ Grain::Installer.installed? # false right after a schema load
449
+ ```
450
+
451
+ In a test suite, put it where the database is prepared:
452
+
453
+ ```ruby
454
+ # test/test_helper.rb
455
+ class ActiveSupport::TestCase
456
+ Grain::Installer.install!
457
+ end
458
+ ```
459
+
460
+ ### Tests that read a rollup have to drain
461
+
462
+ A page backed by a rollup only shows what the worker has already applied, and in a
463
+ test nothing is running one. Any test that writes and then reads a rollup — or
464
+ renders something that does — has to drain in between:
465
+
466
+ ```ruby
467
+ Prediction.create!(...)
468
+ Grain::Worker.drain
469
+ get standings_path
470
+ ```
471
+
472
+ That includes a test that mutates data *after* its own setup already drained. This
473
+ is real friction, and the first thing to check when a test that should show new
474
+ numbers shows the old ones.
475
+
476
+ The other option is `config.active_record.schema_format = :sql`, which dumps
477
+ through `pg_dump` and keeps functions and triggers. That is the more thorough fix
478
+ and a bigger change to how an application works, so Grain does not assume it.
479
+
371
480
  ## Configuration
372
481
 
373
482
  ```ruby
@@ -376,6 +485,7 @@ Grain.configure do |config|
376
485
  config.change_log_table = "grain_change_log" # baked into the trigger function
377
486
  config.batch_size = 1_000 # change log rows claimed per transaction
378
487
  config.max_run_seconds = 30 # how long a drain may run before yielding
488
+ config.queue = :default # queue Grain::DrainJob runs on
379
489
  config.time_zone = "UTC" # the zone day buckets are cut in
380
490
  config.logger = Rails.logger
381
491
  end
@@ -403,15 +513,16 @@ Stated plainly, because finding these out later is worse than reading them now.
403
513
  no percentiles: neither can be maintained without reading the rest of the cell's
404
514
  source rows, which needs sketches (HyperLogLog, t-digest) rather than a column.
405
515
  - **`belongs_to` chains only**, three hops deep. No `has_many`, no join tables.
516
+ This applies to a measure's `through:` as much as to a dimension's `via:`.
406
517
  - **Daily grain only.** No hourly buckets yet.
407
518
  - **One fact per rollup.** No joins between facts, and no rollups built on rollups.
408
519
  - **No deltas.** The worker recomputes affected cells rather than incrementing
409
520
  them. Batching makes this fine in ordinary use — a thousand inserts landing in
410
521
  ten cells cost ten recomputes — but a single enormous cell is recomputed in full
411
522
  every time it is touched.
412
- - **No job integration.** There is no ActiveJob class yet: run `rake grain:drain`
413
- on a schedule or call `Grain::Worker.drain` from a job of your own.
414
523
  - **`pause:` is a fixed wait**, not adaptive throttling on replication lag.
524
+ - **Triggers do not survive a `schema.rb` load** and have to be re-attached with
525
+ `rake grain:triggers`. See above; this catches everyone once.
415
526
  - **A rollup with a broken model reference is skipped with a warning** rather than
416
527
  raising, so one bad rollup cannot stop the log from draining. Watch your logs.
417
528
 
@@ -35,8 +35,8 @@ module Grain
35
35
  say ""
36
36
  say "Grain installed. Next:", :green
37
37
  say " 1. bin/rails db:migrate"
38
- say " 2. Declare a rollup in app/rollups"
39
- say " 3. bin/rails generate grain:rollup <name> (to build its table and triggers)"
38
+ say " 2. bin/rails generate grain:rollup <name> (writes a definition to fill in)"
39
+ say " 3. bin/rails generate grain:table <name> (builds its table and triggers)"
40
40
  say ""
41
41
  end
42
42
 
@@ -14,4 +14,7 @@ Grain.configure do |config|
14
14
  # monopolise a queue slot.
15
15
  config.max_run_seconds = 30
16
16
 
17
+ # Queue Grain::DrainJob runs on.
18
+ config.queue = :default
19
+
17
20
  end
@@ -15,6 +15,9 @@ module Grain
15
15
  # monopolise a job queue slot.
16
16
  attr_accessor :max_run_seconds
17
17
 
18
+ # ActiveJob queue Grain::DrainJob runs on.
19
+ attr_accessor :queue
20
+
18
21
  # Time zone the day buckets are cut in. A timestamp has to be resolved to a
19
22
  # calendar day in some zone, and leaving it to the database session would make
20
23
  # the same row land in different buckets depending on who ran the query.
@@ -27,6 +30,7 @@ module Grain
27
30
  @change_log_table = "grain_change_log"
28
31
  @batch_size = 1_000
29
32
  @max_run_seconds = 30
33
+ @queue = :default
30
34
  @time_zone = "UTC"
31
35
  @logger = nil
32
36
  end
@@ -93,6 +93,14 @@ module Grain
93
93
  watched_paths.flat_map(&:hops).uniq
94
94
  end
95
95
 
96
+ # Associations a measure's expression reads columns from. Their joins have to
97
+ # exist for the expression to resolve, and their tables have to be watched: a
98
+ # match's score changing alters what a measure computes even though no fact
99
+ # row moved.
100
+ def measure_paths
101
+ measures.flat_map(&:through).uniq
102
+ end
103
+
96
104
  # Associations the fact's filter reaches through. A change there adds or
97
105
  # removes fact rows entirely, which no delta can express.
98
106
  def filter_associations
@@ -0,0 +1,19 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Grain
4
+ # Drains the change log from a background job, so keeping rollups fresh is a
5
+ # scheduled entry rather than a cron line invoking rake.
6
+ #
7
+ # Overlapping runs are safe and need no guard. Claiming uses FOR UPDATE SKIP
8
+ # LOCKED, so two of these at once split the work instead of fighting over it or
9
+ # doing it twice — a slow run caught by the next tick is not a problem.
10
+ #
11
+ # Loaded only when ActiveJob is, so the gem does not depend on it.
12
+ class DrainJob < ActiveJob::Base
13
+ queue_as { Grain.config.queue }
14
+
15
+ def perform(limit: nil, max_seconds: nil)
16
+ Worker.drain(**{ limit: limit, max_seconds: max_seconds }.compact)
17
+ end
18
+ end
19
+ end
data/lib/grain/errors.rb CHANGED
@@ -16,6 +16,10 @@ module Grain
16
16
  # `verify` found rows where the rollup disagrees with its source.
17
17
  class VerificationError < Error; end
18
18
 
19
+ # A rollup keyed by tenant was read without one. Not a wrong number: somebody
20
+ # else's number, which is why it raises instead of warning.
21
+ class MissingTenantError < Error; end
22
+
19
23
  # No rollup class matches the name a command was given.
20
24
  class RollupNotFoundError < Error; end
21
25
 
@@ -0,0 +1,46 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Grain
4
+ # Recreates the trigger function and every trigger from the current definitions.
5
+ #
6
+ # This is needed far more often than it looks. Rails' schema.rb cannot represent
7
+ # a function or a trigger — the dumper only knows tables, columns and indexes —
8
+ # so loading the schema creates every table and silently drops everything that
9
+ # keeps them correct. That is how test databases are built by default, and what
10
+ # `db:reset` and restoring a dump both do. Without re-attaching, a rollup simply
11
+ # never updates, and the tests that should have caught it pass.
12
+ #
13
+ # Safe to run at any time: the function is CREATE OR REPLACE and each trigger is
14
+ # dropped before being created.
15
+ module Installer
16
+ class << self
17
+ # Returns the tables triggers were attached to.
18
+ def install!(definitions = default_definitions)
19
+ return [] if definitions.empty?
20
+
21
+ triggers = Triggers.new(definitions)
22
+ connection.execute(ChangeLog.function_sql)
23
+ triggers.up_statements.each { |statement| connection.execute(statement) }
24
+ triggers.specs.map(&:table)
25
+ end
26
+
27
+ # True when the change log's function is present, which is the cheap way to
28
+ # tell a schema load has stripped Grain out.
29
+ def installed?
30
+ connection.select_value(<<~SQL).present?
31
+ SELECT proname FROM pg_proc WHERE proname = '#{ChangeLog::FUNCTION_NAME}'
32
+ SQL
33
+ end
34
+
35
+ private
36
+
37
+ def default_definitions
38
+ Registry.all.map(&:definition)
39
+ end
40
+
41
+ def connection
42
+ ActiveRecord::Base.connection
43
+ end
44
+ end
45
+ end
46
+ end
@@ -11,6 +11,12 @@ module Grain
11
11
  # verbatim, so this is the name they can qualify columns with.
12
12
  FACT = "f"
13
13
 
14
+ # Joined tables are aliased by the association path that reaches them, so an
15
+ # expression can read `g_match.home_score` rather than a generated j0. The
16
+ # prefix keeps the alias clear of SQL keywords: `belongs_to :order` would
17
+ # otherwise produce `order`, which does not parse.
18
+ ALIAS_PREFIX = "g_"
19
+
14
20
  Joined = Struct.new(:name, :model, :on, keyword_init: true)
15
21
 
16
22
  attr_reader :definition, :types
@@ -53,11 +59,16 @@ module Grain
53
59
  private
54
60
 
55
61
  def register_all
56
- definition.key_dimensions.reject { |dimension| dimension.path.local? }
57
- .each { |dimension| register(dimension.path.hops) }
62
+ dimension_hops.each { |hops| register(hops) }
63
+ definition.measure_paths.each { |path| register(path.hops) }
58
64
  definition.filter_associations.each { |association| register([association]) }
59
65
  end
60
66
 
67
+ def dimension_hops
68
+ definition.key_dimensions.reject { |dimension| dimension.path.local? }
69
+ .map { |dimension| dimension.path.hops }
70
+ end
71
+
61
72
  # A local column needs no join, so an empty path is a caller's mistake rather
62
73
  # than a base case: treating it as one recurses forever.
63
74
  def register(hops)
@@ -69,7 +80,9 @@ module Grain
69
80
 
70
81
  def join_for(hops, parent)
71
82
  reflection = types.reflection!(parent.model, hops.last, hops.join("."))
72
- name = "j#{@joined.size}"
83
+ # Named by the whole path rather than the last hop: two different routes can
84
+ # end at the same table, and one alias for both would be ambiguous.
85
+ name = "#{ALIAS_PREFIX}#{hops.join("_")}"
73
86
  Joined.new(name: name, model: reflection.klass, on: "#{name}.id = #{parent.name}.#{reflection.foreign_key}")
74
87
  end
75
88
 
data/lib/grain/measure.rb CHANGED
@@ -20,15 +20,17 @@ module Grain
20
20
  # revenue. The declaration is one word, and it is cheap insurance.
21
21
  COUNT_TYPE = :bigint
22
22
 
23
- attr_reader :name, :aggregate, :expression, :type
23
+ attr_reader :name, :aggregate, :expression, :type, :through
24
24
 
25
25
  def self.from_options(name, options)
26
26
  options = options.dup
27
27
  type = options.delete(:type)
28
+ through = options.delete(:through)
28
29
  reject_ambiguous_aggregate!(name, options)
29
30
 
30
31
  aggregate, value = options.first
31
- new(name: name, aggregate: aggregate, expression: value == true ? nil : value, type: type)
32
+ new(name: name, aggregate: aggregate, expression: value == true ? nil : value,
33
+ type: type, through: through)
32
34
  end
33
35
 
34
36
  def self.reject_ambiguous_aggregate!(name, options)
@@ -39,11 +41,12 @@ module Grain
39
41
  end
40
42
  private_class_method :reject_ambiguous_aggregate!
41
43
 
42
- def initialize(name:, aggregate:, expression: nil, type: nil)
44
+ def initialize(name:, aggregate:, expression: nil, type: nil, through: nil)
43
45
  @name = name.to_sym
44
46
  @aggregate = aggregate.to_sym
45
47
  @expression = expression
46
48
  @type = (type || (@aggregate == :count ? COUNT_TYPE : nil))&.to_sym
49
+ @through = wrap(through).map { |path| Path.parse_association(path) }.freeze
47
50
  validate!
48
51
  freeze
49
52
  end
@@ -71,6 +74,16 @@ module Grain
71
74
 
72
75
  private
73
76
 
77
+ # Not Array(), which turns a Hash into a list of pairs and would take
78
+ # `through: { order: :store }` apart into two unrelated associations.
79
+ def wrap(through)
80
+ case through
81
+ when nil then []
82
+ when Array then through
83
+ else [through]
84
+ end
85
+ end
86
+
74
87
  def validate!
75
88
  validate_aggregate!
76
89
  validate_expression!
data/lib/grain/path.rb CHANGED
@@ -26,6 +26,22 @@ module Grain
26
26
  new(hops, column)
27
27
  end
28
28
 
29
+ # A path with no terminal column: just the associations a measure's expression
30
+ # needs joined in, so it can read their columns.
31
+ #
32
+ # parse_association(:match) # hops [:match]
33
+ # parse_association(order: :store) # hops [:order, :store]
34
+ def self.parse_association(through)
35
+ case through
36
+ when Symbol, String then new([through], nil)
37
+ when Hash
38
+ hops, column = walk(through, [])
39
+ new(hops + [column], nil)
40
+ else
41
+ raise InvalidDefinitionError, "through takes an association name or a nested hash, got #{through.inspect}"
42
+ end
43
+ end
44
+
29
45
  def self.walk(via, hops)
30
46
  case via
31
47
  when Symbol, String then [hops, via.to_sym]
@@ -50,7 +66,7 @@ module Grain
50
66
 
51
67
  def initialize(hops, column)
52
68
  @hops = hops.map(&:to_sym).freeze
53
- @column = column.to_sym
69
+ @column = column&.to_sym
54
70
  validate!
55
71
  freeze
56
72
  end
@@ -66,7 +82,7 @@ module Grain
66
82
  end
67
83
 
68
84
  def to_s
69
- (hops + [column]).join(".")
85
+ (hops + [column]).compact.join(".")
70
86
  end
71
87
 
72
88
  def inspect
data/lib/grain/query.rb CHANGED
@@ -13,12 +13,13 @@ module Grain
13
13
  class Query
14
14
  attr_reader :rollup, :definition
15
15
 
16
- def initialize(rollup, filters: {}, range: nil, groups: {})
16
+ def initialize(rollup, filters: {}, range: nil, groups: {}, across_tenants: false)
17
17
  @rollup = rollup
18
18
  @definition = rollup.definition.validate!
19
19
  @filters = filters
20
20
  @range = range
21
21
  @groups = groups
22
+ @across_tenants = across_tenants
22
23
  end
23
24
 
24
25
  # Filters on any dimension. A value may be an id, an ActiveRecord object, an
@@ -46,7 +47,15 @@ module Grain
46
47
  merge(groups: @groups.merge(groups))
47
48
  end
48
49
 
50
+ # Reads every tenant at once, which is a real thing to want — a platform-wide
51
+ # total, an admin dashboard — and has to be asked for by name. The dangerous
52
+ # read is the one nobody can see in the code.
53
+ def across_tenants
54
+ merge(across_tenants: true)
55
+ end
56
+
49
57
  def sql
58
+ require_tenant!
50
59
  QuerySql.new(definition, filters: @filters, range: @range, groups: @groups).to_s
51
60
  end
52
61
 
@@ -82,7 +91,15 @@ module Grain
82
91
  private
83
92
 
84
93
  def merge(**changes)
85
- self.class.new(rollup, filters: @filters, range: @range, groups: @groups, **changes)
94
+ self.class.new(rollup,
95
+ filters: @filters, range: @range, groups: @groups,
96
+ across_tenants: @across_tenants, **changes)
97
+ end
98
+
99
+ # Checked where the statement is built rather than left to the caller: the
100
+ # mistake this catches produces a plausible number, not an error.
101
+ def require_tenant!
102
+ TenantGuard.new(rollup, @filters).check! unless @across_tenants
86
103
  end
87
104
 
88
105
  def readable?(name)
@@ -62,11 +62,35 @@ module Grain
62
62
  column = quote_column(name)
63
63
  case value
64
64
  when nil then "#{column} IS NULL"
65
- when Array then "#{column} IN (#{value.map { |item| quote(item) }.join(", ")})"
65
+ when Array then any_of(column, value)
66
66
  else "#{column} = #{quote(value)}"
67
67
  end
68
68
  end
69
69
 
70
+ # A list is a set of coordinates to match, and a null is one of them: the
71
+ # cells with no value for a dimension are the ones a dashboard shows as
72
+ # "uncategorised". IN would treat that null as an unknown rather than a
73
+ # coordinate, so those cells would drop out of the answer without a word.
74
+ #
75
+ # An empty list is not a mistake either — it is `current_user.churches.ids`
76
+ # coming back empty — and it means nothing matches. `IN ()` is not valid SQL,
77
+ # so before this it raised a syntax error from inside a dashboard.
78
+ # empty? rather than any?: [nil].any? is false, and so is [false].any?, so
79
+ # asking whether there is anything in a list of values by truthiness drops
80
+ # exactly the values this method exists to handle.
81
+ def any_of(column, values)
82
+ nulls, present = values.partition(&:nil?)
83
+ parts = []
84
+ parts << "#{column} IN (#{present.map { |item| quote(item) }.join(", ")})" unless present.empty?
85
+ parts << "#{column} IS NULL" unless nulls.empty?
86
+
87
+ case parts.length
88
+ when 0 then "FALSE"
89
+ when 1 then parts.first
90
+ else "(#{parts.join(" OR ")})"
91
+ end
92
+ end
93
+
70
94
  def range_condition
71
95
  return nil if @range.nil?
72
96
 
data/lib/grain/railtie.rb CHANGED
@@ -3,19 +3,22 @@
3
3
  require "rails/railtie"
4
4
 
5
5
  module Grain
6
- # Hooks Grain into a Rails application: rake tasks, autoloading of
7
- # app/rollups, and the default logger.
6
+ # Hooks Grain into a Rails application.
7
+ #
8
+ # Nothing here touches autoload_paths. Rails already autoloads and eager loads
9
+ # every directory under app/, so app/rollups needs no help — and by the time a
10
+ # railtie initializer runs, autoload_paths is frozen anyway.
8
11
  class Railtie < ::Rails::Railtie
9
12
  initializer "grain.logger" do
10
13
  Grain.config.logger ||= Rails.logger
11
14
  end
12
15
 
13
- rake_tasks do
14
- load File.expand_path("../tasks/grain.rake", __dir__)
16
+ initializer "grain.drain_job" do
17
+ ActiveSupport.on_load(:active_job) { require "grain/drain_job" }
15
18
  end
16
19
 
17
- initializer "grain.autoload_rollups" do |app|
18
- app.config.autoload_paths << app.root.join("app/rollups") if app.root.join("app/rollups").exist?
20
+ rake_tasks do
21
+ load File.expand_path("../tasks/grain.rake", __dir__)
19
22
  end
20
23
  end
21
24
  end
@@ -48,10 +48,18 @@ module Grain
48
48
  Rollup.subclasses.select { |rollup| rollup.name && valid?(rollup) }
49
49
  end
50
50
 
51
+ # Constantized rather than required, so the autoloader stays in charge.
52
+ # Requiring a file Zeitwerk manages either double-defines the class or fails
53
+ # outright, and in development nothing has referenced these yet.
51
54
  def eager_load_rollups
52
55
  return unless defined?(Rails) && Rails.respond_to?(:root) && Rails.root
53
56
 
54
- Dir[Rails.root.join("app/rollups/**/*.rb")].sort.each { |path| require path }
57
+ directory = Rails.root.join("app/rollups")
58
+ return unless directory.exist?
59
+
60
+ Dir[directory.join("**/*.rb")].sort.each do |path|
61
+ path.delete_prefix("#{directory}/").delete_suffix(".rb").camelize.safe_constantize
62
+ end
55
63
  end
56
64
 
57
65
  # A rollup that cannot be used is left out rather than allowed to take the
@@ -76,8 +84,12 @@ module Grain
76
84
  rollup.definition.fact.model.table_name
77
85
  end
78
86
 
87
+ # Both the tables dimensions are resolved through and the tables measures
88
+ # read from. Missing the second set is a silent failure: the trigger fires,
89
+ # the log takes the row, and nothing routes it to a rollup that cares.
79
90
  def watched_tables(rollup)
80
- WatchedColumns.new(rollup.definition).to_h.keys
91
+ columns = WatchedColumns.new(rollup.definition)
92
+ columns.to_h.keys + columns.unnarrowable_tables
81
93
  end
82
94
  end
83
95
  end
data/lib/grain/rollup.rb CHANGED
@@ -74,6 +74,12 @@ module Grain
74
74
  query.by(*names, **coarse)
75
75
  end
76
76
 
77
+ # Lifts the tenant requirement for a read that is meant to span every one
78
+ # of them. Reads are keyed by tenant and refuse to run without it.
79
+ def across_tenants
80
+ query.across_tenants
81
+ end
82
+
77
83
  # Populates the rollup from data that already exists. A new rollup is empty
78
84
  # until this runs: its triggers only see what happens next.
79
85
  def backfill(from: nil, pause: 0, &progress)
@@ -0,0 +1,43 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Grain
4
+ # Refuses a read that does not say which tenant it is for.
5
+ #
6
+ # Every rollup is keyed by a tenant, so leaving it out does not make a number
7
+ # slightly wrong, it makes it somebody else's: the read totals every tenant at
8
+ # once and nothing about the result looks unusual. Grain cannot pick a default,
9
+ # since it has no idea whose data the caller is entitled to, so the only two
10
+ # options are to raise or to leak.
11
+ class TenantGuard
12
+ attr_reader :rollup, :filters
13
+
14
+ def initialize(rollup, filters)
15
+ @rollup = rollup
16
+ @filters = filters
17
+ end
18
+
19
+ def check!
20
+ name = rollup.definition.tenant.name
21
+ return if filters.key?(name) && !filters[name].nil?
22
+
23
+ raise MissingTenantError, complaint(name)
24
+ end
25
+
26
+ private
27
+
28
+ def complaint(name)
29
+ return null_tenant(name) if filters.key?(name)
30
+
31
+ "#{rollup} is keyed by #{name} and was read without it, which would total every tenant " \
32
+ "at once. Narrow it with .for(#{name}: ...), or ask for .across_tenants explicitly."
33
+ end
34
+
35
+ # A tenant column is never nullable, so a null filter matches no cell at all
36
+ # and the read comes back a clean zero — a lie about the data rather than a
37
+ # leak, and just as quiet.
38
+ def null_tenant(name)
39
+ "#{rollup} was read with #{name}: nil. A tenant is never null, so this matches no cell " \
40
+ "and reads as zero; pass a real #{name}, or .across_tenants to span all of them."
41
+ end
42
+ end
43
+ end
@@ -28,9 +28,9 @@ module Grain
28
28
  @definitions = Array(definitions).map(&:validate!).uniq
29
29
  end
30
30
 
31
- # Fact tables first, then the tables reached through them.
31
+ # The tables that cannot be narrowed first, then the ones that can.
32
32
  def specs
33
- fact_specs + related_specs
33
+ unnarrowed_specs + related_specs
34
34
  end
35
35
 
36
36
  # One statement per entry, so a migration can execute them individually
@@ -53,20 +53,23 @@ module Grain
53
53
 
54
54
  private
55
55
 
56
- # Measures aggregate arbitrary SQL, so which of a fact table's columns feed
57
- # them cannot be known. Narrowing here risks missing an update and letting a
58
- # rollup drift, so every update on a fact table is logged. Being a fact for
59
- # any one rollup is enough to disqualify the table from narrowing.
60
- def fact_specs
61
- fact_tables.map { |table| Spec.new(table: table, update_columns: nil) }
56
+ # Measures aggregate arbitrary SQL, so which columns feed them cannot be known
57
+ # not on the fact table, and not on any table a measure reads through. Both
58
+ # therefore log every update: narrowing would risk missing one and letting a
59
+ # rollup drift in silence. Being unnarrowable for any single rollup is enough
60
+ # to disqualify a table for all of them.
61
+ def unnarrowed_specs
62
+ unnarrowed_tables.map { |table| Spec.new(table: table, update_columns: nil) }
62
63
  end
63
64
 
64
- def fact_tables
65
- definitions.map { |definition| TypeResolver.new(definition).fact_table }.uniq
65
+ def unnarrowed_tables
66
+ definitions.flat_map do |definition|
67
+ [TypeResolver.new(definition).fact_table] + WatchedColumns.new(definition).unnarrowable_tables
68
+ end.uniq
66
69
  end
67
70
 
68
71
  def related_specs
69
- union_of_related_columns.reject { |table, _| fact_tables.include?(table) }
72
+ union_of_related_columns.reject { |table, _| unnarrowed_tables.include?(table) }
70
73
  .map { |table, columns| Spec.new(table: table, update_columns: columns.sort) }
71
74
  end
72
75
 
@@ -12,6 +12,12 @@ module Grain
12
12
  # the row falls into, so its type comes from the grain rather than the source.
13
13
  BUCKET_TYPES = { day: :date }.freeze
14
14
 
15
+ # ActiveRecord reports a bigint column as :integer carrying a limit of 8, so
16
+ # taking its type at face value would key a rollup on four bytes while the
17
+ # source uses eight. That holds until an id passes two billion and then stops,
18
+ # silently, on a table nobody thinks to look at.
19
+ INTEGER_WIDTHS = { 8 => :bigint, 2 => :integer, 1 => :integer }.freeze
20
+
15
21
  attr_reader :definition
16
22
 
17
23
  def initialize(definition)
@@ -21,7 +27,7 @@ module Grain
21
27
  def dimension_type(dimension)
22
28
  return BUCKET_TYPES.fetch(dimension.grain) if dimension.time?
23
29
 
24
- source_column(dimension).type
30
+ widen(source_column(dimension))
25
31
  end
26
32
 
27
33
  # The type of the column a dimension is read from, before any bucketing. A
@@ -33,9 +39,12 @@ module Grain
33
39
 
34
40
  # Whether the source column can be null, which decides whether the rollup can
35
41
  # use a plain composite primary key: Postgres will not accept a null in one.
42
+ #
43
+ # A time dimension is not exempt. It stores a bucket rather than the source
44
+ # timestamp, but the bucket of a null timestamp is null, and declaring that
45
+ # column NOT NULL inside the primary key makes the first such row fail to
46
+ # insert — on the fact table, at write time, far from anything about Grain.
36
47
  def dimension_nullable?(dimension)
37
- return false if dimension.time?
38
-
39
48
  source_column(dimension).null
40
49
  end
41
50
 
@@ -68,6 +77,12 @@ module Grain
68
77
  "value per dimension, or it would be counted in several cells at once."
69
78
  end
70
79
 
80
+ def widen(column)
81
+ return column.type unless column.type == :integer && column.limit == 8
82
+
83
+ INTEGER_WIDTHS.fetch(column.limit, :integer)
84
+ end
85
+
71
86
  def source_column(dimension)
72
87
  model = walk(definition.fact.model, dimension.path)
73
88
  model.columns_hash.fetch(dimension.path.column.to_s) do
data/lib/grain/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Grain
4
- VERSION = "0.0.1"
4
+ VERSION = "0.0.2"
5
5
  end
@@ -28,8 +28,23 @@ module Grain
28
28
  collected
29
29
  end
30
30
 
31
+ # Tables a measure's expression reads from, which cannot be narrowed at all.
32
+ #
33
+ # Same reasoning as the fact table: the expression is arbitrary SQL and there
34
+ # is no way to know which of its columns feed the measure. Guessing risks
35
+ # missing an update and drifting in silence, so every update is logged.
36
+ def unnarrowable_tables
37
+ definition.measure_paths.flat_map { |path| tables_along(path) }.uniq
38
+ end
39
+
31
40
  private
32
41
 
42
+ def tables_along(path)
43
+ model = definition.fact.model
44
+ path.hops.map { |hop| model = resolver.reflection!(model, hop, path).klass }
45
+ .map(&:table_name)
46
+ end
47
+
33
48
  def add_path(path, into)
34
49
  path.hops.each_with_index.reduce(definition.fact.model) do |model, (hop, index)|
35
50
  add_hop(model, hop, path, index, into)
data/lib/grain.rb CHANGED
@@ -21,10 +21,12 @@ require_relative "grain/migration"
21
21
  require_relative "grain/watched_columns"
22
22
  require_relative "grain/triggers"
23
23
  require_relative "grain/registry"
24
+ require_relative "grain/installer"
24
25
  require_relative "grain/join_graph"
25
26
  require_relative "grain/projection"
26
27
  require_relative "grain/cells"
27
28
  require_relative "grain/recompute"
29
+ require_relative "grain/tenant_guard"
28
30
  require_relative "grain/query_sql"
29
31
  require_relative "grain/query"
30
32
  require_relative "grain/backfill"
data/lib/tasks/grain.rake CHANGED
@@ -12,6 +12,16 @@ namespace :grain do
12
12
  abort("grain:verify found disagreements") if reports.any? { |report| !report.clean? } && !repair
13
13
  end
14
14
 
15
+ desc "Re-attach the trigger function and every trigger. Needed after a schema load."
16
+ task triggers: :environment do
17
+ tables = Grain::Installer.install!
18
+ if tables.empty?
19
+ puts "grain: no rollups found, nothing to attach"
20
+ else
21
+ puts "grain: triggers attached to #{tables.join(", ")}"
22
+ end
23
+ end
24
+
15
25
  desc "Populate a rollup from data that already exists. ROLLUP=Name [FROM=2026-08-01] [PAUSE=0.5]"
16
26
  task backfill: :environment do
17
27
  name = ENV.fetch("ROLLUP") { abort("grain:backfill needs ROLLUP=SomeRollup") }
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: grain
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Oscar Ortega
@@ -38,10 +38,10 @@ dependencies:
38
38
  - !ruby/object:Gem::Version
39
39
  version: '7.1'
40
40
  description: |-
41
- Grain keeps dashboard aggregates pre-computed and incrementally up to date
42
- inside your own Postgres database. You declare the grain of an aggregate
43
- once — tenant, time bucket and dimensions — and Grain maintains it with
44
- deltas driven by database triggers instead of recomputing everything on a
41
+ Grain keeps dashboard aggregates pre-computed and up to date inside your own
42
+ Postgres database. You declare the grain of an aggregate once — tenant, time
43
+ bucket and dimensions — and database triggers record what changed so a worker
44
+ can rebuild just the affected cells, instead of recomputing everything on a
45
45
  schedule. No new infrastructure, no materialized view refresh storms, and a
46
46
  verify command that proves the aggregate still matches its source.
47
47
  email:
@@ -51,6 +51,7 @@ extensions: []
51
51
  extra_rdoc_files: []
52
52
  files:
53
53
  - CHANGELOG.md
54
+ - CLAUDE.md
54
55
  - CODE_OF_CONDUCT.md
55
56
  - LICENSE.txt
56
57
  - README.md
@@ -71,8 +72,10 @@ files:
71
72
  - lib/grain/definition_validator.rb
72
73
  - lib/grain/dimension.rb
73
74
  - lib/grain/discrepancy.rb
75
+ - lib/grain/drain_job.rb
74
76
  - lib/grain/errors.rb
75
77
  - lib/grain/fact.rb
78
+ - lib/grain/installer.rb
76
79
  - lib/grain/join_graph.rb
77
80
  - lib/grain/measure.rb
78
81
  - lib/grain/migration.rb
@@ -87,6 +90,7 @@ files:
87
90
  - lib/grain/rollup.rb
88
91
  - lib/grain/rollup_lookup.rb
89
92
  - lib/grain/schema.rb
93
+ - lib/grain/tenant_guard.rb
90
94
  - lib/grain/triggers.rb
91
95
  - lib/grain/type_resolver.rb
92
96
  - lib/grain/verification.rb