mensa 0.6.17 → 0.6.19

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 40f5c40722b8f985ed92ec054bffb1495379772be630b967fae55f6fc77bc8b9
-  data.tar.gz: 1b5e668bb60387b99492c4eaf12f21fe548bdc913c4b6f9ecaa02dd3f8659b44
+  metadata.gz: b38d819bbd96ef6b2cb38e7b89c07c9a487eb7b92e8a191a92f4fd7e09cf1f7f
+  data.tar.gz: 52856b77830db55218a63213c4dc90a685686046175059cd78c53d8fb40d5527
 SHA512:
-  metadata.gz: f36c28f26ae32c725cd0f2696e8fcd2bb57fe77cd75552f7daf24e4737e397eb91e8484a1d18b25cf6093cdef3e69a433e756e7815c10592dd29b635e4ee43f2
-  data.tar.gz: 516d7dfe990004a70d878561f8697e46dada6323db733eeb640625a03ff43d7e006f08d7385aaaf89b84f306636cdea2d5d1b2796d5a23aa3ed4ee88d9939cd1
+  metadata.gz: a6dc8ccce0bed9be30b8b9c4b8d5f255b1d808d48a32c88e1813c40043378b0dd8d485abc5a34aa9b7b41f21335300f0f79f9cf6f7b1f1d7982f7f312027dad1
+  data.tar.gz: 36262b2128d07c1919dee7c08a82f0d15af44be791b012757afe41c28dadf5aafcc75e3be4907c0d9369d319217273297c98a20e2ab15380c03766da7b2e43ee

data/.shipkit.yml

@@ -1,2 +1,2 @@
 ---
-previous_version: 0.6.16
+previous_version: 0.6.18

data/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 0.6.19
+
+- `Mensa::Verifier` checks a table against the database: scope, column SQL,
+  sorting, rendering, filters, search, CSV export, grouping/aggregates and
+  views. Run it with `bin/rails "mensa:verify[table]"` / `mensa:verify_all`,
+  or from tests after `require "mensa/testing"` with
+  `assert_mensa_table_valid` or `be_a_valid_mensa_table`. Set defaults for
+  its context with `config.verifier_context`.
+- The verifier samples records from the first and last page and records with
+  empty values, tries filters with invalid input, checks search, filters, sorting
+  and grouping together, and warns about N+1 queries (with a fix) and slow
+  queries. `strict: true` / `STRICT=1` makes warnings fail.
+  `config.verifier_context` can return an array to verify tables in several
+  contexts, `config.verifier_slow_query` sets when a query is slow.
+- Columns without SQL (only a `render` block or a `method`, no `attribute` or
+  database column) are no longer sortable or filterable by default, sorting or
+  filtering on them produced invalid SQL. Use `sortable true` for a column whose
+  name is an alias the scope selects, or a filter `scope` to filter it.
+- Filters with a value their column's type can't hold (e.g. "abc" for a number,
+  or an invalid date or uuid) match nothing instead of raising a database error.
+- `Mensa::ExportJob.write_csv(io, table, rows)` writes a table's CSV export and
+  `Mensa::ExportJob#generate_file(export)` generates an export's file without
+  saving it.
+- Exports of tables with a `csv` render block no longer fail: the export job
+  renders with a view context of `config.base_controller`
+  (`Mensa.offline_view_context`).
+- Row actions without an `icon` show their title instead of raising.
+- An unknown helper in a render block raises `NoMethodError` instead of
+  `SystemStackError`. Only the app's `*_path`/`*_url` helpers are forwarded to
+  `main_app`.
+- Search leaves out columns with a `having` filter (aggregates), which broke the
+  query.
+- `is_duplicate` works for columns with a `having` filter.
+
 ## 0.6.16
 
 - Custom filter `scope` blocks can take the operator as a second argument,

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mensa (0.6.16)
+    mensa (0.6.18)
       csv
       importmap-rails
       pagy (>= 43)

data/README.md

@@ -118,11 +118,49 @@ You can show your tables on the page using the following:
 Custom views are views not defined by the developer (SystemViews) but by the end-user by adding/removing filters.
 When you enable these, they are stored in the database and can be used across sessions, by the user who created them.
 
+### Sortable and filterable columns
+
+Sorting and filtering happen in SQL, so whether a column can be sorted or filtered
+depends on whether it has SQL: a database column of the model, or an `attribute`.
+
+- **Columns with SQL** are sortable and filterable by default. Turn that off with
+  `sortable false` or `filter false`.
+- **Columns without SQL**, which only show something with a `render` block or a
+  `method`, are *not* sortable or filterable by default. Their headers have no sort
+  link, they aren't offered in the filter menu, and sorting or filtering on them
+  from a URL or saved state is ignored.
+
+```ruby
+column(:email)                      # sortable and filterable
+column(:customer_name) do           # sortable and filterable, it has SQL
+  attribute "customers.name"
+end
+column(:avatar) do                  # render only: not sortable, not filterable
+  render do
+    html { |user| image_tag(user.avatar_url) }
+  end
+end
+```
+
+To make a column without SQL filterable, give its filter a `scope` (see "Custom
+filter scopes" below). Add `sortable true` only when its name can be sorted on in
+SQL, e.g. an alias your table's `scope` selects:
+
+```ruby
+column(:orders_count) do
+  sortable true # ORDER BY orders_count, selected in the scope below
+end
+
+scope { User.left_joins(:orders).group(:id).select("users.*, COUNT(orders.id) AS orders_count") }
+```
+
+Better still, give such a column an `attribute`, so it's sortable and filterable.
+
 ### Filters
 
-Add `filter` to a column to let users filter on it. By default Mensa compares the
-column's attribute with the chosen value, using operators that fit the column type.
-The filter block takes these options:
+Columns with SQL are filterable (see above). Use `filter` to configure how. By
+default Mensa compares the column's attribute with the chosen value, using
+operators that fit the column type. The filter block takes these options:
 
 ```ruby
 column(:role) do
@@ -320,6 +358,115 @@ If you're just using cron, you can add the following to your crontab:
 0 0 * * * rails runner "Mensa::RecurringExportsJob.perform_later"
 ```
 
+### Verifying tables
+
+`Mensa::Verifier` checks that everything a table configures works against your
+real database, using the same code paths as requests do:
+
+- **Loads**: the table builds, its SQL builds and the first and last pages load. A
+  default `order` or `column_order` naming an unknown column fails. Records are
+  sampled for the checks below: some from the first and the last page, and one with
+  an empty value for each column, which often breaks render blocks.
+- **Columns**: every column (internal ones too) can be selected and, when sortable,
+  sorted both ways. Its html and csv renders run for the sampled records, and a
+  failure names the record. Columns hidden by a `visible` lambda are reported as
+  skipped.
+- **Filters**: every filterable column is applied with each of its operators, using
+  the first values of its `collection` or a value that fits the column's type, and
+  with input the column's type can't hold (e.g. "not a number"). The filter pill's
+  text is built. `is_current` is skipped when `Current` has no value for the column.
+- **Search**: basic and fuzzy search (when `pg_trgm` is enabled) for a word taken
+  from the records.
+- **Rendering**: the `table` helper as your views call it, the rows and filter pills
+  as Mensa's controller renders them, the row link and every row action.
+- **Exports**: `Mensa::ExportJob` generates the first page of the export, also for
+  each view and as a password protected zip. It needs the expected headers
+  (skipped for `exportable false`).
+- **Combined**: search, a filter on every column, sorting on every column and
+  grouping with aggregates, all at once.
+- **Grouping**: every groupable column and every aggregate, rendered with their
+  group headers. A `group_by` or `aggregates` naming a column that doesn't allow
+  it fails, because Mensa would silently ignore it.
+- **Views**: system views load, render and only refer to existing columns. With
+  custom views enabled, saved views load too.
+
+It also warns, without failing:
+
+- **N+1 queries**: a query that a render block, the row link or a row action runs for
+  each record. The warning shows the query and how to avoid it for the association
+  it loads, e.g. preload it with `includes` (and `internal :customer_id`, since Mensa
+  only selects the table's columns), select the value in SQL with `attribute`, or
+  count in SQL.
+- **Slow queries**: queries that take `config.verifier_slow_query` seconds (1.0 by
+  default) or more.
+
+Pass `strict: true` (or `STRICT=1` to rake) to make warnings fail.
+
+Each check runs on its own, so one failure doesn't hide the others. Everything runs
+in a read-only transaction that is rolled back, with small limits, so it is safe to
+run against staging. It never writes.
+
+#### From rake
+
+```sh
+bin/rails "mensa:verify[inventories]"   # or InventoriesTable
+bin/rails mensa:verify_all              # every table in app/tables
+```
+
+The tasks print a report (✓ passed, ✗ failed with the error and SQL, - skipped) and
+exit non-zero when a check failed. Classes and modules that don't descend from
+`Mensa::Base` (e.g. helpers in `app/tables`) are reported as skipped.
+
+#### From tests
+
+```ruby
+# test/test_helper.rb or spec/rails_helper.rb
+require "mensa/testing"
+
+# Minitest
+assert_mensa_table_valid InventoriesTable, user: users(:admin)
+
+# RSpec
+expect(InventoriesTable).to be_a_valid_mensa_table(user: admin)
+
+# Plain Ruby
+result = Mensa::Verifier.new(InventoriesTable, user: admin).call
+result.success?  # => true
+result.failures  # => [#<struct Mensa::Verifier::Check section: :filters, subject: :status, ...>]
+puts result      # the report, also the failure message of the helpers
+```
+
+#### Context
+
+Tables often depend on the request: `Current.user`, policy scopes, or the table's
+params. Give the verifier that context:
+
+- `user:` and `account:` set `Current.user` and `Current.account` while verifying.
+- `context: {organization: org}` sets any other `Current` attributes.
+- `params: {customer_id: 1}` sets the table's params (see "Table params").
+- A block is called with the verifier before the checks, for any other setup.
+- `label: "admin"` names the context in the report.
+
+Html render blocks run in a view context of your `ApplicationController` (see
+`config.base_controller`), so url helpers and your helpers work.
+
+From rake, set `USER_ID=` (finds a `User`), `ACCOUNT_ID=` (finds an `Account`) and
+`PARAMS='{"customer_id":1}'` (JSON), or configure defaults for every table in
+your initializer. The options you pass yourself override these defaults. Return an array of them to
+verify every table once for each, e.g. as different kinds of users:
+
+```ruby
+Mensa.setup do |config|
+  config.verifier_context = ->(table_class) {
+    params = {customer_id: Customer.first.id} if table_class == CustomerUsersTable
+    [
+      {label: "admin", user: User.find_by!(email: "admin@example.com"), params: params},
+      {label: "customer", user: User.find_by!(email: "customer@example.com"), params: params}
+    ]
+  }
+end
+```
+
 ## Contributing
 
 ```

data/app/components/mensa/row_action/component.html.erb

@@ -4,7 +4,7 @@
     <% if icon = action.icon(row.record) %>
       <i class="fa <%= icon %>"></i>
     <% else %>
-      <%= name %>
+      <%= action.title %>
     <% end %>
   <% end %>
 <% end %>

data/app/helpers/mensa/application_helper.rb

@@ -6,16 +6,21 @@ module Mensa
       render(component)
     end
 
+    # The app's url helpers, also inside Mensa's engine views
     def respond_to_missing?(method, include_private = false)
-      if method.to_s.ends_with?("_url") || method.to_s.ends_with?("_path") && main_app.respond_to?(method)
-        true
-      else
-        super
-      end
+      main_app_url_helper?(method) || super
     end
 
     def method_missing(method, *args, **kwargs, &block)
-      main_app.send(method, *args, **kwargs, &block)
+      return super unless main_app_url_helper?(method)
+
+      main_app.public_send(method, *args, **kwargs, &block)
+    end
+
+    private
+
+    def main_app_url_helper?(method)
+      method.to_s.end_with?("_url", "_path") && main_app.respond_to?(method)
     end
   end
 end

data/app/jobs/mensa/export_job.rb

@@ -9,20 +9,30 @@ module Mensa
   class ExportJob < ApplicationJob
     queue_as :default
 
+    # Writes the CSV for +rows+ (Mensa::Row) of +table+ to +io+: a header row
+    # with the display column names, then one line per row.
+    def self.write_csv(io, table, rows)
+      csv = CSV.new(io)
+      csv << table.display_columns.map(&:name)
+      rows.each do |row|
+        csv << table.display_columns.map { |column| Mensa::Cell.new(row: row, column: column).render(:csv) }
+      end
+      csv.flush
+      io
+    end
+
     def perform(export)
       return unless export
 
       export.update(status: "processing")
       Mensa.config.run_callback(:export_started, export)
 
-      table = build_table(export)
-      unless table.exportable?
+      tempfile, filename, content_type = generate_file(export)
+      unless tempfile
         finalize(export, status: "failed")
         return
       end
 
-      tempfile, filename, content_type = generate(table, export)
-
       export.asset.purge if export.asset.attached?
       export.asset.attach(io: tempfile, filename: filename, content_type: content_type)
       finalize(export, status: "completed", filename: filename)
@@ -36,6 +46,16 @@ module Mensa
       tempfile&.close!
     end
 
+    # Builds the table for +export+ and generates its file. Returns
+    # [tempfile, filename, content_type], or nil when the table isn't
+    # exportable. Doesn't touch the export's record or asset.
+    def generate_file(export)
+      table = build_table(export)
+      return unless table.exportable?
+
+      generate(table, export)
+    end
+
     private
 
     # Rebuilds the table the export was requested for, layering the view
@@ -54,6 +74,8 @@ module Mensa
       table = Mensa.for_name(export.table_name, config)
       table.request = ActionDispatch::Request.new({})
       table.request.set_header("action_dispatch.request.query_parameters", {"page" => config[:page]})
+      # Render blocks (e.g. csv) run in a view context
+      table.original_view_context = Mensa.offline_view_context
       table
     end
 
@@ -78,12 +100,8 @@ module Mensa
       # with programmatic consumers.
       tempfile.write("\uFEFF") if export.format == "csv_excel"
 
-      csv = CSV.new(tempfile)
-      csv << table.display_columns.map(&:name)
-      export_rows(table, export).each do |row|
-        csv << table.display_columns.map { |column| Mensa::Cell.new(row: row, column: column).render(:csv) }
-      end
-      csv.close
+      self.class.write_csv(tempfile, table, export_rows(table, export))
+      tempfile.close
       tempfile.open
       tempfile.binmode
       tempfile.rewind

data/app/tables/mensa/base.rb

@@ -110,7 +110,9 @@ module Mensa
     def active_filters
       (config[:filters] || {}).filter_map do |column_name, filter_config|
         col = column(column_name)
-        next unless col
+        # Columns without SQL can only be filtered with a custom scope, skip
+        # filters on them, e.g. from browser state saved earlier
+        next if col.nil? || !(col.sql? || col.filter?)
 
         definition = config.dig(:columns, col.name, :filter)
         definition = {} unless definition.is_a?(Hash)

data/app/tables/mensa/column.rb

@@ -14,7 +14,6 @@ module Mensa
       @table = table
     end
 
-    config_reader :sortable?
     config_reader :sanitize?
     config_reader :visible?
     config_reader :internal?
@@ -96,9 +95,25 @@ module Mensa
       aggregates.any?
     end
 
-    # Returns true if the column supports filtering
+    # Whether the column has SQL (an attribute or a database column). Columns
+    # without, e.g. only a render block or a method, can't be sorted or
+    # filtered in SQL.
+    def sql?
+      attribute_for_condition.present?
+    end
+
+    # Columns with SQL are sortable unless `sortable false`, others only with
+    # an explicit `sortable true`
+    def sortable?
+      config[:sortable].nil? ? sql? : !!config[:sortable]
+    end
+
+    # Returns true if the column supports filtering: columns with SQL unless
+    # `filter false`, others only with a filter that has its own scope
     def filter?
-      config[:filter] != false
+      return false if config[:filter] == false
+
+      sql? || (config[:filter].is_a?(Hash) && config[:filter][:scope].present?)
     end
 
     def filter

data/app/tables/mensa/config/column_dsl.rb

@@ -4,7 +4,8 @@ module Mensa::Config
   class ColumnDsl
     include DslLogic
 
-    option :sortable, default: true
+    # Defaults to whether the column has SQL, see Mensa::Column#sortable?
+    option :sortable
     option :sanitize, default: true
     # Allows for sql-parts too
     #

data/app/tables/mensa/filter.rb

@@ -86,6 +86,9 @@ module Mensa
 
         # Nothing selected in a multi-select means no filter
         return record_scope if value.is_a?(Array) && value.compact_blank.empty?
+        # A value the database can't compare with the column, e.g. "abc" for
+        # a number, matches nothing instead of raising
+        return record_scope.none if operator_with_value? && invalid_value?
 
         query, hash = query_and_hash_for_operator
         record_scope = (having? ? record_scope.having(query, hash) : record_scope.where(query, hash)) if query.present?
@@ -211,6 +214,7 @@ module Mensa
     def apply_duplicate_filter(record_scope)
       column_sql = column.attribute_for_condition.to_s
       base_scope = record_scope.except(:select, :order, :limit, :offset)
+      return apply_duplicate_having_filter(record_scope, base_scope, column_sql) if having?
 
       duplicate_values_scope = base_scope
         .where("#{column_sql} IS NOT NULL")
@@ -223,6 +227,42 @@ module Mensa
         .where("#{column_sql} IN (#{duplicate_values_scope.to_sql})")
     end
 
+    UUID_FORMAT = /\A\h{8}-\h{4}-\h{4}-\h{4}-\h{12}\z/
+
+    # Whether a value can't be read as the column's type. Values are sent to
+    # the database as they are, this only catches the ones it would reject.
+    def invalid_value?
+      Array(value).compact_blank.any? { |entry| !valid_for_type?(entry) }
+    end
+
+    def valid_for_type?(entry)
+      return true unless entry.is_a?(String)
+
+      case column.type
+      when :integer, :bigint then entry.strip.match?(/\A[+-]?\d+\z/)
+      when :decimal, :float then entry.strip.match?(/\A[+-]?(\d+\.?\d*|\.\d+)\z/)
+      when :date then Date.parse(entry).present?
+      when :datetime, :timestamp then Time.zone.parse(entry).present?
+      when :uuid then entry.strip.match?(UUID_FORMAT)
+      else true
+      end
+    rescue ArgumentError, RangeError
+      false
+    end
+
+    # Aggregates are only known per group, so the duplicate values come from
+    # a subquery of the grouped rows and are compared in the HAVING
+    def apply_duplicate_having_filter(record_scope, base_scope, column_sql)
+      duplicate_values_scope = base_scope.model.unscoped
+        .from(base_scope.select(Arel.sql("#{column_sql} AS mensa_duplicate_value")), :mensa_values)
+        .where("mensa_values.mensa_duplicate_value IS NOT NULL")
+        .group("mensa_values.mensa_duplicate_value")
+        .having("COUNT(*) > 1")
+        .select("mensa_values.mensa_duplicate_value")
+
+      record_scope.having("#{column_sql} IN (#{duplicate_values_scope.to_sql})")
+    end
+
     # Unused at the moment
     def normalize(query)
       query

data/app/tables/mensa/scope.rb

@@ -129,6 +129,9 @@ module Mensa
         # Skip orders on unknown names, e.g. from browser state saved for
         # different params.
         next unless col || model_attribute?(k)
+        # Columns without SQL can't be sorted on, unless they're explicitly
+        # sortable, e.g. on an alias the scope selects
+        next if col && !col.sql? && !col.sortable?
         next unless %w[asc desc].include?(v)
 
         attribute = col&.attribute_for_condition || k

data/app/tables/mensa/search.rb

@@ -10,7 +10,8 @@ module Mensa
       return record_scope if query.blank?
       return record_scope unless record_scope.is_a?(ActiveRecord::Relation)
 
-      searchable_attributes = columns.filter_map(&:attribute_for_condition).uniq
+      # Aggregates (filter having) can't be in a WHERE
+      searchable_attributes = columns.reject { |column| column.filter&.having? }.filter_map(&:attribute_for_condition).uniq
       return record_scope if searchable_attributes.empty?
 
       @search_order_clause = nil

data/lib/generators/mensa/templates/config/initializers/mensa.rb

@@ -7,6 +7,10 @@ Mensa.setup do |config|
     end
   }
 
+  # Context for `rails mensa:verify` and the test helpers (user:, account:,
+  # context:, params:)
+  # config.verifier_context = ->(table_class) { {user: User.find_by!(email: "admin@example.com")} }
+
   # Override icons in use
   config.icons = {
     add_filter_trigger: "fa-solid fa-circle-plus",

data/lib/mensa/configuration.rb

@@ -144,6 +144,15 @@ module Mensa
     option :row_actions_position, default: :back
     # It's either :basic or :fuzzy, for fuzzy search you need to have `pg_trgm` extension installed
     option :search, default: :basic
+    # Default options for Mensa::Verifier (and so `rails mensa:verify`), called
+    # with the table class. Return a hash with any of user:, account:,
+    # context: (Current attributes), params: and label:, or an array of them
+    # to verify the table in each of those contexts, e.g.
+    #
+    #   config.verifier_context = ->(table_class) { {user: User.find_by!(email: "admin@example.com")} }
+    option :verifier_context, default: nil
+    # Mensa::Verifier warns about queries that take this many seconds or more
+    option :verifier_slow_query, default: 1.0
 
     def initialize
       set_defaults!

data/lib/mensa/testing.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require "mensa/verifier"
+
+module Mensa
+  # Test helpers on top of Mensa::Verifier, opt in with `require "mensa/testing"`.
+  #
+  #   assert_mensa_table_valid UsersTable, user: users(:admin)        # Minitest
+  #   expect(UsersTable).to be_a_valid_mensa_table(user: admin)        # RSpec
+  #
+  # Both take the verifier's options (user:, account:, context:, params:,
+  # strict:), verify the table in every context of
+  # Mensa.config.verifier_context and fail with the reports.
+  module Testing
+    module Assertions
+      def assert_mensa_table_valid(table, message = nil, **options, &setup)
+        results = Mensa::Verifier.verify(table, **options, &setup)
+        assert results.all?(&:success?), -> { [message, *results.reject(&:success?).map(&:to_s)].compact.join("\n\n") }
+        results
+      end
+    end
+
+    class ValidTableMatcher
+      attr_reader :results
+
+      def initialize(**options, &setup)
+        @options = options
+        @setup = setup
+      end
+
+      def matches?(table)
+        @results = Mensa::Verifier.verify(table, **@options, &@setup)
+        @results.all?(&:success?)
+      end
+
+      def failure_message
+        results.reject(&:success?).join("\n\n")
+      end
+
+      def failure_message_when_negated
+        "expected #{results.first.table_class} not to be a valid Mensa table, but all its checks passed"
+      end
+
+      def description
+        "be a valid Mensa table"
+      end
+    end
+
+    module Matchers
+      def be_a_valid_mensa_table(**options, &setup)
+        ValidTableMatcher.new(**options, &setup)
+      end
+    end
+  end
+end
+
+if defined?(Minitest::Test)
+  Minitest::Test.include(Mensa::Testing::Assertions)
+else
+  ActiveSupport.on_load(:active_support_test_case) { include Mensa::Testing::Assertions }
+end
+
+RSpec.configure { |config| config.include(Mensa::Testing::Matchers) } if defined?(RSpec) && RSpec.respond_to?(:configure)