mensa 0.6.18 → 0.6.19

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a9bbb252521ea00a7d6197e9ae12208d121ec069652b45b729095609a596bda2
-  data.tar.gz: 92345f45e5b1336f1e101c2c64d9736c58637cb9792c333cc21f4620dadabf94
+  metadata.gz: b38d819bbd96ef6b2cb38e7b89c07c9a487eb7b92e8a191a92f4fd7e09cf1f7f
+  data.tar.gz: 52856b77830db55218a63213c4dc90a685686046175059cd78c53d8fb40d5527
 SHA512:
-  metadata.gz: 2b97589e71ee52505369eb9cac3af5e49672c8af62d076d2f17465ea76b2c556ca2c5adbede3af5ad850db502bd47271e031c96a5292eb2ef191818e24e60435
-  data.tar.gz: ba31a706fa85f69d30a1b4bfcb17d2c6674df561daf8379341b3f7dc31d21bd10ecc94fc36192f0f2302e602556898c72d9b3d203e330f2a60494c3741b004f2
+  metadata.gz: a6dc8ccce0bed9be30b8b9c4b8d5f255b1d808d48a32c88e1813c40043378b0dd8d485abc5a34aa9b7b41f21335300f0f79f9cf6f7b1f1d7982f7f312027dad1
+  data.tar.gz: 36262b2128d07c1919dee7c08a82f0d15af44be791b012757afe41c28dadf5aafcc75e3be4907c0d9369d319217273297c98a20e2ab15380c03766da7b2e43ee

data/.shipkit.yml

@@ -1,2 +1,2 @@
 ---
-previous_version: 0.6.17
+previous_version: 0.6.18

data/CHANGELOG.md

@@ -1,6 +1,6 @@
 # Changelog
 
-## Unreleased
+## 0.6.19
 
 - `Mensa::Verifier` checks a table against the database: scope, column SQL,
   sorting, rendering, filters, search, CSV export, grouping/aggregates and
@@ -8,6 +8,18 @@
   or from tests after `require "mensa/testing"` with
   `assert_mensa_table_valid` or `be_a_valid_mensa_table`. Set defaults for
   its context with `config.verifier_context`.
+- The verifier samples records from the first and last page and records with
+  empty values, tries filters with invalid input, checks search, filters, sorting
+  and grouping together, and warns about N+1 queries (with a fix) and slow
+  queries. `strict: true` / `STRICT=1` makes warnings fail.
+  `config.verifier_context` can return an array to verify tables in several
+  contexts, `config.verifier_slow_query` sets when a query is slow.
+- Columns without SQL (only a `render` block or a `method`, no `attribute` or
+  database column) are no longer sortable or filterable by default, sorting or
+  filtering on them produced invalid SQL. Use `sortable true` for a column whose
+  name is an alias the scope selects, or a filter `scope` to filter it.
+- Filters with a value their column's type can't hold (e.g. "abc" for a number,
+  or an invalid date or uuid) match nothing instead of raising a database error.
 - `Mensa::ExportJob.write_csv(io, table, rows)` writes a table's CSV export and
   `Mensa::ExportJob#generate_file(export)` generates an export's file without
   saving it.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mensa (0.6.17)
+    mensa (0.6.18)
       csv
       importmap-rails
       pagy (>= 43)

data/README.md

@@ -118,11 +118,49 @@ You can show your tables on the page using the following:
 Custom views are views not defined by the developer (SystemViews) but by the end-user by adding/removing filters.
 When you enable these, they are stored in the database and can be used across sessions, by the user who created them.
 
+### Sortable and filterable columns
+
+Sorting and filtering happen in SQL, so whether a column can be sorted or filtered
+depends on whether it has SQL: a database column of the model, or an `attribute`.
+
+- **Columns with SQL** are sortable and filterable by default. Turn that off with
+  `sortable false` or `filter false`.
+- **Columns without SQL**, which only show something with a `render` block or a
+  `method`, are *not* sortable or filterable by default. Their headers have no sort
+  link, they aren't offered in the filter menu, and sorting or filtering on them
+  from a URL or saved state is ignored.
+
+```ruby
+column(:email)                      # sortable and filterable
+column(:customer_name) do           # sortable and filterable, it has SQL
+  attribute "customers.name"
+end
+column(:avatar) do                  # render only: not sortable, not filterable
+  render do
+    html { |user| image_tag(user.avatar_url) }
+  end
+end
+```
+
+To make a column without SQL filterable, give its filter a `scope` (see "Custom
+filter scopes" below). Add `sortable true` only when its name can be sorted on in
+SQL, e.g. an alias your table's `scope` selects:
+
+```ruby
+column(:orders_count) do
+  sortable true # ORDER BY orders_count, selected in the scope below
+end
+
+scope { User.left_joins(:orders).group(:id).select("users.*, COUNT(orders.id) AS orders_count") }
+```
+
+Better still, give such a column an `attribute`, so it's sortable and filterable.
+
 ### Filters
 
-Add `filter` to a column to let users filter on it. By default Mensa compares the
-column's attribute with the chosen value, using operators that fit the column type.
-The filter block takes these options:
+Columns with SQL are filterable (see above). Use `filter` to configure how. By
+default Mensa compares the column's attribute with the chosen value, using
+operators that fit the column type. The filter block takes these options:
 
 ```ruby
 column(:role) do
@@ -325,15 +363,18 @@ If you're just using cron, you can add the following to your crontab:
 `Mensa::Verifier` checks that everything a table configures works against your
 real database, using the same code paths as requests do:
 
-- **Loads**: the table builds, its SQL builds and the first two pages load. A default
-  `order` or `column_order` naming an unknown column fails.
+- **Loads**: the table builds, its SQL builds and the first and last pages load. A
+  default `order` or `column_order` naming an unknown column fails. Records are
+  sampled for the checks below: some from the first and the last page, and one with
+  an empty value for each column, which often breaks render blocks.
 - **Columns**: every column (internal ones too) can be selected and, when sortable,
-  sorted both ways. Its html and csv renders run for a few records. Columns hidden
-  by a `visible` lambda are reported as skipped.
+  sorted both ways. Its html and csv renders run for the sampled records, and a
+  failure names the record. Columns hidden by a `visible` lambda are reported as
+  skipped.
 - **Filters**: every filterable column is applied with each of its operators, using
   the first values of its `collection` or a value that fits the column's type, and
-  the filter pill's text is built. `is_current` is skipped when `Current` has no
-  value for the column.
+  with input the column's type can't hold (e.g. "not a number"). The filter pill's
+  text is built. `is_current` is skipped when `Current` has no value for the column.
 - **Search**: basic and fuzzy search (when `pg_trgm` is enabled) for a word taken
   from the records.
 - **Rendering**: the `table` helper as your views call it, the rows and filter pills
@@ -341,12 +382,26 @@ real database, using the same code paths as requests do:
 - **Exports**: `Mensa::ExportJob` generates the first page of the export, also for
   each view and as a password protected zip. It needs the expected headers
   (skipped for `exportable false`).
+- **Combined**: search, a filter on every column, sorting on every column and
+  grouping with aggregates, all at once.
 - **Grouping**: every groupable column and every aggregate, rendered with their
   group headers. A `group_by` or `aggregates` naming a column that doesn't allow
   it fails, because Mensa would silently ignore it.
 - **Views**: system views load, render and only refer to existing columns. With
   custom views enabled, saved views load too.
 
+It also warns, without failing:
+
+- **N+1 queries**: a query that a render block, the row link or a row action runs for
+  each record. The warning shows the query and how to avoid it for the association
+  it loads, e.g. preload it with `includes` (and `internal :customer_id`, since Mensa
+  only selects the table's columns), select the value in SQL with `attribute`, or
+  count in SQL.
+- **Slow queries**: queries that take `config.verifier_slow_query` seconds (1.0 by
+  default) or more.
+
+Pass `strict: true` (or `STRICT=1` to rake) to make warnings fail.
+
 Each check runs on its own, so one failure doesn't hide the others. Everything runs
 in a read-only transaction that is rolled back, with small limits, so it is safe to
 run against staging. It never writes.
@@ -390,20 +445,24 @@ params. Give the verifier that context:
 - `context: {organization: org}` sets any other `Current` attributes.
 - `params: {customer_id: 1}` sets the table's params (see "Table params").
 - A block is called with the verifier before the checks, for any other setup.
+- `label: "admin"` names the context in the report.
 
 Html render blocks run in a view context of your `ApplicationController` (see
 `config.base_controller`), so url helpers and your helpers work.
 
 From rake, set `USER_ID=` (finds a `User`), `ACCOUNT_ID=` (finds an `Account`) and
 `PARAMS='{"customer_id":1}'` (JSON), or configure defaults for every table in
-your initializer. The options you pass yourself override these defaults:
+your initializer. The options you pass yourself override these defaults. Return an array of them to
+verify every table once for each, e.g. as different kinds of users:
 
 ```ruby
 Mensa.setup do |config|
   config.verifier_context = ->(table_class) {
-    options = {user: User.find_by!(email: "admin@example.com")}
-    options[:params] = {customer_id: Customer.first.id} if table_class == CustomerUsersTable
-    options
+    params = {customer_id: Customer.first.id} if table_class == CustomerUsersTable
+    [
+      {label: "admin", user: User.find_by!(email: "admin@example.com"), params: params},
+      {label: "customer", user: User.find_by!(email: "customer@example.com"), params: params}
+    ]
   }
 end
 ```

data/app/components/mensa/row_action/component.html.erb

@@ -6,4 +6,5 @@
     <% else %>
       <%= action.title %>
     <% end %>
-  <% end %>
+  <% end %>
+<% end %>

data/app/tables/mensa/base.rb

@@ -110,7 +110,9 @@ module Mensa
     def active_filters
       (config[:filters] || {}).filter_map do |column_name, filter_config|
         col = column(column_name)
-        next unless col
+        # Columns without SQL can only be filtered with a custom scope, skip
+        # filters on them, e.g. from browser state saved earlier
+        next if col.nil? || !(col.sql? || col.filter?)
 
         definition = config.dig(:columns, col.name, :filter)
         definition = {} unless definition.is_a?(Hash)

data/app/tables/mensa/column.rb

@@ -14,7 +14,6 @@ module Mensa
       @table = table
     end
 
-    config_reader :sortable?
     config_reader :sanitize?
     config_reader :visible?
     config_reader :internal?
@@ -96,9 +95,25 @@ module Mensa
       aggregates.any?
     end
 
-    # Returns true if the column supports filtering
+    # Whether the column has SQL (an attribute or a database column). Columns
+    # without, e.g. only a render block or a method, can't be sorted or
+    # filtered in SQL.
+    def sql?
+      attribute_for_condition.present?
+    end
+
+    # Columns with SQL are sortable unless `sortable false`, others only with
+    # an explicit `sortable true`
+    def sortable?
+      config[:sortable].nil? ? sql? : !!config[:sortable]
+    end
+
+    # Returns true if the column supports filtering: columns with SQL unless
+    # `filter false`, others only with a filter that has its own scope
     def filter?
-      config[:filter] != false
+      return false if config[:filter] == false
+
+      sql? || (config[:filter].is_a?(Hash) && config[:filter][:scope].present?)
     end
 
     def filter

data/app/tables/mensa/config/column_dsl.rb

@@ -4,7 +4,8 @@ module Mensa::Config
   class ColumnDsl
     include DslLogic
 
-    option :sortable, default: true
+    # Defaults to whether the column has SQL, see Mensa::Column#sortable?
+    option :sortable
     option :sanitize, default: true
     # Allows for sql-parts too
     #

data/app/tables/mensa/filter.rb

@@ -86,6 +86,9 @@ module Mensa
 
         # Nothing selected in a multi-select means no filter
         return record_scope if value.is_a?(Array) && value.compact_blank.empty?
+        # A value the database can't compare with the column, e.g. "abc" for
+        # a number, matches nothing instead of raising
+        return record_scope.none if operator_with_value? && invalid_value?
 
         query, hash = query_and_hash_for_operator
         record_scope = (having? ? record_scope.having(query, hash) : record_scope.where(query, hash)) if query.present?
@@ -224,6 +227,29 @@ module Mensa
         .where("#{column_sql} IN (#{duplicate_values_scope.to_sql})")
     end
 
+    UUID_FORMAT = /\A\h{8}-\h{4}-\h{4}-\h{4}-\h{12}\z/
+
+    # Whether a value can't be read as the column's type. Values are sent to
+    # the database as they are, this only catches the ones it would reject.
+    def invalid_value?
+      Array(value).compact_blank.any? { |entry| !valid_for_type?(entry) }
+    end
+
+    def valid_for_type?(entry)
+      return true unless entry.is_a?(String)
+
+      case column.type
+      when :integer, :bigint then entry.strip.match?(/\A[+-]?\d+\z/)
+      when :decimal, :float then entry.strip.match?(/\A[+-]?(\d+\.?\d*|\.\d+)\z/)
+      when :date then Date.parse(entry).present?
+      when :datetime, :timestamp then Time.zone.parse(entry).present?
+      when :uuid then entry.strip.match?(UUID_FORMAT)
+      else true
+      end
+    rescue ArgumentError, RangeError
+      false
+    end
+
     # Aggregates are only known per group, so the duplicate values come from
     # a subquery of the grouped rows and are compared in the HAVING
     def apply_duplicate_having_filter(record_scope, base_scope, column_sql)

data/app/tables/mensa/scope.rb

@@ -129,6 +129,9 @@ module Mensa
         # Skip orders on unknown names, e.g. from browser state saved for
         # different params.
         next unless col || model_attribute?(k)
+        # Columns without SQL can't be sorted on, unless they're explicitly
+        # sortable, e.g. on an alias the scope selects
+        next if col && !col.sql? && !col.sortable?
         next unless %w[asc desc].include?(v)
 
         attribute = col&.attribute_for_condition || k

data/lib/mensa/configuration.rb

@@ -146,10 +146,13 @@ module Mensa
     option :search, default: :basic
     # Default options for Mensa::Verifier (and so `rails mensa:verify`), called
     # with the table class. Return a hash with any of user:, account:,
-    # context: (Current attributes) and params:, e.g.
+    # context: (Current attributes), params: and label:, or an array of them
+    # to verify the table in each of those contexts, e.g.
     #
     #   config.verifier_context = ->(table_class) { {user: User.find_by!(email: "admin@example.com")} }
     option :verifier_context, default: nil
+    # Mensa::Verifier warns about queries that take this many seconds or more
+    option :verifier_slow_query, default: 1.0
 
     def initialize
       set_defaults!

data/lib/mensa/testing.rb

@@ -8,19 +8,20 @@ module Mensa
   #   assert_mensa_table_valid UsersTable, user: users(:admin)        # Minitest
   #   expect(UsersTable).to be_a_valid_mensa_table(user: admin)        # RSpec
   #
-  # Both take the verifier's options (user:, account:, context:, params:)
-  # and fail with its report.
+  # Both take the verifier's options (user:, account:, context:, params:,
+  # strict:), verify the table in every context of
+  # Mensa.config.verifier_context and fail with the reports.
   module Testing
     module Assertions
       def assert_mensa_table_valid(table, message = nil, **options, &setup)
-        result = Mensa::Verifier.new(table, **options, &setup).call
-        assert result.success?, -> { [message, result.to_s].compact.join("\n\n") }
-        result
+        results = Mensa::Verifier.verify(table, **options, &setup)
+        assert results.all?(&:success?), -> { [message, *results.reject(&:success?).map(&:to_s)].compact.join("\n\n") }
+        results
       end
     end
 
     class ValidTableMatcher
-      attr_reader :result
+      attr_reader :results
 
       def initialize(**options, &setup)
         @options = options
@@ -28,16 +29,16 @@ module Mensa
       end
 
       def matches?(table)
-        @result = Mensa::Verifier.new(table, **@options, &@setup).call
-        @result.success?
+        @results = Mensa::Verifier.verify(table, **@options, &@setup)
+        @results.all?(&:success?)
       end
 
       def failure_message
-        result.to_s
+        results.reject(&:success?).join("\n\n")
       end
 
       def failure_message_when_negated
-        "expected #{result.table_class} not to be a valid Mensa table, but all its checks passed"
+        "expected #{results.first.table_class} not to be a valid Mensa table, but all its checks passed"
       end
 
       def description

data/lib/mensa/verifier/result.rb

@@ -5,7 +5,8 @@ module Mensa
     # One verified aspect of a table, e.g. sorting a column descending.
     #
     # +section+ groups checks in the report (:loads, :columns, :filters, ...),
-    # +subject+ is the column, filter or view it relates to.
+    # +subject+ is the column, filter or view it relates to. Warnings (N+1
+    # queries, slow queries) don't fail the table unless it's verified strict.
     Check = Struct.new(:section, :subject, :name, :status, :message, :sql, :error) do
       def passed?
         status == :passed
@@ -19,6 +20,10 @@ module Mensa
         status == :skipped
       end
 
+      def warning?
+        status == :warning
+      end
+
       def label
         [subject, name].compact.join(": ")
       end
@@ -31,17 +36,21 @@ module Mensa
         filters: "Filters",
         search: "Search",
         render: "Rendering",
+        combined: "Combined",
         exports: "Exports",
         grouping: "Grouping",
         views: "Views"
       }.freeze
 
-      SYMBOLS = {passed: "✓", failed: "✗", skipped: "-"}.freeze
+      SYMBOLS = {passed: "✓", failed: "✗", skipped: "-", warning: "!"}.freeze
 
-      attr_reader :table_class, :checks
+      attr_reader :table_class, :checks, :label
 
-      def initialize(table_class)
+      # +label+ names the context it was verified in, e.g. "user: User#1"
+      def initialize(table_class, strict: false, label: nil)
         @table_class = table_class
+        @strict = strict
+        @label = label
         @checks = []
       end
 
@@ -50,8 +59,12 @@ module Mensa
         table_class.is_a?(Class) && table_class < Mensa::Base
       end
 
+      def strict?
+        @strict
+      end
+
       def success?
-        failures.empty?
+        failures.empty? && (!strict? || warnings.empty?)
       end
 
       def failures
@@ -66,20 +79,28 @@ module Mensa
         checks.select(&:skipped?)
       end
 
+      def warnings
+        checks.select(&:warning?)
+      end
+
       def summary
-        "#{passes.size} passed, #{failures.size} failed, #{skips.size} skipped"
+        "#{passes.size} passed, #{failures.size} failed, #{skips.size} skipped, #{warnings.size} #{"warning".pluralize(warnings.size)}"
+      end
+
+      def title
+        [table_class.name || table_class.to_s, ("(#{label})" if label)].compact.join(" ")
       end
 
       def to_s
-        return "#{table_class}\n  - skipped: #{checks.first&.message}" unless mensa_table?
+        return "#{title}\n  - skipped: #{checks.first&.message}" unless mensa_table?
 
-        lines = [(table_class.name || table_class).to_s]
+        lines = [title]
 
         checks.group_by(&:section).each do |section, section_checks|
           lines << "  #{SECTIONS.fetch(section, section.to_s.humanize)}"
           section_checks.each do |check|
             lines << "    #{SYMBOLS[check.status]} #{check.label}#{" (skipped: #{check.message})" if check.skipped?}"
-            next unless check.failed?
+            next unless check.failed? || check.warning?
 
             check.message.to_s.each_line { |line| lines << "        #{line.chomp}" }
             lines << "        SQL: #{check.sql}" if check.sql.present?
@@ -87,13 +108,13 @@ module Mensa
         end
 
         lines << ""
-        lines << "#{success? ? "OK" : "FAILED"}: #{summary}"
+        lines << "#{success? ? "OK" : "FAILED"}: #{summary}#{" (strict: warnings fail)" if strict? && warnings.any?}"
         lines.join("\n")
       end
       alias_method :report, :to_s
 
       def inspect
-        "#<#{self.class.name} #{table_class} #{summary}>"
+        "#<#{self.class.name} #{title} #{summary}>"
       end
     end
   end

data/lib/mensa/verifier.rb

@@ -8,31 +8,45 @@ require "mensa/verifier/result"
 module Mensa
   # Checks that everything a table configures works against the real database:
   # its scope, every column's SQL, sorting, rendering, filters, search, the CSV
-  # export, grouping/aggregates and its views.
+  # export, grouping/aggregates and its views. It also warns about N+1 queries
+  # and slow queries.
   #
   #   result = Mensa::Verifier.new(UsersTable, user: User.first).call
   #   result.success? # => true
   #   puts result     # per-check report
   #
   # Checks go through the same code paths as requests do (filtered_scope,
-  # ordered_scope, Mensa::Cell, Mensa::ExportJob.write_csv, ...). Everything
-  # runs in a read-only transaction that is rolled back, and every check in its
-  # own savepoint so one failing query doesn't stop the others.
+  # ordered_scope, Mensa::Cell, the components, Mensa::ExportJob, ...).
+  # Everything runs in a read-only transaction that is rolled back, and every
+  # check in its own savepoint so one failing query doesn't stop the others.
   #
   # Tables often depend on request context, give it with user:, account: and
   # context: (all set on Current while verifying), params: (the table's
   # params) or a block, which is called with the verifier before the checks.
-  # Mensa.config.verifier_context supplies defaults for these.
+  # Mensa.config.verifier_context supplies defaults for these, or a list of
+  # them to verify a table in several contexts (see Verifier.verify).
   class Verifier
-    # Records loaded for record-level checks, and per query
+    # Records sampled from the first and the last page
     SAMPLE_SIZE = 5
+    LAST_PAGE_SAMPLE = 2
+
+    # Statements that aren't the table's queries
+    IGNORED_QUERY = /\A\s*(SAVEPOINT|RELEASE|ROLLBACK|BEGIN|COMMIT|SET|SHOW)\b/i
 
     class Skip < StandardError; end
 
     # A problem the verifier found itself, reported without the class name
     class Failure < StandardError; end
 
-    attr_reader :table_class, :user, :account, :context, :params, :result
+    # A record to render, and why it was sampled
+    Sample = Struct.new(:row, :reason) do
+      def description
+        id = row.record.id if row.record.respond_to?(:id)
+        [("record #{id}" if id), "(#{reason})"].compact.join(" ")
+      end
+    end
+
+    attr_reader :table_class, :user, :account, :context, :params, :label, :result
 
     class << self
       # Accepts a table class, its name (e.g. "UsersTable") or Mensa's table
@@ -55,21 +69,44 @@ module Mensa
         }.uniq.sort_by(&:name)
       end
 
+      # Verifies +table+ in every context Mensa.config.verifier_context
+      # returns for it (a hash, or an array of them), with +options+ on top.
+      # Returns one result per context.
+      def verify(table, **options, &setup)
+        table_class = resolve_table_class(table)
+        options = options.compact
+
+        contexts_for(table_class).map do |defaults|
+          new(table_class, **options, defaults: defaults, &setup).call
+        end
+      end
+
+      # The option hashes Mensa.config.verifier_context gives for a table
+      def contexts_for(table_class)
+        return [{}] unless table_class.is_a?(Class) && table_class < Mensa::Base
+
+        contexts = Mensa.config.verifier_context&.call(table_class)
+        contexts = [contexts] unless contexts.is_a?(Array)
+        contexts.map { |context| context.is_a?(Hash) ? context.symbolize_keys : {} }.presence || [{}]
+      end
+
       # Verifies the tables (all under app/tables when none are given), prints
       # their reports to +io+ and returns true when all of them passed.
-      # Context comes from USER_ID, ACCOUNT_ID and PARAMS (JSON) in +env+, used
-      # by the mensa:verify rake tasks.
+      # Context comes from USER_ID, ACCOUNT_ID and PARAMS (JSON) in +env+, and
+      # STRICT=1 makes warnings fail, used by the mensa:verify rake tasks.
       def run(tables = nil, env: ENV, io: $stdout)
         classes = tables.present? ? Array(tables).map { |table| resolve_table_class(table) } : table_classes
         options = options_from_env(env)
 
-        results = classes.map do |klass|
-          new(klass, **options).call.tap { |result| io.puts(result, "") }
+        results = classes.flat_map do |klass|
+          verify(klass, **options).each { |result| io.puts(result, "") }
         end
 
-        tables, others = results.partition(&:mensa_table?)
+        by_table = results.select(&:mensa_table?).group_by(&:table_class)
+        others = results.reject(&:mensa_table?)
         if results.size > 1
-          io.puts ["#{tables.count(&:success?)}/#{tables.size} tables passed", ("#{others.size} skipped, not a Mensa::Base subclass" if others.any?)].compact.join(", ")
+          passed = by_table.count { |_table, table_results| table_results.all?(&:success?) }
+          io.puts ["#{passed}/#{by_table.size} tables passed", ("#{others.size} skipped, not a Mensa::Base subclass" if others.any?)].compact.join(", ")
         end
         results.all?(&:success?)
       end
@@ -79,20 +116,26 @@ module Mensa
         options[:user] = "User".constantize.find(env["USER_ID"]) if env["USER_ID"].present?
         options[:account] = "Account".constantize.find(env["ACCOUNT_ID"]) if env["ACCOUNT_ID"].present?
         options[:params] = JSON.parse(env["PARAMS"]) if env["PARAMS"].present?
+        options[:strict] = true if %w[1 true yes].include?(env["STRICT"].to_s.downcase)
         options
       end
     end
 
-    def initialize(table, user: nil, account: nil, context: {}, params: nil, view_context: nil, &setup)
+    # +strict+ makes warnings fail, +slow_query+ is the duration in seconds
+    # from which a query is reported (Mensa.config.verifier_slow_query).
+    # +defaults+ replaces the first context of Mensa.config.verifier_context.
+    def initialize(table, user: nil, account: nil, context: {}, params: nil, label: nil, strict: false, slow_query: nil, view_context: nil, defaults: nil, &setup)
       @table_class = self.class.resolve_table_class(table)
-      defaults = default_options
+      defaults ||= self.class.contexts_for(@table_class).first
       @user = user || defaults[:user]
       @account = account || defaults[:account]
       @context = (defaults[:context] || {}).merge(context || {})
       @params = params || defaults[:params] || {}
+      @label = label || defaults[:label] || context_label
+      @slow_query = slow_query || defaults[:slow_query] || Mensa.config.verifier_slow_query
       @view_context = view_context
       @setup = setup
-      @result = Result.new(@table_class)
+      @result = Result.new(@table_class, strict: strict || defaults[:strict], label: @label)
     end
 
     def call
@@ -104,10 +147,15 @@ module Mensa
         return result
       end
 
-      read_only_transaction do
-        with_current_attributes do
-          @setup&.call(self)
-          run_checks
+      @queries = []
+      @reported_n_plus_one = Set.new
+      thread = Thread.current
+      ActiveSupport::Notifications.subscribed(->(event) { record_query(event) if Thread.current == thread }, "sql.active_record") do
+        read_only_transaction do
+          with_current_attributes do
+            @setup&.call(self)
+            run_checks
+          end
         end
       end
       result
@@ -139,15 +187,18 @@ module Mensa
 
     private
 
-    def default_options
-      options = Mensa.config.verifier_context&.call(table_class)
-      options.is_a?(Hash) ? options.symbolize_keys : {}
-    end
-
     def default_host
       Rails.application.routes.default_url_options[:host] || "localhost"
     end
 
+    # Names the context in the report, e.g. "user: User#1, params: {...}"
+    def context_label
+      values = context.symbolize_keys.merge({user: user, account: account}.compact)
+      parts = values.map { |key, value| "#{key}: #{value.is_a?(ActiveRecord::Base) ? "#{value.class.name}##{value.id}" : value.inspect}" }
+      parts << "params: #{params.to_json}" if params.present?
+      parts.join(", ").presence
+    end
+
     def run_checks
       @table = check(:loads, nil, "build table") { build_table }
       unless @table
@@ -158,23 +209,31 @@ module Mensa
       # A table of its own, a relation can't be counted after its to_sql
       check(:loads, nil, "build SQL") { |c| c.sql = build_table.selected_scope.to_sql }
       @rows = check(:loads, nil, "load first page") { @table.rows }
-      verify_second_page
+      verify_last_page
       verify_defaults
+      @samples = check(:loads, nil, "sample records") { |c| load_samples.tap { |samples| c.name = sample_name(samples) } } if @rows
 
       verify_columns
       verify_filters
       verify_search
       verify_rendering
+      verify_combined
       verify_exports
       verify_grouping
       verify_views
     end
 
+    # Checks
+
     # Runs the block as one check in its own savepoint and records the
-    # outcome. Returns the block's value, nil when the check didn't pass.
-    # Quiet checks are only reported when they fail.
+    # outcome, then any warnings found while running it. Returns the block's
+    # value, nil when the check didn't pass. Quiet checks are only reported
+    # when they fail.
     def check(section, subject, name, quiet: false)
       c = Check.new(section: section, subject: subject, name: name)
+      outer_warnings = @warnings
+      @warnings = []
+      start = @queries.size
       value = nil
       savepoint { value = yield(c) }
       c.status = :passed
@@ -183,20 +242,28 @@ module Mensa
       c.status = :skipped
       c.message = e.message
       nil
-    rescue StandardError, NotImplementedError => e
+    rescue StandardError, ScriptError => e
       c.status = :failed
       c.error = e
-      c.message = e.is_a?(Failure) ? e.message : "#{e.class}: #{e.message}"
+      c.message = e.is_a?(Failure) ? e.message : "#{e.class}: #{e.message}#{error_hint(e)}"
       c.sql = e.sql if e.respond_to?(:sql) && e.sql.present?
       nil
     ensure
       result.checks << c unless quiet && c.passed?
+      warn_slow_queries(section, subject, @queries[start..]) if c.passed?
+      result.checks.concat(@warnings)
+      @warnings = outer_warnings
     end
 
     def skip!(reason)
       raise Skip, reason
     end
 
+    def warn!(section, subject, name, message, sql = nil)
+      warning = Check.new(section: section, subject: subject, name: name, status: :warning, message: message, sql: sql)
+      @warnings ? @warnings << warning : result.checks << warning
+    end
+
     def savepoint
       ActiveRecord::Base.transaction(requires_new: true) do
         yield
@@ -204,22 +271,159 @@ module Mensa
       end
     end
 
-    # Record-level checks need loaded records
+    # The block's value, or nil when it raises
+    def attempt
+      value = nil
+      savepoint { value = yield }
+      value
+    rescue
+      nil
+    end
+
+    # Fixes for errors that have a common cause
+    def error_hint(error)
+      error = error.cause if error.is_a?(ActionView::Template::Error) && error.cause
+      return unless error.is_a?(ActiveModel::MissingAttributeError)
+
+      attribute = error.message[/missing attribute '?(\w+)'?/, 1]
+      "\nFix: Mensa only selects the table's columns, add `internal :#{attribute}` to select #{attribute} too." if attribute
+    end
+
+    # Queries
+
+    def record_query(event)
+      payload = event.payload
+      return if payload[:cached] || %w[SCHEMA TRANSACTION].include?(payload[:name]) || payload[:sql].to_s.match?(IGNORED_QUERY)
+
+      @queries << {sql: payload[:sql], duration: event.duration / 1000.0}
+    end
+
+    def warn_slow_queries(section, subject, queries)
+      slow = queries.select { |query| query[:duration] >= @slow_query }
+      return if slow.empty?
+
+      slowest = slow.max_by { |query| query[:duration] }
+      warn!(section, subject, "slow query",
+        "took #{format("%.2f", slowest[:duration])}s (reported from #{@slow_query}s#{", #{slow.size} slow queries" if slow.size > 1}).\n" \
+        "Fix: check its plan with EXPLAIN ANALYZE, e.g. for a missing index.", slowest[:sql])
+    end
+
+    # Records
+
+    # Samples to render: records from the first and the last page, and a
+    # record with an empty value for each column, where null values often
+    # break render blocks.
+    def load_samples
+      samples = @rows.first(SAMPLE_SIZE).map { |row| Sample.new(row, "first page") }
+
+      pages = @table.pagy_details.pages
+      build_table_on_page(pages).rows.last(LAST_PAGE_SAMPLE).each { |row| samples << Sample.new(row, "last page") } if pages > 1
+
+      @table.columns.reject(&:internal?).select(&:filter?).each do |column|
+        next if column.attribute_for_condition.blank?
+        next if samples.any? { |sample| attempt { sample.row.value(column).nil? } }
+
+        sample = attempt do
+          table = build_table(filters: {column.name => {operator: :is_empty}})
+          record = table.selected_scope.limit(1).first
+          Sample.new(Mensa::Row.new(table, record), "#{column.name} empty") if record
+        end
+        samples << sample if sample
+      end
+
+      samples.uniq { |sample| sample.row.record.respond_to?(:id) ? sample.row.record.id : sample.row.record.object_id }
+    end
+
+    def sample_name(samples)
+      "sample #{samples.size} #{"record".pluralize(samples.size)}#{" (#{samples.map(&:reason).uniq.join(", ")})" if samples.any?}"
+    end
+
+    def build_table_on_page(page)
+      table = build_table
+      table.request = ActionDispatch::Request.new(Rack::MockRequest.env_for("http://#{default_host}/?page=#{page}"))
+      table
+    end
+
+    # Record-level checks need sampled records
     def sample_rows!
-      skip!("records could not be loaded") if @rows.nil?
-      skip!("no records") if @rows.empty?
+      skip!("records could not be loaded") if @samples.nil?
+      skip!("no records") if @samples.empty?
+
+      @samples
+    end
+
+    # Renders every sample in its own savepoint. Fails naming the records that
+    # failed, and warns about queries that run for each record (N+1).
+    def render_samples(section, subject, what)
+      errors = []
+      queries_per_record = sample_rows!.map do |sample|
+        start = @queries.size
+        begin
+          savepoint { yield sample.row }
+        rescue StandardError, ScriptError => e
+          errors << "#{sample.description}: #{e.class}: #{e.message}#{error_hint(e)}"
+        end
+        @queries[start..].map { |query| query[:sql] }.uniq
+      end
 
-      @rows.first(SAMPLE_SIZE)
+      warn_n_plus_one(section, subject, what, queries_per_record)
+      raise Failure, errors.join("\n") if errors.any?
     end
 
-    def verify_second_page
-      check(:loads, nil, "load page 2") do
+    def warn_n_plus_one(section, subject, what, queries_per_record)
+      return if queries_per_record.size < 2
+
+      queries_per_record.flatten.tally.each do |sql, records|
+        next if records < 2
+        next unless @reported_n_plus_one.add?([subject, sql])
+
+        warn!(section, subject, "N+1 query in #{what}",
+          "runs this query for each record (#{records} of #{queries_per_record.size} sampled records).#{n_plus_one_hint(sql)}", sql)
+      end
+    end
+
+    # How to avoid a query per record, from the association it loads
+    def n_plus_one_hint(sql)
+      model = @table.model
+      table = sql[/\bFROM\s+"?(\w+)"?/i, 1]
+      reflection = model.reflect_on_all_associations.find do |association|
+        !association.polymorphic? && association.klass.table_name == table
+      rescue NameError
+        false
+      end
+      return "\nFix: select the value in SQL with `attribute`, or preload what the block uses in `scope`." unless reflection
+
+      preload = "`scope { #{model.name}.includes(:#{reflection.name}) }`"
+      hints = if reflection.belongs_to?
+        foreign_key = reflection.foreign_key.to_s
+        selected = @table.columns.any? { |column| column.name.to_s == foreign_key }
+        [
+          "preload it with #{preload}#{" and add `internal :#{foreign_key}`, Mensa only selects the table's columns and the preload needs the foreign key" unless selected}",
+          "or select the value in SQL with `attribute \"#{table}.<column>\"` and `left_outer_joins(:#{reflection.name})` in `scope`"
+        ]
+      else
+        hints = ["preload it with #{preload}"]
+        if sql.match?(/\bCOUNT\(/i) && reflection.collection?
+          hints << "and use `.size` instead of `.count`, which always queries"
+          unless reflection.through_reflection?
+            hints << "or count in SQL with `attribute \"(SELECT COUNT(*) FROM #{table} WHERE #{table}.#{reflection.foreign_key} = #{model.table_name}.#{model.primary_key})\"` and `type :integer`"
+          end
+        end
+        hints
+      end
+      "\nFix: #{hints.join(",\n     ")}."
+    end
+
+    # Loads
+
+    def verify_last_page
+      check(:loads, nil, "load last page") do
         skip!("records could not be loaded") if @rows.nil?
-        skip!("only one page") if @table.pagy_details.pages < 2
 
-        table = build_table
-        table.request = ActionDispatch::Request.new(Rack::MockRequest.env_for("http://#{default_host}/?page=2"))
-        table.rows
+        pages = @table.pagy_details.pages
+        skip!("only one page") if pages < 2
+
+        build_table_on_page(pages).rows
       end
     end
 
@@ -248,7 +452,11 @@ module Mensa
       check(:columns, column.name, "select") do |c|
         skip!("no SQL attribute, the value comes from #{column.method ? "method :#{column.method}" : "the record"}") if column.attribute.blank?
 
-        relation = build_table.filtered_scope.select(Arel.sql(column.attribute)).limit(1)
+        # What Mensa selects besides the column: the id and internal columns,
+        # which preloads in the scope need
+        table = build_table
+        selects = (table.send(:id_column?) ? [:id] : []) + table.columns.select(&:internal?).filter_map(&:attribute)
+        relation = table.filtered_scope.select(*selects, Arel.sql(column.attribute)).limit(1)
         c.sql = relation.to_sql
         relation.load
       end
@@ -269,7 +477,7 @@ module Mensa
         check(:columns, column.name, "render #{format}") do
           skip!("not visible") unless visible
 
-          sample_rows!.each { |row| Mensa::Cell.new(row: row, column: column).render(format) }
+          render_samples(:columns, column.name, "render #{format}") { |row| Mensa::Cell.new(row: row, column: column).render(format) }
         end
       end
     end
@@ -292,8 +500,12 @@ module Mensa
           end
         end
 
-        check(:filters, column.name, filter_check_name(column, operators, values)) do |c|
+        invalid = invalid_value(column)
+        check(:filters, column.name, filter_check_name(column, operators, values, invalid)) do |c|
           errors = operators.filter_map { |operator, _label, with_value| filter_error(column, operator, filter_value(column, operator, with_value, values)) }
+          if invalid
+            errors += operators.select { |_operator, _label, with_value| with_value }.filter_map { |operator, *| filter_error(column, operator, invalid) }
+          end
           next if errors.empty?
 
           c.sql = errors.first[:sql]
@@ -302,8 +514,9 @@ module Mensa
       end
     end
 
-    def filter_check_name(column, operators, values)
-      "#{operators.map(&:first).join(", ")} (#{(column.filter.multiple? ? values : values.first).inspect})"
+    def filter_check_name(column, operators, values, invalid)
+      name = "#{operators.map(&:first).join(", ")} (#{(column.filter.multiple? ? values : values.first).inspect})"
+      invalid ? "#{name}, invalid input (#{invalid.inspect})" : name
     end
 
     # Applies the filter with one operator in its own savepoint, returns
@@ -318,7 +531,7 @@ module Mensa
         table.active_filters.each(&:to_s)
       end
       nil
-    rescue StandardError, NotImplementedError => e
+    rescue StandardError, ScriptError => e
       sql = e.sql if e.respond_to?(:sql)
       sql ||= relation.to_sql if relation
       {message: "#{operator}#{" #{value.inspect}" if value}: #{e.class}: #{e.message}", sql: sql}
@@ -361,6 +574,15 @@ module Mensa
       end
     end
 
+    # A value a user could type that the column's type can't hold
+    def invalid_value(column)
+      case column.type
+      when :integer, :bigint, :decimal, :float then "not a number"
+      when :date, :datetime, :timestamp then "not a date"
+      when :uuid then "not a uuid"
+      end
+    end
+
     # Search
 
     # Both search modes, the app may switch between them
@@ -373,7 +595,8 @@ module Mensa
             table = build_table(query: term)
             skip!("needs the pg_trgm extension") if mode == :fuzzy && !table.send(:fuzzy_search?, table.scope)
 
-            relation = table.ordered_scope.limit(SAMPLE_SIZE)
+            # Search uses every column, so load what requests load
+            relation = table.selected_scope.limit(SAMPLE_SIZE)
             c.sql = relation.to_sql
             relation.load
           end
@@ -391,15 +614,14 @@ module Mensa
 
     # A word from the first string value in the records, so it finds something
     def search_term
-      (@rows || []).first(SAMPLE_SIZE).each do |row|
-        @table.display_columns.each do |column|
-          value = row.value(column)
-          return value.split.first.first(20) if value.is_a?(String) && value.split.first.present?
-        end
+      @search_term ||= begin
+        words = (@rows || []).first(SAMPLE_SIZE).lazy.flat_map { |row|
+          @table.display_columns.map { |column| row.value(column) }
+        }.select { |value| value.is_a?(String) && value.split.first.present? }
+        words.first&.split&.first&.first(20) || "a"
+      rescue
+        "a"
       end
-      "a"
-    rescue
-      "a"
     end
 
     # Rendering, like the app's views and Mensa's controllers do
@@ -408,13 +630,15 @@ module Mensa
       check(:render, nil, "table component") { view_context.table(table_name, params: params) }
       check(:render, nil, "rows and filter pills") { render_page(build_table, view: @table.default_system_view) }
 
-      check(:render, nil, "row link") { sample_rows!.each(&:link) } if @table.link
+      check(:render, nil, "row link") { render_samples(:render, nil, "row link", &:link) } if @table.link
 
       return unless @table.actions?
 
       @table.actions.each do |action|
         check(:render, :actions, action.name.to_s) do
-          sample_rows!.each { |row| view_context.render(Mensa::RowAction::Component.new(table: @table, row: row, action: action)) }
+          render_samples(:render, :actions, action.name.to_s) do |row|
+            view_context.render(Mensa::RowAction::Component.new(table: @table, row: row, action: action))
+          end
         end
       end
     end
@@ -426,6 +650,37 @@ module Mensa
       view_context.render(Mensa::View::Component.new(table))
     end
 
+    # Everything a user can combine at once: search, a filter on every
+    # column, sorting on every column and grouping with aggregates
+    def verify_combined
+      check(:combined, nil, "search, filters, sorting and grouping together") do |c|
+        config = {query: search_term, filters: combined_filters}
+        config[:order] = @table.columns.select { |column| column.sortable? && !column.internal? }.to_h { |column| [column.name, :asc] }
+
+        groupable = @table.columns.find(&:groupable?)
+        if groupable && @table.scope.is_a?(ActiveRecord::Relation)
+          config[:group_by] = groupable.name
+          config[:aggregates] = @table.columns.select(&:aggregatable?).reject(&:internal?).to_h { |column| [column.name, column.aggregates.first] }
+        end
+
+        table = build_table(config)
+        c.sql = table.selected_scope.to_sql
+        render_page(build_table(config))
+      end
+    end
+
+    def combined_filters
+      @table.columns.select(&:filter?).each_with_object({}) do |column, filters|
+        operators = attempt { column.filter.operators } || []
+        operator, _label, with_value = operators.find { |_operator, _label, with_value| with_value } ||
+          operators.find { |operator, *| %i[is_empty isnt_empty].include?(operator) }
+        next unless operator
+
+        values = attempt { filter_values(column) } || [synthetic_value(column)]
+        filters[column.name] = {operator: operator, value: filter_value(column, operator, with_value, values)}.compact
+      end
+    end
+
     # Exports, generated by Mensa::ExportJob like a user's export request,
     # for the first page of the table and of each view
 
@@ -473,7 +728,7 @@ module Mensa
 
       expected = build_table((table_view&.config || {}).deep_symbolize_keys.merge(config.deep_symbolize_keys))
       headers = expected.display_columns.map { |column| column.name.to_s }
-      lines = CSV.parse(csv.force_encoding(Encoding::UTF_8).delete_prefix("\uFEFF"))
+      lines = CSV.parse(csv.force_encoding(Encoding::UTF_8).delete_prefix(""))
       raise Failure, "the export has no columns" if headers.empty?
       raise Failure, "the export has headers #{lines.first.inspect}, expected #{headers.inspect}" unless lines.first == headers
 

data/lib/mensa/version.rb

@@ -1,3 +1,3 @@
 module Mensa
-  VERSION = "0.6.18"
+  VERSION = "0.6.19"
 end

data/lib/tasks/mensa.rake

@@ -1,12 +1,12 @@
 namespace :mensa do
-  desc "Verify a Mensa table against the database, e.g. mensa:verify[users] (context: USER_ID=, ACCOUNT_ID=, PARAMS='{...}')"
+  desc "Verify a Mensa table against the database, e.g. mensa:verify[users] (context: USER_ID=, ACCOUNT_ID=, PARAMS='{...}', STRICT=1 fails on warnings)"
   task :verify, [:table] => :environment do |_task, args|
     abort "Usage: bin/rails \"mensa:verify[table_name]\"" if args[:table].blank?
 
     exit(1) unless Mensa::Verifier.run(args[:table])
   end
 
-  desc "Verify all Mensa tables in app/tables against the database (context: USER_ID=, ACCOUNT_ID=, PARAMS='{...}')"
+  desc "Verify all Mensa tables in app/tables against the database (context: USER_ID=, ACCOUNT_ID=, PARAMS='{...}', STRICT=1 fails on warnings)"
   task verify_all: :environment do
     exit(1) unless Mensa::Verifier.run
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mensa
 version: !ruby/object:Gem::Version
-  version: 0.6.18
+  version: 0.6.19
 platform: ruby
 authors:
 - Tom de Grunt