mensa 0.6.19 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b38d819bbd96ef6b2cb38e7b89c07c9a487eb7b92e8a191a92f4fd7e09cf1f7f
-  data.tar.gz: 52856b77830db55218a63213c4dc90a685686046175059cd78c53d8fb40d5527
+  metadata.gz: 064c836f400dd147fc8fde3fb1f38a49f7bd447c44e6d7493c055cd6dede6b19
+  data.tar.gz: b6fdbbdce0951fb43fdfd8f574cf6888ebe1c9f4e369752f4a497de85ca8b8f4
 SHA512:
-  metadata.gz: a6dc8ccce0bed9be30b8b9c4b8d5f255b1d808d48a32c88e1813c40043378b0dd8d485abc5a34aa9b7b41f21335300f0f79f9cf6f7b1f1d7982f7f312027dad1
-  data.tar.gz: 36262b2128d07c1919dee7c08a82f0d15af44be791b012757afe41c28dadf5aafcc75e3be4907c0d9369d319217273297c98a20e2ab15380c03766da7b2e43ee
+  metadata.gz: f96aca8ee7039304c3292456e0be10722faba2eecac2ba41030f632317998bbf0d1e4d3f649f5a46d675a4d06268dae07d216e536b80294c4990c805160df0b3
+  data.tar.gz: f629a69b4a6177d8aedc58f931fa00d328af953dca50cfd80c282989c40fd2434e2e655b5c1f55611568f893d016b5467b16c2925e8fff293b977be9f53d36fc

data/.gitignore

@@ -6,7 +6,7 @@
 /tmp/
 /test/dummy/db/*.sqlite3
 /test/dummy/db/*.sqlite3-*
-/test/dummy/log/*.log
+/test/dummy/log/*
 /test/dummy/storage/
 /test/dummy/app/assets/builds/
 /test/dummy/tmp/

data/.shipkit.yml

@@ -1,2 +1,2 @@
 ---
-previous_version: 0.6.18
+previous_version: 0.6.19

data/CHANGELOG.md

@@ -1,5 +1,31 @@
 # Changelog
 
+## 0.7.0
+
+- Filters only offer the operators that work for the column's type in SQL,
+  picking one no longer raises. `matches`/`does_not_match` (`LIKE`) are only
+  offered for string, text, citext and untyped attribute columns, so boolean,
+  float, decimal, uuid, time and other columns no longer get them.
+  `gt`/`gteq`/`lt`/`lteq` are only offered for number, date and time columns,
+  so boolean, uuid and text columns no longer get them. json/jsonb columns only
+  get `is_empty`/`isnt_empty` (and jsonb `is_duplicate`). Host apps no longer
+  need to patch `Mensa::Filter#operators` or list `operators` per column to
+  avoid these. An explicit `operators` list is used as given.
+- Time columns use a time input, and a filter value that isn't a time matches
+  nothing instead of raising.
+- The verifier warns when an `attribute` has no `type` but its SQL isn't text,
+  e.g. a boolean `EXISTS (...)`, and says which `type` to add.
+- Columns a `column_order` doesn't list are kept and shown after the listed
+  ones, instead of being dropped. Browser state or views saved before a column
+  was added no longer hide that column (also from the column customizer),
+  ignore filters and sorting on it, or leave its attribute unselected.
+- The verifier checks column order and visibility: a reversed order, an order
+  that lists only one column (like state saved before columns were added),
+  hiding each column and hiding all of them, each rendered and exported.
+- The verifier creates a custom view like a user saving the table's state
+  would, without saving it, and validates, replays and exports it.
+- The verifier confirms when it found no N+1 queries.
+
 ## 0.6.19
 
 - `Mensa::Verifier` checks a table against the database: scope, column SQL,

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    mensa (0.6.18)
+    mensa (0.6.19)
       csv
       importmap-rails
       pagy (>= 43)

data/README.md

@@ -175,7 +175,31 @@ end
 ```
 
 The operators are `is`, `isnt`, `matches`, `does_not_match`, `gt`, `gteq`, `lt`,
-`lteq`, `is_duplicate`, `is_current`, `is_empty` and `isnt_empty`.
+`lteq`, `is_duplicate`, `is_current`, `is_empty` and `isnt_empty`. Without an
+`operators` list, a column gets the ones that work for its type in SQL:
+
+| Column type | Operators besides `is_duplicate`, `is_empty` and `isnt_empty` |
+|---|---|
+| string, text, citext, or an `attribute` without `type` | `is`, `isnt`, `matches`, `does_not_match` |
+| integer, float, decimal, time | `is`, `isnt`, `gt`, `gteq`, `lt`, `lteq` |
+| date, datetime | `gt`, `gteq`, `lt`, `lteq` |
+| boolean, uuid, enums and other types | `is`, `isnt` |
+| jsonb, json | none (json also has no `is_duplicate`) |
+
+`matches` and `does_not_match` are left out when the filter has a `collection`, and
+`is_current` is only offered when `Current` has a method named after the column.
+
+Mensa reads the type from the database column. Give an `attribute` that isn't text a
+`type`, otherwise it's treated as text and gets the text operators:
+
+```ruby
+column(:has_orders) do
+  attribute "EXISTS (SELECT 1 FROM orders WHERE orders.user_id = users.id)"
+  type :boolean
+end
+```
+
+An `operators` list is used as given, also when it lists operators the type can't run.
 
 #### Custom filter scopes
 
@@ -382,13 +406,20 @@ real database, using the same code paths as requests do:
 - **Exports**: `Mensa::ExportJob` generates the first page of the export, also for
   each view and as a password protected zip. It needs the expected headers
   (skipped for `exportable false`).
+- **Column order and visibility**: the columns shown, rendered and exported in a
+  reversed order, with an order that lists only one column (like browser state or a
+  view saved before the other columns were added, which must still show them all),
+  with each column hidden and with all of them hidden.
 - **Combined**: search, a filter on every column, sorting on every column and
   grouping with aggregates, all at once.
 - **Grouping**: every groupable column and every aggregate, rendered with their
   group headers. A `group_by` or `aggregates` naming a column that doesn't allow
   it fails, because Mensa would silently ignore it.
 - **Views**: system views load, render and only refer to existing columns. With
-  custom views enabled, saved views load too.
+  custom views enabled, saved views load too, and a custom view is created like a
+  user saving the table's state would (search, filters, sorting, column order, a
+  hidden column and grouping), then validated, replayed and exported. It's never
+  saved, and needs a user to own it (`user:`).
 
 It also warns, without failing:
 
@@ -396,7 +427,11 @@ It also warns, without failing:
   each record. The warning shows the query and how to avoid it for the association
   it loads, e.g. preload it with `includes` (and `internal :customer_id`, since Mensa
   only selects the table's columns), select the value in SQL with `attribute`, or
-  count in SQL.
+  count in SQL. When none is found, the report confirms it under Rendering, e.g.
+  `✓ N+1 queries: none in 18 renders of 7 sampled records`. It needs at least two
+  records to tell.
+- **Untyped attributes**: an `attribute` without `type` whose SQL isn't text, e.g. a
+  boolean `EXISTS (...)`. The warning names the `type` to add.
 - **Slow queries**: queries that take `config.verifier_slow_query` seconds (1.0 by
   default) or more.
 

data/app/tables/mensa/base.rb

@@ -38,18 +38,16 @@ module Mensa
       order = order&.map(&:to_sym)
       return order if order.nil?
 
-      # Internal columns are never shown in the column customizer UI, so they
-      # are absent from any URL-supplied column_order. Always append them so
-      # that columns and selected_scope include their attributes.
       all_keys = (config[:columns]&.keys || []).map(&:to_sym)
-      # A column_order can come from browser state saved for different params
-      # (with different columns), so drop names that aren't columns here.
-      if all_keys.any?
-        order &= all_keys
-        order = all_keys if order.empty?
-      end
-      internal_keys = all_keys.select { |key| config.dig(:columns, key, :internal) }
-      (order | internal_keys)
+      return order if all_keys.empty?
+
+      # A column_order can come from browser state or a view saved for
+      # different params, or before columns were added, so drop names that
+      # aren't columns here and append the columns it doesn't list. Internal
+      # columns are never in it, as the column customizer doesn't show them.
+      # Appending keeps every column in columns and selected_scope, so no
+      # column disappears and their filters and attributes keep working.
+      (order & all_keys) | all_keys
     end
 
     # Returns all columns

data/app/tables/mensa/filter.rb

@@ -14,6 +14,15 @@ module Mensa
     config_reader :multiple
     config_reader :having?
 
+    # Types that LIKE works on, nil is an attribute without a type
+    TEXT_TYPES = [:string, :text, :citext, nil].freeze
+    # Types with a meaningful order, for gt, gteq, lt and lteq
+    RANGE_TYPES = %i[integer bigint float decimal money date datetime timestamp timestamptz time interval].freeze
+    # Types where an exact match on a value is of little use
+    POINT_IN_TIME_TYPES = %i[date datetime timestamp timestamptz].freeze
+    # Types that can't be compared with a value the user types
+    STRUCTURED_TYPES = %i[json jsonb].freeze
+
     class << self
       def OPERATORS
         [
@@ -102,10 +111,15 @@ module Mensa
         operators = operators.select { |op| config[:operators].include?(op[0]) }
       else
         operators.delete_if { |op| op[0] == :is_current } unless Current.method_defined?(column.name, false)
+        type = column.type
         operators.delete_if { |op| [:matches, :does_not_match].include?(op[0]) } if collection.present?
-        operators.delete_if { |op| [:matches, :does_not_match].include?(op[0]) } if column.type == :integer || column.type == :date || column.type == :datetime
-        operators.delete_if { |op| [:is, :isnt].include?(op[0]) } if column.type == :date || column.type == :datetime
-        operators.delete_if { |op| [:gt, :lt, :gteq, :lteq].include?(op[0]) } if column.type == :string || column.type.blank?
+        # Only these operators work for the column's type in SQL, the others
+        # raise, e.g. LIKE on a boolean
+        operators.delete_if { |op| [:matches, :does_not_match].include?(op[0]) } unless TEXT_TYPES.include?(type)
+        operators.delete_if { |op| [:gt, :lt, :gteq, :lteq].include?(op[0]) } unless RANGE_TYPES.include?(type)
+        operators.delete_if { |op| [:is, :isnt].include?(op[0]) } if POINT_IN_TIME_TYPES.include?(type) || STRUCTURED_TYPES.include?(type)
+        # json has no equality, so duplicates can't be grouped
+        operators.delete_if { |op| op[0] == :is_duplicate } if type == :json
       end
       operators
     end
@@ -162,6 +176,8 @@ module Mensa
         "date"
       when :datetime
         "datetime-local"
+      when :time
+        "time"
       else
         "text"
       end
@@ -242,7 +258,8 @@ module Mensa
       when :integer, :bigint then entry.strip.match?(/\A[+-]?\d+\z/)
       when :decimal, :float then entry.strip.match?(/\A[+-]?(\d+\.?\d*|\.\d+)\z/)
       when :date then Date.parse(entry).present?
-      when :datetime, :timestamp then Time.zone.parse(entry).present?
+      when :datetime, :timestamp, :timestamptz then Time.zone.parse(entry).present?
+      when :time then entry.strip.match?(/\A\d{1,2}:\d{2}(:\d{2}(\.\d+)?)?\z/)
       when :uuid then entry.strip.match?(UUID_FORMAT)
       else true
       end

data/lib/mensa/verifier/result.rb

@@ -36,6 +36,7 @@ module Mensa
         filters: "Filters",
         search: "Search",
         render: "Rendering",
+        layout: "Column order and visibility",
         combined: "Combined",
         exports: "Exports",
         grouping: "Grouping",

data/lib/mensa/verifier.rb

@@ -149,6 +149,7 @@ module Mensa
 
       @queries = []
       @reported_n_plus_one = Set.new
+      @per_record_renders = 0
       thread = Thread.current
       ActiveSupport::Notifications.subscribed(->(event) { record_query(event) if Thread.current == thread }, "sql.active_record") do
         read_only_transaction do
@@ -217,6 +218,8 @@ module Mensa
       verify_filters
       verify_search
       verify_rendering
+      verify_n_plus_one
+      verify_layout
       verify_combined
       verify_exports
       verify_grouping
@@ -373,6 +376,8 @@ module Mensa
     def warn_n_plus_one(section, subject, what, queries_per_record)
       return if queries_per_record.size < 2
 
+      @per_record_renders += 1
+
       queries_per_record.flatten.tally.each do |sql, records|
         next if records < 2
         next unless @reported_n_plus_one.add?([subject, sql])
@@ -458,10 +463,24 @@ module Mensa
         selects = (table.send(:id_column?) ? [:id] : []) + table.columns.select(&:internal?).filter_map(&:attribute)
         relation = table.filtered_scope.select(*selects, Arel.sql(column.attribute)).limit(1)
         c.sql = relation.to_sql
-        relation.load
+        result = table.model.connection.select_all(relation)
+        warn_untyped_attribute(column, result.column_types[column.name.to_s]&.type)
       end
     end
 
+    # The SQL type of an `attribute` decides which filter operators and values
+    # fit it. Without `type`, Mensa treats it as text: it offers LIKE and
+    # compares with text, which fails for e.g. a boolean expression.
+    def warn_untyped_attribute(column, sql_type)
+      return if column.config[:attribute].blank? || column.config[:type].present?
+      return if sql_type.nil? || Mensa::Filter::TEXT_TYPES.include?(sql_type)
+
+      type = {timestamptz: :datetime, timestamp: :datetime}.fetch(sql_type, sql_type)
+      warn!(:columns, column.name, "untyped attribute",
+        "its SQL is #{sql_type}, but without `type` Mensa treats it as text: filters offer text operators and compare with text.\n" \
+        "Fix: add `type :#{type}` to the column.")
+    end
+
     def verify_column_sort(column)
       check(:columns, column.name, "sort asc/desc") do
         %i[asc desc].each { |direction| build_table(order: {column.name => direction}).ordered_scope.limit(1).load }
@@ -567,7 +586,8 @@ module Mensa
       case column.type
       when :integer, :bigint, :decimal, :float then "1"
       when :date then Date.current.iso8601
-      when :datetime, :timestamp, :time then Time.current.strftime("%Y-%m-%dT%H:%M")
+      when :datetime, :timestamp, :timestamptz then Time.current.strftime("%Y-%m-%dT%H:%M")
+      when :time then "12:00"
       when :boolean then "true"
       when :uuid then SecureRandom.uuid
       else "a"
@@ -578,7 +598,8 @@ module Mensa
     def invalid_value(column)
       case column.type
       when :integer, :bigint, :decimal, :float then "not a number"
-      when :date, :datetime, :timestamp then "not a date"
+      when :date, :datetime, :timestamp, :timestamptz then "not a date"
+      when :time then "not a time"
       when :uuid then "not a uuid"
       end
     end
@@ -643,6 +664,21 @@ module Mensa
       end
     end
 
+    # Confirms no N+1 query was found, after everything rendered per record
+    # (column renders, the row link, row actions). Found ones are warnings.
+    def verify_n_plus_one
+      return if @reported_n_plus_one.any?
+
+      check(:render, nil, "N+1 queries") do |c|
+        skip!("records could not be loaded") if @samples.nil?
+        skip!("no records") if @samples.empty?
+        skip!("needs at least 2 sampled records, found 1") if @samples.one?
+        skip!("nothing is rendered per record") if @per_record_renders.zero?
+
+        c.name = "N+1 queries: none in #{@per_record_renders} #{"render".pluralize(@per_record_renders)} of #{@samples.size} sampled records"
+      end
+    end
+
     # What Mensa::TablesController#show renders into the table's turbo frame
     def render_page(table, view: nil)
       table.table_view = view
@@ -650,6 +686,57 @@ module Mensa
       view_context.render(Mensa::View::Component.new(table))
     end
 
+    # Column order and visibility, as users change them with the column
+    # customizer, or as saved views and browser state replay them
+
+    def verify_layout
+      names = @table.display_columns.map(&:name)
+
+      check(:layout, nil, "reversed column order") do
+        skip!("no columns are shown") if names.empty?
+
+        verify_layout_config({column_order: names.reverse}, names.reverse)
+      end
+
+      check(:layout, nil, "column order listing only :#{names.last} (e.g. saved before the others were added)") do
+        skip!("fewer than 2 columns are shown") if names.size < 2
+
+        verify_layout_config({column_order: [names.last]}, [names.last] + names[0..-2],
+          hint: "columns a column_order doesn't list should follow the listed ones")
+      end
+
+      check(:layout, nil, "hide each column (#{names.size})") do
+        skip!("no columns are shown") if names.empty?
+
+        errors = names.filter_map do |name|
+          savepoint { verify_layout_config({hidden_columns: [name]}, names - [name]) }
+          nil
+        rescue StandardError, ScriptError => e
+          ":#{name}: #{e.is_a?(Failure) ? e.message : "#{e.class}: #{e.message}"}"
+        end
+        raise Failure, errors.join("\n") if errors.any?
+      end
+
+      check(:layout, nil, "hide all columns") do
+        skip!("no columns are shown") if names.empty?
+
+        verify_layout_config({hidden_columns: names}, [])
+      end
+    end
+
+    # Builds the table with +config+, checks the columns it shows, renders it
+    # and exports it
+    def verify_layout_config(config, expected, hint: nil)
+      table = build_table(config)
+      shown = table.display_columns.map(&:name)
+      unless shown == expected
+        raise Failure, "shows columns #{shown.inspect}, expected #{expected.inspect}#{"\n#{hint}" if hint}"
+      end
+
+      render_page(table)
+      verify_export(config, columns_required: expected.any?) if @table.exportable?
+    end
+
     # Everything a user can combine at once: search, a filter on every
     # column, sorting on every column and grouping with aggregates
     def verify_combined
@@ -711,7 +798,7 @@ module Mensa
       @table.export_with_password? ? "csv (zip with password)" : "csv"
     end
 
-    def verify_export(config, table_view: nil)
+    def verify_export(config, table_view: nil, columns_required: true)
       config = JSON.parse({params: params.presence, page: 1}.compact.merge(config).to_json)
       export = Mensa::Export.new(
         table_name: table_name, table_view_id: table_view&.id, format: "plain_csv", scope: "current_page",
@@ -724,12 +811,14 @@ module Mensa
       data = file.read
       file.close!
       csv = (content_type == "application/zip") ? unzip(data, export.password) : data
-      raise Failure, "#{filename} is empty" if csv.blank?
+      raise Failure, "#{filename} is empty" if csv.blank? && columns_required
 
       expected = build_table((table_view&.config || {}).deep_symbolize_keys.merge(config.deep_symbolize_keys))
       headers = expected.display_columns.map { |column| column.name.to_s }
       lines = CSV.parse(csv.force_encoding(Encoding::UTF_8).delete_prefix(""))
-      raise Failure, "the export has no columns" if headers.empty?
+      raise Failure, "the export has no columns" if headers.empty? && columns_required
+      return if headers.empty?
+
       raise Failure, "the export has headers #{lines.first.inspect}, expected #{headers.inspect}" unless lines.first == headers
 
       records = expected.rows.size
@@ -812,13 +901,71 @@ module Mensa
         end
       end
 
-      return unless @table.supports_custom_views?
+      unless @table.supports_custom_views?
+        check(:views, "custom view", "create") { skip!("supports_custom_views is off") }
+        return
+      end
+
+      verify_custom_view
 
       saved_views.each do |view|
         check(:views, "saved view #{view.id}", view.name.to_s) { render_page(build_table((view.config || {}).deep_symbolize_keys), view: view) }
       end
     end
 
+    # Creates a custom view like a user saving the table's state would, but
+    # doesn't save it: it's validated, then replayed and exported like a saved
+    # view.
+    def verify_custom_view
+      check(:views, "custom view", "create and replay (not saved)") do |c|
+        skip!("custom views are owned by Current.user, give the verifier a user:") if @table.current_user.blank?
+
+        view = custom_view
+        raise Failure, "the view is invalid: #{view.errors.full_messages.to_sentence}" unless view.valid?
+
+        c.name = "create and replay #{view.config.keys.join(", ")} (not saved)"
+        # Like Mensa::TablesController#show and the views controller
+        render_page(build_table(view.config.deep_transform_keys(&:to_sym)), view: view)
+      end
+
+      return unless @table.exportable? && @table.current_user.present?
+
+      check(:exports, "custom view", export_format) do
+        verify_export(custom_view.config.deep_symbolize_keys.slice(:filters, :column_order, :hidden_columns, :group_by, :order))
+      end
+    end
+
+    # A view as Mensa::Tables::ViewsController#create builds it from the
+    # table's state in the browser
+    def custom_view
+      Mensa::TableView.new(
+        table_name: table_name,
+        name: "Mensa::Verifier",
+        description: "Created by Mensa::Verifier, not saved",
+        config: JSON.parse(custom_view_config.to_json),
+        user: @table.current_user
+      )
+    end
+
+    def custom_view_config
+      @custom_view_config ||= begin
+        columns = @table.display_columns
+        sortable = columns.find(&:sortable?)
+        groupable = @table.columns.find(&:groupable?)
+
+        config = {query: search_term, filters: combined_filters.first(2).to_h}
+        config[:order] = {sortable.name => :desc} if sortable
+        config[:column_order] = columns.map(&:name).reverse
+        config[:hidden_columns] = [columns.last.name] if columns.size > 1
+        if groupable && @table.scope.is_a?(ActiveRecord::Relation)
+          config[:group_by] = groupable.name
+          aggregatable = @table.columns.reject(&:internal?).find(&:aggregatable?)
+          config[:aggregates] = {aggregatable.name => aggregatable.aggregates.first} if aggregatable
+        end
+        config
+      end
+    end
+
     def saved_views
       Mensa::TableView.where(table_name: table_name, user: [nil, @table.current_user].uniq).order(:created_at).limit(25).to_a
     end

data/lib/mensa/version.rb

@@ -1,3 +1,3 @@
 module Mensa
-  VERSION = "0.6.19"
+  VERSION = "0.7.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mensa
 version: !ruby/object:Gem::Version
-  version: 0.6.19
+  version: 0.7.0
 platform: ruby
 authors:
 - Tom de Grunt