card-mod-lookup 0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: b4215382b1478803a91069e8f71e7a980a64d632c2d5b62e1bd69cf5fb0c7cbe
4
+ data.tar.gz: 746772c19ae1fa95a3d685a8f8befd8d2d2b516033908d99ae82a7ce0175eeed
5
+ SHA512:
6
+ metadata.gz: 03eef8d51212c24dffb854ea9d1727c8accd0cb1371a4d46f0b3c05ae478b209c3bd3cd86946ee9a4637e18513126fc342b26d9a7395f5656baaa45972df8c07
7
+ data.tar.gz: 3eab763cc4378322b543659203326ab436a4990b6ec8be550e5b2b97bd2ed4714c74e54db61c310b9eae44cb6e8573d426f4ee45baa0a370c53ebcd7f686ca3f
data/README.md ADDED
@@ -0,0 +1,118 @@
1
+ <!--
2
+ # @title README - mod: lookup
3
+ -->
4
+
5
+ # Lookup mod
6
+
7
+ This mod manages lookup tables that help decko sites scale as their decks
8
+ grow increasingly large and their queries increasingly complex.
9
+
10
+ ## Why
11
+
12
+ CQL is a powerful language for navigating card relationships. It is easy, for
13
+ example, to find all the cards of a given type that have a field card of a given
14
+ name that link to a third card with given content and on and on and on.
15
+
16
+ But to work, these CQL queries must be translated into SQL and executed,
17
+ and when a database grows large, it can become quite expensive to execute all
18
+ the implied self joins.
19
+
20
+ For example, consider the use case that first inspired this code: _answers_ on
21
+ WikiRate.org. WikiRate is a site that collects answers to questions about
22
+ companies. A given answer is a response to a given metric for a given year
23
+ for a given company. When you consider all the kinds of companies and metrics
24
+ and metric designers that WikiRate supports, you can imagine the queries
25
+ becoming quite complex if we have to re-join the cards table to itself every
26
+ time we want to consider a different variable.
27
+
28
+ The solution was to make a "lookup" table for answers that employs much more
29
+ conventional relational database design.
30
+
31
+ _An increasingly common Decko pattern is to design structures organically and
32
+ fluidly with cards and then to optimize those structures with lookup tables once
33
+ the structures have matured. This pattern can be a surprisingly pleasant change
34
+ to those accustomed to having to try to perfect their data structure before
35
+ they've collected any data._
36
+
37
+ ## How
38
+
39
+ ### LookupTable
40
+
41
+ To create a lookup table, you will need to create a database table (most
42
+ often by using Rails migrations) and a ruby class like the following `Country`
43
+ lookup:
44
+
45
+ in lib/country.rb:
46
+ ```
47
+ # class for lookup table named "countries"
48
+ class Country < Cardio::Record
49
+
50
+ # country_id in countries table corresponds to id of company card
51
+ @card_column = :country_id
52
+
53
+ # query of all cards in lookup
54
+ @card_query = { type_id: :company, trash: false }
55
+
56
+ include LookupTable
57
+
58
+ # The following three are equivalent.
59
+ # Each would populate a `continent_id` column based on a value returned
60
+ # by the continent_id method on the country card.
61
+
62
+ # explicit fetch method
63
+ def fetch_continent_id
64
+ card.continent_d
65
+ end
66
+
67
+ # fetcher with hash argument. hash value becomes method
68
+ fetcher continent_id: :continent_id
69
+
70
+ # fetcher with symbol arguments. column name and method must be the same
71
+ fetcher :continent_id
72
+
73
+ ```
74
+
75
+ ### Abstract::Lookup
76
+
77
+ in set/type/country.rb:
78
+ ```
79
+ # methods for accessing lookup entries
80
+ include_set Abstract::Lookup
81
+
82
+ # events for maintaining lookup sync
83
+ include_set Abstract::LookupEvents
84
+
85
+ # record class from above
86
+ def lookup_class
87
+ ::Country
88
+ end
89
+
90
+ # called by lookup instance; uses cards
91
+ def continent_id
92
+ fetch(:continent)&.id
93
+ end
94
+
95
+ # uses lookup table
96
+ def continent_id_from_lookup
97
+ lookup.continent_id
98
+ end
99
+
100
+ ```
101
+
102
+ ### Abstract::LookupField
103
+
104
+ in set/right/continent.rb
105
+ ```
106
+ # methods for maintaing companies table when continent changes
107
+ # (admittedly an infrequent occurence)
108
+ include_set Abstract::LookupField
109
+ ```
110
+
111
+
112
+ ### Abstract::LookupSearch
113
+
114
+ Include this set to gain a helpful framework for developing a filter interface
115
+ for lookup tables.
116
+
117
+
118
+
@@ -0,0 +1,43 @@
1
+ class Card
2
+ class LookupFilterQuery
3
+ # support methods for sort and page
4
+ module ActiveRecordExtension
5
+ # @params hash [Hash] key1: dir1, key2: dir2
6
+ def sort hash
7
+ hash.present? ? sort_by_hash(hash) : self
8
+ end
9
+
10
+ def paging args
11
+ return self unless valid_page_args? args
12
+ limit(args[:limit]).offset(args[:offset])
13
+ end
14
+
15
+ private
16
+
17
+ def valid_page_args? args
18
+ args.present? && args[:limit].to_i.positive?
19
+ end
20
+
21
+ def sort_by_hash hash
22
+ rel = self
23
+ hash.each do |fld, dir|
24
+ rel, fld = interpret_sort_field rel, fld
25
+ rel = rel.order Arel.sql("#{fld} #{dir}")
26
+ end
27
+ rel
28
+ end
29
+
30
+ def interpret_sort_field rel, fld
31
+ if (match = fld.match(/^(\w+)_bookmarkers$/))
32
+ sort_by_bookmarkers match[1], rel
33
+ else
34
+ [rel, fld]
35
+ end
36
+ end
37
+
38
+ def sort_by_bookmarkers type, rel
39
+ [Card::Bookmark.add_sort_join(rel, "#{table.name}.#{type}_id"), "cts.value"]
40
+ end
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,98 @@
1
+ class Card
2
+ class LookupFilterQuery
3
+ # shared filtering methods for FilterQuery classes built on lookup tables
4
+ module Filtering
5
+ def process_filters
6
+ normalize_filter_args
7
+ return if @empty_result
8
+ @filter_args.each { |k, v| process_filter_option k, v if v.present? }
9
+ @restrict_to_ids.each { |k, v| filter k, v }
10
+ end
11
+
12
+ def normalize_filter_args
13
+ # override
14
+ end
15
+
16
+ def process_filter_option key, value
17
+ if (method = filter_method key)
18
+ send method, key, value
19
+ else
20
+ try "#{key}_query", value
21
+ end
22
+ end
23
+
24
+ def filter_method key
25
+ case key
26
+ when *simple_filters
27
+ :filter_exact_match
28
+ when *card_id_filters
29
+ :filter_card_id
30
+ end
31
+ end
32
+
33
+ def filter_exact_match key, value
34
+ filter key, value if value.present?
35
+ end
36
+
37
+ def filter_card_id key, value
38
+ return unless (card_id = to_card_id value)
39
+
40
+ filter card_id_map[key], card_id
41
+ end
42
+
43
+ def not_ids_query value
44
+ add_condition "#{lookup_class.card_column} not in (?)", value.split(",")
45
+ end
46
+
47
+ def to_card_id value
48
+ if value.is_a?(Array)
49
+ value.map { |v| Card.fetch_id(v) }
50
+ else
51
+ Card.fetch_id(value)
52
+ end
53
+ end
54
+
55
+ def restrict_to_ids col, ids
56
+ ids = Array(ids)
57
+ @empty_result ||= ids.empty?
58
+ restrict_lookup_ids col, ids
59
+ end
60
+
61
+ def restrict_lookup_ids col, ids
62
+ existing = @restrict_to_ids[col]
63
+ @restrict_to_ids[col] = existing ? (existing & ids) : ids
64
+ end
65
+
66
+ def restrict_by_cql col, cql
67
+ cql.reverse_merge! return: :id, limit: 0
68
+ restrict_to_ids col, Card.search(cql)
69
+ end
70
+
71
+ def filter field, value, operator=nil
72
+ condition = "#{filter_table field}.#{field} #{op_and_val operator, value}"
73
+ add_condition condition, value
74
+ end
75
+
76
+ def filter_table _field
77
+ lookup_table
78
+ end
79
+
80
+ def op_and_val op, val
81
+ "#{db_operator op, val} #{db_value val}"
82
+ end
83
+
84
+ def add_condition condition, value
85
+ @conditions << condition
86
+ @values << value
87
+ end
88
+
89
+ def db_operator operator, value
90
+ operator || (value.is_a?(Array) ? "IN" : "=")
91
+ end
92
+
93
+ def db_value value
94
+ value.is_a?(Array) ? "(?)" : "?"
95
+ end
96
+ end
97
+ end
98
+ end
@@ -0,0 +1,115 @@
1
+ class Card
2
+ # base class for FilterQuery classes built on lookup tables
3
+ class LookupFilterQuery
4
+ include Filtering
5
+
6
+ attr_accessor :filter_args, :sort_args, :paging_args
7
+ class_attribute :card_id_map, :card_id_filters, :simple_filters
8
+
9
+ def initialize filter, sorting={}, paging={}
10
+ @filter_args = filter
11
+ @sort_args = sorting
12
+ @paging_args = paging
13
+
14
+ @conditions = []
15
+ @joins = []
16
+ @values = []
17
+ @restrict_to_ids = {}
18
+
19
+ process_sort
20
+ process_filters
21
+ end
22
+
23
+ def lookup_query
24
+ q = lookup_class.where lookup_conditions
25
+ q = q.joins(@joins.uniq) if @joins.present?
26
+ q
27
+ end
28
+
29
+ def lookup_table
30
+ @lookup_table ||= lookup_class.arel_table.name
31
+ end
32
+
33
+ def condition_sql conditions
34
+ lookup_class.sanitize_sql_for_conditions conditions
35
+ end
36
+
37
+ def lookup_relation
38
+ sort_and_page { lookup_query }
39
+ end
40
+
41
+ # @return args for AR's where method
42
+ def lookup_conditions
43
+ condition_sql([@conditions.join(" AND ")] + @values)
44
+ end
45
+
46
+ # TODO: support optionally returning lookup objects
47
+
48
+ # @return array of metric answer card objects
49
+ # if filtered by missing values then the card objects
50
+ # are newly instantiated and not in the database
51
+ def run
52
+ @empty_result ? [] : main_results
53
+ end
54
+
55
+ # @return [Array]
56
+ def count
57
+ @empty_result ? 0 : main_query.count
58
+ end
59
+
60
+ def limit
61
+ @paging_args[:limit]
62
+ end
63
+
64
+ def main_query
65
+ @main_query ||= lookup_query
66
+ end
67
+
68
+ def main_results
69
+ # puts "SQL: #{lookup_relation.to_sql}"
70
+ lookup_relation.map(&:card)
71
+ end
72
+
73
+ private
74
+
75
+ def sort_and_page
76
+ relation = yield
77
+ @sort_joins.uniq.each { |j| relation = relation.joins(j) }
78
+
79
+ relation.sort(@sort_hash).paging(@paging_args)
80
+ end
81
+
82
+ def process_sort
83
+ @sort_joins = []
84
+ @sort_hash = @sort_args.each_with_object({}) do |(by, dir), h|
85
+ h[sort_by(by)] = sort_dir(dir)
86
+ end
87
+ end
88
+
89
+ def sort_by sort_by
90
+ if (id_field = sort_by_cardname[sort_by])
91
+ sort_by_join sort_by, lookup_table, id_field
92
+ else
93
+ simple_sort_by sort_by
94
+ end
95
+ end
96
+
97
+ def sort_by_cardname
98
+ {}
99
+ end
100
+
101
+ def sort_dir dir
102
+ dir
103
+ end
104
+
105
+ def simple_sort_by sort_by
106
+ sort_by
107
+ end
108
+
109
+ def sort_by_join sort_by, from_table, from_id_field
110
+ @sort_joins <<
111
+ "JOIN cards as #{sort_by} ON #{sort_by}.id = #{from_table}.#{from_id_field}"
112
+ "#{sort_by}.key"
113
+ end
114
+ end
115
+ end
@@ -0,0 +1,104 @@
1
+ module LookupTable
2
+ # shared class methods for lookup tables.
3
+ module ClassMethods
4
+ attr_reader :card_column,
5
+ :card_query # cql that finds all items in the cards table
6
+
7
+ def for_card cardish
8
+ card_id = Card.id cardish
9
+ card_id ? where(card_column => card_id).take : nil
10
+ end
11
+ alias_method :find_for_card, :for_card
12
+
13
+ def new_for_card cardish
14
+ new.tap do |lookup|
15
+ lookup.card_id = Card.id cardish
16
+ end
17
+ end
18
+
19
+ # TODO: change to create_for_card for consistency
20
+ def create cardish
21
+ new_for_card(cardish).refresh
22
+ end
23
+
24
+ def create! cardish
25
+ lookup = new_for_card cardish
26
+ raise ActiveRecord::RecordInvalid, lookup if lookup.invalid?
27
+ lookup.refresh
28
+ end
29
+
30
+ def create_or_update cardish, *fields
31
+ lookup = for_card(cardish) || new_for_card(cardish)
32
+ fields = nil if lookup.new_record? # update all fields if record is new
33
+ lookup.refresh(*fields)
34
+ end
35
+
36
+ # @param ids [Array<Integer>] ids of answers in the answer table (NOT card ids)
37
+ def update_by_ids ids, *fields
38
+ Array(ids).each do |id|
39
+ next unless (entry = find_by_id(id))
40
+ entry.refresh(*fields)
41
+ end
42
+ end
43
+
44
+ def delete_for_card cardish
45
+ for_card(cardish)&.destroy
46
+ end
47
+
48
+ # @param ids [Integer, Array<Integer>] card ids of metric answer cards
49
+ def refresh ids=nil, *fields
50
+ Array(ids).compact.each do |ma_id|
51
+ refresh_entry fields, ma_id
52
+ end
53
+ end
54
+
55
+ def refresh_entry fields, card_id
56
+ if Card.exists? card_id
57
+ create_or_update card_id, *fields
58
+ else
59
+ delete_for_card card_id
60
+ end
61
+ rescue StandardError => e
62
+ raise e, "failed to refresh #{name} lookup table " \
63
+ "for card id #{card_id}: #{e.message}"
64
+ end
65
+
66
+ def refresh_all fields=nil
67
+ count = 0
68
+ Card.where(card_query).pluck_in_batches(:id) do |batch|
69
+ count += batch.size
70
+ puts "#{batch.first} - #{count}"
71
+ refresh(batch, *fields)
72
+ end
73
+ end
74
+
75
+ # define standard lookup fetch methods.
76
+ #
77
+ # Eg. fetcher(:company_id) defines #fetch_company_id to fetch from card.company_id
78
+ #
79
+ # And fetcher(foo: :bar) defines #fetch_foo to fetch from card.bar
80
+ def fetcher *args
81
+ fetcher_hash(*args).each { |col, method| define_fetch_method col, method }
82
+ end
83
+
84
+ def define_main_fetcher
85
+ define_fetch_method @card_column, :id
86
+ end
87
+
88
+ private
89
+
90
+ def define_fetch_method column, card_method
91
+ define_method "fetch_#{column}" do
92
+ card.send card_method
93
+ end
94
+ end
95
+
96
+ def fetcher_hash *args
97
+ if args.first.is_a?(Hash)
98
+ args.first
99
+ else
100
+ args.each_with_object({}) { |item, h| h[item] = item }
101
+ end
102
+ end
103
+ end
104
+ end
@@ -0,0 +1,64 @@
1
+ # lookup table to optimize complex card systems
2
+ #
3
+ # TODO: make this a class and have lookup classes inherit from it
4
+ module LookupTable
5
+ def self.included host_class
6
+ host_class.extend LookupTable::ClassMethods
7
+ host_class.define_main_fetcher
8
+ end
9
+
10
+ def card_column
11
+ self.class.card_column
12
+ end
13
+
14
+ def card
15
+ @card ||= Card.fetch send(card_column), look_in_trash: true
16
+ end
17
+
18
+ def card_id
19
+ send card_column
20
+ end
21
+
22
+ def card_id= id
23
+ send "#{card_column}=", id
24
+ end
25
+
26
+ def delete_on_refresh?
27
+ !card || card.trash
28
+ end
29
+
30
+ def refresh *fields
31
+ return delete if delete_on_refresh?
32
+
33
+ refresh_fields fields
34
+ raise Card::Error, "invalid #{self.class} lookup" if invalid?
35
+
36
+ save!
37
+ end
38
+
39
+ def refresh_fields fields=nil
40
+ keys = fields.present? ? fields : attribute_names
41
+ keys.delete("id")
42
+ keys.each { |method_name| refresh_value method_name }
43
+ end
44
+
45
+ def refresh_value method_name
46
+ send "#{method_name}=", send("fetch_#{method_name}")
47
+ end
48
+
49
+ def method_missing method_name, *args, &block
50
+ if card.respond_to? method_name
51
+ card.send method_name, *args, &block
52
+ else
53
+ super
54
+ end
55
+ end
56
+
57
+ def respond_to_missing? *args
58
+ card.respond_to?(*args) || super
59
+ end
60
+
61
+ def is_a? klass
62
+ klass == Card || super
63
+ end
64
+ end
@@ -0,0 +1,7 @@
1
+ def lookup_class
2
+ raise StandardError, "must define #lookup_class"
3
+ end
4
+
5
+ def lookup
6
+ lookup_class.for_card id
7
+ end
@@ -0,0 +1,12 @@
1
+ event :create_lookup, :finalize, on: :create do
2
+ lookup_class.create self
3
+ end
4
+
5
+ # lookup fields are often based on cards' compound names
6
+ event :refresh_lookup, :integrate, changed: :name, on: :update do
7
+ lookup.refresh
8
+ end
9
+
10
+ event :delete_lookup, :finalize, on: :delete do
11
+ lookup_class.delete_for_card id
12
+ end
@@ -0,0 +1,23 @@
1
+ def lookup_card
2
+ left
3
+ end
4
+
5
+ def lookup
6
+ lookup_card.lookup
7
+ end
8
+
9
+ def lookup_columns
10
+ Codename[right_id]
11
+ end
12
+
13
+ event :update_lookup_field, :finalize, changed: :content do
14
+ lookup_field_update do
15
+ lookup.refresh(*Array.wrap(lookup_columns))
16
+ end
17
+ end
18
+
19
+ private
20
+
21
+ def lookup_field_update
22
+ yield unless lookup_card.action.in? %i[create delete]
23
+ end
@@ -0,0 +1,94 @@
1
+ def search args={}
2
+ return_type = args.delete :return
3
+ query = args.delete(:query) || query(args)
4
+ run_query_returning query, return_type
5
+ end
6
+
7
+ def run_query_returning query, return_type
8
+ case return_type
9
+ when :name then query.run.map(&:name)
10
+ when :count then query.count
11
+ else query.run
12
+ end
13
+ end
14
+
15
+ def filter_class
16
+ raise Error::ServerError, "filter_class required!"
17
+ end
18
+
19
+ def query paging={}
20
+ filter_class.new query_hash, {}, paging
21
+ end
22
+
23
+ def query_hash
24
+ {}
25
+ end
26
+
27
+ def count
28
+ query.count
29
+ end
30
+
31
+ format do
32
+ delegate :filter_class, to: :card
33
+
34
+ def search_with_params
35
+ @search_with_params ||= card.search query: query
36
+ end
37
+
38
+ def count_with_params
39
+ @count_with_params ||= card.search query: count_query, return: :count
40
+ end
41
+
42
+ def query
43
+ filter_class.new query_hash, sort_hash, paging_params
44
+ end
45
+
46
+ def count_query
47
+ filter_class.new query_hash
48
+ end
49
+
50
+ def query_hash
51
+ (filter_hash || {}).merge card.query_hash
52
+ end
53
+
54
+ def default_filter_hash
55
+ card.query_hash
56
+ end
57
+
58
+ def sort_hash
59
+ { sort_by.to_sym => sort_dir }
60
+ end
61
+
62
+ def sort_dir
63
+ return unless sort_by
64
+ @sort_dir ||= safe_sql_param("sort_dir") || default_sort_dir(sort_by)
65
+ end
66
+
67
+ def default_sort_dir sort_by
68
+ if default_desc_sort_dir.include? sort_by.to_sym
69
+ :desc
70
+ else
71
+ :asc
72
+ end
73
+ end
74
+
75
+ def default_desc_sort_dir
76
+ []
77
+ end
78
+
79
+ def sort_by
80
+ @sort_by ||= sort_by_from_param || default_sort_option
81
+ end
82
+
83
+ def default_sort_option
84
+ # override
85
+ end
86
+
87
+ def sort_by_from_param
88
+ safe_sql_param(:sort_by)&.to_sym
89
+ end
90
+
91
+ def default_limit
92
+ Auth.signed_in? ? 5000 : 500
93
+ end
94
+ end
metadata ADDED
@@ -0,0 +1,69 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: card-mod-lookup
3
+ version: !ruby/object:Gem::Version
4
+ version: '0.1'
5
+ platform: ruby
6
+ authors:
7
+ - Philipp Kühl
8
+ - Ethan McCutchen
9
+ autorequire:
10
+ bindir: bin
11
+ cert_chain: []
12
+ date: 2021-10-26 00:00:00.000000000 Z
13
+ dependencies:
14
+ - !ruby/object:Gem::Dependency
15
+ name: card
16
+ requirement: !ruby/object:Gem::Requirement
17
+ requirements:
18
+ - - ">="
19
+ - !ruby/object:Gem::Version
20
+ version: '0'
21
+ type: :runtime
22
+ prerelease: false
23
+ version_requirements: !ruby/object:Gem::Requirement
24
+ requirements:
25
+ - - ">="
26
+ - !ruby/object:Gem::Version
27
+ version: '0'
28
+ description: ''
29
+ email:
30
+ - info@decko.org
31
+ executables: []
32
+ extensions: []
33
+ extra_rdoc_files: []
34
+ files:
35
+ - README.md
36
+ - lib/card/lookup_filter_query.rb
37
+ - lib/card/lookup_filter_query/active_record_extension.rb
38
+ - lib/card/lookup_filter_query/filtering.rb
39
+ - lib/lookup_table.rb
40
+ - lib/lookup_table/class_methods.rb
41
+ - set/abstract/lookup.rb
42
+ - set/abstract/lookup_events.rb
43
+ - set/abstract/lookup_field.rb
44
+ - set/abstract/lookup_search.rb
45
+ homepage: http://decko.org
46
+ licenses:
47
+ - GPL-3.0
48
+ metadata:
49
+ card-mod: lookup
50
+ post_install_message:
51
+ rdoc_options: []
52
+ require_paths:
53
+ - lib
54
+ required_ruby_version: !ruby/object:Gem::Requirement
55
+ requirements:
56
+ - - ">="
57
+ - !ruby/object:Gem::Version
58
+ version: '2.5'
59
+ required_rubygems_version: !ruby/object:Gem::Requirement
60
+ requirements:
61
+ - - ">="
62
+ - !ruby/object:Gem::Version
63
+ version: '0'
64
+ requirements: []
65
+ rubygems_version: 3.2.28
66
+ signing_key:
67
+ specification_version: 4
68
+ summary: lookup
69
+ test_files: []