fast_count 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3432010a9c6d6f616b7341df1dffbf3ea02b9fb22e846ab59c6562609ff109ff
+  data.tar.gz: 42b6a79b370de8a2c919bdc9744bee718436b7ecba75adf6c95cb82a3bafbe69
+SHA512:
+  metadata.gz: 0b61215c4ce6eb05626baba644ff34ba11e4c2fe08deec601f5313bdc4c6805594a4ee9dc1637345fdc1891d9bb17e12a251bfe57a17c96f807913ecfb64c83b
+  data.tar.gz: 9172798b85f35b0b82b1e91917feba59fc317697cea6786cf94549e23aa2625b284c482f7c7d7411ba21111f83c59c9af976d2afe10d4c7fdff0fd3cc5966f3c

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## master (unreleased)
+
+## 0.1.0 (2023-04-26)
+
+- First release

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 fatkodima
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,107 @@
+# FastCount
+
+[![Build Status](https://github.com/fatkodima/fast_count/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/fatkodima/fast_count/actions/workflows/ci.yml)
+
+Unfortunately, it's currently notoriously difficult and expensive to get an exact count on large tables.
+
+Luckily, there are [some tricks](https://www.citusdata.com/blog/2016/10/12/count-performance) for quickly getting fairly accurate estimates. For example, on a PostgreSQL table with over 450 million records, you can get a 99.82% accurate count within a fraction of the time. See the table below for an example dataset.
+
+| SQL | Result | Accuracy | Time |
+| --- | --- | --- | --- |
+| `SELECT count(*) FROM small_table;` | `2037104` | `100.000%` | `4.900s` |
+| `SELECT fast_count('small_table');` | `2036407` | `99.965%` | `0.050s` |
+| `SELECT count(*) FROM medium_table;` | `81716243` | `100.000%` | `257.5s` |
+| `SELECT fast_count('medium_table');` | `81600513` | `99.858%` | `0.048s` |
+| `SELECT count(*) FROM large_table;` | `455270802` | `100.000%` | `310.6s` |
+| `SELECT fast_count('large_table');` | `454448393` | `99.819%` | `0.046s` |
+
+*These metrics were pulled from real PostgreSQL databases being used in a production environment.*
+
+For MySQL, this gem uses internal statistics to return the estimated table's size. And as [per documentation](https://dev.mysql.com/doc/refman/8.0/en/show-table-status.html), it may vary from the actual value by as much as 40% to 50%.
+But still is useful to get a rough idea of the number of rows in very large tables (where `COUNT(*)` can literally take hours).
+
+Supports PostgreSQL, MySQL, MariaDB, and SQLite.
+
+## Requirements
+
+- Ruby 2.7+
+- ActiveRecord 6+
+
+If you need support for older versions, [open an issue](https://github.com/fatkodima/fast_count/issues/new).
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'fast_count'
+```
+
+And then execute:
+
+```sh
+$ bundle
+```
+
+Or install it yourself as:
+
+```sh
+$ gem install fast_count
+```
+
+If you are using PostgreSQL, you need to create a database function, used internally:
+
+```ruby
+class InstallFastCount < ActiveRecord::Migration[7.0]
+  def up
+    FastCount.install
+  end
+
+  def down
+    FastCount.uninstall
+  end
+end
+```
+
+## Usage
+
+To get an estimated count of the rows in a table:
+
+```ruby
+User.fast_count # => 1_254_312_219
+```
+
+If you want to quickly get an estimation of how many rows will the query return, without actually executing it, yo can run:
+
+```ruby
+User.where.missing(:avatar).estimated_count # => 324_200
+```
+
+**Note**: `estimated_count` relies on the database query planner estimations (basically on the output of `EXPLAIN`) to get its results and can be very imprecise. It is better be used to get an idea of the order of magnitude of the future result.
+
+## Configuration
+
+You can override the following default options:
+
+```ruby
+# Determines for how large tables this gem should get the exact row count using SELECT COUNT.
+# If the approximate row count is smaller than this value, SELECT COUNT will be used,
+# otherwise the approximate count will be used.
+FastCount.threshold = 100_000
+```
+
+## Credits
+
+Thanks to [quick_count gem](https://github.com/TwilightCoders/quick_count) for the original idea.
+
+## Development
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/fatkodima/fast_count.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/fast_count/adapters/base_adapter.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module FastCount
+  module Adapters
+    # @private
+    class BaseAdapter
+      def initialize(connection)
+        @connection = connection
+      end
+
+      def install; end
+      def uninstall; end
+    end
+  end
+end

data/lib/fast_count/adapters/mysql_adapter.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module FastCount
+  module Adapters
+    # @private
+    class MysqlAdapter < BaseAdapter
+      # Documentation says, that this value may vary from
+      # the actual value by as much as 40% to 50%.
+      def fast_count(table_name, threshold)
+        estimate = @connection.select_one("SHOW TABLE STATUS LIKE #{@connection.quote(table_name)}")["Rows"]
+        if estimate >= threshold
+          estimate
+        else
+          @connection.select_value("SELECT COUNT(*) FROM #{@connection.quote_table_name(table_name)}")
+        end
+      end
+
+      # Tree format was added in MySQL 8.0.16.
+      # For other formats I wasn't able to find an easy way to get this count.
+      def estimated_count(sql)
+        query_plan = @connection.select_value("EXPLAIN format=tree #{sql}")
+        query_plan.match(/rows=(\d+)/)[1].to_i
+      end
+    end
+  end
+end

data/lib/fast_count/adapters/postgresql_adapter.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+module FastCount
+  module Adapters
+    # @private
+    class PostgresqlAdapter < BaseAdapter
+      def install
+        @connection.execute(<<~SQL)
+          CREATE FUNCTION fast_count(table_name text, threshold bigint) RETURNS bigint AS $$
+          DECLARE count bigint;
+            BEGIN
+              EXECUTE '
+                WITH tables_counts AS (
+                  -- inherited and partitioned tables counts
+                  SELECT
+                    ((SUM(child.reltuples::float) / greatest(SUM(child.relpages), 1))) *
+                      (SUM(pg_relation_size(child.oid))::float / (current_setting(''block_size'')::float))::integer AS estimate
+                  FROM pg_inherits
+                    INNER JOIN pg_class parent ON pg_inherits.inhparent = parent.oid
+                    INNER JOIN pg_class child  ON pg_inherits.inhrelid  = child.oid
+                  WHERE parent.relname = ''' || table_name || '''
+
+                  UNION ALL
+
+                  -- table count
+                  SELECT
+                    (reltuples::float / greatest(relpages, 1)) *
+                      (pg_relation_size(pg_class.oid)::float / (current_setting(''block_size'')::float))::integer AS estimate
+                  FROM pg_class
+                  WHERE relname = '''|| table_name ||'''
+                )
+
+                SELECT
+                  CASE
+                  WHEN SUM(estimate) < '|| threshold ||' THEN (SELECT COUNT(*) FROM "'|| table_name ||'")
+                  ELSE SUM(estimate)
+                  END AS count
+                FROM tables_counts' INTO count;
+              RETURN count;
+            END
+          $$ LANGUAGE plpgsql;
+        SQL
+      end
+
+      def uninstall
+        @connection.execute("DROP FUNCTION IF EXISTS fast_count(text, bigint)")
+      end
+
+      def fast_count(table_name, threshold)
+        @connection.select_value(
+          "SELECT fast_count(#{@connection.quote(table_name)}, #{@connection.quote(threshold)})"
+        ).to_i
+      end
+
+      def estimated_count(sql)
+        query_plan = @connection.select_value("EXPLAIN #{sql}")
+        query_plan.match(/rows=(\d+)/)[1].to_i
+      end
+    end
+  end
+end

data/lib/fast_count/adapters/sqlite_adapter.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module FastCount
+  module Adapters
+    # @private
+    # No one should use sqlite in production and moreover with lots of data,
+    # so we can just use `SELECT COUNT(*)`. Support for it is technically not needed,
+    # but was added for convenience in development.
+    #
+    class SqliteAdapter < BaseAdapter
+      def fast_count(table_name, _threshold)
+        @connection.select_value("SELECT COUNT(*) FROM #{@connection.quote_table_name(table_name)}")
+      end
+
+      def estimated_count(sql)
+        @connection.select_value("SELECT COUNT(*) FROM (#{sql})")
+      end
+    end
+  end
+end

data/lib/fast_count/adapters.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require_relative "adapters/base_adapter"
+require_relative "adapters/postgresql_adapter"
+require_relative "adapters/mysql_adapter"
+require_relative "adapters/sqlite_adapter"
+
+module FastCount
+  # @private
+  module Adapters
+    def self.for_connection(connection)
+      adapter_name = Utils.adapter_name(connection)
+      lookup(adapter_name).new(connection)
+    end
+
+    def self.lookup(name)
+      const_get("#{name.to_s.camelize}Adapter")
+    end
+  end
+end

data/lib/fast_count/extensions.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module FastCount
+  module Extensions
+    module ModelExtension
+      # @example
+      #   User.fast_count
+      #   User.fast_count(threshold: 50_000)
+      #
+      def fast_count(threshold: FastCount.threshold)
+        adapter = Adapters.for_connection(connection)
+        adapter.fast_count(table_name, threshold)
+      end
+    end
+
+    module RelationExtension
+      # @example
+      #   User.where.missing(:avatar).estimated_count
+      #
+      def estimated_count
+        adapter = Adapters.for_connection(connection)
+        adapter.estimated_count(to_sql)
+      end
+    end
+  end
+end

data/lib/fast_count/utils.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module FastCount
+  # @private
+  module Utils
+    def self.adapter_name(connection)
+      case connection.adapter_name
+      when /postg/i # PostgreSQL, PostGIS
+        :postgresql
+      when /mysql/i
+        :mysql
+      when /sqlite/i
+        :sqlite
+      end
+    end
+  end
+end

data/lib/fast_count/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module FastCount
+  VERSION = "0.1.0"
+end

data/lib/fast_count.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "active_record"
+
+require_relative "fast_count/utils"
+require_relative "fast_count/adapters"
+require_relative "fast_count/extensions"
+require_relative "fast_count/version"
+
+module FastCount
+  class << self
+    def install(connection: ActiveRecord::Base.connection)
+      adapter = Adapters.for_connection(connection)
+      adapter.install
+    end
+
+    def uninstall(connection: ActiveRecord::Base.connection)
+      adapter = Adapters.for_connection(connection)
+      adapter.uninstall
+    end
+
+    # Determines for how large tables this gem should get the exact row count using SELECT COUNT.
+    # If the approximate row count is smaller than this value, SELECT COUNT will be used,
+    # otherwise the approximate count will be used.
+    attr_accessor :threshold
+  end
+
+  self.threshold = 100_000
+end
+
+ActiveSupport.on_load(:active_record) do
+  extend FastCount::Extensions::ModelExtension
+  ActiveRecord::Relation.include(FastCount::Extensions::RelationExtension)
+end

metadata

@@ -0,0 +1,73 @@
+--- !ruby/object:Gem::Specification
+name: fast_count
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- fatkodima
+- Dale Stevens
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2023-04-26 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+description:
+email:
+- fatkodima123@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/fast_count.rb
+- lib/fast_count/adapters.rb
+- lib/fast_count/adapters/base_adapter.rb
+- lib/fast_count/adapters/mysql_adapter.rb
+- lib/fast_count/adapters/postgresql_adapter.rb
+- lib/fast_count/adapters/sqlite_adapter.rb
+- lib/fast_count/extensions.rb
+- lib/fast_count/utils.rb
+- lib/fast_count/version.rb
+homepage: https://github.com/fatkodima/fast_count
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/fatkodima/fast_count
+  source_code_uri: https://github.com/fatkodima/fast_count
+  changelog_uri: https://github.com/fatkodima/fast_count/blob/master/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.12
+signing_key:
+specification_version: 4
+summary: Quickly get a count estimation for large tables.
+test_files: []