trifle-stats 1.6.0 → 2.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eece6621a74a0dfc38e7e1932897ad0481e3b85744ec85d016320cc6c7e5b395
-  data.tar.gz: a90a4b7bf5e885fdc77af3638b2fcb0fdef0f48e17001b2da925161164fd31f4
+  metadata.gz: cace74102291a5e8b9d1de50cc489ea5ca0514e7b23a4a73fb218f9626569f44
+  data.tar.gz: 7149e5a8928348b7631897abb94c33beca2963b9e1a4d8c1490168e2a2bf4e4b
 SHA512:
-  metadata.gz: 7bb55a42c115a0e353277b7445f33db93bc16a6011202de515adb63126d0d154615bbc2b13bfad45bd6ca990aedec32de4911c5843553b2c5148676c01d21b99
-  data.tar.gz: 475e37cab204a8b07ea90ead1c209cb784844ba3683884362f284d41c0f2e8170f9f7ec858125dece018312561d934efd555a41ac6457ec16af2d31e36be0695
+  metadata.gz: 6578def9715b7cdee5c236ea592f482888c0c29304224bc5cdae419df49900613cd115a0c5f3e2e920f04aea02cdbe433ad0742b6ceaf981a3a46ecd5c3c541b
+  data.tar.gz: 4d257ea260be8adc9d8a1868199cf1ceb983fc240312b13dc63444bf21ff1c0c2493aa2118a80001c3419758536a2d7e7678bb852b2583930f1e0b58f72b17ad

data/.devops/docker/local/docker-compose.yml

@@ -12,7 +12,7 @@ services:
   app:
     command: /bin/sh -c "while sleep 1000; do :; done"
     build:
-      context: ../../..
+      context: .
       dockerfile: .devops/docker/local/Dockerfile
     depends_on:
       - postgres
@@ -26,5 +26,5 @@ services:
     expose:
       - 4000
     volumes:
-      - ../../..:/workspaces/stats
-    working_dir: /workspaces/stats
+      - ..:/workspaces
+    working_dir: /workspaces/trifle-stats

data/.gitignore

@@ -13,3 +13,5 @@
 stats.db
 .DS_Store
 spec/performance/.byebug_history
+stats_joined.db
+stats_separated.db

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    trifle-stats (1.6.0)
+    trifle-stats (2.3.0)
       tzinfo (~> 2.0)
 
 GEM

data/README.md

@@ -1,18 +1,13 @@
 # Trifle::Stats
 
-[![Gem Version](https://badge.fury.io/rb/trifle-stats.svg)](https://badge.fury.io/rb/trifle-stats)
-![Ruby](https://github.com/trifle-io/trifle-stats/workflows/Ruby/badge.svg?branch=main)
-[![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/trifle-io/trifle-stats)
+[![Gem Version](https://badge.fury.io/rb/trifle-stats.svg)](https://rubygems.org/gems/trifle-stats)
+[![Ruby](https://github.com/trifle-io/trifle-stats/workflows/Ruby/badge.svg?branch=main)](https://github.com/trifle-io/trifle-stats)
 
-Simple analytics backed by Redis, Postgres, MongoDB, Google Analytics, Segment, or whatever. [^1]
-
-`Trifle::Stats` is a _way too_ simple timeline analytics that helps you track custom metrics. Automatically increments counters for each enabled range. It supports timezones and different week beginning.
-
-[^1]: TBH only Redis, Postgres and MongoDB for now 💔.
+Simple analytics backed by Redis, Postgres, MongoDB, Google Analytics, Segment, or whatever. It gets you from having bunch of events occuring within few minutes to being able to say what happened on 25th January 2021.
 
 ## Documentation
 
-You can find guides and documentation at https://trifle.io/trifle-stats
+For comprehensive guides, API reference, and examples, visit [trifle.io/trifle-stats-rb](https://trifle.io/trifle-stats-rb)
 
 ## Installation
 
@@ -24,137 +19,119 @@ gem 'trifle-stats'
 
 And then execute:
 
-```sh
+```bash
 $ bundle install
 ```
 
 Or install it yourself as:
 
-```sh
+```bash
 $ gem install trifle-stats
 ```
 
-Depending on driver you would like to use, make sure you add required gems into your `Gemfile`.
-```ruby
-gem 'mongo', '>= 2.14.0'
-gem 'pg', '>= 1.2'
-gem 'redis', '>= 4.2'
-```
-
-## Usage
+## Quick Start
 
-You don't need to use it with Rails, but you still need to run `Trifle::Stats.configure`. If youre running it with Rails, create `config/initializers/trifle-stats.rb` and configure the gem.
+### 1. Configure
 
 ```ruby
+require 'trifle/stats'
+
 Trifle::Stats.configure do |config|
-  config.driver = Trifle::Stats::Driver::Redis.new
-  config.track_ranges = [:hour, :day]
-  config.time_zone = 'Europe/Bratislava'
-  config.beginning_of_week = :monday
+  config.driver = Trifle::Stats::Driver::Redis.new(Redis.new)
+  config.granularities = ['1m', '1h', '1d', '1w', '1mo', '1q', '1y']
 end
 ```
 
-### Track values
-
-Track your first metrics
+### 2. Track events
 
 ```ruby
-Trifle::Stats.track(key: 'event::logs', at: Time.now, values: {count: 1, duration: 2, lines: 241})
-=> [{2021-01-25 16:00:00 +0100=>{:count=>1, :duration=>2, :lines=>241}}, {2021-01-25 00:00:00 +0100=>{:count=>1, :duration=>2, :lines=>241}}]
+Trifle::Stats.track(key: 'event::logs', at: Time.now, values: { count: 1, duration: 2.11 })
 ```
 
-Then do it few more times
+### 3. Retrieve values
 
 ```ruby
-Trifle::Stats.track(key: 'event::logs', at: Time.now, values: {count: 1, duration: 1, lines: 56})
-=> [{2021-01-25 16:00:00 +0100=>{:count=>1, :duration=>1, :lines=>56}}, {2021-01-25 00:00:00 +0100=>{:count=>1, :duration=>1, :lines=>56}}]
-Trifle::Stats.track(key: 'event::logs', at: Time.now, values: {count: 1, duration: 5, lines: 361})
-=> [{2021-01-25 16:00:00 +0100=>{:count=>1, :duration=>5, :lines=>361}}, {2021-01-25 00:00:00 +0100=>{:count=>1, :duration=>5, :lines=>361}}]
+Trifle::Stats.values(key: 'event::logs', from: 1.month.ago, to: Time.now, granularity: :day)
+#=> {:at=>[Wed, 25 Jan 2023 00:00:00 +0000], :values=>[{"count"=>1, "duration"=>2.11}]}
 ```
 
-You can also store nested counters like
+## Drivers
 
-```ruby
-Trifle::Stats.track(key: 'event::logs', at: Time.now, values: {
-  count: 1,
-  duration: {
-    parsing: 21,
-    compression: 8,
-    upload: 1
-  },
-  lines: 25432754
-})
-```
+Trifle::Stats supports multiple backends:
 
-#### Get values
+- **Redis** - Fast, in-memory storage
+- **Postgres** - SQL database with JSONB support
+- **SQLite** - SQL database in a file
+- **MongoDB** - Document database
+- **Process** - Thread-safe in-memory storage (development/testing)
+- **Dummy** - No-op driver for disabled analytics
 
-Retrieve your values for specific `range`. Adding increments above will return sum of all the values you've tracked.
+## Features
 
-```ruby
-Trifle::Stats.values(key: 'event::logs', from: Time.now, to: Time.now, range: :day)
-=> {:at=>[2021-01-25 00:00:00 +0200], :values=>[{"count"=>3, "duration"=>8, "lines"=>658}]}
-```
+- **Multiple time granularities** - Track data across different time periods
+- **Custom aggregators** - Sum, average, min, max with custom logic
+- **Series operations** - Advanced data manipulation and calculations
+- **Performance optimized** - Efficient storage and retrieval patterns
+- **Buffered writes** - Queue metrics locally before flushing to the driver
+- **Driver flexibility** - Switch between storage backends easily
 
-### Assert values
+## Buffered Persistence
 
-Asserting values works same way like incrementing, but instead of increment, it sets the value. Duh.
+Every `track/assert/assort` call can be buffered before touching the driver. The buffer is enabled by
+default and flushes on an interval, when the queue reaches a configurable size, and again on shutdown
+(`SIGTERM`/`at_exit`).
 
-Set your first metrics
+Available configuration options:
 
-```ruby
-Trifle::Stats.assert(key: 'event::logs', at: Time.now, values: {count: 1, duration: 2, lines: 241})
-=> [{2021-01-25 16:00:00 +0100=>{:count=>1, :duration=>2, :lines=>241}}, {2021-01-25 00:00:00 +0100=>{:count=>1, :duration=>2, :lines=>241}}]
-```
+- `buffer_enabled` (default: `true`) – Disable to write-through synchronously
+- `buffer_duration` (default: `1` second) – Maximum time between automatic flushes
+- `buffer_size` (default: `256`) – Maximum queued actions before forcing a flush
+- `buffer_aggregate` (default: `true`) – Combine repeated operations on the same key set
 
-Then do it few more times
+Example:
 
 ```ruby
-Trifle::Stats.assert(key: 'event::logs', at: Time.now, values: {count: 1, duration: 1, lines: 56})
-=> [{2021-01-25 16:00:00 +0100=>{:count=>1, :duration=>1, :lines=>56}}, {2021-01-25 00:00:00 +0100=>{:count=>1, :duration=>1, :lines=>56}}]
-Trifle::Stats.assert(key: 'event::logs', at: Time.now, values: {count: 1, duration: 5, lines: 361})
-=> [{2021-01-25 16:00:00 +0100=>{:count=>1, :duration=>5, :lines=>361}}, {2021-01-25 00:00:00 +0100=>{:count=>1, :duration=>5, :lines=>361}}]
+Trifle::Stats.configure do |config|
+  config.driver = Trifle::Stats::Driver::Redis.new(Redis.new)
+  config.buffer_duration = 5   # flush every ~5 seconds
+  config.buffer_size = 100     # ...or sooner when 100 actions are enqueued
+  config.buffer_aggregate = true
+end
 ```
 
-#### Get values
-
-Retrieve your values for specific `range`. As you just used `assert` above, it will return latest value you've asserted.
-
-```ruby
-Trifle::Stats.values(key: 'event::logs', from: Time.now, to: Time.now, range: :day)
-=> {:at=>[2021-01-25 00:00:00 +0200], :values=>[{"count"=>1, "duration"=>5, "lines"=>361}]}
-```
+If your application manages database connections manually (e.g. ActiveRecord with a pool size of 1),
+increase the pool size or disable buffering to avoid starving other threads.
 
 ## Testing
 
-### Testing Principles
+Tests are run against all supported drivers. To run the test suite:
 
-Tests are structured to be simple, isolated, and mirror the class structure. Each test is independent and self-contained.
+```bash
+$ bundle exec rspec
+```
 
-#### Key Rules:
+Ensure Redis, Postgres, and MongoDB are running locally. The test suite will handle database setup automatically.
 
-1. **Keep tests simple and isolated** - Each test should focus on a single class/method
-2. **Independent tests** - Tests should not depend on each other and can be run in any order
-3. **Self-contained setup** - Every test configures its own variables and dependencies
-4. **Single layer testing** - Test only the specific class, not multiple layers of functionality
-5. **Use appropriate stubbing** - When testing operations, stub driver methods. Let driver tests verify driver behavior
-6. **Repeat yourself** - It's okay to repeat setup code for clarity and independence
+Tests are meant to be **simple and isolated**. Every test should be **independent** and able to run in any order. Tests should be **self-contained** and set up their own configuration. This makes it easier to debug and maintain the test suite.
 
-#### Driver Testing:
+Use **single layer testing** to focus on testing a specific class or module in isolation. Use **appropriate stubbing** for driver methods when testing higher-level operations.
 
-- Driver tests use **real database connections** (Redis, PostgreSQL, MongoDB, SQLite)
-- Clean data between tests to ensure isolation
-- Use appropriate test databases (e.g., Redis database 15, test-specific DB names)
-- The **Process driver** is ideal for testing environments as it uses in-memory storage
+Driver tests use real database connections for accurate behavior validation. The `Process` driver is preferred for in-memory testing environments.
 
-#### Test Structure:
+**Repeat yourself** in test setup for clarity rather than complex shared setups that can hide dependencies.
 
-Tests follow the same structure as the classes they test:
-- `spec/stats/driver/` - Driver class tests
-- `spec/stats/operations/` - Operation class tests  
-- `spec/stats/mixins/` - Mixin tests
+For performance testing:
 
-This approach makes it easier to see initial configuration and expected results for each test.
+```bash
+$ cd specs/performance
+$ bundle install
+$ ruby run.rb 100 '{"a":1}'
+```
 
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/trifle-io/trifle-stats.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/docker-compose.yml

@@ -0,0 +1 @@
+.devops/docker/local/docker-compose.yml

data/lib/trifle/stats/aggregator/{avg.rb → mean.rb}

@@ -3,8 +3,8 @@
 module Trifle
   module Stats
     class Aggregator
-      class Avg
-        Trifle::Stats::Series.register_aggregator(:avg, self)
+      class Mean
+        Trifle::Stats::Series.register_aggregator(:mean, self)
 
         def aggregate(series:, path:, slices: 1)
           return [] if series[:at].empty?

data/lib/trifle/stats/buffer.rb

@@ -0,0 +1,380 @@
+# frozen_string_literal: true
+
+module Trifle
+  module Stats
+    module BufferRegistry
+      class << self
+        def register(buffer)
+          registry_mutex.synchronize do
+            registry << buffer
+            install_shutdown_hooks
+          end
+        end
+
+        def unregister(buffer)
+          registry_mutex.synchronize do
+            registry.delete(buffer)
+          end
+          pending_mutex.synchronize do
+            pending.delete(buffer)
+          end
+        end
+
+        def enqueue_pending(buffer)
+          pending_mutex.synchronize do
+            pending << buffer unless pending.include?(buffer)
+          end
+        end
+
+        def cancel_pending(buffer)
+          pending_mutex.synchronize do
+            pending.delete(buffer)
+          end
+        end
+
+        def run_pending!
+          snapshot = pending_mutex.synchronize do
+            buffers = pending.dup
+            pending.clear
+            buffers
+          end
+          snapshot.each(&:flush_from_registry!)
+        end
+
+        def pending?
+          pending_mutex.synchronize { pending.any? }
+        end
+
+        def flush_all
+          run_pending!
+          snapshot = registry_mutex.synchronize { registry.dup }
+          snapshot.each(&:shutdown!)
+        end
+
+        private
+
+        def registry
+          @registry ||= []
+        end
+
+        def registry_mutex
+          @registry_mutex ||= Mutex.new
+        end
+
+        def pending
+          @pending ||= []
+        end
+
+        def pending_mutex
+          @pending_mutex ||= Mutex.new
+        end
+
+        def install_shutdown_hooks
+          return if @shutdown_hooks_installed
+
+          at_exit { flush_all }
+          install_sigterm_trap if Signal.list.key?('TERM')
+          @shutdown_hooks_installed = true
+        end
+
+        def install_sigterm_trap
+          previous = Signal.trap('TERM') do
+            flush_all
+            invoke_previous_handler(previous)
+          end
+          @previous_sigterm_handler = previous
+        end
+
+        def invoke_previous_handler(previous)
+          case previous
+          when Proc
+            previous.call
+          when Symbol, String
+            Signal.trap('TERM', previous)
+            Process.kill('TERM', Process.pid)
+          end
+        rescue StandardError
+          nil
+        end
+      end
+    end
+
+    class BufferQueue
+      def initialize(aggregate:)
+        @aggregate = aggregate
+        reset!
+      end
+
+      def store(operation, keys, values)
+        aggregate? ? store_aggregate(operation, keys, values) : store_linear(operation, keys, values)
+        @operation_count += 1
+      end
+
+      def size
+        @operation_count
+      end
+
+      def empty?
+        size.zero?
+      end
+
+      def drain
+        drained = aggregate? ? @actions.values : @actions.dup
+        reset!
+        drained
+      end
+
+      private
+
+      def aggregate?
+        @aggregate
+      end
+
+      def reset!
+        @actions = aggregate? ? {} : []
+        @operation_count = 0
+      end
+
+      def store_linear(operation, keys, values)
+        @actions << { operation: operation, keys: keys, values: duplicate(values), count: 1 }
+      end
+
+      def store_aggregate(operation, keys, values)
+        signature = signature_for(operation, keys)
+        if (entry = @actions[signature])
+          entry[:values] = merge_values(operation, entry[:values], values)
+          entry[:count] += 1
+        else
+          @actions[signature] = { operation: operation, keys: keys, values: duplicate(values), count: 1 }
+        end
+      end
+
+      def merge_values(operation, current, incoming)
+        case operation
+        when :inc
+          merge_increment(current, incoming)
+        when :set
+          duplicate(incoming)
+        else
+          duplicate(incoming)
+        end
+      end
+
+      def merge_increment(current, incoming)
+        incoming.each do |key, value|
+          current[key] =
+            if value.is_a?(Hash)
+              merge_increment(current.fetch(key, {}), value)
+            else
+              current.fetch(key, 0).to_i + value.to_i
+            end
+        end
+        current
+      end
+
+      def signature_for(operation, keys)
+        identifiers = keys.map do |key|
+          [key.prefix, key.key, key.granularity, key.at&.to_i].join(':')
+        end
+        "#{operation}-#{identifiers.join('|')}"
+      end
+
+      def duplicate(value)
+        case value
+        when Hash
+          value.transform_values { |entry| duplicate(entry) }
+        when Array
+          value.map { |entry| duplicate(entry) }
+        else
+          value
+        end
+      end
+    end
+
+    class Buffer # rubocop:disable Metrics/ClassLength
+      DEFAULT_DURATION = 1
+      DEFAULT_SIZE = 256
+
+      class << self
+        def register(buffer)
+          BufferRegistry.register(buffer)
+        end
+
+        def unregister(buffer)
+          BufferRegistry.unregister(buffer)
+        end
+
+        def flush_all
+          BufferRegistry.flush_all
+        end
+
+        def run_pending!
+          BufferRegistry.run_pending!
+        end
+
+        def pending_flushes?
+          BufferRegistry.pending?
+        end
+      end
+
+      def initialize(driver:, duration: DEFAULT_DURATION, size: DEFAULT_SIZE, aggregate: true, async: true) # rubocop:disable Metrics/MethodLength
+        @driver = driver
+        @duration = duration.to_f
+        @size = size.to_i.positive? ? size.to_i : 1
+        @async = async
+        @queue = BufferQueue.new(aggregate: aggregate)
+        @mutex = Mutex.new
+        @stopped = false
+        @flush_pending = false
+        @pending_condition = ConditionVariable.new
+        @worker = start_worker if async && @duration.positive?
+        self.class.register(self)
+      end
+
+      def inc(keys:, values:)
+        enqueue(:inc, keys: keys, values: values)
+      end
+
+      def set(keys:, values:)
+        enqueue(:set, keys: keys, values: values)
+      end
+
+      def flush!
+        actions = drain_actions(reset_pending: true)
+        return if actions.nil?
+
+        process(actions)
+      end
+
+      def shutdown!
+        return if @shutdown
+
+        @shutdown = true
+        stop_worker
+        BufferRegistry.cancel_pending(self)
+        flush!
+        self.class.unregister(self)
+      end
+
+      def flush_from_registry!
+        actions = drain_pending_actions
+        process(actions) if actions
+      end
+
+      private
+
+      def enqueue(operation, keys:, values:)
+        should_flush = false
+        @mutex.synchronize do
+          @queue.store(operation, keys, values)
+          should_flush = @queue.size >= @size
+        end
+
+        flush! if should_flush
+      end
+
+      def request_async_flush
+        return unless mark_flush_pending
+
+        BufferRegistry.enqueue_pending(self)
+        wait_for_pending_flush
+      end
+
+      def mark_flush_pending
+        @mutex.synchronize do
+          return false if @queue.empty? || @flush_pending
+
+          @flush_pending = true
+          true
+        end
+      end
+
+      def drain_actions(reset_pending: false)
+        @mutex.synchronize do
+          return if @queue.empty?
+
+          mark_flush_serviced if reset_pending
+          @queue.drain
+        end
+      end
+
+      def drain_pending_actions
+        @mutex.synchronize do
+          return unless @flush_pending
+          return if @queue.empty?
+
+          mark_flush_serviced
+          @queue.drain
+        end
+      end
+
+      def wait_for_pending_flush # rubocop:disable Metrics/MethodLength
+        should_force = false
+        timeout = @duration.positive? ? @duration : DEFAULT_DURATION
+        @mutex.synchronize do
+          while @flush_pending && timeout.positive?
+            @pending_condition.wait(@mutex, timeout)
+            break unless @flush_pending
+
+            timeout = 0
+          end
+          should_force = @flush_pending
+        end
+
+        return unless should_force
+
+        BufferRegistry.cancel_pending(self)
+        flush!
+      end
+
+      def mark_flush_serviced
+        return unless @flush_pending
+
+        @flush_pending = false
+        BufferRegistry.cancel_pending(self)
+        @pending_condition.broadcast
+      end
+
+      def process(actions)
+        actions.each do |action|
+          @driver.public_send(
+            action[:operation], keys: action[:keys], values: action[:values], count: action[:count] || 1
+          )
+        end
+      ensure
+        release_active_record_connection
+      end
+
+      def start_worker
+        Thread.new do
+          loop do
+            break if @stopped
+
+            sleep(@duration)
+            request_async_flush
+          end
+        end
+      end
+
+      def stop_worker
+        return if @worker.nil?
+
+        @stopped = true
+        begin
+          @worker.wakeup
+        rescue ThreadError
+          nil
+        end
+        @worker.join
+      end
+
+      def release_active_record_connection
+        # Workers run on dedicated threads, so make sure ActiveRecord connections
+        # are released back to the shared pool once a flush finishes.
+        return unless defined?(::ActiveRecord::Base)
+
+        ::ActiveRecord::Base.clear_active_connections!
+      end
+    end
+  end
+end