recapi 0.1.4 → 0.1.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dfb3a3778d3f1733138f9f09c388edfd2e7cb946f6a2f38647b1070aa885d865
-  data.tar.gz: cc7d5eed12f41976d3e898a43bfee08364eaf2224b23f36b062b101577f7cb06
+  metadata.gz: f8edb463693be71a5cac3a101441d5fe6c045e5e58f4003025291995ec5c9e44
+  data.tar.gz: 818ea20b257819a76f15c832af44b83694681fc36995f08b91987ac2b9e8afdf
 SHA512:
-  metadata.gz: b48b41679b86910bb139237e630e5495bebfc4d1aa4fd9f924c2202344e5fbb903fb4c3966e5c608ffa406a90a2cb97c6a3fb75a989091509a41709f257e614f
-  data.tar.gz: b711bee7d27e579947e075e02a3e4487a97a2b7fe433e4d067d035d5b747157cd2cec7efa1adf34f6bd62bc1dd4d0e63ceb8f5612eb90bb929b1282a981f4051
+  metadata.gz: cb54199fc5b2ff480e94be20d594598a2cbda5fb2b1b8ffb854b3dd4ac0dd3d9d2858b05e67cc79b264d93f41f4e5132b57c50812aa5c626582583d369763df7
+  data.tar.gz: 33fbcbfc1cca21caf75849d106e9500236d6adf2ac897c74b49e9731eb79b6680cb1054e637677babbd0ebc52fc853aa7a999ff10a1e11911da5ff01bb538c41

data/README.md

@@ -30,7 +30,7 @@ line with your address): save it as `lib/recapi.rb` and
 `require Rails.root.join("lib/recapi").to_s` instead.
 
 ```sh
-curl -fLo lib/recapi.rb https://api.<your-domain>/sdk/ruby/recapi-0.1.4.rb
+curl -fLo lib/recapi.rb https://api.<your-domain>/sdk/ruby/recapi-0.1.7.rb
 ```
 
 ## Quick start
@@ -41,7 +41,7 @@ require "recapi"
 
 # A secret key (rk_sec_…): it stays on your server.
 RECAPI = Recapi::Client.new(ENV.fetch("RECAPI_KEY"), "https://api.example.com",
-                            store: Rails.cache, logger: Rails.logger)
+                            logger: Rails.logger)
 ```
 
 ```ruby
@@ -79,10 +79,10 @@ API failure `Recapi::Error` (`code`, `status`, `retry_after`, `retryable?`,
 
 `Recapi::Client.new(api_key, base_url, store:, timeout:, flush_interval:,
 max_batch:, logger:)` sets the options. The default `Recapi::MemoryStore`
-keeps the 10,000 most recent users in the process; pass `Rails.cache` (it
-has `read`/`write`/`delete`) or a Redis client (`get`/`set`/`del`) when
-several processes serve the same users. It stores the same JSON as the
-Python library, so both can share one store. Written to `Rails.cache` or a
+keeps the 10,000 most recent users in the process; pass a Redis client
+(`get`/`set`/`del`) or a `Rails.cache` on Redis, Memcached or Solid Cache (it
+has `read`/`write`/`delete`) when several processes serve the same users. It
+stores the same JSON as the Python library, so both can share one store. Written to `Rails.cache` or a
 redis-rb client, a user's state lasts 30 days from its last write, with no
 setting to choose: well past the 24 hours a click earns credit, and recapi gives
 a known `user_id` its history back afterwards. Any other store gets a plain
@@ -93,7 +93,29 @@ client; a forked worker (Puma cluster, Unicorn, Passenger) starts its own
 queue and sender at its first call. Proxies come from `HTTPS_PROXY`,
 `HTTP_PROXY` and `NO_PROXY`, as `Net::HTTP` reads them. Failures are logged
 to `logger:`, anything with `warn` and `error` (`Rails.logger`, a
-`Logger`); without one they go to `$stderr` as `recapi: …` lines.
+`Logger`); without one they go to `$stderr` as `recapi: …` lines, with lost
+events (a refused batch, rejected events) at most one line a minute for each
+error code and for rejected events, so a revoked key on a busy server doesn't
+flood the log, and a full queue one line until a send works. A failed write to
+`$stderr` (a full disk, a closed stream) is ignored. A logger you give gets
+every line, and if it raises, that reaches your code.
+
+### Which Rails caches work
+
+Not every `Rails.cache` can hold this, and Rails' defaults often can't. The
+library recognises ActiveSupport's own classes (a subclass of one is left alone)
+and logs one line at start:
+
+| `Rails.cache` | Where Rails sets it | What happens | Shared between |
+|---|---|---|---|
+| `RedisCacheStore`, `MemCacheStore`, Solid Cache | your choice; Solid Cache in new 8.x apps' production | used, no line | servers |
+| `FileStore` | production on 7.2 and older | used, with a warning | one server's processes |
+| `MemoryStore` | development on 8.x | the library's own store instead, with a warning | nothing |
+| `NullStore` | development on 7.2 and older, without `rails dev:cache` | the library's own store instead, with a warning | nothing |
+
+`FileStore` frees an expired entry only when it is read again or `cleanup` runs,
+so a visitor who never returns stays on disk (about 8 KB each): run
+`Rails.cache.cleanup` on a schedule, or leave `store:` out.
 
 ## Development
 

data/lib/recapi.rb

@@ -25,7 +25,7 @@ rescue LoadError
 end
 
 module Recapi
-  VERSION = "0.1.4"
+  VERSION = "0.1.7"
 
   # Decayed user history, exactly as docs/history-spec.md (times in unix ms).
   # A port of sdk/python's history.py, snapshot merging included.
@@ -319,7 +319,8 @@ module Recapi
   end
 
   # In-process store keeping the most recently used max_users users. Pass a
-  # shared one (Redis, Rails.cache) when several processes serve the same users.
+  # shared one (Redis, a Rails.cache on Redis, Memcached or Solid Cache) when
+  # several processes serve the same users.
   class MemoryStore
     def initialize(max_users: 10_000)
       @max = [1, max_users].max
@@ -353,11 +354,21 @@ module Recapi
   # Bundler app without it in its Gemfile can't require.
   class StderrLogger
     def warn(message)
-      $stderr.write("#{message}\n")
+      write_line(message)
     end
 
     def error(message)
+      write_line(message)
+    end
+
+    private
+
+    # A full disk or a closed stderr must not reach the app's track or kill the
+    # sender thread: a line nobody can read is no reason to stop sending.
+    def write_line(message)
       $stderr.write("#{message}\n")
+    rescue StandardError
+      nil
     end
   end
 
@@ -377,6 +388,7 @@ module Recapi
     LEVEL_NAMES = { "view" => 3, "cart" => 5, "purchase" => 10, "reject" => -10 }.freeze
     # Events waiting for a send: a minute-long outage at 150 events/s fits.
     MAX_QUEUE = 10_000
+    LOST_LINE_MS = 60_000
     MAX_FLUSH_BACKOFF = 60.0
     RETRIES = 3
     RETRY_BASE = 0.5
@@ -396,6 +408,16 @@ module Recapi
     # starts afresh. It bounds a Redis at about one month's visitors instead of
     # growing for ever.
     STATE_LIFETIME_SECONDS = 30 * 24 * 60 * 60
+    # ActiveSupport's caches that lose visitors' state, by exact class: a subclass
+    # may have changed what makes them so. They keep nothing or only in this
+    # process, where our MemoryStore is as local, holds more and competes with
+    # nothing else in the app's cache.
+    USE_OURS = {
+      "ActiveSupport::Cache::NullStore" => "keeps nothing",
+      "ActiveSupport::Cache::MemoryStore" => "keeps its entries in this process only"
+    }.freeze
+    FILE_STORE = "ActiveSupport::Cache::FileStore"
+    private_constant :USE_OURS, :FILE_STORE
     # Where redis-rb defines set: 4.x and 5 on, older versions, Redis::Distributed.
     REDIS_RB_SET = %w[Redis::Commands::Strings Redis Redis::Distributed].freeze
     # Open connections kept between calls: one per call in flight at the busiest moment.
@@ -426,7 +448,8 @@ module Recapi
     # api_key: secret key (rk_sec_...): a server can then restore a user's
     # history from recapi when the store lacks it. base_url: your recapi API
     # origin. store: per-user state, anything with get/set/delete (a Redis
-    # client) or read/write/delete (Rails.cache); defaults to a MemoryStore.
+    # client) or read/write/delete (Rails.cache); defaults to a MemoryStore,
+    # which also replaces ActiveSupport's NullStore and MemoryStore.
     # timeout: per-request seconds (recommend() uses at most 1.5).
     # flush_interval: seconds between background sends. max_batch: events per
     # request, at most the API's 1,000. logger: where failures are logged,
@@ -443,12 +466,14 @@ module Recapi
         raise ArgumentError, "base_url must be an http(s) URL"
       end
 
-      @store = store || MemoryStore.new
+      @logger = logger || StderrLogger.new
+      @limit_lost_lines = logger.nil?
+      @lost_printed = {}
+      @store = store ? store_to_use(store) : MemoryStore.new
       @store_lifetime = store_lifetime
       @timeout = timeout.to_f
       @interval = flush_interval.to_f
       @max_batch = max_batch.to_i.clamp(1, MAX_EVENTS_PER_REQUEST)
-      @logger = logger || StderrLogger.new
       # Two environments' keys may share one Redis: their users must not mix.
       @prefix = "recapi:#{Digest::SHA256.hexdigest(api_key)[0, 12]}:"
       # Indirections the tests replace (clock, backoff sleep, jitter).
@@ -718,6 +743,7 @@ module Recapi
       @send_lock = Mutex.new
       @thread = nil
       @failures = 0
+      @overflow_printed = false
       @closed = false
       @idle = []
       @idle_lock = Mutex.new
@@ -777,6 +803,13 @@ module Recapi
     def drop_overflow
       dropped = @queue.size - MAX_QUEUE
       @queue.shift(dropped)
+      # A full queue drops on every track through an outage: on $stderr, which
+      # the operator can't filter, one line until a send works is enough, and
+      # the "sending failed" lines keep showing the outage. A logger given gets
+      # them all. Runs under @mutex.
+      return if @limit_lost_lines && @overflow_printed
+
+      @overflow_printed = true
       @logger.warn("recapi: event queue full, dropped the #{dropped} oldest events")
     end
 
@@ -809,7 +842,7 @@ module Recapi
             send_events(batch, retry_errors)
           rescue StandardError => e
             if e.is_a?(Error) && !e.retryable?
-              @logger.error("recapi: dropped #{batch.size} events: #{e.message}")
+              log_lost(e.code, "recapi: dropped #{batch.size} events: #{e.message}")
               next
             end
             # Anything else keeps the batch, a bug of ours included: event ids
@@ -823,7 +856,10 @@ module Recapi
             @logger.warn("recapi: sending #{batch.size} events failed, kept: #{e.message}")
             return
           end
-          @mutex.synchronize { @failures = 0 }
+          @mutex.synchronize do
+            @failures = 0
+            @overflow_printed = false
+          end
         end
       end
     end
@@ -850,7 +886,23 @@ module Recapi
       detail = rejected.first(5).map do |r|
         r.is_a?(Hash) && r["index"].is_a?(Integer) ? r.merge("event" => events[r["index"]]) : r
       end
-      @logger.error("recapi: #{rejected.size} event(s) rejected: #{JSON.generate(detail)}")
+      log_lost(:rejected, "recapi: #{rejected.size} event(s) rejected: #{JSON.generate(detail)}")
+    end
+
+    # A revoked key refuses every batch and the sender runs on with no backoff,
+    # so $stderr gets one line a minute per kind (error code, or rejected
+    # events): one line per outage would have aged out of the log by the time
+    # someone looks. A logger given gets them all. Runs under @send_lock.
+    def log_lost(kind, message)
+      if @limit_lost_lines
+        now = @clock.call
+        last = @lost_printed[kind]
+        # abs: a clock set back must not mute the kind until it catches up.
+        return if last && (now - last).abs < LOST_LINE_MS
+
+        @lost_printed[kind] = now
+      end
+      @logger.error(message)
     end
 
     # Calls the block, retrying retryable errors with jittered exponential
@@ -1020,10 +1072,30 @@ module Recapi
       @logger.error("recapi: store write failed: #{e.class}: #{e.message}")
     end
 
+    # The store, or our own in its place if it is an ActiveSupport cache that
+    # loses state. Says so in one line, at start only. Any doubt, or a store
+    # that fails here, means the store as given.
+    def store_to_use(store)
+      name = store.class.name
+      if USE_OURS.key?(name)
+        @logger.warn("recapi: #{name} #{USE_OURS[name]}, so recapi uses its own in-process store (10,000 users). " \
+                     "For several processes or servers, use Redis, Memcached or Solid Cache.")
+        return MemoryStore.new
+      end
+      if name == FILE_STORE
+        @logger.warn("recapi: #{name} isn't shared between servers, and frees an expired entry only when it is " \
+                     "read again or cleanup runs. Run Rails.cache.cleanup on a schedule; for several servers, " \
+                     "use Redis, Memcached or Solid Cache.")
+      end
+      store
+    rescue StandardError
+      store
+    end
+
     # What makes this store's write expire the entry. By class name, since the
     # library requires neither gem. Any doubt (a subclass's own set, a wrapper
-    # such as ConnectionPool::Wrapper) means no option, as for a store of the
-    # customer's.
+    # such as ConnectionPool::Wrapper, a store that fails here) means no option,
+    # as for a store of the customer's.
     def store_lifetime
       klass = @store.class
       if klass.public_method_defined?(:set)
@@ -1033,6 +1105,8 @@ module Recapi
       else
         {}
       end
+    rescue StandardError
+      {}
     end
 
     # Redis clients have get/set/del, Rails.cache read/write/delete.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: recapi
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.7
 platform: ruby
 authors:
 - recapi