recapi 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE +21 -0
- data/README.md +104 -0
- data/lib/recapi.rb +1192 -0
- metadata +46 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: d12730fb7a1d92afff5bc3b2b2383830df795b24fada7104fd8e66c3ac84dc9a
|
|
4
|
+
data.tar.gz: 5accff282fae359cb6fe664dbebf5957b351df0d155199ce04fa145049ffe3d4
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: eb5e428dd2fe97d7bdc35da5838b84bbcf68b1ece0ee137b8bbdd0dd4b75f24d0b3f30f01d392416a2e4034efa94ac8adc1aaceb570366b46fbf8b9012ea5646
|
|
7
|
+
data.tar.gz: 71cc4af3ea8dd8b424c54e4cf69b94ec50c288fc0dbb7370797064d67cd9d40d06b56c46f94c8d92dd87a237653ef8a808ece2a0c523950ee994d14af2f2ba8b
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 recapi
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# recapi for Ruby
|
|
2
|
+
|
|
3
|
+
Ruby client for the recapi recommendation API, for Rails, Solidus, Sinatra
|
|
4
|
+
and background jobs. Ruby 3.1 or newer (tested on 3.1 and 4.0), standard
|
|
5
|
+
library only (`net/http`, `json`), one file, no dependency: it requires only
|
|
6
|
+
default gems, so it loads in a Bundler app whose Gemfile lists nothing else.
|
|
7
|
+
|
|
8
|
+
Like the [Python](../python/README.md) and [Go](../go/README.md) libraries,
|
|
9
|
+
it keeps each user's decayed **history**
|
|
10
|
+
([docs/history-spec.md](../../docs/history-spec.md)) and sends it with every
|
|
11
|
+
recommendation request, **sends events in the background** in batches of up
|
|
12
|
+
to 1,000 with **retries**, **falls back to popular items** when recapi is
|
|
13
|
+
slow or failing, and **attributes** events on recommended items to the
|
|
14
|
+
recommendation that showed them (the dashboard's clicks and conversions). It
|
|
15
|
+
passes the shared vectors in [`spec/`](../../spec).
|
|
16
|
+
|
|
17
|
+
## Install
|
|
18
|
+
|
|
19
|
+
Until the library is published as a gem, recapi serves it as one file (the
|
|
20
|
+
dashboard's quick start shows the line with your address). In a Rails app,
|
|
21
|
+
save it as `lib/recapi.rb`:
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
curl -fLo lib/recapi.rb https://api.<your-domain>/sdk/ruby/recapi-0.1.3.rb
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Once it is published on RubyGems (production releases publish it:
|
|
28
|
+
[docs/deployment.md](../../docs/deployment.md#publishing-the-ruby-library)),
|
|
29
|
+
`gem "recapi"` in your Gemfile, or `gem install recapi`, installs the same
|
|
30
|
+
file as a gem, and `require "recapi"` loads it.
|
|
31
|
+
|
|
32
|
+
## Quick start
|
|
33
|
+
|
|
34
|
+
```ruby
|
|
35
|
+
# config/initializers/recapi.rb: one client per process, shared by its threads.
|
|
36
|
+
require Rails.root.join("lib/recapi").to_s
|
|
37
|
+
|
|
38
|
+
# A secret key (rk_sec_…): it stays on your server.
|
|
39
|
+
RECAPI = Recapi::Client.new(ENV.fetch("RECAPI_KEY"), "https://api.example.com",
|
|
40
|
+
store: Rails.cache, logger: Rails.logger)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
```ruby
|
|
44
|
+
# Queued, sent in the background. Levels: :view, :cart, :purchase, :reject or -10..10.
|
|
45
|
+
RECAPI.track(product.id, :view, user_id: current_user&.id, anon_id: cookies[:visitor])
|
|
46
|
+
|
|
47
|
+
recs = RECAPI.recommend(user_id: current_user.id, count: 4)
|
|
48
|
+
recs.items.each do |item|
|
|
49
|
+
item.item_id # a String, as every id recapi returns
|
|
50
|
+
item.metadata # the item's stored object (a Hash), or nil
|
|
51
|
+
# why it is listed ("for_you", "similar", "trending"...), safe to show anyone: map the codes you
|
|
52
|
+
# want to your own label (docs/api.md, "Reason codes"), and show none for a code you don't map.
|
|
53
|
+
label = LABELS[item.why]
|
|
54
|
+
end
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Ids may be strings or integers (an ActiveRecord id): they travel as strings.
|
|
58
|
+
|
|
59
|
+
## Calls
|
|
60
|
+
|
|
61
|
+
A refused argument (an invalid id or level) raises `ArgumentError`, and an
|
|
62
|
+
API failure `Recapi::Error` (`code`, `status`, `retry_after`, `retryable?`,
|
|
63
|
+
`outage?`).
|
|
64
|
+
|
|
65
|
+
| Call | What it does |
|
|
66
|
+
|---|---|
|
|
67
|
+
| `track(item_id, level, user_id:, anon_id:, recommendation_id:)` | Records an interaction: `level` -10 to 10 or `:view` (3), `:cart` (5), `:purchase` (10), `:reject` (-10: never recommend the item to this user again until a positive event on it). The history changes at once; a background thread sends the event within 2 s (at once when a batch of 1,000 is full). |
|
|
68
|
+
| `flush` | Sends the queued events now, with retries (end of a job). A network error, 429 or 5xx is retried 3 times (0.5 s, 1 s, 2 s with jitter, or the `Retry-After`, at most 8 s), then the events stay queued and the sender backs off up to a minute; any other error drops them (logged). |
|
|
69
|
+
| `close` | Stops the sender and sends what is queued, one try per batch. It runs by itself at exit. |
|
|
70
|
+
| `recommend(user_id:, anon_id:, count:, placement:, tags_any:, tags_all:, exclude:, context_item_id:, discovery:)` | A `Recapi::Recommendations` (`recommendation_id`, `fallback?`, `items`). At most 1.5 s; on a network error, timeout or 5xx it returns popular items (`fallback?`, no id), or no item when recapi is down: hide the slot. A 4xx or 429 raises. |
|
|
71
|
+
| `identify(anon_id, user_id)` | At login: the visitor's history and shown items become the user's. The user's first restore from recapi then leaves that visitor's browsing out, so it doesn't count twice. |
|
|
72
|
+
| `popular(count:, tags_any:, tags_all:)` | Popular items, for pages without a user. |
|
|
73
|
+
| `upsert_items(items)` | Creates or replaces items (hashes of `item_id`, `title`, `tags`, `metadata`…), 1,000 per request, retried like events. Returns how many were stored. |
|
|
74
|
+
| `delete_user(user_id)` | Erases a user's events in recapi and their state in the store, retried like events. |
|
|
75
|
+
|
|
76
|
+
`Recapi::Client.new(api_key, base_url, store:, timeout:, flush_interval:,
|
|
77
|
+
max_batch:, logger:)` sets the options. The default `Recapi::MemoryStore`
|
|
78
|
+
keeps the 10,000 most recent users in the process; pass `Rails.cache` (it
|
|
79
|
+
has `read`/`write`/`delete`) or a Redis client (`get`/`set`/`del`) when
|
|
80
|
+
several processes serve the same users. It stores the same JSON as the
|
|
81
|
+
Python library, so both can share one store. Puma's threads share one
|
|
82
|
+
client; a forked worker (Puma cluster, Unicorn, Passenger) starts its own
|
|
83
|
+
queue and sender at its first call. Proxies come from `HTTPS_PROXY`,
|
|
84
|
+
`HTTP_PROXY` and `NO_PROXY`, as `Net::HTTP` reads them. Failures are logged
|
|
85
|
+
to `logger:`, anything with `warn` and `error` (`Rails.logger`, a
|
|
86
|
+
`Logger`); without one they go to `$stderr` as `recapi: …` lines.
|
|
87
|
+
|
|
88
|
+
## Development
|
|
89
|
+
|
|
90
|
+
`../../scripts/ruby.sh -w test/run.rb [filter]` runs the tests (`make
|
|
91
|
+
test-sdk` does) against a fake API on a local socket. The autopilot
|
|
92
|
+
containers have no Ruby: `scripts/ruby.sh` then fetches Debian's ruby3.1
|
|
93
|
+
once and runs it without RubyGems, so the tests have their own small runner
|
|
94
|
+
(`test/test_case.rb`), no minitest; `test/gem_test.rb` loads RubyGems itself
|
|
95
|
+
to build the gem from `recapi.gemspec` (`lib/recapi.rb`, this README and the
|
|
96
|
+
licence). The API embeds `lib/recapi.rb` and
|
|
97
|
+
serves it (`server/internal/rubysdk`): any change to it needs a new
|
|
98
|
+
`Recapi::VERSION` and the line `go test ./internal/rubysdk` (in `server/`)
|
|
99
|
+
prints for `published.json`.
|
|
100
|
+
|
|
101
|
+
`RECAPI_KEY=rk_sec_… ../../scripts/ruby.sh sample.rb [item ids]` runs
|
|
102
|
+
[`sample.rb`](sample.rb) against staging (`RECAPI_URL` picks another API):
|
|
103
|
+
it tracks views for a new visitor, then prints their recommendations and the
|
|
104
|
+
popular items.
|
data/lib/recapi.rb
ADDED
|
@@ -0,0 +1,1192 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The recapi client for Ruby servers (Rails, Solidus, Sinatra, jobs): history,
|
|
4
|
+
# batching, retries and fallback. A port of sdk/python, which behaves like the
|
|
5
|
+
# JS SDK; spec/client_vectors.json and spec/history_vectors.json pin what they
|
|
6
|
+
# must agree on. Standard library only: it adds no gem to an app.
|
|
7
|
+
#
|
|
8
|
+
# require_relative "recapi" # or require "recapi"
|
|
9
|
+
# RECAPI = Recapi::Client.new(ENV.fetch("RECAPI_KEY"), "https://api.example.com")
|
|
10
|
+
# RECAPI.track("sku-1", :view, user_id: current_user.id)
|
|
11
|
+
# RECAPI.recommend(user_id: current_user.id, count: 4).items
|
|
12
|
+
|
|
13
|
+
require "json"
|
|
14
|
+
require "net/http"
|
|
15
|
+
require "openssl"
|
|
16
|
+
require "securerandom"
|
|
17
|
+
require "uri"
|
|
18
|
+
require "digest"
|
|
19
|
+
require "weakref"
|
|
20
|
+
begin
|
|
21
|
+
# Net::HTTP inflates gzip replies with it, when the Ruby has it.
|
|
22
|
+
require "zlib"
|
|
23
|
+
rescue LoadError
|
|
24
|
+
nil
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
module Recapi
|
|
28
|
+
VERSION = "0.1.3"
|
|
29
|
+
|
|
30
|
+
# Decayed user history, exactly as docs/history-spec.md (times in unix ms).
|
|
31
|
+
# A port of sdk/python's history.py, snapshot merging included.
|
|
32
|
+
class History
|
|
33
|
+
HALF_LIFE_SECONDS = 604_800
|
|
34
|
+
TOP_K = 50
|
|
35
|
+
RECENT_SIZE = 20
|
|
36
|
+
REJECTED_SIZE = 200
|
|
37
|
+
MAX_ID_BYTES = 256
|
|
38
|
+
RATE = Math.log(2) / HALF_LIFE_SECONDS
|
|
39
|
+
# Go's unicode.IsControl, which the API applies to every id: C0, DEL and C1.
|
|
40
|
+
CONTROL = /[\u0000-\u001f\u007f-\u009f]/.freeze
|
|
41
|
+
|
|
42
|
+
# Whether the API accepts value as an id: 1-256 bytes of UTF-8, no controls.
|
|
43
|
+
def self.id?(value)
|
|
44
|
+
return false unless value.is_a?(String) && !value.empty? && value.bytesize <= MAX_ID_BYTES
|
|
45
|
+
|
|
46
|
+
utf8 = value.encoding == Encoding::UTF_8 ? value : value.dup.force_encoding(Encoding::UTF_8)
|
|
47
|
+
utf8.valid_encoding? && !CONTROL.match?(utf8)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def self.decay(dt_ms)
|
|
51
|
+
Math.exp(-RATE * (dt_ms / 1000.0))
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def self.time?(value)
|
|
55
|
+
(value.is_a?(Integer) || value.is_a?(Float)) && value.finite?
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def self.weight?(value)
|
|
59
|
+
time?(value) && value >= 0
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# top: item => [score, t]; recent: [[item, t]] oldest first; rejected:
|
|
63
|
+
# item => time of the newest rejection; last_t: the newest event applied
|
|
64
|
+
# to the recent list (nil when unknown).
|
|
65
|
+
attr_accessor :top, :recent, :rejected, :last_t
|
|
66
|
+
|
|
67
|
+
def initialize
|
|
68
|
+
@top = {}
|
|
69
|
+
@recent = []
|
|
70
|
+
@rejected = {}
|
|
71
|
+
@last_t = nil
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Nothing to send, so the server uses its stored history. Rejections alone
|
|
75
|
+
# don't count: the server still has the user's positive events.
|
|
76
|
+
def empty?
|
|
77
|
+
@top.empty? && @recent.empty?
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
# Applies one event: level 0 changes nothing, a negative level is a rejection.
|
|
81
|
+
def apply(item, level, t)
|
|
82
|
+
return if level.zero?
|
|
83
|
+
return reject(item, t) if level.negative?
|
|
84
|
+
|
|
85
|
+
mark = @rejected[item]
|
|
86
|
+
unless mark.nil?
|
|
87
|
+
return if t < mark
|
|
88
|
+
|
|
89
|
+
@rejected.delete(item)
|
|
90
|
+
end
|
|
91
|
+
current = @top[item]
|
|
92
|
+
@top[item] =
|
|
93
|
+
if current.nil?
|
|
94
|
+
[level.to_f, t]
|
|
95
|
+
elsif t >= current[1]
|
|
96
|
+
[current[0] * History.decay(t - current[1]) + level, t]
|
|
97
|
+
else
|
|
98
|
+
[current[0] + level * History.decay(current[1] - t), current[1]]
|
|
99
|
+
end
|
|
100
|
+
if @last_t.nil? || t >= @last_t
|
|
101
|
+
@recent = @recent.reject { |entry| entry[0] == item }
|
|
102
|
+
@recent << [item, t]
|
|
103
|
+
@recent = @recent.last(RECENT_SIZE)
|
|
104
|
+
@last_t = t
|
|
105
|
+
end
|
|
106
|
+
prune(t)
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
# [item, weight at now, t] by weight desc, ties by item id asc.
|
|
110
|
+
def ranked(now)
|
|
111
|
+
entries = @top.map { |item, (s, t)| [item, s * History.decay(now - t), t] }
|
|
112
|
+
entries.sort_by { |item, w, _| [-w, item] }
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# The rejected items, oldest mark first, ties by item id.
|
|
116
|
+
def rejected_order
|
|
117
|
+
@rejected.keys.sort_by { |item| [@rejected[item], item] }
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# The request's history and recent, and the rejected items. Weights are
|
|
121
|
+
# decayed to now and rounded half up to 4 decimals like JS Math.round
|
|
122
|
+
# (Ruby's round rounds half away from zero); ages are whole seconds, never negative.
|
|
123
|
+
def wire(now)
|
|
124
|
+
history = ranked(now).first(TOP_K).map do |item, weight, t|
|
|
125
|
+
[item, (weight * 1e4 + 0.5).floor / 1e4, [0, ((now - t) / 1000.0).floor].max]
|
|
126
|
+
end
|
|
127
|
+
[history, @recent.last(RECENT_SIZE).map(&:first), rejected_order]
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
# A history from the server's snapshot, skipping invalid entries. Each top
|
|
131
|
+
# entry is a score at as_of, each rejected item a mark at as_of. last_t
|
|
132
|
+
# stays unknown: as_of is the server clock, and a local clock behind it
|
|
133
|
+
# would see every new event as out of order.
|
|
134
|
+
def self.from_snapshot(snapshot)
|
|
135
|
+
state = new
|
|
136
|
+
return state unless snapshot.is_a?(Hash) && time?(snapshot["as_of"])
|
|
137
|
+
|
|
138
|
+
as_of = snapshot["as_of"]
|
|
139
|
+
list(snapshot["top"]).each do |e|
|
|
140
|
+
state.top[e[0]] = [e[1].to_f, as_of] if e.is_a?(Array) && e.size >= 2 && id?(e[0]) && weight?(e[1])
|
|
141
|
+
end
|
|
142
|
+
list(snapshot["recent"]).each do |item|
|
|
143
|
+
state.recent << [item, as_of] if id?(item) && state.recent.none? { |entry| entry[0] == item }
|
|
144
|
+
end
|
|
145
|
+
state.recent = state.recent.last(RECENT_SIZE)
|
|
146
|
+
list(snapshot["rejected"]).each { |item| state.rejected[item] = as_of if id?(item) }
|
|
147
|
+
state.send(:prune, as_of)
|
|
148
|
+
state.send(:prune_rejected)
|
|
149
|
+
state
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# The union of two histories built from disjoint events. Top scores add
|
|
153
|
+
# up, expressed at the later of the two times; the recent lists are
|
|
154
|
+
# interleaved by time (self first on ties); rejection marks are united, and
|
|
155
|
+
# a mark older than the item's merged entry is dropped, otherwise the item
|
|
156
|
+
# leaves the top and the recent list.
|
|
157
|
+
def merge(other, now)
|
|
158
|
+
out = History.new
|
|
159
|
+
out.top = @top.dup
|
|
160
|
+
other.top.each do |item, (s, t)|
|
|
161
|
+
mine = out.top[item]
|
|
162
|
+
if mine.nil?
|
|
163
|
+
out.top[item] = [s, t]
|
|
164
|
+
next
|
|
165
|
+
end
|
|
166
|
+
later = [t, mine[1]].max
|
|
167
|
+
out.top[item] = [s * History.decay(later - t) + mine[0] * History.decay(later - mine[1]), later]
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
out.rejected = @rejected.dup
|
|
171
|
+
other.rejected.each { |item, t| out.rejected[item] = [t, out.rejected.fetch(item, t)].max }
|
|
172
|
+
out.rejected.to_a.each do |item, t|
|
|
173
|
+
entry = out.top[item]
|
|
174
|
+
if !entry.nil? && entry[1] > t
|
|
175
|
+
out.rejected.delete(item)
|
|
176
|
+
else
|
|
177
|
+
out.top.delete(item)
|
|
178
|
+
end
|
|
179
|
+
end
|
|
180
|
+
|
|
181
|
+
by_item = {}
|
|
182
|
+
(@recent + other.recent).each_with_index do |(item, t), order|
|
|
183
|
+
existing = by_item[item]
|
|
184
|
+
by_item[item] = [t, order] if existing.nil? || t > existing[0]
|
|
185
|
+
end
|
|
186
|
+
# Rejected items go before the trim, so they don't cost the list its places.
|
|
187
|
+
kept = by_item.reject { |item, _| out.rejected.key?(item) }.map { |item, (t, order)| [t, order, item] }.sort
|
|
188
|
+
out.recent = kept.last(RECENT_SIZE).map { |t, _, item| [item, t] }
|
|
189
|
+
|
|
190
|
+
times = [@last_t, other.last_t].compact
|
|
191
|
+
out.last_t = times.max
|
|
192
|
+
out.send(:prune, now)
|
|
193
|
+
out.send(:prune_rejected)
|
|
194
|
+
out
|
|
195
|
+
end
|
|
196
|
+
|
|
197
|
+
# Merges a server snapshot, keeping the items rejected here rejected: a
|
|
198
|
+
# local rejection is usually one the server hasn't stored yet, so it wins
|
|
199
|
+
# over the snapshot's score.
|
|
200
|
+
def merge_snapshot(snapshot, now)
|
|
201
|
+
trimmed = snapshot.dup
|
|
202
|
+
trimmed.top = snapshot.top.reject { |item, _| @rejected.key?(item) }
|
|
203
|
+
merge(trimmed, now)
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
# The JSON-safe stored form (raw scores), shaped like sdk/python's.
|
|
207
|
+
def encode
|
|
208
|
+
{
|
|
209
|
+
"top" => @top.map { |item, (s, t)| [item, s, t] },
|
|
210
|
+
"recent" => @recent.map { |item, t| [item, t] },
|
|
211
|
+
"rejected" => @rejected.map { |item, t| [item, t] },
|
|
212
|
+
"lastT" => @last_t
|
|
213
|
+
}
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
# Inverse of encode; nil for any malformed field. A state stored before
|
|
217
|
+
# rejections existed has no "rejected".
|
|
218
|
+
def self.decode(raw)
|
|
219
|
+
return nil unless valid_encoding?(raw)
|
|
220
|
+
|
|
221
|
+
state = new
|
|
222
|
+
raw["top"].each { |item, s, t| state.top[item] = [s.to_f, t] }
|
|
223
|
+
state.recent = raw["recent"].map { |item, t| [item, t] }
|
|
224
|
+
raw.fetch("rejected", []).each { |item, t| state.rejected[item] = t }
|
|
225
|
+
state.last_t = raw["lastT"]
|
|
226
|
+
state.send(:prune_rejected)
|
|
227
|
+
state
|
|
228
|
+
end
|
|
229
|
+
|
|
230
|
+
def self.valid_encoding?(raw)
|
|
231
|
+
return false unless raw.is_a?(Hash) && raw.key?("lastT")
|
|
232
|
+
|
|
233
|
+
top = raw["top"]
|
|
234
|
+
recent = raw["recent"]
|
|
235
|
+
rejected = raw.fetch("rejected", [])
|
|
236
|
+
return false unless [top, recent, rejected].all? { |v| v.is_a?(Array) }
|
|
237
|
+
|
|
238
|
+
(raw["lastT"].nil? || time?(raw["lastT"])) &&
|
|
239
|
+
top.all? { |e| pair?(e, 3) && weight?(e[1]) && time?(e[2]) } &&
|
|
240
|
+
(recent + rejected).all? { |e| pair?(e, 2) && time?(e[1]) }
|
|
241
|
+
end
|
|
242
|
+
|
|
243
|
+
def self.pair?(entry, size)
|
|
244
|
+
entry.is_a?(Array) && entry.size == size && id?(entry[0])
|
|
245
|
+
end
|
|
246
|
+
|
|
247
|
+
def self.list(value)
|
|
248
|
+
value.is_a?(Array) ? value : []
|
|
249
|
+
end
|
|
250
|
+
|
|
251
|
+
private_class_method :valid_encoding?, :pair?, :list
|
|
252
|
+
|
|
253
|
+
private
|
|
254
|
+
|
|
255
|
+
def reject(item, t)
|
|
256
|
+
# Dropped, not lowered: a lowered score would decay back towards 0.
|
|
257
|
+
current = @top[item]
|
|
258
|
+
return if !current.nil? && current[1] > t
|
|
259
|
+
|
|
260
|
+
@top.delete(item)
|
|
261
|
+
@recent = @recent.reject { |entry| entry[0] == item }
|
|
262
|
+
@rejected[item] = [t, @rejected.fetch(item, t)].max
|
|
263
|
+
prune_rejected
|
|
264
|
+
end
|
|
265
|
+
|
|
266
|
+
def prune(now)
|
|
267
|
+
return unless @top.size > TOP_K
|
|
268
|
+
|
|
269
|
+
@top = ranked(now).first(TOP_K).to_h { |item, _, _| [item, @top[item]] }
|
|
270
|
+
end
|
|
271
|
+
|
|
272
|
+
def prune_rejected
|
|
273
|
+
return unless @rejected.size > REJECTED_SIZE
|
|
274
|
+
|
|
275
|
+
@rejected = rejected_order.last(REJECTED_SIZE).to_h { |item| [item, @rejected[item]] }
|
|
276
|
+
end
|
|
277
|
+
end
|
|
278
|
+
|
|
279
|
+
# A failed call. status is 0 when no HTTP response was received. code is the
|
|
280
|
+
# API's error code (invalid_request, rate_limited, quota_exceeded...) or a
|
|
281
|
+
# client one: network, timeout, invalid_response.
|
|
282
|
+
class Error < StandardError
|
|
283
|
+
attr_reader :code, :status, :retry_after
|
|
284
|
+
|
|
285
|
+
def initialize(code, message, status: 0, retry_after: nil)
|
|
286
|
+
super(message)
|
|
287
|
+
@code = code
|
|
288
|
+
@status = status
|
|
289
|
+
@retry_after = retry_after
|
|
290
|
+
end
|
|
291
|
+
|
|
292
|
+
# Whether trying again later can succeed (quota_exceeded can't until it resets).
|
|
293
|
+
def retryable?
|
|
294
|
+
return true if %w[network timeout].include?(@code)
|
|
295
|
+
return @code != "quota_exceeded" if @status == 429
|
|
296
|
+
|
|
297
|
+
@status >= 500
|
|
298
|
+
end
|
|
299
|
+
|
|
300
|
+
# Whether recapi couldn't answer in time or failed on its side. A 429 is
|
|
301
|
+
# the tenant's own limit and a 4xx an integration bug: a fallback would hide both.
|
|
302
|
+
def outage?
|
|
303
|
+
%w[network timeout].include?(@code) || @status >= 500
|
|
304
|
+
end
|
|
305
|
+
end
|
|
306
|
+
|
|
307
|
+
# One recommended or popular item. metadata is the object stored with the
|
|
308
|
+
# item (name, picture, link, price...), nil when it has none.
|
|
309
|
+
#
|
|
310
|
+
# why is the reason code to turn into a label for visitors: "for_you",
|
|
311
|
+
# "similar", "next", "bought_together", "trending", "popular", "new",
|
|
312
|
+
# "explore" or "more" (docs/api.md "Reason codes"), nil when the API sent
|
|
313
|
+
# none. New codes may be added: show nothing for an unknown one.
|
|
314
|
+
Item = Struct.new(:item_id, :score, :slot, :metadata, :why, keyword_init: true)
|
|
315
|
+
|
|
316
|
+
# A recommendation list. fallback lists are popular items with no id.
|
|
317
|
+
Recommendations = Struct.new(:recommendation_id, :fallback, :items, keyword_init: true) do
|
|
318
|
+
alias_method :fallback?, :fallback
|
|
319
|
+
end
|
|
320
|
+
|
|
321
|
+
# In-process store keeping the most recently used max_users users. Pass a
|
|
322
|
+
# shared one (Redis, Rails.cache) when several processes serve the same users.
|
|
323
|
+
class MemoryStore
|
|
324
|
+
def initialize(max_users: 10_000)
|
|
325
|
+
@max = [1, max_users].max
|
|
326
|
+
@data = {}
|
|
327
|
+
@lock = Mutex.new
|
|
328
|
+
end
|
|
329
|
+
|
|
330
|
+
def get(key)
|
|
331
|
+
@lock.synchronize do
|
|
332
|
+
value = @data.delete(key)
|
|
333
|
+
@data[key] = value unless value.nil?
|
|
334
|
+
value
|
|
335
|
+
end
|
|
336
|
+
end
|
|
337
|
+
|
|
338
|
+
def set(key, value)
|
|
339
|
+
@lock.synchronize do
|
|
340
|
+
@data.delete(key)
|
|
341
|
+
@data[key] = value
|
|
342
|
+
@data.delete(@data.first[0]) while @data.size > @max
|
|
343
|
+
end
|
|
344
|
+
end
|
|
345
|
+
|
|
346
|
+
def delete(key)
|
|
347
|
+
@lock.synchronize { @data.delete(key) }
|
|
348
|
+
end
|
|
349
|
+
end
|
|
350
|
+
|
|
351
|
+
# The default logger: warnings and errors as "recapi: ..." lines on $stderr.
|
|
352
|
+
# Not the standard Logger: Ruby 4.0 made logger a bundled gem, which a
|
|
353
|
+
# Bundler app without it in its Gemfile can't require.
|
|
354
|
+
class StderrLogger
|
|
355
|
+
def warn(message)
|
|
356
|
+
$stderr.write("#{message}\n")
|
|
357
|
+
end
|
|
358
|
+
|
|
359
|
+
def error(message)
|
|
360
|
+
$stderr.write("#{message}\n")
|
|
361
|
+
end
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
# A recapi client, safe to share between threads (Puma); create one per
|
|
365
|
+
# process. In a forked worker (Unicorn, Puma cluster) it starts afresh.
|
|
366
|
+
class Client
|
|
367
|
+
# API limits (docs/api.md).
|
|
368
|
+
MAX_EVENTS_PER_REQUEST = 1000
|
|
369
|
+
MAX_ITEMS_PER_REQUEST = 1000
|
|
370
|
+
MAX_EXCLUDE = 500
|
|
371
|
+
DEFAULT_COUNT = 10
|
|
372
|
+
MAX_COUNT = 100
|
|
373
|
+
# A widget still empty after 1.5 s has lost the user: recommend() gives up
|
|
374
|
+
# then and falls back, whatever the general timeout (which suits events).
|
|
375
|
+
RECOMMEND_TIMEOUT = 1.5
|
|
376
|
+
# The recommended level scale (docs/api.md "Interaction levels"), by name.
|
|
377
|
+
LEVEL_NAMES = { "view" => 3, "cart" => 5, "purchase" => 10, "reject" => -10 }.freeze
|
|
378
|
+
# Events waiting for a send: a minute-long outage at 150 events/s fits.
|
|
379
|
+
MAX_QUEUE = 10_000
|
|
380
|
+
MAX_FLUSH_BACKOFF = 60.0
|
|
381
|
+
RETRIES = 3
|
|
382
|
+
RETRY_BASE = 0.5
|
|
383
|
+
RETRY_MAX = 8.0
|
|
384
|
+
# Attribution windows, as in the JS SDK: an event on a shown item within
|
|
385
|
+
# 30 minutes of the display is the click; after a click, that item's
|
|
386
|
+
# events are attributed for 24 hours.
|
|
387
|
+
DISPLAY_WINDOW_MS = 30 * 60 * 1000
|
|
388
|
+
CLICK_WINDOW_MS = 24 * 60 * 60 * 1000
|
|
389
|
+
# Shown items remembered per user.
|
|
390
|
+
MAX_SHOWN_ITEMS = 100
|
|
391
|
+
PUBLIC_KEY_PREFIX = "rk_pub_"
|
|
392
|
+
# Open connections kept between calls: one per call in flight at the busiest moment.
|
|
393
|
+
MAX_IDLE_CONNECTIONS = 16
|
|
394
|
+
CODE_BY_STATUS = {
|
|
395
|
+
400 => "invalid_request", 401 => "unauthorized", 403 => "forbidden",
|
|
396
|
+
404 => "not_found", 413 => "too_large", 429 => "rate_limited"
|
|
397
|
+
}.freeze
|
|
398
|
+
# Failures before any usable reply: the request may not have reached the API.
|
|
399
|
+
NETWORK_ERRORS = [
|
|
400
|
+
IOError, EOFError, SocketError, SystemCallError, OpenSSL::SSL::SSLError,
|
|
401
|
+
Net::HTTPBadResponse, Net::ProtocolError, *(Zlib::Error if defined?(Zlib::Error))
|
|
402
|
+
].freeze
|
|
403
|
+
TIMEOUT_ERRORS = [Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout].freeze
|
|
404
|
+
# A connection the server closed while idle fails this way, before any reply.
|
|
405
|
+
STALE_ERRORS = [EOFError, Errno::ECONNRESET, Errno::EPIPE, Errno::ECONNABORTED].freeze
|
|
406
|
+
# Made here, not in initialize, whose block would hold the client through
|
|
407
|
+
# self: the weak reference lets a dropped client be collected.
|
|
408
|
+
CLOSE_AT_EXIT = lambda do |ref|
|
|
409
|
+
at_exit do
|
|
410
|
+
ref.__getobj__.close
|
|
411
|
+
rescue WeakRef::RefError
|
|
412
|
+
nil
|
|
413
|
+
end
|
|
414
|
+
end
|
|
415
|
+
private_constant :CLOSE_AT_EXIT
|
|
416
|
+
|
|
417
|
+
# api_key: secret key (rk_sec_...): a server can then restore a user's
|
|
418
|
+
# history from recapi when the store lacks it. base_url: your recapi API
|
|
419
|
+
# origin. store: per-user state, anything with get/set/delete (a Redis
|
|
420
|
+
# client) or read/write/delete (Rails.cache); defaults to a MemoryStore.
|
|
421
|
+
# timeout: per-request seconds (recommend() uses at most 1.5).
|
|
422
|
+
# flush_interval: seconds between background sends. max_batch: events per
|
|
423
|
+
# request, at most the API's 1,000. logger: where failures are logged,
|
|
424
|
+
# anything with warn and error (a Logger, Rails.logger); $stderr by default.
|
|
425
|
+
def initialize(api_key, base_url, store: nil, timeout: 5.0, flush_interval: 2.0,
|
|
426
|
+
max_batch: MAX_EVENTS_PER_REQUEST, logger: nil)
|
|
427
|
+
raise ArgumentError, "api_key is required" unless api_key.is_a?(String) && !api_key.empty?
|
|
428
|
+
raise ArgumentError, "base_url is required" unless base_url.is_a?(String) && !base_url.empty?
|
|
429
|
+
|
|
430
|
+
@api_key = api_key
|
|
431
|
+
@base = base_url.chomp("/")
|
|
432
|
+
@url = URI.parse(@base)
|
|
433
|
+
unless @url.is_a?(URI::HTTP) && !@url.host.to_s.empty?
|
|
434
|
+
raise ArgumentError, "base_url must be an http(s) URL"
|
|
435
|
+
end
|
|
436
|
+
|
|
437
|
+
@store = store || MemoryStore.new
|
|
438
|
+
@timeout = timeout.to_f
|
|
439
|
+
@interval = flush_interval.to_f
|
|
440
|
+
@max_batch = max_batch.to_i.clamp(1, MAX_EVENTS_PER_REQUEST)
|
|
441
|
+
@logger = logger || StderrLogger.new
|
|
442
|
+
# Two environments' keys may share one Redis: their users must not mix.
|
|
443
|
+
@prefix = "recapi:#{Digest::SHA256.hexdigest(api_key)[0, 12]}:"
|
|
444
|
+
# Indirections the tests replace (clock, backoff sleep, jitter).
|
|
445
|
+
@clock = -> { Process.clock_gettime(Process::CLOCK_REALTIME, :millisecond) }
|
|
446
|
+
@sleeper = method(:wait)
|
|
447
|
+
@random = -> { rand }
|
|
448
|
+
init_threading
|
|
449
|
+
CLOSE_AT_EXIT.call(WeakRef.new(self))
|
|
450
|
+
end
|
|
451
|
+
|
|
452
|
+
# Records an interaction: the user's history changes at once, the event
|
|
453
|
+
# is queued. level is an integer -10 to 10 or :view (3), :cart (5),
|
|
454
|
+
# :purchase (10), :reject (-10: never recommend the item to this user
|
|
455
|
+
# again until a positive event on it). recommendation_id overrides the
|
|
456
|
+
# automatic attribution. Ids may be strings or integers.
|
|
457
|
+
def track(item_id, level, user_id: nil, anon_id: nil, recommendation_id: nil)
|
|
458
|
+
level = level_value(level)
|
|
459
|
+
item_id = id_string(item_id)
|
|
460
|
+
raise ArgumentError, "item_id #{ID_RULE}" unless History.id?(item_id)
|
|
461
|
+
|
|
462
|
+
user_id, anon_id, recommendation_id = check_ids(user_id: user_id, anon_id: anon_id,
|
|
463
|
+
recommendation_id: recommendation_id)
|
|
464
|
+
t = now_ms
|
|
465
|
+
attributed = recommendation_id
|
|
466
|
+
key = key_for(user_id, anon_id)
|
|
467
|
+
unless key.nil?
|
|
468
|
+
with_locks(key) do
|
|
469
|
+
state = load(key)
|
|
470
|
+
state.history.apply(item_id, level, t)
|
|
471
|
+
entry = state.shown[item_id]
|
|
472
|
+
# An explicit id of the recommendation that showed the item is
|
|
473
|
+
# still the click on it; another id leaves the entry alone.
|
|
474
|
+
if recommendation_id.nil? || (entry && entry[0] == recommendation_id)
|
|
475
|
+
attributed = attribute(entry, level, t) || recommendation_id
|
|
476
|
+
end
|
|
477
|
+
save(key, state, t)
|
|
478
|
+
end
|
|
479
|
+
end
|
|
480
|
+
enqueue(compact(
|
|
481
|
+
"item_id" => item_id, "level" => level, "user_id" => user_id, "anon_id" => anon_id,
|
|
482
|
+
"recommendation_id" => attributed, "timestamp" => iso(t), "event_id" => SecureRandom.uuid
|
|
483
|
+
))
|
|
484
|
+
nil
|
|
485
|
+
end
|
|
486
|
+
|
|
487
|
+
# Sends every queued event now, with retries (end of a job or a request in
|
|
488
|
+
# a worker). Never raises: failures are logged, retryable batches stay queued.
|
|
489
|
+
def flush
|
|
490
|
+
drain(retry_errors: true)
|
|
491
|
+
nil
|
|
492
|
+
end
|
|
493
|
+
|
|
494
|
+
# Stops the sender and sends what is queued, one try per batch (runs at
|
|
495
|
+
# exit). No retries, so a deploy never waits on a recapi outage.
|
|
496
|
+
def close
|
|
497
|
+
# In a forked worker, the parent's queue and closed state are not this one's.
|
|
498
|
+
forked
|
|
499
|
+
closing = false
|
|
500
|
+
@mutex.synchronize do
|
|
501
|
+
closing = !@closed
|
|
502
|
+
# Also ends a send's retry sleep, so close waits for at most one try.
|
|
503
|
+
@closed = true
|
|
504
|
+
@cond.broadcast
|
|
505
|
+
end
|
|
506
|
+
drain(retry_errors: false) if closing
|
|
507
|
+
# Also on a later close (the one at exit): calls made after the first
|
|
508
|
+
# may have opened connections again.
|
|
509
|
+
idle = @idle_lock.synchronize { @idle.slice!(0..) }
|
|
510
|
+
idle.each { |conn| finish(conn) }
|
|
511
|
+
nil
|
|
512
|
+
end
|
|
513
|
+
|
|
514
|
+
# Personalised recommendations for a user (or a visitor's anon_id). Sends
|
|
515
|
+
# the history from the store; without one, recapi uses its stored history
|
|
516
|
+
# for user_id and returns it for the store. A history the store started
|
|
517
|
+
# without it (a track() or identify() came first) gets recapi's fetched
|
|
518
|
+
# once, within the same deadline, and merged in. No retry, at most 1.5 s:
|
|
519
|
+
# on a network error, timeout or 5xx it returns popular items (fallback,
|
|
520
|
+
# no id) and logs the error. If those fail too (recapi is down), the list
|
|
521
|
+
# is empty: hide the slot. A 4xx or 429 raises: a fallback would hide an
|
|
522
|
+
# integration bug or a plan limit.
|
|
523
|
+
def recommend(user_id: nil, anon_id: nil, count: 0, placement: nil, tags_any: nil, tags_all: nil,
|
|
524
|
+
exclude: nil, context_item_id: nil, discovery: nil)
|
|
525
|
+
deadline = monotonic + recommend_timeout
|
|
526
|
+
user_id, anon_id = check_ids(user_id: user_id, anon_id: anon_id)
|
|
527
|
+
context_item_id = id_string(context_item_id)
|
|
528
|
+
exclude = exclude&.map { |item| id_string(item) }
|
|
529
|
+
key = key_for(user_id, anon_id)
|
|
530
|
+
t = now_ms
|
|
531
|
+
state = key.nil? ? UserState.new : load(key)
|
|
532
|
+
if user_id && !state.restored && !state.history.empty? && !@api_key.start_with?(PUBLIC_KEY_PREFIX)
|
|
533
|
+
state = restore(key, user_id, state, deadline, t)
|
|
534
|
+
end
|
|
535
|
+
history, recent, rejected = state.history.wire(t)
|
|
536
|
+
excluded = with_rejected(exclude, rejected)
|
|
537
|
+
begin
|
|
538
|
+
response = request(
|
|
539
|
+
"POST", "/v1/recommendations",
|
|
540
|
+
timeout: deadline - monotonic,
|
|
541
|
+
body: compact(
|
|
542
|
+
"user_id" => user_id, "anon_id" => anon_id, "placement" => placement,
|
|
543
|
+
"count" => count.to_i.zero? ? nil : count, "tags_any" => non_empty(tags_any),
|
|
544
|
+
"tags_all" => non_empty(tags_all), "exclude" => excluded, "context_item_id" => context_item_id,
|
|
545
|
+
"history" => history, "recent" => recent, "discovery" => discovery
|
|
546
|
+
)
|
|
547
|
+
)
|
|
548
|
+
rec_id, items = parse_recommendations(response)
|
|
549
|
+
rescue Error => e
|
|
550
|
+
raise unless e.outage?
|
|
551
|
+
|
|
552
|
+
return fallback(e, count, excluded, context_item_id, tags_any, tags_all)
|
|
553
|
+
end
|
|
554
|
+
|
|
555
|
+
unless key.nil?
|
|
556
|
+
with_locks(key) do
|
|
557
|
+
state = load(key)
|
|
558
|
+
items.each do |item|
|
|
559
|
+
state.shown.delete(item.item_id)
|
|
560
|
+
state.shown[item.item_id] = [rec_id, t]
|
|
561
|
+
end
|
|
562
|
+
state.shown.delete(state.shown.first[0]) while state.shown.size > MAX_SHOWN_ITEMS
|
|
563
|
+
# Only trusted when we sent nothing; merged, never replacing what
|
|
564
|
+
# was tracked while the request was in flight, and only once: a
|
|
565
|
+
# concurrent first call may have merged it already.
|
|
566
|
+
snapshot = response["history"]
|
|
567
|
+
done = state.restored && !state.history.empty?
|
|
568
|
+
if history.empty? && recent.empty? && snapshot.is_a?(Hash) && !done
|
|
569
|
+
state.history = state.history.merge_snapshot(History.from_snapshot(snapshot), t)
|
|
570
|
+
state.restored = true
|
|
571
|
+
end
|
|
572
|
+
save(key, state, t)
|
|
573
|
+
end
|
|
574
|
+
end
|
|
575
|
+
Recommendations.new(recommendation_id: rec_id, fallback: false, items: items)
|
|
576
|
+
end
|
|
577
|
+
|
|
578
|
+
# Call at login: the visitor's history and shown items become the user's.
|
|
579
|
+
# Merged into what the store has for user_id (scores add up, as in the
|
|
580
|
+
# history spec), then the visitor's entry is deleted. Without it, a
|
|
581
|
+
# visitor's clicks stop counting once they log in.
|
|
582
|
+
def identify(anon_id, user_id)
|
|
583
|
+
user_id, anon_id = check_ids(user_id: user_id, anon_id: anon_id)
|
|
584
|
+
raise ArgumentError, "anon_id and user_id are required" if anon_id.nil? || user_id.nil?
|
|
585
|
+
|
|
586
|
+
visitor = key_for(nil, anon_id)
|
|
587
|
+
user = key_for(user_id, nil)
|
|
588
|
+
t = now_ms
|
|
589
|
+
with_locks(visitor, user) do
|
|
590
|
+
guest = load(visitor)
|
|
591
|
+
next if guest.history.empty? && guest.history.rejected.empty? && guest.shown.empty?
|
|
592
|
+
|
|
593
|
+
state = load(user)
|
|
594
|
+
state.history = state.history.merge(guest.history, t)
|
|
595
|
+
# recapi's snapshot, fetched later, holds this visitor's browsing too.
|
|
596
|
+
state.anon_id = anon_id unless state.restored
|
|
597
|
+
# Oldest display first, as recommend() keeps them; an item's later display wins.
|
|
598
|
+
shown = (state.shown.to_a + guest.shown.to_a).each_with_index.sort_by { |(_, e), i| [e[1], i] }
|
|
599
|
+
state.shown = {}
|
|
600
|
+
shown.each do |(item, entry), _|
|
|
601
|
+
state.shown.delete(item)
|
|
602
|
+
state.shown[item] = entry
|
|
603
|
+
end
|
|
604
|
+
save(user, state, t)
|
|
605
|
+
begin
|
|
606
|
+
store_call(:delete, visitor)
|
|
607
|
+
rescue StandardError => e
|
|
608
|
+
@logger.error("recapi: could not delete the visitor's stored state: #{e.message}")
|
|
609
|
+
end
|
|
610
|
+
end
|
|
611
|
+
nil
|
|
612
|
+
end
|
|
613
|
+
|
|
614
|
+
# Non-personalised popular items, for pages without a user (no retry).
|
|
615
|
+
def popular(count: DEFAULT_COUNT, tags_any: nil, tags_all: nil)
|
|
616
|
+
fetch_popular(count, tags_any, tags_all, nil)
|
|
617
|
+
end
|
|
618
|
+
|
|
619
|
+
# Creates or replaces items (hashes of the API's item fields: item_id,
|
|
620
|
+
# tags, metadata...), 1,000 per request. Retried like events. Returns how
|
|
621
|
+
# many were stored; an invalid item raises, and nothing of its batch of
|
|
622
|
+
# 1,000 is stored.
|
|
623
|
+
def upsert_items(items)
|
|
624
|
+
upserted = 0
|
|
625
|
+
items.each_slice(MAX_ITEMS_PER_REQUEST) do |batch|
|
|
626
|
+
body = { "items" => batch.map { |item| item.to_h.transform_keys(&:to_s) } }
|
|
627
|
+
response = with_retry { request("POST", "/v1/items:batch", body: body) }
|
|
628
|
+
upserted += response.is_a?(Hash) ? response.fetch("upserted", 0).to_i : 0
|
|
629
|
+
end
|
|
630
|
+
upserted
|
|
631
|
+
end
|
|
632
|
+
|
|
633
|
+
# Erases a user's events in recapi (GDPR) and their state in the store.
|
|
634
|
+
def delete_user(user_id)
|
|
635
|
+
user_id = id_string(user_id)
|
|
636
|
+
raise ArgumentError, "user_id #{ID_RULE}" unless History.id?(user_id)
|
|
637
|
+
|
|
638
|
+
with_retry { request("DELETE", "/v1/users/#{escape(user_id)}") }
|
|
639
|
+
begin
|
|
640
|
+
store_call(:delete, key_for(user_id, nil))
|
|
641
|
+
rescue StandardError => e
|
|
642
|
+
@logger.error("recapi: could not delete the user's stored state: #{e.message}")
|
|
643
|
+
end
|
|
644
|
+
nil
|
|
645
|
+
end
|
|
646
|
+
|
|
647
|
+
ID_RULE = "must be 1-256 bytes of UTF-8 without control characters"
|
|
648
|
+
|
|
649
|
+
# What the store keeps per user: the history and the shown items. shown
|
|
650
|
+
# maps item => [recommendation id, shown at, clicked at?], oldest first.
|
|
651
|
+
# restored is set once the history holds recapi's stored one. anon_id is
|
|
652
|
+
# the visitor identify merged in before that: recapi's snapshot also holds
|
|
653
|
+
# that visitor's browsing, so the restore asks it to leave it out.
|
|
654
|
+
class UserState
|
|
655
|
+
attr_accessor :history, :shown, :restored, :anon_id
|
|
656
|
+
|
|
657
|
+
def initialize(history = nil, shown = nil, restored = false, anon_id = nil)
|
|
658
|
+
@history = history || History.new
|
|
659
|
+
@shown = shown || {}
|
|
660
|
+
@restored = restored
|
|
661
|
+
@anon_id = anon_id
|
|
662
|
+
end
|
|
663
|
+
|
|
664
|
+
def dump(now)
|
|
665
|
+
shown = @shown.select { |_, entry| Client.attributable?(entry, now) }.map { |item, entry| [item, *entry] }
|
|
666
|
+
state = { "history" => @history.encode, "shown" => shown.last(MAX_SHOWN_ITEMS) }
|
|
667
|
+
state["restored"] = true if @restored
|
|
668
|
+
state["anon_id"] = @anon_id if @anon_id && !@anon_id.empty?
|
|
669
|
+
JSON.generate(state)
|
|
670
|
+
end
|
|
671
|
+
|
|
672
|
+
# nil when raw is malformed.
|
|
673
|
+
def self.parse(raw)
|
|
674
|
+
data = JSON.parse(raw)
|
|
675
|
+
return nil unless data.is_a?(Hash) && data["shown"].is_a?(Array)
|
|
676
|
+
|
|
677
|
+
history = History.decode(data["history"])
|
|
678
|
+
return nil if history.nil?
|
|
679
|
+
|
|
680
|
+
shown = {}
|
|
681
|
+
data["shown"].each do |e|
|
|
682
|
+
shown[e[0]] = e[1..] if e.is_a?(Array) && e.size.between?(3, 4) && History.id?(e[0])
|
|
683
|
+
end
|
|
684
|
+
anon_id = data["anon_id"]
|
|
685
|
+
new(history, shown, data["restored"] == true, History.id?(anon_id) ? anon_id : nil)
|
|
686
|
+
rescue JSON::ParserError, TypeError
|
|
687
|
+
nil
|
|
688
|
+
end
|
|
689
|
+
end
|
|
690
|
+
|
|
691
|
+
def self.attributable?(entry, t)
|
|
692
|
+
return t - entry[1] <= DISPLAY_WINDOW_MS if entry.size < 3
|
|
693
|
+
|
|
694
|
+
t - entry[2] <= CLICK_WINDOW_MS
|
|
695
|
+
end
|
|
696
|
+
|
|
697
|
+
private
|
|
698
|
+
|
|
699
|
+
def init_threading
|
|
700
|
+
# Also runs in a forked worker: the parent's queue is the parent's to
|
|
701
|
+
# send, its sender thread did not survive the fork, and its connections
|
|
702
|
+
# must not be shared.
|
|
703
|
+
@pid = Process.pid
|
|
704
|
+
@user_locks = Array.new(64) { Mutex.new }
|
|
705
|
+
@queue = []
|
|
706
|
+
@mutex = Mutex.new
|
|
707
|
+
@cond = ConditionVariable.new
|
|
708
|
+
@send_lock = Mutex.new
|
|
709
|
+
@thread = nil
|
|
710
|
+
@failures = 0
|
|
711
|
+
@closed = false
|
|
712
|
+
@idle = []
|
|
713
|
+
@idle_lock = Mutex.new
|
|
714
|
+
end
|
|
715
|
+
|
|
716
|
+
def forked
|
|
717
|
+
init_threading if @pid != Process.pid
|
|
718
|
+
end
|
|
719
|
+
|
|
720
|
+
def now_ms
|
|
721
|
+
@clock.call
|
|
722
|
+
end
|
|
723
|
+
|
|
724
|
+
def monotonic
|
|
725
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
726
|
+
end
|
|
727
|
+
|
|
728
|
+
def recommend_timeout
|
|
729
|
+
[@timeout, RECOMMEND_TIMEOUT].min
|
|
730
|
+
end
|
|
731
|
+
|
|
732
|
+
# Waits seconds; returns true, at once, when interruptible and the client
|
|
733
|
+
# is or becomes closed.
|
|
734
|
+
def wait(seconds, interruptible)
|
|
735
|
+
return sleep(seconds) && false unless interruptible
|
|
736
|
+
|
|
737
|
+
deadline = monotonic + seconds
|
|
738
|
+
@mutex.synchronize do
|
|
739
|
+
until @closed
|
|
740
|
+
left = deadline - monotonic
|
|
741
|
+
return false if left <= 0
|
|
742
|
+
|
|
743
|
+
@cond.wait(@mutex, left)
|
|
744
|
+
end
|
|
745
|
+
true
|
|
746
|
+
end
|
|
747
|
+
end
|
|
748
|
+
|
|
749
|
+
# --- events ---------------------------------------------------------------
|
|
750
|
+
|
|
751
|
+
def enqueue(event)
|
|
752
|
+
forked
|
|
753
|
+
closed = @mutex.synchronize do
|
|
754
|
+
if @thread.nil? && !@closed
|
|
755
|
+
@thread = Thread.new { run }
|
|
756
|
+
@thread.name = "recapi-sender"
|
|
757
|
+
@thread.report_on_exception = false
|
|
758
|
+
end
|
|
759
|
+
@queue << event
|
|
760
|
+
drop_overflow if @queue.size > MAX_QUEUE
|
|
761
|
+
@cond.broadcast if @queue.size >= @max_batch
|
|
762
|
+
@closed
|
|
763
|
+
end
|
|
764
|
+
drain(retry_errors: false) if closed
|
|
765
|
+
end
|
|
766
|
+
|
|
767
|
+
def drop_overflow
|
|
768
|
+
dropped = @queue.size - MAX_QUEUE
|
|
769
|
+
@queue.shift(dropped)
|
|
770
|
+
@logger.warn("recapi: event queue full, dropped the #{dropped} oldest events")
|
|
771
|
+
end
|
|
772
|
+
|
|
773
|
+
def run
|
|
774
|
+
loop do
|
|
775
|
+
@mutex.synchronize do
|
|
776
|
+
deadline = monotonic + [@interval * 2**@failures, MAX_FLUSH_BACKOFF].min
|
|
777
|
+
# A full batch goes at once, except after a failure: then the
|
|
778
|
+
# backoff holds, or a down API would be hammered.
|
|
779
|
+
until @closed || (@failures.zero? && @queue.size >= @max_batch)
|
|
780
|
+
left = deadline - monotonic
|
|
781
|
+
break if left <= 0
|
|
782
|
+
|
|
783
|
+
@cond.wait(@mutex, left)
|
|
784
|
+
end
|
|
785
|
+
return if @closed
|
|
786
|
+
end
|
|
787
|
+
drain(retry_errors: true)
|
|
788
|
+
end
|
|
789
|
+
end
|
|
790
|
+
|
|
791
|
+
def drain(retry_errors:)
|
|
792
|
+
forked
|
|
793
|
+
@send_lock.synchronize do
|
|
794
|
+
loop do
|
|
795
|
+
batch = @mutex.synchronize { @queue.shift(@max_batch) }
|
|
796
|
+
return if batch.empty?
|
|
797
|
+
|
|
798
|
+
begin
|
|
799
|
+
send_events(batch, retry_errors)
|
|
800
|
+
rescue StandardError => e
|
|
801
|
+
if e.is_a?(Error) && !e.retryable?
|
|
802
|
+
@logger.error("recapi: dropped #{batch.size} events: #{e.message}")
|
|
803
|
+
next
|
|
804
|
+
end
|
|
805
|
+
# Anything else keeps the batch, a bug of ours included: event ids
|
|
806
|
+
# make a resend harmless, and a dead sender would lose every event.
|
|
807
|
+
@logger.error("recapi: unexpected error sending events: #{e.class}: #{e.message}") unless e.is_a?(Error)
|
|
808
|
+
@mutex.synchronize do
|
|
809
|
+
@queue.unshift(*batch)
|
|
810
|
+
drop_overflow if @queue.size > MAX_QUEUE
|
|
811
|
+
@failures += 1
|
|
812
|
+
end
|
|
813
|
+
@logger.warn("recapi: sending #{batch.size} events failed, kept: #{e.message}")
|
|
814
|
+
return
|
|
815
|
+
end
|
|
816
|
+
@mutex.synchronize { @failures = 0 }
|
|
817
|
+
end
|
|
818
|
+
end
|
|
819
|
+
end
|
|
820
|
+
|
|
821
|
+
def send_events(events, retry_errors)
|
|
822
|
+
send = -> { request("POST", "/v1/events", body: { "events" => events }) }
|
|
823
|
+
begin
|
|
824
|
+
response = retry_errors ? with_retry(interruptible: true, &send) : send.call
|
|
825
|
+
rescue Error => e
|
|
826
|
+
# Over the API's body limit (ids mostly quotes or backslashes, which
|
|
827
|
+
# JSON escapes): halves fit where the whole didn't. If the second half
|
|
828
|
+
# fails, the batch is kept whole; event ids make the resend harmless.
|
|
829
|
+
raise if e.status != 413 || events.size < 2
|
|
830
|
+
|
|
831
|
+
half = events.size / 2
|
|
832
|
+
send_events(events[0, half], retry_errors)
|
|
833
|
+
send_events(events[half..], retry_errors)
|
|
834
|
+
return
|
|
835
|
+
end
|
|
836
|
+
rejected = response.is_a?(Hash) ? response["rejected"] : nil
|
|
837
|
+
return unless rejected.is_a?(Array) && !rejected.empty?
|
|
838
|
+
|
|
839
|
+
# Invalid events are reported, never retried: they would fail again.
|
|
840
|
+
detail = rejected.first(5).map do |r|
|
|
841
|
+
r.is_a?(Hash) && r["index"].is_a?(Integer) ? r.merge("event" => events[r["index"]]) : r
|
|
842
|
+
end
|
|
843
|
+
@logger.error("recapi: #{rejected.size} event(s) rejected: #{JSON.generate(detail)}")
|
|
844
|
+
end
|
|
845
|
+
|
|
846
|
+
# Calls the block, retrying retryable errors with jittered exponential
|
|
847
|
+
# backoff. Retry-After wins over the backoff, and both are capped at 8 s.
|
|
848
|
+
# Once the client closes (interruptible), the last error is raised instead.
|
|
849
|
+
def with_retry(interruptible: false)
|
|
850
|
+
attempt = 0
|
|
851
|
+
begin
|
|
852
|
+
yield
|
|
853
|
+
rescue Error => e
|
|
854
|
+
raise if attempt >= RETRIES || !e.retryable?
|
|
855
|
+
|
|
856
|
+
backoff = RETRY_BASE * 2**attempt * (0.5 + @random.call / 2.0)
|
|
857
|
+
attempt += 1
|
|
858
|
+
raise if @sleeper.call([RETRY_MAX, e.retry_after || backoff].min, interruptible)
|
|
859
|
+
|
|
860
|
+
retry
|
|
861
|
+
end
|
|
862
|
+
end
|
|
863
|
+
|
|
864
|
+
# --- recommendations ------------------------------------------------------
|
|
865
|
+
|
|
866
|
+
def fallback(error, count, excluded, context_item_id, tags_any, tags_all)
|
|
867
|
+
# The requested length without the excluded items, so a few more are
|
|
868
|
+
# asked for. 0 is the recommendations default; popular refuses it.
|
|
869
|
+
n = count.to_i.zero? ? DEFAULT_COUNT : count.to_i
|
|
870
|
+
skip = (excluded || []).to_h { |item| [item, true] }
|
|
871
|
+
skip[context_item_id] = true unless context_item_id.nil?
|
|
872
|
+
begin
|
|
873
|
+
popular = fetch_popular([n + skip.size, MAX_COUNT].min, tags_any, tags_all, recommend_timeout)
|
|
874
|
+
rescue Error => e
|
|
875
|
+
# recapi is down altogether: an empty list the page can hide beats an
|
|
876
|
+
# exception that fails the whole page.
|
|
877
|
+
@logger.warn("recapi: recommendations failed (#{error.message}), popular too, served no items: #{e.message}")
|
|
878
|
+
return Recommendations.new(recommendation_id: nil, fallback: true, items: [])
|
|
879
|
+
end
|
|
880
|
+
@logger.warn("recapi: recommendations failed, served popular items: #{error.message}")
|
|
881
|
+
items = popular.reject { |item| skip.key?(item.item_id) }.first(n)
|
|
882
|
+
Recommendations.new(recommendation_id: nil, fallback: true, items: items)
|
|
883
|
+
end
|
|
884
|
+
|
|
885
|
+
# Merges recapi's stored history into a local one that lacks it (a fresh
|
|
886
|
+
# store). The recommendation's reply carries it only when the request
|
|
887
|
+
# sends no history, which would drop this one. So it is asked for apart,
|
|
888
|
+
# once: a failure leaves the state as it was, to be tried on the next call.
|
|
889
|
+
def restore(key, user_id, state, deadline, t)
|
|
890
|
+
begin
|
|
891
|
+
# Half the time left, so the recommendation itself still fits.
|
|
892
|
+
query = state.anon_id ? { "anon_id" => state.anon_id } : nil
|
|
893
|
+
path = "/v1/users/#{escape(user_id)}/history"
|
|
894
|
+
snapshot = request("GET", path, query: query, timeout: (deadline - monotonic) / 2)
|
|
895
|
+
raise Error.new("invalid_response", "malformed history response") unless snapshot.is_a?(Hash)
|
|
896
|
+
rescue Error => e
|
|
897
|
+
@logger.warn("recapi: could not restore the user's stored history: #{e.message}")
|
|
898
|
+
return state
|
|
899
|
+
end
|
|
900
|
+
with_locks(key) do
|
|
901
|
+
state = load(key)
|
|
902
|
+
# A concurrent first call may have merged it already.
|
|
903
|
+
unless state.restored
|
|
904
|
+
state.history = state.history.merge_snapshot(History.from_snapshot(snapshot), t)
|
|
905
|
+
state.restored = true
|
|
906
|
+
state.anon_id = nil
|
|
907
|
+
save(key, state, t)
|
|
908
|
+
end
|
|
909
|
+
end
|
|
910
|
+
state
|
|
911
|
+
end
|
|
912
|
+
|
|
913
|
+
def fetch_popular(count, tags_any, tags_all, timeout)
|
|
914
|
+
query = {}
|
|
915
|
+
query["tags_any"] = tags_any.join(",") if tags_any && !tags_any.empty?
|
|
916
|
+
query["tags_all"] = tags_all.join(",") if tags_all && !tags_all.empty?
|
|
917
|
+
query["count"] = count.to_s
|
|
918
|
+
response = request("GET", "/v1/popular", timeout: timeout, query: query)
|
|
919
|
+
raw = response.is_a?(Hash) ? response["items"] : nil
|
|
920
|
+
raise Error.new("invalid_response", "malformed popular response") unless raw.is_a?(Array)
|
|
921
|
+
|
|
922
|
+
raw.map { |entry| parse_item(entry, "backfill") }
|
|
923
|
+
end
|
|
924
|
+
|
|
925
|
+
def parse_recommendations(response)
|
|
926
|
+
unless response.is_a?(Hash) && response["recommendation_id"].is_a?(String) && response["items"].is_a?(Array)
|
|
927
|
+
raise Error.new("invalid_response", "malformed recommendations response")
|
|
928
|
+
end
|
|
929
|
+
|
|
930
|
+
[response["recommendation_id"], response["items"].map { |entry| parse_item(entry, "exploit") }]
|
|
931
|
+
end
|
|
932
|
+
|
|
933
|
+
def parse_item(entry, slot)
|
|
934
|
+
unless entry.is_a?(Hash) && entry["item_id"].is_a?(String)
|
|
935
|
+
raise Error.new("invalid_response", "malformed item in response")
|
|
936
|
+
end
|
|
937
|
+
|
|
938
|
+
metadata = entry["metadata"]
|
|
939
|
+
why = entry["why"]
|
|
940
|
+
score = entry["score"]
|
|
941
|
+
Item.new(
|
|
942
|
+
item_id: entry["item_id"],
|
|
943
|
+
score: score.is_a?(Numeric) ? score.to_f : 0.0,
|
|
944
|
+
slot: entry["slot"].is_a?(String) && !entry["slot"].empty? ? entry["slot"] : slot,
|
|
945
|
+
metadata: metadata.is_a?(Hash) && !metadata.empty? ? metadata : nil,
|
|
946
|
+
why: why.is_a?(String) && !why.empty? ? why : nil
|
|
947
|
+
)
|
|
948
|
+
end
|
|
949
|
+
|
|
950
|
+
# The recommendation an event at t belongs to; a first click is recorded.
|
|
951
|
+
def attribute(entry, level, t)
|
|
952
|
+
return nil if entry.nil? || !Client.attributable?(entry, t)
|
|
953
|
+
|
|
954
|
+
entry << t if level.positive? && entry.size < 3
|
|
955
|
+
entry[0]
|
|
956
|
+
end
|
|
957
|
+
|
|
958
|
+
# The caller's exclude plus the rejected items, newest first, up to the API limit.
|
|
959
|
+
def with_rejected(exclude, rejected)
|
|
960
|
+
return exclude&.dup if rejected.empty?
|
|
961
|
+
|
|
962
|
+
out = (exclude || []).uniq
|
|
963
|
+
seen = out.to_h { |item| [item, true] }
|
|
964
|
+
rejected.reverse_each do |item|
|
|
965
|
+
break if out.size >= MAX_EXCLUDE
|
|
966
|
+
next if seen.key?(item)
|
|
967
|
+
|
|
968
|
+
seen[item] = true
|
|
969
|
+
out << item
|
|
970
|
+
end
|
|
971
|
+
out
|
|
972
|
+
end
|
|
973
|
+
|
|
974
|
+
# --- state ----------------------------------------------------------------
|
|
975
|
+
|
|
976
|
+
def key_for(user_id, anon_id)
|
|
977
|
+
return "#{@prefix}u:#{user_id}" if user_id
|
|
978
|
+
return "#{@prefix}a:#{anon_id}" if anon_id
|
|
979
|
+
|
|
980
|
+
nil
|
|
981
|
+
end
|
|
982
|
+
|
|
983
|
+
# Always in the same order, or two logins could each hold the other's lock.
|
|
984
|
+
def with_locks(*keys, &block)
|
|
985
|
+
locks = keys.map { |key| key.hash % @user_locks.size }.uniq.sort.map { |i| @user_locks[i] }
|
|
986
|
+
locks.reverse.reduce(block) { |inner, lock| -> { lock.synchronize(&inner) } }.call
|
|
987
|
+
end
|
|
988
|
+
|
|
989
|
+
# The store is a cache: when it fails, recapi's stored history is still
|
|
990
|
+
# there, so the call goes on without it.
|
|
991
|
+
def load(key)
|
|
992
|
+
raw = begin
|
|
993
|
+
store_call(:get, key)
|
|
994
|
+
rescue StandardError => e
|
|
995
|
+
@logger.error("recapi: store read failed: #{e.class}: #{e.message}")
|
|
996
|
+
return UserState.new
|
|
997
|
+
end
|
|
998
|
+
return UserState.new if raw.nil? || raw.empty?
|
|
999
|
+
|
|
1000
|
+
state = UserState.parse(raw)
|
|
1001
|
+
return state unless state.nil?
|
|
1002
|
+
|
|
1003
|
+
@logger.warn("recapi: discarding a malformed stored user state")
|
|
1004
|
+
UserState.new
|
|
1005
|
+
end
|
|
1006
|
+
|
|
1007
|
+
def save(key, state, now)
|
|
1008
|
+
store_call(:set, key, state.dump(now))
|
|
1009
|
+
rescue StandardError => e
|
|
1010
|
+
@logger.error("recapi: store write failed: #{e.class}: #{e.message}")
|
|
1011
|
+
end
|
|
1012
|
+
|
|
1013
|
+
# Redis clients have get/set/del, Rails.cache read/write/delete.
|
|
1014
|
+
def store_call(action, key, value = nil)
|
|
1015
|
+
case action
|
|
1016
|
+
when :get then @store.respond_to?(:get) ? @store.get(key) : @store.read(key)
|
|
1017
|
+
when :set then @store.respond_to?(:set) ? @store.set(key, value) : @store.write(key, value)
|
|
1018
|
+
else @store.respond_to?(:delete) ? @store.delete(key) : @store.del(key)
|
|
1019
|
+
end
|
|
1020
|
+
end
|
|
1021
|
+
|
|
1022
|
+
# --- HTTP -----------------------------------------------------------------
|
|
1023
|
+
|
|
1024
|
+
def request(method, path, body: nil, query: nil, timeout: nil)
|
|
1025
|
+
params = (query || {}).dup
|
|
1026
|
+
headers = { "Accept" => "application/json", "User-Agent" => "recapi-ruby/#{VERSION}" }
|
|
1027
|
+
# A public key travels in the URL only on GET /v1/popular (CDN caching);
|
|
1028
|
+
# the API refuses a secret key there.
|
|
1029
|
+
if method == "GET" && @api_key.start_with?(PUBLIC_KEY_PREFIX)
|
|
1030
|
+
params["key"] = @api_key
|
|
1031
|
+
else
|
|
1032
|
+
headers["Authorization"] = "Bearer #{@api_key}"
|
|
1033
|
+
end
|
|
1034
|
+
target = @url.path + path
|
|
1035
|
+
target += "?#{URI.encode_www_form(params)}" unless params.empty?
|
|
1036
|
+
data = nil
|
|
1037
|
+
unless body.nil?
|
|
1038
|
+
# Raw UTF-8: escaped, non-Latin ids would take up to 3 times the bytes.
|
|
1039
|
+
data = JSON.generate(body)
|
|
1040
|
+
headers["Content-Type"] = "application/json"
|
|
1041
|
+
end
|
|
1042
|
+
limit = timeout.nil? ? @timeout : timeout
|
|
1043
|
+
raise Error.new("timeout", "the deadline passed before the request was sent") if limit <= 0
|
|
1044
|
+
|
|
1045
|
+
begin
|
|
1046
|
+
response = exchange(method, target, data, headers, limit)
|
|
1047
|
+
rescue *TIMEOUT_ERRORS
|
|
1048
|
+
raise Error.new("timeout", "request timed out after #{format('%.3g', limit)} s")
|
|
1049
|
+
rescue *NETWORK_ERRORS => e
|
|
1050
|
+
raise Error.new("network", "#{e.class}: #{e.message}")
|
|
1051
|
+
end
|
|
1052
|
+
status = response.code.to_i
|
|
1053
|
+
# Net::HTTP follows no redirect: a 3xx (an http:// base_url sent on to
|
|
1054
|
+
# https) fails like a 4xx, or a DELETE would look done.
|
|
1055
|
+
raise http_error(status, response) unless (200..299).cover?(status)
|
|
1056
|
+
|
|
1057
|
+
parse_json(response.body)
|
|
1058
|
+
end
|
|
1059
|
+
|
|
1060
|
+
# Sends one request on an idle connection, or a new one. A connection the
|
|
1061
|
+
# server closed while idle fails before any reply comes: it is replaced
|
|
1062
|
+
# once. A failure once the reply has started is never resent: the API has
|
|
1063
|
+
# the request (a recommendation would count twice).
|
|
1064
|
+
def exchange(method, target, data, headers, limit)
|
|
1065
|
+
forked
|
|
1066
|
+
start = monotonic
|
|
1067
|
+
# The most recently used: the least likely to have timed out.
|
|
1068
|
+
conn = @idle_lock.synchronize { @idle.pop }
|
|
1069
|
+
2.times do |attempt|
|
|
1070
|
+
remaining = limit - (monotonic - start)
|
|
1071
|
+
raise Net::OpenTimeout, "deadline" if remaining <= 0
|
|
1072
|
+
|
|
1073
|
+
reused = !conn.nil?
|
|
1074
|
+
conn ||= connection
|
|
1075
|
+
conn.open_timeout = conn.read_timeout = conn.write_timeout = remaining
|
|
1076
|
+
req = Net::HTTPGenericRequest.new(method, !data.nil?, method != "HEAD", target, headers)
|
|
1077
|
+
req.body = data unless data.nil?
|
|
1078
|
+
begin
|
|
1079
|
+
conn.start unless conn.started?
|
|
1080
|
+
response = conn.request(req)
|
|
1081
|
+
rescue *STALE_ERRORS
|
|
1082
|
+
finish(conn)
|
|
1083
|
+
raise if !reused || attempt.positive?
|
|
1084
|
+
|
|
1085
|
+
conn = nil
|
|
1086
|
+
next
|
|
1087
|
+
rescue Exception # rubocop:disable Lint/RescueException
|
|
1088
|
+
finish(conn)
|
|
1089
|
+
raise
|
|
1090
|
+
end
|
|
1091
|
+
# A body cut short of its Content-Length reads as whole here.
|
|
1092
|
+
length = response["Content-Length"]
|
|
1093
|
+
if length && response.body.to_s.bytesize < length.to_i && response["Content-Encoding"].nil?
|
|
1094
|
+
finish(conn)
|
|
1095
|
+
raise EOFError, "the reply was cut short"
|
|
1096
|
+
end
|
|
1097
|
+
keep(conn, response)
|
|
1098
|
+
return response
|
|
1099
|
+
end
|
|
1100
|
+
end
|
|
1101
|
+
|
|
1102
|
+
def connection
|
|
1103
|
+
# Net::HTTP honours HTTPS_PROXY, HTTP_PROXY and NO_PROXY itself.
|
|
1104
|
+
conn = Net::HTTP.new(@url.host, @url.port)
|
|
1105
|
+
conn.use_ssl = @url.scheme == "https"
|
|
1106
|
+
# No silent resend: a recommendation would count twice.
|
|
1107
|
+
conn.max_retries = 0
|
|
1108
|
+
# Reused until the server closes it (Net::HTTP's 2 s would handshake again).
|
|
1109
|
+
conn.keep_alive_timeout = 60
|
|
1110
|
+
conn
|
|
1111
|
+
end
|
|
1112
|
+
|
|
1113
|
+
# Returns a connection whose last reply was read whole to the idle ones.
|
|
1114
|
+
def keep(conn, response)
|
|
1115
|
+
return finish(conn) if response["Connection"].to_s.downcase == "close"
|
|
1116
|
+
|
|
1117
|
+
kept = @idle_lock.synchronize { @idle.size < MAX_IDLE_CONNECTIONS && @idle.push(conn) }
|
|
1118
|
+
finish(conn) unless kept
|
|
1119
|
+
end
|
|
1120
|
+
|
|
1121
|
+
def finish(conn)
|
|
1122
|
+
conn.finish if conn.started?
|
|
1123
|
+
rescue IOError, SystemCallError
|
|
1124
|
+
nil
|
|
1125
|
+
end
|
|
1126
|
+
|
|
1127
|
+
def http_error(status, response)
|
|
1128
|
+
payload = parse_json(response.body)
|
|
1129
|
+
body = payload.is_a?(Hash) && payload["error"].is_a?(Hash) ? payload["error"] : {}
|
|
1130
|
+
code = body["code"]
|
|
1131
|
+
code = CODE_BY_STATUS[status] || (status >= 500 ? "internal" : "http_error") if code.nil? || code == ""
|
|
1132
|
+
message = body["message"]
|
|
1133
|
+
message = "HTTP #{status}" if message.nil? || message == ""
|
|
1134
|
+
Error.new(code.to_s, message.to_s, status: status, retry_after: retry_after(response["Retry-After"]))
|
|
1135
|
+
end
|
|
1136
|
+
|
|
1137
|
+
def retry_after(header)
|
|
1138
|
+
seconds = Float(header.to_s.strip)
|
|
1139
|
+
seconds.finite? && seconds >= 0 ? seconds : nil
|
|
1140
|
+
rescue ArgumentError
|
|
1141
|
+
nil
|
|
1142
|
+
end
|
|
1143
|
+
|
|
1144
|
+
def parse_json(raw)
|
|
1145
|
+
return nil if raw.nil? || raw.empty?
|
|
1146
|
+
|
|
1147
|
+
JSON.parse(raw)
|
|
1148
|
+
rescue JSON::ParserError
|
|
1149
|
+
nil
|
|
1150
|
+
end
|
|
1151
|
+
|
|
1152
|
+
# --- arguments ------------------------------------------------------------
|
|
1153
|
+
|
|
1154
|
+
def level_value(level)
|
|
1155
|
+
name = level.is_a?(Symbol) ? level.to_s : level
|
|
1156
|
+
return LEVEL_NAMES[name] if name.is_a?(String) && LEVEL_NAMES.key?(name)
|
|
1157
|
+
return level if level.is_a?(Integer) && level.between?(-10, 10)
|
|
1158
|
+
|
|
1159
|
+
raise ArgumentError, "level must be an integer from -10 to 10 or one of #{LEVEL_NAMES.keys.join(', ')}"
|
|
1160
|
+
end
|
|
1161
|
+
|
|
1162
|
+
# Rails ids are integers: they travel as strings, which the API requires.
|
|
1163
|
+
def id_string(value)
|
|
1164
|
+
value.is_a?(Integer) ? value.to_s : value
|
|
1165
|
+
end
|
|
1166
|
+
|
|
1167
|
+
def check_ids(**ids)
|
|
1168
|
+
ids.map do |name, value|
|
|
1169
|
+
value = id_string(value)
|
|
1170
|
+
raise ArgumentError, "#{name} #{ID_RULE}" unless value.nil? || History.id?(value)
|
|
1171
|
+
|
|
1172
|
+
value
|
|
1173
|
+
end
|
|
1174
|
+
end
|
|
1175
|
+
|
|
1176
|
+
def escape(id)
|
|
1177
|
+
URI.encode_www_form_component(id).gsub("+", "%20")
|
|
1178
|
+
end
|
|
1179
|
+
|
|
1180
|
+
def non_empty(values)
|
|
1181
|
+
values.nil? || values.empty? ? nil : values.to_a
|
|
1182
|
+
end
|
|
1183
|
+
|
|
1184
|
+
def compact(fields)
|
|
1185
|
+
fields.compact
|
|
1186
|
+
end
|
|
1187
|
+
|
|
1188
|
+
def iso(t_ms)
|
|
1189
|
+
Time.at(t_ms / 1000, t_ms % 1000, :millisecond).utc.strftime("%Y-%m-%dT%H:%M:%S.%LZ")
|
|
1190
|
+
end
|
|
1191
|
+
end
|
|
1192
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: recapi
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.3
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- recapi
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: bin
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2026-11-02 00:00:00.000000000 Z
|
|
12
|
+
dependencies: []
|
|
13
|
+
description: 'Ruby client for the recapi recommendation API: per-user history, event
|
|
14
|
+
batching, retries, popular fallback. Standard library only.'
|
|
15
|
+
email:
|
|
16
|
+
executables: []
|
|
17
|
+
extensions: []
|
|
18
|
+
extra_rdoc_files: []
|
|
19
|
+
files:
|
|
20
|
+
- LICENSE
|
|
21
|
+
- README.md
|
|
22
|
+
- lib/recapi.rb
|
|
23
|
+
homepage: https://recapi.io
|
|
24
|
+
licenses:
|
|
25
|
+
- MIT
|
|
26
|
+
metadata: {}
|
|
27
|
+
post_install_message:
|
|
28
|
+
rdoc_options: []
|
|
29
|
+
require_paths:
|
|
30
|
+
- lib
|
|
31
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
32
|
+
requirements:
|
|
33
|
+
- - ">="
|
|
34
|
+
- !ruby/object:Gem::Version
|
|
35
|
+
version: '3.1'
|
|
36
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
37
|
+
requirements:
|
|
38
|
+
- - ">="
|
|
39
|
+
- !ruby/object:Gem::Version
|
|
40
|
+
version: '0'
|
|
41
|
+
requirements: []
|
|
42
|
+
rubygems_version: 3.3.27
|
|
43
|
+
signing_key:
|
|
44
|
+
specification_version: 4
|
|
45
|
+
summary: Ruby client for the recapi recommendation API
|
|
46
|
+
test_files: []
|