sitegoose 0.3.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: f88d90ea071f9dde430e079a88e42a22dabd4f0faeaa4a662da5c94eb8cd942c
4
+ data.tar.gz: 1033458a91be1a78a2464a9439d2d021eaedce386b0cfb7eb7662db4927a3e50
5
+ SHA512:
6
+ metadata.gz: e20cd12422c5830f3a710340198b634c2c192e9d26a830879fa756ebd1901a95c28054ab5d74d73812a80b7243653078592ddd59d9df02048760039b6e9f543b
7
+ data.tar.gz: 5190a6b466bdf730a41265537aa1fdf01090aaeb77966d0bf720532ec12db816ebcb15b9ee96561e0a1f45c4d5f9d83256a1adf049f05873f94e57118896c635
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 steviec87
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,342 @@
1
+ # sitegoose (Ruby)
2
+
3
+ One package for your Ruby server, run from your analytics dashboard: a
4
+ **firewall**, **error monitoring** and **server analytics** (payments,
5
+ sign-ups, leads). Rails, Sinatra and any Rack app, with Sidekiq jobs. Only
6
+ the standard library and `rack`; Ruby 3.0 or later.
7
+
8
+ ```ruby
9
+ # Gemfile
10
+ gem "sitegoose"
11
+ ```
12
+
13
+ ## Rails
14
+
15
+ Nothing to add to the middleware stack: the gem's Railtie puts it first.
16
+
17
+ ```ruby
18
+ # config/initializers/sitegoose.rb
19
+ SiteGoose.configure(
20
+ site_id: "YOUR_SITE_ID",
21
+ secret: ENV["ANALYTICS_SITE_SECRET"], # site settings; keep it on the server
22
+ endpoint: "https://YOUR_COLLECTOR_URL",
23
+ # trust_proxy: 1, # behind Nginx or a load balancer
24
+ # trust_proxy: "cloudflare", # every request through Cloudflare
25
+ )
26
+ ```
27
+
28
+ Or keep it in the credentials (`bin/rails credentials:edit`), under
29
+ `sitegoose:` with the same keys, or in `config.sitegoose`.
30
+ The environment works too: `ANALYTICS_SITE_ID`, `ANALYTICS_SITE_SECRET`,
31
+ `ANALYTICS_ENDPOINT`, `ANALYTICS_SERVER_NAME`, `ANALYTICS_TRUST_PROXY`.
32
+ Lowest first: the environment, credentials, `config.sitegoose`,
33
+ `configure`.
34
+
35
+ Then, on their own:
36
+
37
+ - Request errors, with the route pattern (`/orders/:id`), including the ones
38
+ Rails turns into its 500 page.
39
+ - `Rails.error.handle`, `.record` and `.report`: the errors your code handles.
40
+ - Devise (Warden) failed logins count towards bot protection, and logins and
41
+ logouts go to the audit log.
42
+ - The controller action, mail and queued job before an error are its
43
+ breadcrumbs.
44
+
45
+ In your code: `SiteGoose.track(...)`, or `SiteGoose.platform`.
46
+
47
+ ## Sinatra and other Rack apps
48
+
49
+ ```ruby
50
+ use SiteGoose::Middleware,
51
+ site_id: "YOUR_SITE_ID", secret: ENV["ANALYTICS_SITE_SECRET"], endpoint: "https://YOUR_COLLECTOR_URL"
52
+ ```
53
+
54
+ Put it first; added twice (by hand next to the Railtie, or by an engine),
55
+ the second copy passes requests straight on. Sinatra's route patterns (`/users/:id`) and its error pages are
56
+ read on their own; set `set :raise_errors, false` (production's default) or
57
+ `set :show_exceptions, false` as usual. Hanami, Roda and plain Rack work the
58
+ same way. A full example is in `examples/sinatra/`.
59
+
60
+ ## Sidekiq
61
+
62
+ ```ruby
63
+ # config/initializers/sidekiq.rb
64
+ require "sitegoose/sidekiq"
65
+ SiteGoose::Sidekiq.install
66
+ ```
67
+
68
+ Every job then runs as a [job](#error-monitoring) named after its class (the
69
+ ActiveJob class under ActiveJob): its errors carry the route
70
+ `/(job) ChargeJob`, and each run is timed (Server speed → Jobs).
71
+
72
+ - A failure that will be retried is timed as failed but not sent: the error
73
+ is sent when the last attempt fails, or when Sidekiq gives the job up (the
74
+ Dead set, `:kill`).
75
+ - `Sidekiq::Shutdown` (a job stopped mid-run, to run again) is neither timed
76
+ nor sent.
77
+ - What's waiting is sent when Sidekiq shuts down.
78
+ - Don't also wrap a job in `platform.job`: it would be timed twice.
79
+
80
+ ## Puma, Unicorn and Passenger
81
+
82
+ The platform is made on the first request, so a master that only forks
83
+ workers starts nothing. Its background threads start again in each worker
84
+ after a fork (Puma clustered mode, `preload_app!`, Unicorn, Passenger), a
85
+ fork waits for a send in flight, and what the parent had waiting is the
86
+ parent's to send. Nothing to configure. On Ruby 3.0, which has no fork hook,
87
+ a worker notices it was forked at its first request or job instead, and a
88
+ fork doesn't wait for a send in flight.
89
+
90
+ ## Modules
91
+
92
+ | Module | What it does | Default |
93
+ |---|---|---|
94
+ | `firewall` | Blocks addresses, known criminal networks, shared blocks and floods (Security → Firewall) | on |
95
+ | `firewall`'s attack patterns | Rules that match the *request* (its path, query string and user agent), not just the address, so an attack is turned away before your code runs. They come with the firewall's rules and are managed on the Security page (Firewall → Attack patterns). Nothing to configure (protocol: `sdk/protocols/firewall.md`). Patterns need Ruby 3.2+, which can stop a match that runs long; on 3.0 and 3.1 a rule's pattern is skipped (its other parts still match) and a warning says so once | with the firewall |
96
+ | `proxy` | Serves the tracker and takes visits under your own domain, so ad blockers can't hide those visitors | on |
97
+ | `errors` | Sends your server's errors to the Errors report | on |
98
+ | `analytics` | `track`, `identify` and `lead` from your server | on |
99
+ | `audit` | `platform.audit("user.role_changed", actor: ..., target: ..., request: ...)`; Devise logins and logouts on their own | on |
100
+ | `flags` | `platform.enabled?("new-checkout", user: user.id)`: features on or off from the dashboard | on |
101
+ | `maintenance` | A "back soon" page while maintenance mode is on in the dashboard | on |
102
+ | `headers` | Adds the security headers you pick on the Security page; your own headers win | on |
103
+ | `forms` | Checks forms before you accept them: `platform.check_form(request)` | on |
104
+ | `performance` | Response times and errors of every route (Health → Server speed): Rails and Sinatra route patterns. With it, the database queries each route runs (below) | on |
105
+ | `crawlers` | Which search engines and AI bots fetch your pages (Health → Crawlers), counted by the middleware | on |
106
+ | `resources` | This machine's CPU, load, memory, disks and network, once a minute (Health → Server resources). Linux | on |
107
+ | `inventory` | The installed gems and Ruby version, once a day, so the Security check can flag known security holes (`{host: ...}`: the server's name) | on |
108
+ | `posture` | How the server is set up, once a day, for the Security check: Rails' environment, `consider_all_requests_local`, `secret_key_base`, `config.hosts`, `force_ssl` and session cookie settings (when it's a Rails app), whether the app runs as root, code anyone on the machine can write, and files in the folders you serve to the web that shouldn't be public (a dump, a `.env`, a `.bak`). Reads only, and sends no file's contents. `{roots: ["public", {dir: "uploads", prefix: "files"}]}` (default: Rails' `public/`), `{checks: -> { [...] }}` for checks of your own (protocol: `sdk/protocols/posture.md`) | on |
109
+
110
+ Each takes `true`, `false` or its options as a hash:
111
+
112
+ ```ruby
113
+ SiteGoose.configure(site_id: ..., secret: ..., endpoint: ...,
114
+ firewall: { mode: "monitor", cache_file: "tmp/firewall-rules.json" },
115
+ errors: { capture_uncaught: false },
116
+ analytics: false)
117
+ ```
118
+
119
+ `enabled: false` turns the whole package off (every request passes).
120
+
121
+ **Switch modules off in the dashboard** (Security → Server package) without a
122
+ deploy: the package checks about once a minute. Modules are independent: one
123
+ failing to start, or the collector being down, never stops the others, and
124
+ never slows your requests; a failure of the package itself passes the request
125
+ to your app.
126
+
127
+ ## Database queries
128
+
129
+ Server speed → Queries: per route and per query, how many ran, how many
130
+ failed, and how long they took, so a slow route shows which query made it
131
+ slow. Every value is replaced by `?` before anything is counted
132
+ (`WHERE email = ?`), and bound parameters are never read
133
+ (protocol: `sdk/protocols/queries.md`).
134
+
135
+ - **Active Record** (Rails, or on its own): automatic, from its
136
+ `sql.active_record` notification. Schema and cached queries are left out.
137
+ - **Sequel**: one line, after `Sequel.connect`:
138
+
139
+ ```ruby
140
+ DB = Sequel.connect(ENV["DATABASE_URL"])
141
+ SiteGoose.watch_database(DB)
142
+ ```
143
+
144
+ - **MongoDB** (the `mongo` gem, Mongoid): automatic, through the driver's
145
+ command monitoring.
146
+
147
+ A query counts under the request (or Sidekiq or `platform.job` run) it ran
148
+ in, with the same method and route Server speed shows; one outside them,
149
+ like a console or a thread of your own, counts as `JOB (background)`. A
150
+ query is counted once: one run by Sequel on Active Record's connection is
151
+ Sequel's. It rides on `performance`: switched off there, nothing is
152
+ recorded; `queries: false` keeps Server speed without the queries:
153
+
154
+ ```ruby
155
+ SiteGoose.configure(site_id: ..., secret: ..., endpoint: ..., queries: false)
156
+ ```
157
+
158
+ ## Firewall
159
+
160
+ Rules, monitor or block mode and reports are in the dashboard (Security →
161
+ Firewall). New sites start in monitor mode.
162
+
163
+ Options: `mode` (`"monitor"` or `"block"`, overrides the dashboard),
164
+ `cache_file` (keep rules between restarts), `trust_proxy`, `counter`.
165
+
166
+ **The visitor's address:** behind a proxy, every request comes from the proxy
167
+ unless you set `trust_proxy`: `1` for one proxy or load balancer, `2` for
168
+ Cloudflare in front of a load balancer, `"cloudflare"` when every request
169
+ comes through Cloudflare (`CF-Connecting-IP` is believed only from
170
+ Cloudflare's own addresses: anyone who reaches your server directly can send
171
+ it). The package reads the connection's address itself, not Rails'
172
+ `request.remote_ip`, so set it here even if Rails already trusts the proxy.
173
+
174
+ **Who is counted:** an IPv4 address, or an IPv6 address's /64 (one machine
175
+ usually holds the whole /64); `::ffff:a.b.c.d` is the IPv4 address. Files a
176
+ page load asks for by the dozen (`/assets/`, `/packs/`, `.css`, `.js`, images,
177
+ fonts) don't count against the rate limit. A forwarded address that isn't one
178
+ (`unknown`, a name) means the connection's address is checked instead, and a
179
+ port on it (`1.2.3.4:5678`, `[2001:db8::1]:443`) is dropped.
180
+
181
+ **Rate limit with several workers:** each worker counts on its own, so the
182
+ limit is per worker. For one count, pass a shared counter:
183
+
184
+ ```ruby
185
+ counter = lambda do |key, ttl|
186
+ hits = REDIS.incr(key)
187
+ REDIS.expire(key, ttl) if hits == 1
188
+ hits
189
+ end
190
+
191
+ SiteGoose.configure(..., firewall: { counter: counter })
192
+ ```
193
+
194
+ **Bot protection:** Devise's failed logins are counted on their own;
195
+ elsewhere call `platform.login_failed(request)` when a login fails. Past the
196
+ site's limit the address is banned here straight away and on your other
197
+ servers within a minute or two. Floods (over the rate limit) are banned the
198
+ same way.
199
+
200
+ **Probes.** Requests for files no visitor ever asks for (`/.env`, `/.git/…`,
201
+ backups: the list comes with the rules) are rule `probe`; in block mode the
202
+ address is also banned on this server, and the collector bans it on the
203
+ site's other servers.
204
+
205
+ **Browser check** (paid plans, switched on in the dashboard). A visitor's
206
+ first page of the day is a short "checking your browser" page that sets a
207
+ cookie with JavaScript and reloads; after that the site answers as usual for
208
+ about a day. Only a `GET` or `HEAD` that asks for a web page is checked:
209
+ forms, webhooks, payment callbacks, API calls (`Accept: application/json`),
210
+ the paths you exclude, your allowed addresses, search engines and the AI
211
+ crawlers that bring visits never see it. It stops cheap floods (scripts that
212
+ don't run JavaScript) and nothing more: the pass is bound to the visitor's
213
+ address and the day. The cookie, `__sg_pass`, holds a signed day and nothing
214
+ about the visitor; it is strictly necessary and needs no consent. Checked
215
+ requests show on the Security page as rule `challenge`.
216
+
217
+ ## Error monitoring
218
+
219
+ Request errors are sent by the middleware, and the exception that ends the
220
+ program (a crash in a script or rake task) at exit. Errors you catch:
221
+
222
+ ```ruby
223
+ begin
224
+ charge_card(order)
225
+ rescue PaymentError => e
226
+ SiteGoose.capture_exception(e, status: 502)
227
+ raise
228
+ end
229
+ ```
230
+
231
+ In Rails, `Rails.error.handle { ... }` and `Rails.error.report(e)` do the
232
+ same. Exceptions your code only logs: `errors: { capture_logging: true }`
233
+ sends an exception logged at ERROR or above (`logger.error(e)`); it is still
234
+ logged. Rails' logger is hooked on its own; elsewhere
235
+ `platform.capture_logger(logger)`. An exception is sent once however many of
236
+ these see it, and one sent inside a request keeps that request's route. A
237
+ batch the collector couldn't take is tried once more a few seconds later.
238
+
239
+ What still can't be sent waits in a file in the temp folder and is sent when
240
+ it can be, after a restart too; each error keeps the time it happened.
241
+
242
+ An error in a request or job also shows the steps before it. Add them with
243
+ `SiteGoose.add_breadcrumb("Order saved")`; a step added outside a
244
+ request or job is dropped. Outgoing HTTP calls aren't recorded on their own
245
+ (Ruby has no public hook for them); in Rails the controller, mailer and job
246
+ notifications are.
247
+
248
+ Background jobs other than Sidekiq's:
249
+
250
+ ```ruby
251
+ SiteGoose.job("send-receipts") { send_receipts }
252
+ ```
253
+
254
+ Their errors carry the job's name as the route, a job that raises is sent
255
+ (status 500) and raised on, and each run is timed (Server speed → Jobs).
256
+
257
+ **Several apps on one site** (a Rails app, a Sidekiq process, a Node service
258
+ next to it): install the package in each, with the same `site_id`, `secret`
259
+ and `endpoint`, and a `server_name` of its own. Errors, Server speed,
260
+ Crawlers and Security → Updates then tell the apps apart, and Health → All
261
+ systems lists each with a light: the package checks in about once a minute
262
+ (`sdk/protocols/checkin.md`). A service only your other apps call takes
263
+ `firewall: false, proxy: false, maintenance: false`; on a machine shared by
264
+ several apps, keep `resources` on in only one of them.
265
+
266
+ Options: `server_name`, `root` (backtrace paths relative to it; default
267
+ `Rails.root` or the working directory), `release` (default
268
+ `ANALYTICS_RELEASE`), `capture_uncaught`, `capture_logging`, `spool_dir`
269
+ (where unsent errors wait; `false` for none), `before_send` (a lambda taking
270
+ the event and the error: return the event, changed or not, as a Hash, or
271
+ `nil` to drop it; anything else, or a raise, drops it with a warning).
272
+ Messages are cut at 2,000 characters, and the file of unsent errors stops at
273
+ 1 MB.
274
+
275
+ ## Server analytics
276
+
277
+ For what only your server knows for sure. Not pageviews: the tracker counts those.
278
+
279
+ ```ruby
280
+ platform = SiteGoose.platform
281
+ session = platform.session_id(request) # the visit, from the X-Analytics-Session header
282
+
283
+ platform.track("purchase", { "numeric_value" => "49.90", "currency" => "EUR", "event_uid" => "order-#{order.id}" }, session_id: session)
284
+ platform.identify(user.id, session_id: session)
285
+ platform.lead({ "email" => "ana@example.com", "name" => "Ana", "status" => "qualified" })
286
+ ```
287
+
288
+ `track` sends in the background; `identify` and `lead` wait for the answer
289
+ (up to 5 seconds).
290
+
291
+ **Link to the visit:** send the tracker's session ID with your site's requests:
292
+
293
+ ```js
294
+ fetch("/api/orders", { headers: { "x-analytics-session": window.analyticsSession?.getId() || "" }, ... });
295
+ ```
296
+
297
+ ## Ad blockers
298
+
299
+ This package already runs on your own domain, so it serves the tracker and
300
+ takes the visits itself. Use the two addresses it gives you in your layout:
301
+
302
+ ```erb
303
+ <% proxy = SiteGoose.platform&.proxy %>
304
+ <script async src="<%= proxy&.script_url %>" data-siteid="..." data-domain="example.com"
305
+ data-collect="<%= proxy&.collect_url %>"></script>
306
+ ```
307
+
308
+ - The path comes from your secret, so it isn't in the page and isn't a name
309
+ any blocker lists. Pick your own with `proxy: { prefix: "..." }`.
310
+ - Tracker files are fetched once and then held in memory; if we can't be
311
+ reached, the copy already held is served rather than nothing, and a file
312
+ that couldn't be fetched isn't asked for again for 30 seconds.
313
+ - These run on your app's request threads, so they wait at most 1 second to
314
+ connect and 2 to answer, and after three failures in a row the collector
315
+ is left alone for 30 seconds (beacons get a 502 straight away; the
316
+ tracker counts them as lost). A collector that stops answering can't use
317
+ up Puma's threads.
318
+ - The visitor's address goes on as `X-Forwarded-For` and cookies never leave
319
+ your server.
320
+ - Beacons are answered before the rate limit, the browser check, the
321
+ crawler count and the timing, so your own analytics traffic doesn't
322
+ appear as pages of your site. A denied or banned address gets a 403 here
323
+ too.
324
+ - Switch it off with `proxy: false`, or in the dashboard (Site settings →
325
+ Server). Then load the tracker from us as before.
326
+
327
+ ## Shutting down
328
+
329
+ What's waiting is sent at exit on its own, within 5 seconds in all (a
330
+ container's or PM2's grace period is longer): no send starts after that,
331
+ none is retried, and errors that weren't sent wait in their file for the
332
+ next start. Sidekiq's shutdown flush has the same deadline.
333
+ `SiteGoose.flush` sends it
334
+ now (a short-lived script, a serverless function before returning);
335
+ `platform.close` also stops the background threads.
336
+
337
+ ## Tests
338
+
339
+ ```bash
340
+ bundle install
341
+ ruby -Ilib -Itest -e 'Dir["test/*_test.rb"].each { |f| require_relative f }'
342
+ ```
@@ -0,0 +1,113 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Analytics module: what only the server sees for sure (payments, sign-ups,
4
+ # plan changes), next to the browser tracker's visits. Not pageviews: the
5
+ # tracker counts those, and counting them here too would count them twice.
6
+ #
7
+ # - track(): events, batched to POST /events/server-events in the background.
8
+ # Each gets an event_uid, so a retried batch can be told apart from new events.
9
+ # - identify(): links the browser's visit to your user ID (/events/identify).
10
+ # - lead(): adds or updates a lead (/events/lead).
11
+ # identify() and lead() wait for the answer (up to 5 seconds).
12
+
13
+ require "securerandom"
14
+
15
+ module SiteGoose
16
+ class Analytics
17
+ BATCH = 100
18
+ QUEUE_MAX = 1000
19
+ FLUSH_EVERY = 2
20
+ RETRY_EVERY = 15
21
+
22
+ attr_reader :queue
23
+
24
+ def initialize(core, **_)
25
+ @core = core
26
+ @queue = []
27
+ @lock = Mutex.new
28
+ @job = core.valid? ? Periodic.new(FLUSH_EVERY) { flush ? FLUSH_EVERY : RETRY_EVERY } : nil
29
+ Forking.register(self)
30
+ end
31
+
32
+ def after_fork_in_child
33
+ @lock = Mutex.new
34
+ @queue = []
35
+ end
36
+
37
+ # name: the event's name; fields: value, numeric_value, currency, url,
38
+ # event_label, variant, id, timestamp, event_uid, transaction_id, coupon
39
+ # (expresscollect/docs/server-events.md). Returns the event_uid, or ""
40
+ # when not queued.
41
+ def track(name, fields = nil, session_id = "")
42
+ return "" unless name.is_a?(String) && !name.empty? && @core.on?("analytics")
43
+
44
+ fields = (fields || {}).each_with_object({}) { |(k, v), out| out[k.to_s] = v }
45
+ event = { "timestamp" => SiteGoose.iso_now, "session_id" => session_id.to_s }
46
+ event.merge!(fields)
47
+ event["event_name"] = name[0, 100]
48
+ event["event_uid"] = fields["event_uid"].to_s.empty? ? SecureRandom.uuid : fields["event_uid"].to_s
49
+ event["numeric_value"] = event["numeric_value"].to_s if event["numeric_value"].is_a?(Numeric)
50
+ full = @lock.synchronize do
51
+ if @queue.size >= QUEUE_MAX
52
+ @core.log("analytics", "queue full, event dropped")
53
+ return ""
54
+ end
55
+ @queue << event
56
+ @queue.size >= BATCH
57
+ end
58
+ @job&.run_now if full
59
+ event["event_uid"]
60
+ end
61
+
62
+ # Sends what's waiting; false when some of it must be tried again.
63
+ def flush
64
+ waiting = @lock.synchronize do
65
+ taken = @queue
66
+ @queue = []
67
+ taken
68
+ end
69
+ ok = true
70
+ waiting.each_slice(BATCH) do |batch|
71
+ res = @core.request("analytics", "/events/server-events", body: { "events" => batch })
72
+ if res.nil? || res.status == 429 || res.status >= 500
73
+ ok = false
74
+ @lock.synchronize { @queue = (batch + @queue).first(QUEUE_MAX) }
75
+ elsif !res.ok?
76
+ @core.log("analytics", "events refused:", res.body.to_s[0, 300])
77
+ end
78
+ end
79
+ ok
80
+ end
81
+
82
+ # Links the visit to your user ID. true when stored.
83
+ def identify(user_id, session_id = "")
84
+ return false if user_id.to_s.empty? || session_id.to_s.empty? || !@core.on?("analytics")
85
+
86
+ res = @core.request("analytics", "/events/identify", body: { "user_id" => user_id.to_s, "session_id" => session_id.to_s })
87
+ res ? res.ok? : false
88
+ end
89
+
90
+ # fields: email or phone, and name, status, tags, note, deal fields...
91
+ # Returns {"status", "lead_id"}, or nil.
92
+ def lead(fields, session_id = "")
93
+ return nil unless @core.on?("analytics")
94
+
95
+ body = { "session_id" => session_id.to_s }
96
+ (fields || {}).each { |k, v| body[k.to_s] = v }
97
+ res = @core.request("analytics", "/events/lead", body: body)
98
+ return nil if res.nil?
99
+
100
+ unless res.ok?
101
+ @core.log("analytics", "lead refused:", res.body.to_s[0, 300])
102
+ return nil
103
+ end
104
+ res.json
105
+ end
106
+
107
+ def close
108
+ @job&.stop
109
+ Forking.unregister(self)
110
+ flush
111
+ end
112
+ end
113
+ end
@@ -0,0 +1,177 @@
1
+ # frozen_string_literal: true
2
+
3
+ # The browser check (rule `challenge`, sdk/protocols/firewall.md under "Flood
4
+ # protection"): a GET or HEAD that asks for a page, from an address with no
5
+ # valid pass, gets a small page instead of the site. Its script sets the pass
6
+ # cookie and reloads, which a script that doesn't run JavaScript never does.
7
+ #
8
+ # The pass is <day>.<first 32 hex digits of HMAC-SHA256(key, "<day>|<who>")>,
9
+ # key = HMAC-SHA256 of "sg-challenge-v1" keyed with the site secret, so any of
10
+ # the site's servers accepts a pass another one handed out. It counts for its
11
+ # own day and the next. sdk/protocols/challenge-cases.json holds the cases
12
+ # every client runs; backend/scripts/challenge-cases.mjs is the reference.
13
+
14
+ require "openssl"
15
+ require "securerandom"
16
+ require "ipaddr"
17
+ require "json"
18
+
19
+ module SiteGoose
20
+ module Challenge
21
+ COOKIE = "__sg_pass"
22
+ MAX_AGE = 86_400
23
+ LABEL = "sg-challenge-v1"
24
+ PASS = /\A(\d{1,10})\.([0-9a-f]{32})\z/.freeze
25
+
26
+ module_function
27
+
28
+ # The key every pass is signed with, from the site secret.
29
+ def pass_key(secret)
30
+ OpenSSL::HMAC.digest("SHA256", secret.to_s, LABEL)
31
+ end
32
+
33
+ # What a pass is bound to: the IPv4 address (an IPv4-mapped IPv6 address
34
+ # as IPv4), or an IPv6 address's /64 as four groups of four lower-case hex
35
+ # digits. "*" for the agent's nginx (a package never accepts it); nil for
36
+ # anything that isn't an address.
37
+ def who(ip)
38
+ text = ip.to_s.strip
39
+ return "*" if text == "*"
40
+
41
+ key = IP.key(text)
42
+ return nil if key.nil?
43
+
44
+ if key >> 32 == 0xFFFF
45
+ v4 = key & 0xFFFFFFFF
46
+ return [24, 16, 8, 0].map { |shift| (v4 >> shift) & 0xFF }.join(".")
47
+ end
48
+ format("%032x", key).scan(/.{4}/).first(4).join(":")
49
+ end
50
+
51
+ # Whole days since 1970-01-01 UTC.
52
+ def day_of(now = nil)
53
+ ((now || Time.now.to_f).to_f / 86_400).floor
54
+ end
55
+
56
+ def sign(key, day, bound)
57
+ OpenSSL::HMAC.hexdigest("SHA256", key, "#{day}|#{bound}")[0, 32]
58
+ end
59
+
60
+ # Today's pass for this address, or "" when it isn't one.
61
+ def make_pass(key, ip, now = nil)
62
+ bound = who(ip)
63
+ return "" if bound.nil?
64
+
65
+ day = day_of(now)
66
+ "#{day}.#{sign(key, day, bound)}"
67
+ end
68
+
69
+ def secure_compare(a, b)
70
+ return false unless a.bytesize == b.bytesize
71
+
72
+ if OpenSSL.respond_to?(:fixed_length_secure_compare)
73
+ OpenSSL.fixed_length_secure_compare(a, b)
74
+ else
75
+ diff = 0
76
+ a.bytes.zip(b.bytes) { |x, y| diff |= x ^ y }
77
+ diff.zero?
78
+ end
79
+ end
80
+
81
+ # True when value is this address's pass for today or yesterday.
82
+ def valid?(key, ip, value, now = nil)
83
+ match = PASS.match(value.to_s)
84
+ bound = who(ip)
85
+ return false if match.nil? || bound.nil? || bound == "*"
86
+
87
+ day = match[1].to_i
88
+ today = day_of(now)
89
+ return false unless day == today || day == today - 1
90
+
91
+ secure_compare(match[2], sign(key, day, bound))
92
+ end
93
+
94
+ # Every value the Cookie header carries under name (a browser can send
95
+ # two, from different paths); at most four, like the Node client. Two
96
+ # Cookie headers arrive joined by ", " (Puma), so that separates cookies
97
+ # too: our values ("<day>.<hex>", the preview code) never hold one, and a
98
+ # piece of someone else's value split off this way names no cookie of ours.
99
+ def cookie_values(cookie_header, name = COOKIE)
100
+ found = []
101
+ cookie_header.to_s.split(/;|, /).each do |part|
102
+ k, sep, value = part.strip.partition("=")
103
+ next if sep.empty? || k.strip != name
104
+
105
+ found << value.strip.delete_prefix('"').delete_suffix('"')
106
+ break if found.size == 4
107
+ end
108
+ found
109
+ end
110
+
111
+ def pass?(key, ip, cookie_header, now = nil)
112
+ cookie_values(cookie_header).any? { |value| valid?(key, ip, value, now) }
113
+ end
114
+
115
+ # The rules' skip prefixes, lower-cased.
116
+ def compile_skip(entries)
117
+ Array(entries).select { |p| p.is_a?(String) && !p.empty? }.map(&:downcase).first(200)
118
+ end
119
+
120
+ # Conditions 1 to 3: a GET or HEAD, whose Accept asks for text/html, for a
121
+ # path (lower-cased, no query string) under none of skip.
122
+ def asks_for_page?(method, target, accept, skip)
123
+ return false unless %w[GET HEAD].include?(method.to_s.upcase)
124
+ return false unless accept.to_s.downcase.include?("text/html")
125
+
126
+ path = target.to_s.split("?", 2)[0].to_s.split("#", 2)[0].to_s.downcase
127
+ skip.none? { |prefix| path.start_with?(prefix) }
128
+ end
129
+
130
+ PAGE = <<~HTML
131
+ <!doctype html>
132
+ <html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
133
+ <meta name="robots" content="noindex"><title>Just a moment</title>
134
+ <style nonce="%<nonce>s">body{font:16px/1.5 system-ui,sans-serif;margin:0;display:flex;min-height:100vh;align-items:center;justify-content:center;color:#222;background:#fafafa}main{max-width:28rem;padding:2rem;text-align:center}</style>
135
+ </head><body><main><p id="m">Checking your browser before you continue&hellip;</p>
136
+ <noscript><p>Please turn on JavaScript to continue to this site.</p></noscript></main>
137
+ <script nonce="%<nonce>s">%<script>s</script>
138
+ </body></html>
139
+ HTML
140
+
141
+ # The page's script: sets the pass and reloads, but only when reading the
142
+ # cookie back shows it was set (with cookies off it says so instead of
143
+ # reloading forever), and at most three times a minute, counted in
144
+ # sessionStorage (an address that changes on every request would otherwise
145
+ # never get a pass that holds and reload for ever), or in a __sg_tries
146
+ # cookie lasting a minute where storage is blocked or full. Secure only
147
+ # when the browser is on https: it knows, where the server behind a proxy
148
+ # can only be told. The same script as the Node, PHP and Python clients'.
149
+ def script(value)
150
+ "(function(){var n=#{COOKIE.to_json},v=#{value.to_json},m=document.getElementById(\"m\")," \
151
+ "s=location.protocol===\"https:\";" \
152
+ "document.cookie=n+\"=\"+v+\"; Max-Age=#{MAX_AGE}; Path=/; SameSite=Lax\"+(s?\"; Secure\":\"\");" \
153
+ "if((\" \"+document.cookie+\";\").indexOf(\" \"+n+\"=\"+v+\";\")<0){m.textContent=\"Please allow cookies for this site, then reload the page.\";return;}" \
154
+ "var k=\"__sg_tries\",r=0,t;" \
155
+ "try{t=JSON.parse(sessionStorage.getItem(k)||\"[]\").filter(function(x){return Date.now()-x<60000;});r=t.length;if(r<3){t.push(Date.now());sessionStorage.setItem(k,JSON.stringify(t));}}" \
156
+ "catch(e){t=/(?:^|; )__sg_tries=([0-9]+)/.exec(document.cookie);r=t?+t[1]:0;if(r<3)document.cookie=k+\"=\"+(r+1)+\"; Max-Age=60; Path=/; SameSite=Lax\"+(s?\"; Secure\":\"\");}" \
157
+ "if(r>=3){m.textContent=\"This page couldn't be opened because your connection's address keeps changing. Please try again in a minute.\";return;}" \
158
+ "location.reload();})();"
159
+ end
160
+
161
+ # [503, headers, body] for the check page: its own Content-Security-Policy
162
+ # with a fresh nonce for its one script, and no body for a HEAD.
163
+ def page(key, ip, method, now = nil)
164
+ nonce = SecureRandom.urlsafe_base64(18)
165
+ headers = {
166
+ "content-type" => "text/html; charset=utf-8",
167
+ "cache-control" => "no-store",
168
+ "x-robots-tag" => "noindex",
169
+ "content-security-policy" => "default-src 'none'; script-src 'nonce-#{nonce}'; style-src 'nonce-#{nonce}'; " \
170
+ "base-uri 'none'; form-action 'none'; frame-ancestors 'none'"
171
+ }
172
+ return [503, headers, ""] if method.to_s.upcase == "HEAD"
173
+
174
+ [503, headers, format(PAGE, nonce: nonce, script: script(make_pass(key, ip, now)))]
175
+ end
176
+ end
177
+ end