sitegoose 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE +21 -0
- data/README.md +342 -0
- data/lib/sitegoose/analytics.rb +113 -0
- data/lib/sitegoose/challenge.rb +177 -0
- data/lib/sitegoose/core.rb +523 -0
- data/lib/sitegoose/crawlers.rb +195 -0
- data/lib/sitegoose/errors.rb +456 -0
- data/lib/sitegoose/firewall.rb +417 -0
- data/lib/sitegoose/inventory.rb +131 -0
- data/lib/sitegoose/ip.rb +159 -0
- data/lib/sitegoose/middleware.rb +52 -0
- data/lib/sitegoose/performance.rb +114 -0
- data/lib/sitegoose/platform.rb +557 -0
- data/lib/sitegoose/posture.rb +272 -0
- data/lib/sitegoose/proxy.rb +260 -0
- data/lib/sitegoose/queries.rb +399 -0
- data/lib/sitegoose/query_text.rb +215 -0
- data/lib/sitegoose/railtie.rb +124 -0
- data/lib/sitegoose/request_rules.rb +390 -0
- data/lib/sitegoose/resources.rb +449 -0
- data/lib/sitegoose/sidekiq.rb +95 -0
- data/lib/sitegoose/small_modules.rb +250 -0
- data/lib/sitegoose/version.rb +5 -0
- data/lib/sitegoose.rb +261 -0
- metadata +92 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: f88d90ea071f9dde430e079a88e42a22dabd4f0faeaa4a662da5c94eb8cd942c
|
|
4
|
+
data.tar.gz: 1033458a91be1a78a2464a9439d2d021eaedce386b0cfb7eb7662db4927a3e50
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: e20cd12422c5830f3a710340198b634c2c192e9d26a830879fa756ebd1901a95c28054ab5d74d73812a80b7243653078592ddd59d9df02048760039b6e9f543b
|
|
7
|
+
data.tar.gz: 5190a6b466bdf730a41265537aa1fdf01090aaeb77966d0bf720532ec12db816ebcb15b9ee96561e0a1f45c4d5f9d83256a1adf049f05873f94e57118896c635
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 steviec87
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,342 @@
|
|
|
1
|
+
# sitegoose (Ruby)
|
|
2
|
+
|
|
3
|
+
One package for your Ruby server, run from your analytics dashboard: a
|
|
4
|
+
**firewall**, **error monitoring** and **server analytics** (payments,
|
|
5
|
+
sign-ups, leads). Rails, Sinatra and any Rack app, with Sidekiq jobs. Only
|
|
6
|
+
the standard library and `rack`; Ruby 3.0 or later.
|
|
7
|
+
|
|
8
|
+
```ruby
|
|
9
|
+
# Gemfile
|
|
10
|
+
gem "sitegoose"
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Rails
|
|
14
|
+
|
|
15
|
+
Nothing to add to the middleware stack: the gem's Railtie puts it first.
|
|
16
|
+
|
|
17
|
+
```ruby
|
|
18
|
+
# config/initializers/sitegoose.rb
|
|
19
|
+
SiteGoose.configure(
|
|
20
|
+
site_id: "YOUR_SITE_ID",
|
|
21
|
+
secret: ENV["ANALYTICS_SITE_SECRET"], # site settings; keep it on the server
|
|
22
|
+
endpoint: "https://YOUR_COLLECTOR_URL",
|
|
23
|
+
# trust_proxy: 1, # behind Nginx or a load balancer
|
|
24
|
+
# trust_proxy: "cloudflare", # every request through Cloudflare
|
|
25
|
+
)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Or keep it in the credentials (`bin/rails credentials:edit`), under
|
|
29
|
+
`sitegoose:` with the same keys, or in `config.sitegoose`.
|
|
30
|
+
The environment works too: `ANALYTICS_SITE_ID`, `ANALYTICS_SITE_SECRET`,
|
|
31
|
+
`ANALYTICS_ENDPOINT`, `ANALYTICS_SERVER_NAME`, `ANALYTICS_TRUST_PROXY`.
|
|
32
|
+
Lowest first: the environment, credentials, `config.sitegoose`,
|
|
33
|
+
`configure`.
|
|
34
|
+
|
|
35
|
+
Then, on their own:
|
|
36
|
+
|
|
37
|
+
- Request errors, with the route pattern (`/orders/:id`), including the ones
|
|
38
|
+
Rails turns into its 500 page.
|
|
39
|
+
- `Rails.error.handle`, `.record` and `.report`: the errors your code handles.
|
|
40
|
+
- Devise (Warden) failed logins count towards bot protection, and logins and
|
|
41
|
+
logouts go to the audit log.
|
|
42
|
+
- The controller action, mail and queued job before an error are its
|
|
43
|
+
breadcrumbs.
|
|
44
|
+
|
|
45
|
+
In your code: `SiteGoose.track(...)`, or `SiteGoose.platform`.
|
|
46
|
+
|
|
47
|
+
## Sinatra and other Rack apps
|
|
48
|
+
|
|
49
|
+
```ruby
|
|
50
|
+
use SiteGoose::Middleware,
|
|
51
|
+
site_id: "YOUR_SITE_ID", secret: ENV["ANALYTICS_SITE_SECRET"], endpoint: "https://YOUR_COLLECTOR_URL"
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Put it first; added twice (by hand next to the Railtie, or by an engine),
|
|
55
|
+
the second copy passes requests straight on. Sinatra's route patterns (`/users/:id`) and its error pages are
|
|
56
|
+
read on their own; set `set :raise_errors, false` (production's default) or
|
|
57
|
+
`set :show_exceptions, false` as usual. Hanami, Roda and plain Rack work the
|
|
58
|
+
same way. A full example is in `examples/sinatra/`.
|
|
59
|
+
|
|
60
|
+
## Sidekiq
|
|
61
|
+
|
|
62
|
+
```ruby
|
|
63
|
+
# config/initializers/sidekiq.rb
|
|
64
|
+
require "sitegoose/sidekiq"
|
|
65
|
+
SiteGoose::Sidekiq.install
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Every job then runs as a [job](#error-monitoring) named after its class (the
|
|
69
|
+
ActiveJob class under ActiveJob): its errors carry the route
|
|
70
|
+
`/(job) ChargeJob`, and each run is timed (Server speed → Jobs).
|
|
71
|
+
|
|
72
|
+
- A failure that will be retried is timed as failed but not sent: the error
|
|
73
|
+
is sent when the last attempt fails, or when Sidekiq gives the job up (the
|
|
74
|
+
Dead set, `:kill`).
|
|
75
|
+
- `Sidekiq::Shutdown` (a job stopped mid-run, to run again) is neither timed
|
|
76
|
+
nor sent.
|
|
77
|
+
- What's waiting is sent when Sidekiq shuts down.
|
|
78
|
+
- Don't also wrap a job in `platform.job`: it would be timed twice.
|
|
79
|
+
|
|
80
|
+
## Puma, Unicorn and Passenger
|
|
81
|
+
|
|
82
|
+
The platform is made on the first request, so a master that only forks
|
|
83
|
+
workers starts nothing. Its background threads start again in each worker
|
|
84
|
+
after a fork (Puma clustered mode, `preload_app!`, Unicorn, Passenger), a
|
|
85
|
+
fork waits for a send in flight, and what the parent had waiting is the
|
|
86
|
+
parent's to send. Nothing to configure. On Ruby 3.0, which has no fork hook,
|
|
87
|
+
a worker notices it was forked at its first request or job instead, and a
|
|
88
|
+
fork doesn't wait for a send in flight.
|
|
89
|
+
|
|
90
|
+
## Modules
|
|
91
|
+
|
|
92
|
+
| Module | What it does | Default |
|
|
93
|
+
|---|---|---|
|
|
94
|
+
| `firewall` | Blocks addresses, known criminal networks, shared blocks and floods (Security → Firewall) | on |
|
|
95
|
+
| `firewall`'s attack patterns | Rules that match the *request* (its path, query string and user agent), not just the address, so an attack is turned away before your code runs. They come with the firewall's rules and are managed on the Security page (Firewall → Attack patterns). Nothing to configure (protocol: `sdk/protocols/firewall.md`). Patterns need Ruby 3.2+, which can stop a match that runs long; on 3.0 and 3.1 a rule's pattern is skipped (its other parts still match) and a warning says so once | with the firewall |
|
|
96
|
+
| `proxy` | Serves the tracker and takes visits under your own domain, so ad blockers can't hide those visitors | on |
|
|
97
|
+
| `errors` | Sends your server's errors to the Errors report | on |
|
|
98
|
+
| `analytics` | `track`, `identify` and `lead` from your server | on |
|
|
99
|
+
| `audit` | `platform.audit("user.role_changed", actor: ..., target: ..., request: ...)`; Devise logins and logouts on their own | on |
|
|
100
|
+
| `flags` | `platform.enabled?("new-checkout", user: user.id)`: features on or off from the dashboard | on |
|
|
101
|
+
| `maintenance` | A "back soon" page while maintenance mode is on in the dashboard | on |
|
|
102
|
+
| `headers` | Adds the security headers you pick on the Security page; your own headers win | on |
|
|
103
|
+
| `forms` | Checks forms before you accept them: `platform.check_form(request)` | on |
|
|
104
|
+
| `performance` | Response times and errors of every route (Health → Server speed): Rails and Sinatra route patterns. With it, the database queries each route runs (below) | on |
|
|
105
|
+
| `crawlers` | Which search engines and AI bots fetch your pages (Health → Crawlers), counted by the middleware | on |
|
|
106
|
+
| `resources` | This machine's CPU, load, memory, disks and network, once a minute (Health → Server resources). Linux | on |
|
|
107
|
+
| `inventory` | The installed gems and Ruby version, once a day, so the Security check can flag known security holes (`{host: ...}`: the server's name) | on |
|
|
108
|
+
| `posture` | How the server is set up, once a day, for the Security check: Rails' environment, `consider_all_requests_local`, `secret_key_base`, `config.hosts`, `force_ssl` and session cookie settings (when it's a Rails app), whether the app runs as root, code anyone on the machine can write, and files in the folders you serve to the web that shouldn't be public (a dump, a `.env`, a `.bak`). Reads only, and sends no file's contents. `{roots: ["public", {dir: "uploads", prefix: "files"}]}` (default: Rails' `public/`), `{checks: -> { [...] }}` for checks of your own (protocol: `sdk/protocols/posture.md`) | on |
|
|
109
|
+
|
|
110
|
+
Each takes `true`, `false` or its options as a hash:
|
|
111
|
+
|
|
112
|
+
```ruby
|
|
113
|
+
SiteGoose.configure(site_id: ..., secret: ..., endpoint: ...,
|
|
114
|
+
firewall: { mode: "monitor", cache_file: "tmp/firewall-rules.json" },
|
|
115
|
+
errors: { capture_uncaught: false },
|
|
116
|
+
analytics: false)
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
`enabled: false` turns the whole package off (every request passes).
|
|
120
|
+
|
|
121
|
+
**Switch modules off in the dashboard** (Security → Server package) without a
|
|
122
|
+
deploy: the package checks about once a minute. Modules are independent: one
|
|
123
|
+
failing to start, or the collector being down, never stops the others, and
|
|
124
|
+
never slows your requests; a failure of the package itself passes the request
|
|
125
|
+
to your app.
|
|
126
|
+
|
|
127
|
+
## Database queries
|
|
128
|
+
|
|
129
|
+
Server speed → Queries: per route and per query, how many ran, how many
|
|
130
|
+
failed, and how long they took, so a slow route shows which query made it
|
|
131
|
+
slow. Every value is replaced by `?` before anything is counted
|
|
132
|
+
(`WHERE email = ?`), and bound parameters are never read
|
|
133
|
+
(protocol: `sdk/protocols/queries.md`).
|
|
134
|
+
|
|
135
|
+
- **Active Record** (Rails, or on its own): automatic, from its
|
|
136
|
+
`sql.active_record` notification. Schema and cached queries are left out.
|
|
137
|
+
- **Sequel**: one line, after `Sequel.connect`:
|
|
138
|
+
|
|
139
|
+
```ruby
|
|
140
|
+
DB = Sequel.connect(ENV["DATABASE_URL"])
|
|
141
|
+
SiteGoose.watch_database(DB)
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
- **MongoDB** (the `mongo` gem, Mongoid): automatic, through the driver's
|
|
145
|
+
command monitoring.
|
|
146
|
+
|
|
147
|
+
A query counts under the request (or Sidekiq or `platform.job` run) it ran
|
|
148
|
+
in, with the same method and route Server speed shows; one outside them,
|
|
149
|
+
like a console or a thread of your own, counts as `JOB (background)`. A
|
|
150
|
+
query is counted once: one run by Sequel on Active Record's connection is
|
|
151
|
+
Sequel's. It rides on `performance`: switched off there, nothing is
|
|
152
|
+
recorded; `queries: false` keeps Server speed without the queries:
|
|
153
|
+
|
|
154
|
+
```ruby
|
|
155
|
+
SiteGoose.configure(site_id: ..., secret: ..., endpoint: ..., queries: false)
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
## Firewall
|
|
159
|
+
|
|
160
|
+
Rules, monitor or block mode and reports are in the dashboard (Security →
|
|
161
|
+
Firewall). New sites start in monitor mode.
|
|
162
|
+
|
|
163
|
+
Options: `mode` (`"monitor"` or `"block"`, overrides the dashboard),
|
|
164
|
+
`cache_file` (keep rules between restarts), `trust_proxy`, `counter`.
|
|
165
|
+
|
|
166
|
+
**The visitor's address:** behind a proxy, every request comes from the proxy
|
|
167
|
+
unless you set `trust_proxy`: `1` for one proxy or load balancer, `2` for
|
|
168
|
+
Cloudflare in front of a load balancer, `"cloudflare"` when every request
|
|
169
|
+
comes through Cloudflare (`CF-Connecting-IP` is believed only from
|
|
170
|
+
Cloudflare's own addresses: anyone who reaches your server directly can send
|
|
171
|
+
it). The package reads the connection's address itself, not Rails'
|
|
172
|
+
`request.remote_ip`, so set it here even if Rails already trusts the proxy.
|
|
173
|
+
|
|
174
|
+
**Who is counted:** an IPv4 address, or an IPv6 address's /64 (one machine
|
|
175
|
+
usually holds the whole /64); `::ffff:a.b.c.d` is the IPv4 address. Files a
|
|
176
|
+
page load asks for by the dozen (`/assets/`, `/packs/`, `.css`, `.js`, images,
|
|
177
|
+
fonts) don't count against the rate limit. A forwarded address that isn't one
|
|
178
|
+
(`unknown`, a name) means the connection's address is checked instead, and a
|
|
179
|
+
port on it (`1.2.3.4:5678`, `[2001:db8::1]:443`) is dropped.
|
|
180
|
+
|
|
181
|
+
**Rate limit with several workers:** each worker counts on its own, so the
|
|
182
|
+
limit is per worker. For one count, pass a shared counter:
|
|
183
|
+
|
|
184
|
+
```ruby
|
|
185
|
+
counter = lambda do |key, ttl|
|
|
186
|
+
hits = REDIS.incr(key)
|
|
187
|
+
REDIS.expire(key, ttl) if hits == 1
|
|
188
|
+
hits
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
SiteGoose.configure(..., firewall: { counter: counter })
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
**Bot protection:** Devise's failed logins are counted on their own;
|
|
195
|
+
elsewhere call `platform.login_failed(request)` when a login fails. Past the
|
|
196
|
+
site's limit the address is banned here straight away and on your other
|
|
197
|
+
servers within a minute or two. Floods (over the rate limit) are banned the
|
|
198
|
+
same way.
|
|
199
|
+
|
|
200
|
+
**Probes.** Requests for files no visitor ever asks for (`/.env`, `/.git/…`,
|
|
201
|
+
backups: the list comes with the rules) are rule `probe`; in block mode the
|
|
202
|
+
address is also banned on this server, and the collector bans it on the
|
|
203
|
+
site's other servers.
|
|
204
|
+
|
|
205
|
+
**Browser check** (paid plans, switched on in the dashboard). A visitor's
|
|
206
|
+
first page of the day is a short "checking your browser" page that sets a
|
|
207
|
+
cookie with JavaScript and reloads; after that the site answers as usual for
|
|
208
|
+
about a day. Only a `GET` or `HEAD` that asks for a web page is checked:
|
|
209
|
+
forms, webhooks, payment callbacks, API calls (`Accept: application/json`),
|
|
210
|
+
the paths you exclude, your allowed addresses, search engines and the AI
|
|
211
|
+
crawlers that bring visits never see it. It stops cheap floods (scripts that
|
|
212
|
+
don't run JavaScript) and nothing more: the pass is bound to the visitor's
|
|
213
|
+
address and the day. The cookie, `__sg_pass`, holds a signed day and nothing
|
|
214
|
+
about the visitor; it is strictly necessary and needs no consent. Checked
|
|
215
|
+
requests show on the Security page as rule `challenge`.
|
|
216
|
+
|
|
217
|
+
## Error monitoring
|
|
218
|
+
|
|
219
|
+
Request errors are sent by the middleware, and the exception that ends the
|
|
220
|
+
program (a crash in a script or rake task) at exit. Errors you catch:
|
|
221
|
+
|
|
222
|
+
```ruby
|
|
223
|
+
begin
|
|
224
|
+
charge_card(order)
|
|
225
|
+
rescue PaymentError => e
|
|
226
|
+
SiteGoose.capture_exception(e, status: 502)
|
|
227
|
+
raise
|
|
228
|
+
end
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
In Rails, `Rails.error.handle { ... }` and `Rails.error.report(e)` do the
|
|
232
|
+
same. Exceptions your code only logs: `errors: { capture_logging: true }`
|
|
233
|
+
sends an exception logged at ERROR or above (`logger.error(e)`); it is still
|
|
234
|
+
logged. Rails' logger is hooked on its own; elsewhere
|
|
235
|
+
`platform.capture_logger(logger)`. An exception is sent once however many of
|
|
236
|
+
these see it, and one sent inside a request keeps that request's route. A
|
|
237
|
+
batch the collector couldn't take is tried once more a few seconds later.
|
|
238
|
+
|
|
239
|
+
What still can't be sent waits in a file in the temp folder and is sent when
|
|
240
|
+
it can be, after a restart too; each error keeps the time it happened.
|
|
241
|
+
|
|
242
|
+
An error in a request or job also shows the steps before it. Add them with
|
|
243
|
+
`SiteGoose.add_breadcrumb("Order saved")`; a step added outside a
|
|
244
|
+
request or job is dropped. Outgoing HTTP calls aren't recorded on their own
|
|
245
|
+
(Ruby has no public hook for them); in Rails the controller, mailer and job
|
|
246
|
+
notifications are.
|
|
247
|
+
|
|
248
|
+
Background jobs other than Sidekiq's:
|
|
249
|
+
|
|
250
|
+
```ruby
|
|
251
|
+
SiteGoose.job("send-receipts") { send_receipts }
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Their errors carry the job's name as the route, a job that raises is sent
|
|
255
|
+
(status 500) and raised on, and each run is timed (Server speed → Jobs).
|
|
256
|
+
|
|
257
|
+
**Several apps on one site** (a Rails app, a Sidekiq process, a Node service
|
|
258
|
+
next to it): install the package in each, with the same `site_id`, `secret`
|
|
259
|
+
and `endpoint`, and a `server_name` of its own. Errors, Server speed,
|
|
260
|
+
Crawlers and Security → Updates then tell the apps apart, and Health → All
|
|
261
|
+
systems lists each with a light: the package checks in about once a minute
|
|
262
|
+
(`sdk/protocols/checkin.md`). A service only your other apps call takes
|
|
263
|
+
`firewall: false, proxy: false, maintenance: false`; on a machine shared by
|
|
264
|
+
several apps, keep `resources` on in only one of them.
|
|
265
|
+
|
|
266
|
+
Options: `server_name`, `root` (backtrace paths relative to it; default
|
|
267
|
+
`Rails.root` or the working directory), `release` (default
|
|
268
|
+
`ANALYTICS_RELEASE`), `capture_uncaught`, `capture_logging`, `spool_dir`
|
|
269
|
+
(where unsent errors wait; `false` for none), `before_send` (a lambda taking
|
|
270
|
+
the event and the error: return the event, changed or not, as a Hash, or
|
|
271
|
+
`nil` to drop it; anything else, or a raise, drops it with a warning).
|
|
272
|
+
Messages are cut at 2,000 characters, and the file of unsent errors stops at
|
|
273
|
+
1 MB.
|
|
274
|
+
|
|
275
|
+
## Server analytics
|
|
276
|
+
|
|
277
|
+
For what only your server knows for sure. Not pageviews: the tracker counts those.
|
|
278
|
+
|
|
279
|
+
```ruby
|
|
280
|
+
platform = SiteGoose.platform
|
|
281
|
+
session = platform.session_id(request) # the visit, from the X-Analytics-Session header
|
|
282
|
+
|
|
283
|
+
platform.track("purchase", { "numeric_value" => "49.90", "currency" => "EUR", "event_uid" => "order-#{order.id}" }, session_id: session)
|
|
284
|
+
platform.identify(user.id, session_id: session)
|
|
285
|
+
platform.lead({ "email" => "ana@example.com", "name" => "Ana", "status" => "qualified" })
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
`track` sends in the background; `identify` and `lead` wait for the answer
|
|
289
|
+
(up to 5 seconds).
|
|
290
|
+
|
|
291
|
+
**Link to the visit:** send the tracker's session ID with your site's requests:
|
|
292
|
+
|
|
293
|
+
```js
|
|
294
|
+
fetch("/api/orders", { headers: { "x-analytics-session": window.analyticsSession?.getId() || "" }, ... });
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
## Ad blockers
|
|
298
|
+
|
|
299
|
+
This package already runs on your own domain, so it serves the tracker and
|
|
300
|
+
takes the visits itself. Use the two addresses it gives you in your layout:
|
|
301
|
+
|
|
302
|
+
```erb
|
|
303
|
+
<% proxy = SiteGoose.platform&.proxy %>
|
|
304
|
+
<script async src="<%= proxy&.script_url %>" data-siteid="..." data-domain="example.com"
|
|
305
|
+
data-collect="<%= proxy&.collect_url %>"></script>
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
- The path comes from your secret, so it isn't in the page and isn't a name
|
|
309
|
+
any blocker lists. Pick your own with `proxy: { prefix: "..." }`.
|
|
310
|
+
- Tracker files are fetched once and then held in memory; if we can't be
|
|
311
|
+
reached, the copy already held is served rather than nothing, and a file
|
|
312
|
+
that couldn't be fetched isn't asked for again for 30 seconds.
|
|
313
|
+
- These run on your app's request threads, so they wait at most 1 second to
|
|
314
|
+
connect and 2 to answer, and after three failures in a row the collector
|
|
315
|
+
is left alone for 30 seconds (beacons get a 502 straight away; the
|
|
316
|
+
tracker counts them as lost). A collector that stops answering can't use
|
|
317
|
+
up Puma's threads.
|
|
318
|
+
- The visitor's address goes on as `X-Forwarded-For` and cookies never leave
|
|
319
|
+
your server.
|
|
320
|
+
- Beacons are answered before the rate limit, the browser check, the
|
|
321
|
+
crawler count and the timing, so your own analytics traffic doesn't
|
|
322
|
+
appear as pages of your site. A denied or banned address gets a 403 here
|
|
323
|
+
too.
|
|
324
|
+
- Switch it off with `proxy: false`, or in the dashboard (Site settings →
|
|
325
|
+
Server). Then load the tracker from us as before.
|
|
326
|
+
|
|
327
|
+
## Shutting down
|
|
328
|
+
|
|
329
|
+
What's waiting is sent at exit on its own, within 5 seconds in all (a
|
|
330
|
+
container's or PM2's grace period is longer): no send starts after that,
|
|
331
|
+
none is retried, and errors that weren't sent wait in their file for the
|
|
332
|
+
next start. Sidekiq's shutdown flush has the same deadline.
|
|
333
|
+
`SiteGoose.flush` sends it
|
|
334
|
+
now (a short-lived script, a serverless function before returning);
|
|
335
|
+
`platform.close` also stops the background threads.
|
|
336
|
+
|
|
337
|
+
## Tests
|
|
338
|
+
|
|
339
|
+
```bash
|
|
340
|
+
bundle install
|
|
341
|
+
ruby -Ilib -Itest -e 'Dir["test/*_test.rb"].each { |f| require_relative f }'
|
|
342
|
+
```
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Analytics module: what only the server sees for sure (payments, sign-ups,
|
|
4
|
+
# plan changes), next to the browser tracker's visits. Not pageviews: the
|
|
5
|
+
# tracker counts those, and counting them here too would count them twice.
|
|
6
|
+
#
|
|
7
|
+
# - track(): events, batched to POST /events/server-events in the background.
|
|
8
|
+
# Each gets an event_uid, so a retried batch can be told apart from new events.
|
|
9
|
+
# - identify(): links the browser's visit to your user ID (/events/identify).
|
|
10
|
+
# - lead(): adds or updates a lead (/events/lead).
|
|
11
|
+
# identify() and lead() wait for the answer (up to 5 seconds).
|
|
12
|
+
|
|
13
|
+
require "securerandom"
|
|
14
|
+
|
|
15
|
+
module SiteGoose
|
|
16
|
+
class Analytics
|
|
17
|
+
BATCH = 100
|
|
18
|
+
QUEUE_MAX = 1000
|
|
19
|
+
FLUSH_EVERY = 2
|
|
20
|
+
RETRY_EVERY = 15
|
|
21
|
+
|
|
22
|
+
attr_reader :queue
|
|
23
|
+
|
|
24
|
+
def initialize(core, **_)
|
|
25
|
+
@core = core
|
|
26
|
+
@queue = []
|
|
27
|
+
@lock = Mutex.new
|
|
28
|
+
@job = core.valid? ? Periodic.new(FLUSH_EVERY) { flush ? FLUSH_EVERY : RETRY_EVERY } : nil
|
|
29
|
+
Forking.register(self)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def after_fork_in_child
|
|
33
|
+
@lock = Mutex.new
|
|
34
|
+
@queue = []
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# name: the event's name; fields: value, numeric_value, currency, url,
|
|
38
|
+
# event_label, variant, id, timestamp, event_uid, transaction_id, coupon
|
|
39
|
+
# (expresscollect/docs/server-events.md). Returns the event_uid, or ""
|
|
40
|
+
# when not queued.
|
|
41
|
+
def track(name, fields = nil, session_id = "")
|
|
42
|
+
return "" unless name.is_a?(String) && !name.empty? && @core.on?("analytics")
|
|
43
|
+
|
|
44
|
+
fields = (fields || {}).each_with_object({}) { |(k, v), out| out[k.to_s] = v }
|
|
45
|
+
event = { "timestamp" => SiteGoose.iso_now, "session_id" => session_id.to_s }
|
|
46
|
+
event.merge!(fields)
|
|
47
|
+
event["event_name"] = name[0, 100]
|
|
48
|
+
event["event_uid"] = fields["event_uid"].to_s.empty? ? SecureRandom.uuid : fields["event_uid"].to_s
|
|
49
|
+
event["numeric_value"] = event["numeric_value"].to_s if event["numeric_value"].is_a?(Numeric)
|
|
50
|
+
full = @lock.synchronize do
|
|
51
|
+
if @queue.size >= QUEUE_MAX
|
|
52
|
+
@core.log("analytics", "queue full, event dropped")
|
|
53
|
+
return ""
|
|
54
|
+
end
|
|
55
|
+
@queue << event
|
|
56
|
+
@queue.size >= BATCH
|
|
57
|
+
end
|
|
58
|
+
@job&.run_now if full
|
|
59
|
+
event["event_uid"]
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# Sends what's waiting; false when some of it must be tried again.
|
|
63
|
+
def flush
|
|
64
|
+
waiting = @lock.synchronize do
|
|
65
|
+
taken = @queue
|
|
66
|
+
@queue = []
|
|
67
|
+
taken
|
|
68
|
+
end
|
|
69
|
+
ok = true
|
|
70
|
+
waiting.each_slice(BATCH) do |batch|
|
|
71
|
+
res = @core.request("analytics", "/events/server-events", body: { "events" => batch })
|
|
72
|
+
if res.nil? || res.status == 429 || res.status >= 500
|
|
73
|
+
ok = false
|
|
74
|
+
@lock.synchronize { @queue = (batch + @queue).first(QUEUE_MAX) }
|
|
75
|
+
elsif !res.ok?
|
|
76
|
+
@core.log("analytics", "events refused:", res.body.to_s[0, 300])
|
|
77
|
+
end
|
|
78
|
+
end
|
|
79
|
+
ok
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Links the visit to your user ID. true when stored.
|
|
83
|
+
def identify(user_id, session_id = "")
|
|
84
|
+
return false if user_id.to_s.empty? || session_id.to_s.empty? || !@core.on?("analytics")
|
|
85
|
+
|
|
86
|
+
res = @core.request("analytics", "/events/identify", body: { "user_id" => user_id.to_s, "session_id" => session_id.to_s })
|
|
87
|
+
res ? res.ok? : false
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
# fields: email or phone, and name, status, tags, note, deal fields...
|
|
91
|
+
# Returns {"status", "lead_id"}, or nil.
|
|
92
|
+
def lead(fields, session_id = "")
|
|
93
|
+
return nil unless @core.on?("analytics")
|
|
94
|
+
|
|
95
|
+
body = { "session_id" => session_id.to_s }
|
|
96
|
+
(fields || {}).each { |k, v| body[k.to_s] = v }
|
|
97
|
+
res = @core.request("analytics", "/events/lead", body: body)
|
|
98
|
+
return nil if res.nil?
|
|
99
|
+
|
|
100
|
+
unless res.ok?
|
|
101
|
+
@core.log("analytics", "lead refused:", res.body.to_s[0, 300])
|
|
102
|
+
return nil
|
|
103
|
+
end
|
|
104
|
+
res.json
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def close
|
|
108
|
+
@job&.stop
|
|
109
|
+
Forking.unregister(self)
|
|
110
|
+
flush
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
end
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The browser check (rule `challenge`, sdk/protocols/firewall.md under "Flood
|
|
4
|
+
# protection"): a GET or HEAD that asks for a page, from an address with no
|
|
5
|
+
# valid pass, gets a small page instead of the site. Its script sets the pass
|
|
6
|
+
# cookie and reloads, which a script that doesn't run JavaScript never does.
|
|
7
|
+
#
|
|
8
|
+
# The pass is <day>.<first 32 hex digits of HMAC-SHA256(key, "<day>|<who>")>,
|
|
9
|
+
# key = HMAC-SHA256 of "sg-challenge-v1" keyed with the site secret, so any of
|
|
10
|
+
# the site's servers accepts a pass another one handed out. It counts for its
|
|
11
|
+
# own day and the next. sdk/protocols/challenge-cases.json holds the cases
|
|
12
|
+
# every client runs; backend/scripts/challenge-cases.mjs is the reference.
|
|
13
|
+
|
|
14
|
+
require "openssl"
|
|
15
|
+
require "securerandom"
|
|
16
|
+
require "ipaddr"
|
|
17
|
+
require "json"
|
|
18
|
+
|
|
19
|
+
module SiteGoose
|
|
20
|
+
module Challenge
|
|
21
|
+
COOKIE = "__sg_pass"
|
|
22
|
+
MAX_AGE = 86_400
|
|
23
|
+
LABEL = "sg-challenge-v1"
|
|
24
|
+
PASS = /\A(\d{1,10})\.([0-9a-f]{32})\z/.freeze
|
|
25
|
+
|
|
26
|
+
module_function
|
|
27
|
+
|
|
28
|
+
# The key every pass is signed with, from the site secret.
|
|
29
|
+
def pass_key(secret)
|
|
30
|
+
OpenSSL::HMAC.digest("SHA256", secret.to_s, LABEL)
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# What a pass is bound to: the IPv4 address (an IPv4-mapped IPv6 address
|
|
34
|
+
# as IPv4), or an IPv6 address's /64 as four groups of four lower-case hex
|
|
35
|
+
# digits. "*" for the agent's nginx (a package never accepts it); nil for
|
|
36
|
+
# anything that isn't an address.
|
|
37
|
+
def who(ip)
|
|
38
|
+
text = ip.to_s.strip
|
|
39
|
+
return "*" if text == "*"
|
|
40
|
+
|
|
41
|
+
key = IP.key(text)
|
|
42
|
+
return nil if key.nil?
|
|
43
|
+
|
|
44
|
+
if key >> 32 == 0xFFFF
|
|
45
|
+
v4 = key & 0xFFFFFFFF
|
|
46
|
+
return [24, 16, 8, 0].map { |shift| (v4 >> shift) & 0xFF }.join(".")
|
|
47
|
+
end
|
|
48
|
+
format("%032x", key).scan(/.{4}/).first(4).join(":")
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# Whole days since 1970-01-01 UTC.
|
|
52
|
+
def day_of(now = nil)
|
|
53
|
+
((now || Time.now.to_f).to_f / 86_400).floor
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def sign(key, day, bound)
|
|
57
|
+
OpenSSL::HMAC.hexdigest("SHA256", key, "#{day}|#{bound}")[0, 32]
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
# Today's pass for this address, or "" when it isn't one.
|
|
61
|
+
def make_pass(key, ip, now = nil)
|
|
62
|
+
bound = who(ip)
|
|
63
|
+
return "" if bound.nil?
|
|
64
|
+
|
|
65
|
+
day = day_of(now)
|
|
66
|
+
"#{day}.#{sign(key, day, bound)}"
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def secure_compare(a, b)
|
|
70
|
+
return false unless a.bytesize == b.bytesize
|
|
71
|
+
|
|
72
|
+
if OpenSSL.respond_to?(:fixed_length_secure_compare)
|
|
73
|
+
OpenSSL.fixed_length_secure_compare(a, b)
|
|
74
|
+
else
|
|
75
|
+
diff = 0
|
|
76
|
+
a.bytes.zip(b.bytes) { |x, y| diff |= x ^ y }
|
|
77
|
+
diff.zero?
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
# True when value is this address's pass for today or yesterday.
|
|
82
|
+
def valid?(key, ip, value, now = nil)
|
|
83
|
+
match = PASS.match(value.to_s)
|
|
84
|
+
bound = who(ip)
|
|
85
|
+
return false if match.nil? || bound.nil? || bound == "*"
|
|
86
|
+
|
|
87
|
+
day = match[1].to_i
|
|
88
|
+
today = day_of(now)
|
|
89
|
+
return false unless day == today || day == today - 1
|
|
90
|
+
|
|
91
|
+
secure_compare(match[2], sign(key, day, bound))
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
# Every value the Cookie header carries under name (a browser can send
|
|
95
|
+
# two, from different paths); at most four, like the Node client. Two
|
|
96
|
+
# Cookie headers arrive joined by ", " (Puma), so that separates cookies
|
|
97
|
+
# too: our values ("<day>.<hex>", the preview code) never hold one, and a
|
|
98
|
+
# piece of someone else's value split off this way names no cookie of ours.
|
|
99
|
+
def cookie_values(cookie_header, name = COOKIE)
|
|
100
|
+
found = []
|
|
101
|
+
cookie_header.to_s.split(/;|, /).each do |part|
|
|
102
|
+
k, sep, value = part.strip.partition("=")
|
|
103
|
+
next if sep.empty? || k.strip != name
|
|
104
|
+
|
|
105
|
+
found << value.strip.delete_prefix('"').delete_suffix('"')
|
|
106
|
+
break if found.size == 4
|
|
107
|
+
end
|
|
108
|
+
found
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
def pass?(key, ip, cookie_header, now = nil)
|
|
112
|
+
cookie_values(cookie_header).any? { |value| valid?(key, ip, value, now) }
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# The rules' skip prefixes, lower-cased.
|
|
116
|
+
def compile_skip(entries)
|
|
117
|
+
Array(entries).select { |p| p.is_a?(String) && !p.empty? }.map(&:downcase).first(200)
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# Conditions 1 to 3: a GET or HEAD, whose Accept asks for text/html, for a
|
|
121
|
+
# path (lower-cased, no query string) under none of skip.
|
|
122
|
+
def asks_for_page?(method, target, accept, skip)
|
|
123
|
+
return false unless %w[GET HEAD].include?(method.to_s.upcase)
|
|
124
|
+
return false unless accept.to_s.downcase.include?("text/html")
|
|
125
|
+
|
|
126
|
+
path = target.to_s.split("?", 2)[0].to_s.split("#", 2)[0].to_s.downcase
|
|
127
|
+
skip.none? { |prefix| path.start_with?(prefix) }
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
PAGE = <<~HTML
|
|
131
|
+
<!doctype html>
|
|
132
|
+
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
133
|
+
<meta name="robots" content="noindex"><title>Just a moment</title>
|
|
134
|
+
<style nonce="%<nonce>s">body{font:16px/1.5 system-ui,sans-serif;margin:0;display:flex;min-height:100vh;align-items:center;justify-content:center;color:#222;background:#fafafa}main{max-width:28rem;padding:2rem;text-align:center}</style>
|
|
135
|
+
</head><body><main><p id="m">Checking your browser before you continue…</p>
|
|
136
|
+
<noscript><p>Please turn on JavaScript to continue to this site.</p></noscript></main>
|
|
137
|
+
<script nonce="%<nonce>s">%<script>s</script>
|
|
138
|
+
</body></html>
|
|
139
|
+
HTML
|
|
140
|
+
|
|
141
|
+
# The page's script: sets the pass and reloads, but only when reading the
|
|
142
|
+
# cookie back shows it was set (with cookies off it says so instead of
|
|
143
|
+
# reloading forever), and at most three times a minute, counted in
|
|
144
|
+
# sessionStorage (an address that changes on every request would otherwise
|
|
145
|
+
# never get a pass that holds and reload for ever), or in a __sg_tries
|
|
146
|
+
# cookie lasting a minute where storage is blocked or full. Secure only
|
|
147
|
+
# when the browser is on https: it knows, where the server behind a proxy
|
|
148
|
+
# can only be told. The same script as the Node, PHP and Python clients'.
|
|
149
|
+
def script(value)
|
|
150
|
+
"(function(){var n=#{COOKIE.to_json},v=#{value.to_json},m=document.getElementById(\"m\")," \
|
|
151
|
+
"s=location.protocol===\"https:\";" \
|
|
152
|
+
"document.cookie=n+\"=\"+v+\"; Max-Age=#{MAX_AGE}; Path=/; SameSite=Lax\"+(s?\"; Secure\":\"\");" \
|
|
153
|
+
"if((\" \"+document.cookie+\";\").indexOf(\" \"+n+\"=\"+v+\";\")<0){m.textContent=\"Please allow cookies for this site, then reload the page.\";return;}" \
|
|
154
|
+
"var k=\"__sg_tries\",r=0,t;" \
|
|
155
|
+
"try{t=JSON.parse(sessionStorage.getItem(k)||\"[]\").filter(function(x){return Date.now()-x<60000;});r=t.length;if(r<3){t.push(Date.now());sessionStorage.setItem(k,JSON.stringify(t));}}" \
|
|
156
|
+
"catch(e){t=/(?:^|; )__sg_tries=([0-9]+)/.exec(document.cookie);r=t?+t[1]:0;if(r<3)document.cookie=k+\"=\"+(r+1)+\"; Max-Age=60; Path=/; SameSite=Lax\"+(s?\"; Secure\":\"\");}" \
|
|
157
|
+
"if(r>=3){m.textContent=\"This page couldn't be opened because your connection's address keeps changing. Please try again in a minute.\";return;}" \
|
|
158
|
+
"location.reload();})();"
|
|
159
|
+
end
|
|
160
|
+
|
|
161
|
+
# [503, headers, body] for the check page: its own Content-Security-Policy
|
|
162
|
+
# with a fresh nonce for its one script, and no body for a HEAD.
|
|
163
|
+
def page(key, ip, method, now = nil)
|
|
164
|
+
nonce = SecureRandom.urlsafe_base64(18)
|
|
165
|
+
headers = {
|
|
166
|
+
"content-type" => "text/html; charset=utf-8",
|
|
167
|
+
"cache-control" => "no-store",
|
|
168
|
+
"x-robots-tag" => "noindex",
|
|
169
|
+
"content-security-policy" => "default-src 'none'; script-src 'nonce-#{nonce}'; style-src 'nonce-#{nonce}'; " \
|
|
170
|
+
"base-uri 'none'; form-action 'none'; frame-ancestors 'none'"
|
|
171
|
+
}
|
|
172
|
+
return [503, headers, ""] if method.to_s.upcase == "HEAD"
|
|
173
|
+
|
|
174
|
+
[503, headers, format(PAGE, nonce: nonce, script: script(make_pass(key, ip, now)))]
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
end
|