findxpand 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +325 -0
- data/exe/findxpand +20 -0
- data/findxpand.gemspec +71 -0
- data/lib/findxpand/admin.rb +157 -0
- data/lib/findxpand/auto.rb +607 -0
- data/lib/findxpand/cli.rb +214 -0
- data/lib/findxpand/encoding.rb +160 -0
- data/lib/findxpand/middleware.rb +597 -0
- data/lib/findxpand/railtie.rb +72 -0
- data/lib/findxpand/rewrite.rb +628 -0
- data/lib/findxpand/signature.rb +127 -0
- data/lib/findxpand/store.rb +657 -0
- data/lib/findxpand.rb +255 -0
- metadata +62 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 756ce879fc1079faabc838210e35b0467f721dc3064b4c7f0032df2164ce8f0d
|
|
4
|
+
data.tar.gz: 28c91eb83f1f26942080772c18522cdb7a52fe7bd7c4c652dfa0aba58ac08735
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 84a14740683127e6b8b76b3194c460e9e451f21f28f1d6b02ff81cf6e8d4e4249eb166cd9d4440997660bc5b8e1d78f64075af05557a2d78b849a11a7a25d920
|
|
7
|
+
data.tar.gz: b711ee022f7a8122ea6929a633bbee91ae98612641ee926210511db4ce53dc15761e978bd2a96560749961c6135c5ad60731b2675fc40cc3234edef331c25e0e
|
data/README.md
ADDED
|
@@ -0,0 +1,325 @@
|
|
|
1
|
+
# findxpand
|
|
2
|
+
|
|
3
|
+
Server-side SEO remediation at your origin, for Rack, Rails, Sinatra, Hanami,
|
|
4
|
+
Roda, Grape, or a bare lambda.
|
|
5
|
+
|
|
6
|
+
```bash
|
|
7
|
+
gem install ./findxpand-0.1.0.gem
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
> **Not on RubyGems yet.** We send you the `.gem` file; `findxpand` is not a
|
|
11
|
+
> published name, so installing it by name fails. The version in the filename is
|
|
12
|
+
> what a support conversation refers to.
|
|
13
|
+
|
|
14
|
+
## The whole install
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
# Gemfile
|
|
18
|
+
gem 'findxpand'
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
RUBYOPT="-rfindxpand/auto" bundle exec puma
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Nothing in your application changes. `FINDXPAND_TOKEN` in the same environment is
|
|
26
|
+
the only other thing needed.
|
|
27
|
+
|
|
28
|
+
**The Gemfile line is not optional.** `bundle exec` restricts the load path to
|
|
29
|
+
the Gemfile before `RUBYOPT` is processed, so without it Ruby answers
|
|
30
|
+
`LoadError: cannot load such file -- findxpand/auto` and your server does not
|
|
31
|
+
start at all. That is the one failure this gem cannot soften — every other one
|
|
32
|
+
degrades to *not attached* — and it is why the Gemfile is step one.
|
|
33
|
+
|
|
34
|
+
That is the only deploy. Approved fixes arrive afterwards as a manifest pushed to
|
|
35
|
+
`/__findxpand/manifest`, and no further release is needed.
|
|
36
|
+
|
|
37
|
+
### On Rails, the Gemfile line is enough on its own
|
|
38
|
+
|
|
39
|
+
A Railtie inserts the middleware at position 0 of the Rails stack, which is
|
|
40
|
+
outermost of everything Rails builds — ahead of a
|
|
41
|
+
`config.middleware.use Rack::Deflater`, and ahead of the router. The `RUBYOPT` is
|
|
42
|
+
still the recommended install because it is the same instruction for every
|
|
43
|
+
framework and it survives a `config.ru` that does something unusual.
|
|
44
|
+
|
|
45
|
+
### Why not `use Findxpand::Middleware`
|
|
46
|
+
|
|
47
|
+
Because there is no position in a Rack stack that also uses `Rack::Deflater`
|
|
48
|
+
where mounting by hand does the right thing, and the same measurement holds in
|
|
49
|
+
three languages. 31 Aug 2026:
|
|
50
|
+
|
|
51
|
+
| stack | mount | result |
|
|
52
|
+
|---|---|---|
|
|
53
|
+
| Express + `compression` | findxpand first | `skip-encoded` — page unmodified, `/status` healthy, the fix silently never lands |
|
|
54
|
+
| Express + `compression` | compression first | 2069 bytes of plain HTML on the wire under `content-encoding: gzip` — undecodable |
|
|
55
|
+
| Django 6.1 + `GZipMiddleware` | outermost | `skip-encoded`, original title served, `/status` healthy |
|
|
56
|
+
|
|
57
|
+
In Rack, `use Rack::Deflater` *inside* you means the body you are handed has
|
|
58
|
+
already been gzipped, and there is nowhere further out to stand. So the wrapper
|
|
59
|
+
stops asking to be placed correctly and takes over the negotiation instead: on
|
|
60
|
+
the handful of paths the manifest names, your application is asked for
|
|
61
|
+
`Accept-Encoding: identity` and the rewritten page is compressed on the way out.
|
|
62
|
+
A path with no rule is not touched at all, and your own compression still runs on
|
|
63
|
+
the rest of the site exactly as before.
|
|
64
|
+
|
|
65
|
+
`Findxpand::Middleware` is still exported for embedding, and is what `/auto`
|
|
66
|
+
mounts.
|
|
67
|
+
|
|
68
|
+
### What it reaches, and what it does not
|
|
69
|
+
|
|
70
|
+
`findxpand/auto` prepends `to_app` on the class your server builds its Rack
|
|
71
|
+
application with, and wraps what that returns.
|
|
72
|
+
|
|
73
|
+
**Not `Rack::Builder` alone.** This page used to say "one method —
|
|
74
|
+
`Rack::Builder#to_app` — which every Ruby web server calls", and that was wrong
|
|
75
|
+
on the server in the command at the top of this page: Puma parses `config.ru`
|
|
76
|
+
with its own vendored copy of the builder, so a prepend on `Rack::Builder` never
|
|
77
|
+
ran and nothing said so. What attaches now is two things — every builder class
|
|
78
|
+
already loaded, by name, and every class defined afterwards that carries the
|
|
79
|
+
`config.ru` DSL (`to_app`, `use` and `run` together), which is what a server's
|
|
80
|
+
vendored copy is. See `lib/findxpand/auto.rb`; that file also records that none
|
|
81
|
+
of it has been run against a real Puma.
|
|
82
|
+
|
|
83
|
+
**Reached:** anything booted from a `config.ru` (`rackup`, Puma, Unicorn, Thin,
|
|
84
|
+
Falcon, Passenger, WEBrick); **Rails**, because `rails server` loads `config.ru`
|
|
85
|
+
too, so the wrap sits outside the Rails stack *and* outside anything `config.ru`
|
|
86
|
+
itself used; **Sinatra**, classic and modular, including `ruby app.rb` with no
|
|
87
|
+
`config.ru` at all, because `Sinatra::Base.new` builds through a builder;
|
|
88
|
+
**Hanami**, **Roda**, **Grape**, `Rack::URLMap`, and a plain lambda.
|
|
89
|
+
|
|
90
|
+
**Not reached**, and there is no point pretending otherwise:
|
|
91
|
+
|
|
92
|
+
- a server handed an app object in code rather than through a builder —
|
|
93
|
+
`Puma::Server.new(MyApp).run`;
|
|
94
|
+
- a builder that does not answer to `to_app`, `use` and `run` together — that is
|
|
95
|
+
the whole of what is recognised, and a server with a different DSL is a server
|
|
96
|
+
this does not see;
|
|
97
|
+
- anything that is not a Rack application: a bare `WEBrick::HTTPServer` with a
|
|
98
|
+
servlet, `Async::HTTP::Server` used directly, gRPC;
|
|
99
|
+
- a process that never receives `RUBYOPT` — a `bin/rails` wrapper that re-execs,
|
|
100
|
+
a systemd unit with a scrubbed `Environment=`, a Dockerfile that sets `ENV` in
|
|
101
|
+
a build stage the runtime stage does not inherit. On Rails the Railtie still
|
|
102
|
+
catches this; on Sinatra and Hanami nothing does;
|
|
103
|
+
- **JRuby and TruffleRuby**, in the sense that the mechanism is ordinary Ruby,
|
|
104
|
+
ought to work, and has not been run there. Watching for a server's own builder
|
|
105
|
+
needs `TracePoint`; where that will not enable, the boot log says so in a
|
|
106
|
+
sentence rather than leaving it to be discovered.
|
|
107
|
+
|
|
108
|
+
If it does not attach, `/__findxpand/status` says so rather than reporting
|
|
109
|
+
health — `attach.built: 0` is a hook that was placed and never ran. See *It
|
|
110
|
+
tells you when it is doing nothing*. The one case no endpoint can report is the
|
|
111
|
+
total one: with no middleware anywhere there is nothing to answer `/status`, so
|
|
112
|
+
that one is a line on stderr at boot.
|
|
113
|
+
|
|
114
|
+
## Not a Ruby app, or not one of those? Run it as a sidecar
|
|
115
|
+
|
|
116
|
+
The same behaviour, with a reverse proxy where your application would be. Nothing
|
|
117
|
+
installs into your stack and it does not matter what your stack is. See
|
|
118
|
+
`middleware/node/README.md` for the container; it is the route for PHP, Java,
|
|
119
|
+
.NET, Go, and static files behind nginx.
|
|
120
|
+
|
|
121
|
+
## Environment
|
|
122
|
+
|
|
123
|
+
Two variables matter:
|
|
124
|
+
|
|
125
|
+
| Variable | Meaning |
|
|
126
|
+
|---|---|
|
|
127
|
+
| `FINDXPAND_TOKEN` | **Required.** The shared secret we push manifests with, and the HMAC key. We generate it; nothing attaches without it. |
|
|
128
|
+
| `FINDXPAND_CACHE_FILE` | **Required in practice.** Where a pushed manifest is persisted. Without it, every restart drops every rule until the next push — and on a forking server a push reaches one worker. |
|
|
129
|
+
|
|
130
|
+
And five that have sensible defaults:
|
|
131
|
+
|
|
132
|
+
| Variable | Default | Meaning |
|
|
133
|
+
|---|---|---|
|
|
134
|
+
| `FINDXPAND_MANIFEST_FILE` | — | A manifest committed to your repository, for the no-inbound-endpoint mode. Read at boot; a push overrides it. |
|
|
135
|
+
| `FINDXPAND_ENABLED` | on | `false`, `0`, `no` or `off` leaves the gem installed and stops it touching responses. |
|
|
136
|
+
| `FINDXPAND_MAX_BYTES` | 2 MB | Largest HTML body to buffer. Larger responses stream through untouched. |
|
|
137
|
+
| `FINDXPAND_REQUIRE_SIGNATURE` | on | Verify the signature on every push. See below before turning this off. |
|
|
138
|
+
| `FINDXPAND_RECOMPRESS` | on | On a path with a rule, ask your application for `identity` and gzip the rewritten page on the way out. Turn it **off** only where an intermediary has already taken over content negotiation. Off means a ruled page is served uncompressed. |
|
|
139
|
+
|
|
140
|
+
**All seven are read on every way in** — the `RUBYOPT` attach, the Rails Railtie,
|
|
141
|
+
and a hand-written `use Findxpand::Middleware`. Until 2 Sep 2026 only the first
|
|
142
|
+
of those read them, so on Rails `FINDXPAND_CACHE_FILE` was silently ignored and
|
|
143
|
+
nothing persisted while `/status` reported health.
|
|
144
|
+
|
|
145
|
+
> **Pick a cache path that survives a restart, and that every worker can read.**
|
|
146
|
+
> `/var/tmp` is fine on a VM and wrong on a container or a serverless platform,
|
|
147
|
+
> where it is discarded. Use a mounted volume; if the platform has no durable
|
|
148
|
+
> disk at all, say so — the manifest-in-the-repository mode exists for exactly
|
|
149
|
+
> that and needs none.
|
|
150
|
+
|
|
151
|
+
## Forking servers
|
|
152
|
+
|
|
153
|
+
Puma in cluster mode, Unicorn and Passenger serve from forked workers, and a
|
|
154
|
+
manifest pushed to `/__findxpand/manifest` arrives at **exactly one of them**.
|
|
155
|
+
The worker that took the push writes the cache file; every other worker notices
|
|
156
|
+
the change and adopts it within a second. That is the whole reason
|
|
157
|
+
`FINDXPAND_CACHE_FILE` is listed as required rather than optional — without a
|
|
158
|
+
shared cache file, three workers out of four go on serving the previous rules
|
|
159
|
+
until the next deploy, and a fix verifies or fails to verify depending on which
|
|
160
|
+
worker answers the re-fetch.
|
|
161
|
+
|
|
162
|
+
No network is involved: a worker reads a local file its sibling wrote.
|
|
163
|
+
|
|
164
|
+
That paragraph is true of every way in, and until 2 Sep 2026 it was true of one:
|
|
165
|
+
`FINDXPAND_CACHE_FILE` was read only by the `RUBYOPT` attach, so a Rails app —
|
|
166
|
+
where the Railtie is always the first to build the store — ran with no cache
|
|
167
|
+
file at all whatever the environment said. Nothing was written, nothing was
|
|
168
|
+
adopted, and `/status` reported health. If you are looking at a `/status` from
|
|
169
|
+
before that, `persisted` is the field that would have shown it.
|
|
170
|
+
|
|
171
|
+
## What it does to a response
|
|
172
|
+
|
|
173
|
+
Only when the path has a rule, the status is 200, the content type is HTML, and
|
|
174
|
+
nothing has already compressed it. Everything else is passed through and marked:
|
|
175
|
+
|
|
176
|
+
| `x-findxpand-origin-mw` | Meaning |
|
|
177
|
+
|---|---|
|
|
178
|
+
| `1` | rewritten |
|
|
179
|
+
| `pass` | a rule existed and changed nothing |
|
|
180
|
+
| `error` | the rewrite raised; **your original bytes were served unchanged** |
|
|
181
|
+
| `skip-encoded` | already compressed — something below us gzipped it |
|
|
182
|
+
| `skip-large` | over `FINDXPAND_MAX_BYTES`, served whole rather than truncated |
|
|
183
|
+
| `skip-encoding` | the body was not valid UTF-8, so it was not touched |
|
|
184
|
+
| `redirect` | served from the manifest's redirect table |
|
|
185
|
+
| `status` | a 404/410/451 rule answered before your app was reached |
|
|
186
|
+
|
|
187
|
+
**On any exception, the original response is served.** The rewrite runs inside a
|
|
188
|
+
`rescue` and the buffered bytes are sent untouched if it raises — that is the
|
|
189
|
+
reason for buffering at all. A bug of ours cannot take down a page.
|
|
190
|
+
|
|
191
|
+
Two things happen to the *request*, and only on a path with a rule:
|
|
192
|
+
`Accept-Encoding` becomes `identity`, and `If-None-Match` / `If-Modified-Since`
|
|
193
|
+
are dropped. The second is there because a `304 Not Modified` answered below us
|
|
194
|
+
would leave a returning visitor — and a crawler with the page already cached —
|
|
195
|
+
holding the *old* document indefinitely while the fix counted as applied for
|
|
196
|
+
everybody else.
|
|
197
|
+
|
|
198
|
+
**Every visitor gets identical bytes.** There is no user-agent branch anywhere in
|
|
199
|
+
this gem. Serving crawlers something different is cloaking, and the point of a
|
|
200
|
+
server-side rewrite is that it needs no such trick.
|
|
201
|
+
|
|
202
|
+
## Endpoints
|
|
203
|
+
|
|
204
|
+
Both require `Authorization: Bearer <token>`, are `no-store`, and carry
|
|
205
|
+
`x-robots-tag: noindex, nofollow`.
|
|
206
|
+
|
|
207
|
+
- `GET /__findxpand/status` — version, rule count, age, health, and the manifest
|
|
208
|
+
- `POST /__findxpand/manifest` — replace the rules; echoes the stored version
|
|
209
|
+
|
|
210
|
+
### It tells you when it is doing nothing
|
|
211
|
+
|
|
212
|
+
`/__findxpand/status` reports `counters`, `requests_seen`, `applied_age`,
|
|
213
|
+
`persisted`, `reach_reason`, `attach`, `degraded` and `degraded_reason` as well
|
|
214
|
+
as the rule count. This is the failure worth monitoring: attached below
|
|
215
|
+
compression, attached where no rule matches, unable to persist a push, or not
|
|
216
|
+
attached at all, the middleware serves every page untouched and every other
|
|
217
|
+
signal looks healthy.
|
|
218
|
+
|
|
219
|
+
```json
|
|
220
|
+
{
|
|
221
|
+
"degraded": true,
|
|
222
|
+
"degraded_reason": "12 response(s) arrived already compressed and could not be rewritten - the middleware is mounted after compression, and needs to be mounted before it",
|
|
223
|
+
"requests_seen": 12,
|
|
224
|
+
"persisted": true,
|
|
225
|
+
"counters": { "considered": 12, "applied": 0, "skip_encoded": 12, "unmatched": 0, "error": 0 },
|
|
226
|
+
"attach": { "attach_state": "attached", "builders": ["Rack::Builder"], "built": 1, "middlewares": 1 }
|
|
227
|
+
}
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
Alert on `degraded`. `applied_age` is seconds since a rewrite last landed, or
|
|
231
|
+
`null` if one never has. Four fields answer four different "is it really
|
|
232
|
+
working" questions, and they are not interchangeable:
|
|
233
|
+
|
|
234
|
+
- `counters.unmatched` — requests that reached the middleware on a path the
|
|
235
|
+
manifest does not name. Many of them with `considered: 0` means the manifest's
|
|
236
|
+
keys match nothing this site serves, or the middleware is mounted where the
|
|
237
|
+
pages do not go. It is the same counter the Node package reports, and this
|
|
238
|
+
package votes on it: **twenty unmatched requests, no match at all, and a
|
|
239
|
+
minute of uptime** puts the sentence in `degraded_reason`. One unmatched
|
|
240
|
+
request is a favicon, so it is counted and not voted on — if you alert on this
|
|
241
|
+
yourself, use the same floor rather than `> 0`, or your alert fires on the
|
|
242
|
+
first stylesheet of every deploy.
|
|
243
|
+
- `requests_seen` — every non-admin request that reached the middleware, pages
|
|
244
|
+
and assets and health probes alike. Weaker than `unmatched` on purpose: one
|
|
245
|
+
`GET /api/health` moves it, so it answers "was I in *a* request path" and not
|
|
246
|
+
"was I in the one the pages go down". Up more than a minute with
|
|
247
|
+
`requests_seen: 0` is reported as degraded rather than healthy.
|
|
248
|
+
- `persisted` — `true` a push was written to `FINDXPAND_CACHE_FILE`, `false` one
|
|
249
|
+
was taken and could not be, `null` nothing has been pushed here yet. `false`
|
|
250
|
+
is the expensive one: the rules are live in this process, a restart drops
|
|
251
|
+
them, and the other workers never saw them.
|
|
252
|
+
- `attach.built` — how many applications were built through the `RUBYOPT` hook.
|
|
253
|
+
`0` with the endpoint answering means the Railtie mounted us and the hook
|
|
254
|
+
never fired.
|
|
255
|
+
|
|
256
|
+
Asking `/status` does not count as a request; an auditor that satisfies its own
|
|
257
|
+
question is not an auditor.
|
|
258
|
+
|
|
259
|
+
## The CLI
|
|
260
|
+
|
|
261
|
+
```bash
|
|
262
|
+
findxpand init # the lines to add, ready to paste
|
|
263
|
+
findxpand doctor # this machine's preconditions, and what it cannot check
|
|
264
|
+
findxpand status # the manifest cached on this machine
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
`doctor` runs in a different process from your server, so it cannot tell you
|
|
268
|
+
whether the middleware is in the request path, and it says so instead of implying
|
|
269
|
+
otherwise. It prints the `curl` line for `/status`, which is the thing that
|
|
270
|
+
knows.
|
|
271
|
+
|
|
272
|
+
Nothing in the CLI makes an HTTP request, to us or to localhost.
|
|
273
|
+
|
|
274
|
+
## Pushes are signed, not merely authenticated
|
|
275
|
+
|
|
276
|
+
Every manifest push carries `x-findxpand-signature` and `x-findxpand-timestamp`:
|
|
277
|
+
HMAC-SHA256 over `timestamp.body`, keyed by your token, compared in constant
|
|
278
|
+
time. Unsigned, altered or stale pushes are refused with **401**.
|
|
279
|
+
|
|
280
|
+
The bearer token proves who sent a push and says nothing about what was in it.
|
|
281
|
+
Anything holding it — a logging proxy, a mirrored request, an old CI secret —
|
|
282
|
+
could otherwise replay a captured manifest indefinitely, or swap the manifest
|
|
283
|
+
inside one. The timestamp is inside the digest, so a captured push expires; the
|
|
284
|
+
window is 300 seconds, which means the clock on that host has to be roughly
|
|
285
|
+
right.
|
|
286
|
+
|
|
287
|
+
There is one supported reason to turn this off, and it is an intermediary that
|
|
288
|
+
rewrites or re-encodes request bodies, since that invalidates any signature over
|
|
289
|
+
them.
|
|
290
|
+
|
|
291
|
+
## Why it never calls us
|
|
292
|
+
|
|
293
|
+
It cannot. There is no outbound request anywhere in this gem. If Findxpand is
|
|
294
|
+
unavailable, your site serves the last manifest it received, indefinitely.
|
|
295
|
+
|
|
296
|
+
## Removing the gem removes the rules
|
|
297
|
+
|
|
298
|
+
The corrected titles, descriptions, canonicals and alt text are produced by this
|
|
299
|
+
middleware at request time. They are not written into your templates, your
|
|
300
|
+
database or your CMS. Uninstall the gem, unset `RUBYOPT`, or set
|
|
301
|
+
`FINDXPAND_ENABLED=false`, and your pages go back to exactly what your
|
|
302
|
+
application renders — immediately, with no migration and nothing left behind.
|
|
303
|
+
|
|
304
|
+
That is a property to know before you choose this route rather than a defect, and
|
|
305
|
+
it cuts both ways: it is why a bad fix is one environment variable from being
|
|
306
|
+
gone, and it is why a fix is not "in your codebase" once it is live. Where a
|
|
307
|
+
change should persist in your own source instead, the pull-request route carries
|
|
308
|
+
the same fields into your repository.
|
|
309
|
+
|
|
310
|
+
## Development
|
|
311
|
+
|
|
312
|
+
```bash
|
|
313
|
+
ruby -Ilib -Itest test/conformance_test.rb # the shared cross-language fixture
|
|
314
|
+
ruby -Ilib -Itest test/rewrite_test.rb # Ruby's own ways to get this wrong
|
|
315
|
+
ruby -Ilib -Itest test/middleware_test.rb # the Rack layer, end to end
|
|
316
|
+
ruby -Ilib -Itest test/attach_test.rb # the attach, the env, the store singleton
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
No Bundler and no `gem install`: minitest ships with Ruby and this gem has no
|
|
320
|
+
dependencies, so its suite runs wherever the gem does.
|
|
321
|
+
|
|
322
|
+
`test/conformance_test.rb` runs `middleware/conformance/cases.json`, which is
|
|
323
|
+
generated by the engine and is the same file the Node and Python middlewares run.
|
|
324
|
+
Four hand-written rewriters have to agree byte for byte, and one fixture with
|
|
325
|
+
four consumers is the only thing that makes disagreement visible.
|
data/exe/findxpand
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# The console script. Everything it does is in `lib/findxpand/cli.rb`.
|
|
5
|
+
#
|
|
6
|
+
# `exit` with the CLI's own return value, so that `findxpand doctor` can be the
|
|
7
|
+
# last line of a deploy script and mean something. A tool that reports a failed
|
|
8
|
+
# precondition and exits 0 is the §20.1 rule 3 shape from the outside.
|
|
9
|
+
|
|
10
|
+
begin
|
|
11
|
+
require 'findxpand/cli'
|
|
12
|
+
rescue LoadError
|
|
13
|
+
# Run straight out of a checkout rather than an installed gem. The rescue
|
|
14
|
+
# rather than an unconditional `$LOAD_PATH` push, so that an installed copy
|
|
15
|
+
# never quietly loads a different one that happens to be nearby.
|
|
16
|
+
$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
|
|
17
|
+
require 'findxpand/cli'
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
exit(Findxpand::CLI.run(ARGV))
|
data/findxpand.gemspec
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The gem, and its one unusual property: it has no dependencies at all.
|
|
4
|
+
#
|
|
5
|
+
# Not "few". None. This installs into somebody else's application, and a
|
|
6
|
+
# middleware that drags in a dependency tree is a middleware that can break their
|
|
7
|
+
# build or pin a version they cannot afford — so everything it needs comes out of
|
|
8
|
+
# the standard library: `json`, `openssl`, `zlib`, `stringio`, `fileutils`,
|
|
9
|
+
# `mutex_m`-free plain `Mutex`, and `minitest` for the suite, which ships with
|
|
10
|
+
# Ruby. It is a **Rack** middleware that does not depend on Rack: nothing in it
|
|
11
|
+
# needs more of Rack than the calling convention, and `findxpand/auto` only
|
|
12
|
+
# touches `Rack::Builder` if the host application has already loaded it.
|
|
13
|
+
#
|
|
14
|
+
# `required_ruby_version >= 3.0` for three reasons that are all load-bearing:
|
|
15
|
+
# `OpenSSL.secure_compare` (constant-time digest comparison) ships with the
|
|
16
|
+
# openssl that comes with Ruby 3.0; `Enumerable#to_h` with a block builds the
|
|
17
|
+
# pattern table; and `Hash#transform_keys` shapes the status JSON. None is worth
|
|
18
|
+
# a compatibility shim for Ruby 2.
|
|
19
|
+
|
|
20
|
+
# The version comes from the library rather than being written here as well: two
|
|
21
|
+
# copies of one fact are two that drift, and the one somebody edits is never the
|
|
22
|
+
# one the other reads (§20.1 rule 2).
|
|
23
|
+
#
|
|
24
|
+
# `$LOAD_PATH` rather than `require_relative 'lib/findxpand'`, because that file
|
|
25
|
+
# requires its own siblings by name, and `lib/` is not on the path while a
|
|
26
|
+
# gemspec is being evaluated.
|
|
27
|
+
$LOAD_PATH.unshift(File.expand_path('lib', __dir__))
|
|
28
|
+
require 'findxpand'
|
|
29
|
+
|
|
30
|
+
Gem::Specification.new do |spec|
|
|
31
|
+
spec.name = 'findxpand'
|
|
32
|
+
spec.version = Findxpand::VERSION
|
|
33
|
+
spec.license = 'MIT'
|
|
34
|
+
spec.authors = ['Nextoria']
|
|
35
|
+
spec.summary = 'Server-side SEO remediation at your origin, for Rack, Rails, Sinatra and Hanami.'
|
|
36
|
+
spec.description =
|
|
37
|
+
'One deploy. After that, approved fixes arrive as a manifest pushed to your own ' \
|
|
38
|
+
'origin and no further release is needed. The corrected HTML is produced on the ' \
|
|
39
|
+
'server, so crawlers that do not run JavaScript still see it. Zero runtime ' \
|
|
40
|
+
'dependencies; it never makes an outbound request.'
|
|
41
|
+
spec.homepage = 'https://nextoria.ae/findxpand'
|
|
42
|
+
spec.required_ruby_version = '>= 3.0'
|
|
43
|
+
|
|
44
|
+
spec.metadata = {
|
|
45
|
+
'homepage_uri' => spec.homepage,
|
|
46
|
+
# Deliberately absent: `allowed_push_host`. This gem is not published, and a
|
|
47
|
+
# push host in the metadata is the kind of line somebody trusts later.
|
|
48
|
+
# `Findxpand::PUBLISHED_TO_REGISTRIES` is the flag to flip when it is, and it
|
|
49
|
+
# moves the README, the CLI and the engine's install screen together.
|
|
50
|
+
'rubygems_mfa_required' => 'true'
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
# Listed with `Dir` rather than `git ls-files`, which is the usual line and
|
|
54
|
+
# needs git to be installed and the tree to be a checkout — neither is true of
|
|
55
|
+
# the tarball a customer builds from.
|
|
56
|
+
spec.files = Dir[
|
|
57
|
+
'lib/**/*.rb',
|
|
58
|
+
'exe/*',
|
|
59
|
+
'README.md',
|
|
60
|
+
'findxpand.gemspec'
|
|
61
|
+
]
|
|
62
|
+
spec.bindir = 'exe'
|
|
63
|
+
spec.executables = ['findxpand']
|
|
64
|
+
spec.require_paths = ['lib']
|
|
65
|
+
|
|
66
|
+
# The suite is deliberately not in `spec.files`: it reads
|
|
67
|
+
# `middleware/conformance/cases.json`, which lives outside this directory
|
|
68
|
+
# because three other packages read the same file, and a gem that shipped a
|
|
69
|
+
# test it cannot run would be worse than one that ships none. Run it from the
|
|
70
|
+
# repository with `ruby -Ilib -Itest test/conformance_test.rb`.
|
|
71
|
+
end
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The two endpoints, and nothing else about the request path.
|
|
4
|
+
#
|
|
5
|
+
# GET /__findxpand/status bearer token
|
|
6
|
+
# POST /__findxpand/manifest bearer token *and* HMAC-SHA256
|
|
7
|
+
#
|
|
8
|
+
# Split out of the middleware because these are the only branches in the gem
|
|
9
|
+
# that fail **closed**. §3 rule 7 draws the line exactly here: an audit may
|
|
10
|
+
# degrade and a page may be served unchanged, but a write, a credential and a
|
|
11
|
+
# budget must refuse rather than guess. Every other file in this gem is written
|
|
12
|
+
# so that a failure serves the customer's own bytes; every branch in this one is
|
|
13
|
+
# written so that a failure serves a 401.
|
|
14
|
+
#
|
|
15
|
+
# Both replies carry `cache-control: no-store` and `x-robots-tag: noindex,
|
|
16
|
+
# nofollow`, because the body describes the site's own SEO configuration and
|
|
17
|
+
# belongs to whoever holds the token.
|
|
18
|
+
#
|
|
19
|
+
# Header names are emitted in lower case. Rack 3 requires it, Rack 2 does not
|
|
20
|
+
# care, and HTTP has been case-insensitive about them since 1996 — so lower case
|
|
21
|
+
# is the one spelling that is correct under both versions of the specification.
|
|
22
|
+
#
|
|
23
|
+
# Frozen string literals: on.
|
|
24
|
+
|
|
25
|
+
require 'json'
|
|
26
|
+
|
|
27
|
+
module Findxpand
|
|
28
|
+
class Admin
|
|
29
|
+
ROOT = '/__findxpand'
|
|
30
|
+
MANIFEST_PATH = "#{ROOT}/manifest"
|
|
31
|
+
STATUS_PATH = "#{ROOT}/status"
|
|
32
|
+
|
|
33
|
+
def self.admin_path?(path)
|
|
34
|
+
path == STATUS_PATH || path == MANIFEST_PATH
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def initialize(store:, token:, max_bytes:, require_signature: true, clock: nil)
|
|
38
|
+
@store = store
|
|
39
|
+
@token = token.to_s
|
|
40
|
+
@max_bytes = max_bytes
|
|
41
|
+
@require_signature = require_signature
|
|
42
|
+
@clock = clock || -> { Time.now.to_f }
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def call(env, path)
|
|
46
|
+
unless authorised?(env)
|
|
47
|
+
return json(401, { 'ok' => false, 'error' => 'bad or missing bearer token' })
|
|
48
|
+
end
|
|
49
|
+
# `attach` is merged in here rather than kept in `Store`, because the store
|
|
50
|
+
# holds what this *manifest* has done and these are facts about how this
|
|
51
|
+
# process was assembled — and because a Store that reached into `Auto`
|
|
52
|
+
# would be a Store that cannot be constructed in a suite that never loads
|
|
53
|
+
# it. Additive, and safe to be: the engine spreads whatever this endpoint
|
|
54
|
+
# answers (`engine.fix.origin.status`, `{"ok": True, "installed": True,
|
|
55
|
+
# **payload}`), and `middleware/php/src/Store.php:762-776`'s `runtime`
|
|
56
|
+
# block is the precedent — a package reports the failures only its own
|
|
57
|
+
# runtime can have, under a name that says what they mean.
|
|
58
|
+
#
|
|
59
|
+
# The one it exists for: `attach.built == 0` on a Rails process means the
|
|
60
|
+
# Railtie mounted us and the `RUBYOPT` hook never fired, which until
|
|
61
|
+
# 2 Sep 2026 was the *normal* state under Puma and was reported nowhere.
|
|
62
|
+
if path == STATUS_PATH
|
|
63
|
+
return json(200, @store.status.merge('attach' => Findxpand.attach_report))
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# Anything but POST on the manifest path. Worded as an instruction rather
|
|
67
|
+
# than a refusal because the request that lands here most often is a human
|
|
68
|
+
# with a browser checking whether the install worked.
|
|
69
|
+
unless env['REQUEST_METHOD'] == 'POST'
|
|
70
|
+
return json(405, { 'ok' => false, 'error' => 'POST a manifest here' })
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
store_manifest(env)
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
private
|
|
77
|
+
|
|
78
|
+
# Constant-time, and a length mismatch answers false rather than raising.
|
|
79
|
+
#
|
|
80
|
+
# An empty configured token can never authorise: without this the gem would
|
|
81
|
+
# ship an open manifest endpoint to anybody who installed it and forgot to
|
|
82
|
+
# set `FINDXPAND_TOKEN`, since `'' == ''` is true.
|
|
83
|
+
def authorised?(env)
|
|
84
|
+
return false if @token.empty?
|
|
85
|
+
|
|
86
|
+
Signature.secure_compare(bearer(env), @token)
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def bearer(env)
|
|
90
|
+
header = env['HTTP_AUTHORIZATION'].to_s
|
|
91
|
+
header.start_with?('Bearer ') ? header[7..].to_s : ''
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def store_manifest(env)
|
|
95
|
+
raw = read_body(env)
|
|
96
|
+
return json(400, { 'ok' => false, 'error' => 'manifest too large' }) if raw.nil?
|
|
97
|
+
|
|
98
|
+
if @require_signature
|
|
99
|
+
reason = Signature.problem(env[Signature::SIGNATURE_ENV], env[Signature::TIMESTAMP_ENV],
|
|
100
|
+
raw, @token, now: @clock.call)
|
|
101
|
+
unless reason.empty?
|
|
102
|
+
# 401, not 400: this is an authentication failure, and a 400 would read
|
|
103
|
+
# as "our manifest was malformed" in our own logs.
|
|
104
|
+
return json(401, { 'ok' => false, 'error' => reason })
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# The bytes arrive binary and are only ever interpreted as text here, after
|
|
109
|
+
# the signature has been checked over the bytes themselves. Doing it in the
|
|
110
|
+
# other order — parse, then re-serialise to verify — would reorder keys and
|
|
111
|
+
# reject every legitimate push.
|
|
112
|
+
text = raw.dup.force_encoding('UTF-8')
|
|
113
|
+
unless text.valid_encoding?
|
|
114
|
+
return json(400, { 'ok' => false, 'error' => 'manifest is not valid UTF-8' })
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
version = @store.replace(JSON.parse(text))
|
|
118
|
+
json(200, { 'ok' => true, 'version' => version })
|
|
119
|
+
rescue JSON::ParserError, ArgumentError => e
|
|
120
|
+
# `ArgumentError` is `Store#replace` refusing a payload that parsed but is
|
|
121
|
+
# not a manifest. Both are the sender's mistake and both are a 400.
|
|
122
|
+
json(400, { 'ok' => false, 'error' => e.message })
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
# The raw bytes, unparsed, or `nil` when the push is over the limit.
|
|
126
|
+
#
|
|
127
|
+
# Read with a cap rather than trusting `CONTENT_LENGTH`: the header is what
|
|
128
|
+
# the sender claims and the cap is what we will actually hold. Both are
|
|
129
|
+
# checked, because the header lets us refuse a large push without reading it
|
|
130
|
+
# and the cap catches a chunked one that declares no length at all.
|
|
131
|
+
def read_body(env)
|
|
132
|
+
declared = env['CONTENT_LENGTH'].to_s
|
|
133
|
+
return nil if !declared.empty? && declared.to_i > @max_bytes
|
|
134
|
+
|
|
135
|
+
input = env['rack.input']
|
|
136
|
+
# Rack 3 makes `rack.input` optional, and a GET that reached here has none.
|
|
137
|
+
return ''.b if input.nil?
|
|
138
|
+
|
|
139
|
+
# One byte past the limit, so "exactly at the limit" and "over it" are
|
|
140
|
+
# distinguishable without reading the rest of an enormous body.
|
|
141
|
+
raw = input.read(@max_bytes + 1).to_s
|
|
142
|
+
return nil if raw.bytesize > @max_bytes
|
|
143
|
+
|
|
144
|
+
raw
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
def json(status, payload)
|
|
148
|
+
body = JSON.generate(payload)
|
|
149
|
+
[status,
|
|
150
|
+
{ 'content-type' => 'application/json',
|
|
151
|
+
'content-length' => body.bytesize.to_s,
|
|
152
|
+
'cache-control' => 'no-store',
|
|
153
|
+
'x-robots-tag' => 'noindex, nofollow' },
|
|
154
|
+
[body]]
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
end
|