constile_gate 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE +21 -0
- data/README.md +226 -0
- data/constile_gate.gemspec +21 -0
- data/lib/constile_gate/agents.rb +50 -0
- data/lib/constile_gate/charset.rb +427 -0
- data/lib/constile_gate/charset_tables.rb +284 -0
- data/lib/constile_gate/gate.rb +612 -0
- data/lib/constile_gate/http.rb +235 -0
- data/lib/constile_gate/js.rb +99 -0
- data/lib/constile_gate/page_meta.rb +83 -0
- data/lib/constile_gate/rack.rb +371 -0
- data/lib/constile_gate/railtie.rb +17 -0
- data/lib/constile_gate/url.rb +208 -0
- data/lib/constile_gate/version.rb +5 -0
- data/lib/constile_gate.rb +6 -0
- metadata +62 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 51879f686a8d88a87ab113a5b4b2aaf8747dd8e2092deccd9368c1c714ed75a7
|
|
4
|
+
data.tar.gz: d902ec0beb8a223e0770269d0397b439f168c8a189971d04b408cda82f3bb226
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 93d2b9c2dd1d10276877dc9932d678477b6b3d1a948811eb620e591ad4fc0ae9a4f7be91bf01749fe5fc114ee6326773c714d4b3d8d88be409657b965b3bbffc
|
|
7
|
+
data.tar.gz: 2a2a7ec02af0b62dc27cccaa1e67c23b35bf66392d9cbb72394245b513e767d34215563f669bd30f2efef0407dfe0d1af95e0e607222570544172eda4abfa0d3
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 CCC2C Inc.
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,226 @@
|
|
|
1
|
+
# Constile gate for Ruby
|
|
2
|
+
|
|
3
|
+
The Constile gate runs in front of your site. When an AI agent fetches
|
|
4
|
+
one of your pages for a person, the gate serves the Constile offer
|
|
5
|
+
instead of the article. People, search engines, and crawlers get your pages as
|
|
6
|
+
usual.
|
|
7
|
+
|
|
8
|
+
This gem is the gate as Rack middleware. It works in Rails, Sinatra, and other
|
|
9
|
+
Rack apps on Ruby 2.7 or later, and it uses only Ruby's standard library.
|
|
10
|
+
|
|
11
|
+
## Before you begin
|
|
12
|
+
|
|
13
|
+
You need:
|
|
14
|
+
|
|
15
|
+
- A Rack app on Ruby 2.7 or later.
|
|
16
|
+
- Your site ID, site secret, and the network URL. Find them on your site page in
|
|
17
|
+
the publisher dashboard.
|
|
18
|
+
|
|
19
|
+
The gate answers at the root of your domain, at `/.well-known/constile` and
|
|
20
|
+
`/sponsored/`. Your app must serve the whole domain, not a subpath such as
|
|
21
|
+
`/blog`.
|
|
22
|
+
|
|
23
|
+
## Install the gem
|
|
24
|
+
|
|
25
|
+
Add it to your `Gemfile`, then run `bundle install`:
|
|
26
|
+
|
|
27
|
+
```ruby
|
|
28
|
+
gem "constile_gate", "~> 0.6"
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
If your app doesn't use Bundler, run `gem install constile_gate` on each server
|
|
32
|
+
instead.
|
|
33
|
+
|
|
34
|
+
To use the copy from your site page instead, download
|
|
35
|
+
`constile_gate-0.6.0.gem`, unpack it into your app with
|
|
36
|
+
`gem unpack constile_gate-0.6.0.gem --target vendor/gems`, point your
|
|
37
|
+
`Gemfile` at it with
|
|
38
|
+
`gem "constile_gate", path: "vendor/gems/constile_gate-0.6.0"`, and commit
|
|
39
|
+
`vendor/gems/constile_gate-0.6.0` so your deploys get it too.
|
|
40
|
+
|
|
41
|
+
## Set your site's settings
|
|
42
|
+
|
|
43
|
+
The gate reads its settings from environment variables:
|
|
44
|
+
|
|
45
|
+
| Variable | Value |
|
|
46
|
+
|---|---|
|
|
47
|
+
| `CONSTILE_SITE_ID` | Your site ID. |
|
|
48
|
+
| `CONSTILE_SITE_SECRET` | Your site secret. Keep it out of source control. |
|
|
49
|
+
| `CONSTILE_NETWORK_URL` | The network URL. |
|
|
50
|
+
|
|
51
|
+
If any of the three is missing, the gate passes every request to your app and
|
|
52
|
+
logs one warning when your app starts.
|
|
53
|
+
|
|
54
|
+
You can also pass each setting as an option, which wins over its variable:
|
|
55
|
+
|
|
56
|
+
| Option | Variable | Default | What it does |
|
|
57
|
+
|---|---|---|---|
|
|
58
|
+
| `site_id` | `CONSTILE_SITE_ID` | | Your site ID. |
|
|
59
|
+
| `site_secret` | `CONSTILE_SITE_SECRET` | | Your site secret. |
|
|
60
|
+
| `network_url` | `CONSTILE_NETWORK_URL` | | The network URL. |
|
|
61
|
+
| `timeout` | `CONSTILE_TIMEOUT` | `5` | Seconds the gate waits for each call to the network, from connecting to the last byte. |
|
|
62
|
+
| `config_retry` | `CONSTILE_CONFIG_RETRY` | `30` | Seconds the gate leaves the network alone after it fails to give your site's configuration. |
|
|
63
|
+
| `trusted_proxies` | `CONSTILE_TRUSTED_PROXIES` | none | Addresses or ranges of your proxies, such as `10.0.0.0/8`. See [Behind a proxy](#behind-a-proxy-or-load-balancer). |
|
|
64
|
+
| `enabled` | | `true` | Set it to `false` to turn the gate off without a warning. |
|
|
65
|
+
| | `CONSTILE_DISABLED` | | Set it to `1` to turn the gate off: every request goes to your app unchanged, and nothing goes to the network. Read on every request. |
|
|
66
|
+
| `platform` | | `"rack"`, or `"rails"` in Rails | The name your site page and the install check show. |
|
|
67
|
+
|
|
68
|
+
## Rails
|
|
69
|
+
|
|
70
|
+
The gem adds the gate as the first middleware in your app, in front of
|
|
71
|
+
everything else. After you install the gem and set your settings, there's
|
|
72
|
+
nothing more to add.
|
|
73
|
+
|
|
74
|
+
To confirm that the gate is first, run:
|
|
75
|
+
|
|
76
|
+
```sh
|
|
77
|
+
bin/rails middleware
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
The first line is `use ConstileGate::Rack`.
|
|
81
|
+
|
|
82
|
+
To set options, use `config.constile_gate` in `config/application.rb` or in a file
|
|
83
|
+
under `config/environments/`:
|
|
84
|
+
|
|
85
|
+
```ruby
|
|
86
|
+
# config/environments/production.rb
|
|
87
|
+
config.constile_gate.trusted_proxies = ["10.0.0.0/8"]
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
To turn the gate off in development and test, add this to
|
|
91
|
+
`config/environments/development.rb` and `config/environments/test.rb`:
|
|
92
|
+
|
|
93
|
+
```ruby
|
|
94
|
+
config.constile_gate.enabled = false
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Sinatra
|
|
98
|
+
|
|
99
|
+
Add the gate in `config.ru`, in front of your app:
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
# config.ru
|
|
103
|
+
require "constile_gate"
|
|
104
|
+
require_relative "app"
|
|
105
|
+
|
|
106
|
+
use ConstileGate::Rack, platform: "sinatra"
|
|
107
|
+
run App
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
If your app has no `config.ru`, add `use ConstileGate::Rack, platform: "sinatra"` to
|
|
111
|
+
the app itself. It works the same way, but it runs after Sinatra's own
|
|
112
|
+
middleware, such as sessions, so `config.ru` is the better place.
|
|
113
|
+
|
|
114
|
+
## Any other Rack app
|
|
115
|
+
|
|
116
|
+
Add the gate first in `config.ru`, before any other middleware, so it sees each
|
|
117
|
+
request as it arrived:
|
|
118
|
+
|
|
119
|
+
```ruby
|
|
120
|
+
# config.ru
|
|
121
|
+
require "constile_gate"
|
|
122
|
+
|
|
123
|
+
use ConstileGate::Rack
|
|
124
|
+
run MyApp
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
For another Rack framework, add `ConstileGate::Rack` as the first middleware the way
|
|
128
|
+
that framework adds Rack middleware.
|
|
129
|
+
|
|
130
|
+
## Paywalled content
|
|
131
|
+
|
|
132
|
+
Gate 0.6.0 and later can fetch a paywalled article with the release
|
|
133
|
+
instructions the network sends, so an agent gets the full text and the offer
|
|
134
|
+
fits it. The network gives the gate a page address on your site's own host
|
|
135
|
+
and, optionally, headers to send: `Cookie`, `Authorization`,
|
|
136
|
+
`Constile-Paywall-Key`, or a header starting with `X-`. The gate ignores an
|
|
137
|
+
address on another host and drops other headers. It strips `Set-Cookie` from
|
|
138
|
+
what it releases and never logs the headers. To set up a method, open your
|
|
139
|
+
site's page in the dashboard and choose a content API, a paywall rule, or a
|
|
140
|
+
Constile login. A paywalled page gets offers only after the method works and
|
|
141
|
+
the gate is 0.6.0 or later.
|
|
142
|
+
|
|
143
|
+
## Behind a proxy or load balancer
|
|
144
|
+
|
|
145
|
+
The gate sends the network the address of an agent that claims to be a
|
|
146
|
+
registered agent, so the network can check it against the agent's published
|
|
147
|
+
addresses. The gate reads that address from `REMOTE_ADDR`. It never believes
|
|
148
|
+
`X-Forwarded-For`, because any client can send one.
|
|
149
|
+
|
|
150
|
+
If your app runs behind a proxy or load balancer, `REMOTE_ADDR` is the proxy's
|
|
151
|
+
address. To read the visitor's address, list your proxies in `trusted_proxies`.
|
|
152
|
+
The gate then reads `X-Forwarded-For` from right to left and takes the first
|
|
153
|
+
address that isn't one of your proxies.
|
|
154
|
+
|
|
155
|
+
```ruby
|
|
156
|
+
use ConstileGate::Rack, trusted_proxies: ["10.0.0.0/8", "192.0.2.10"]
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
From one of your trusted proxies, the gate also reads the page's scheme and host
|
|
160
|
+
from `X-Forwarded-Proto` and `X-Forwarded-Host`. From any other client, it
|
|
161
|
+
ignores them. If your proxy terminates TLS, list it in `trusted_proxies`, so the
|
|
162
|
+
gate knows the page is `https`.
|
|
163
|
+
|
|
164
|
+
## Caches in front of your app
|
|
165
|
+
|
|
166
|
+
A cache in front of your app, such as a CDN or Varnish, can serve a cached page
|
|
167
|
+
to an agent without asking your app. That agent then gets the article without an
|
|
168
|
+
offer. To avoid that, configure the cache to send requests to your app when the
|
|
169
|
+
`User-Agent` names an AI agent, or to vary on `User-Agent`. The gate marks each
|
|
170
|
+
offer `Cache-Control: no-store`, so a cache never serves an offer to a person.
|
|
171
|
+
|
|
172
|
+
## Check the install
|
|
173
|
+
|
|
174
|
+
To check the install, open `https://your-site.example/.well-known/constile`. The
|
|
175
|
+
gate answers with your site ID, `"gate_version": "constile-gate 0.6.0"`, and the
|
|
176
|
+
platform. The install check on your site page reads the same file.
|
|
177
|
+
|
|
178
|
+
## How the gate behaves
|
|
179
|
+
|
|
180
|
+
- **Agents.** For a registered agent's request to an eligible page, the gate gets
|
|
181
|
+
the page from your app, sends its title, description, and keywords to the
|
|
182
|
+
network, and serves the offer in its place.
|
|
183
|
+
- **Everyone else.** Every other response goes back from your app untouched. The
|
|
184
|
+
gate reads a response only for an agent's request.
|
|
185
|
+
- **Settings from the network.** The gate gets your site's configuration from the
|
|
186
|
+
network and keeps it for as long as the network says, from 10 seconds to an
|
|
187
|
+
hour. Apart from that, people's requests never reach the network.
|
|
188
|
+
- **Outages.** If the network can't be reached, the gate serves your page as if
|
|
189
|
+
the gate weren't there. Each call to the network takes one `timeout` at most,
|
|
190
|
+
from connecting to the last byte, with no retries. On Ruby 2.7, a DNS lookup
|
|
191
|
+
that hangs can take longer, because Ruby 2.7 can't interrupt the system
|
|
192
|
+
resolver. When a call for your site's configuration fails, the gate
|
|
193
|
+
passes every request through without asking the network for `config_retry`
|
|
194
|
+
seconds, then asks again. The `/sponsored/` callbacks don't fall back to your
|
|
195
|
+
page: they release content only against a recorded commitment, so they answer
|
|
196
|
+
with an error instead.
|
|
197
|
+
- **Errors.** If the gate itself fails, it logs the error and serves your app's
|
|
198
|
+
page, as it does in an outage. The `/sponsored/` routes and the seller file
|
|
199
|
+
answer 503 instead. Errors your app raises go on as they would without the
|
|
200
|
+
gate.
|
|
201
|
+
- **Character sets.** The gate reads a page's title and keywords in the charset
|
|
202
|
+
its `Content-Type` names, or UTF-8 when it names none, and passes a page
|
|
203
|
+
through byte for byte.
|
|
204
|
+
- **Released articles.** When the network grants an agent access, the gate gets
|
|
205
|
+
the article from your app with a fresh request that carries none of the
|
|
206
|
+
agent's headers, cookies, or session, and drops any cookies your app sets on
|
|
207
|
+
it.
|
|
208
|
+
|
|
209
|
+
### Limits
|
|
210
|
+
|
|
211
|
+
- Under Rack 3, a page your app streams (a body that has neither `to_ary` nor
|
|
212
|
+
`to_path`) gets no offer, because Rack 3 doesn't let middleware read it. The
|
|
213
|
+
page goes to the agent as usual. Under Rack 2, the gate reads streamed pages.
|
|
214
|
+
- For an agent's request, the gate asks your app for the page without
|
|
215
|
+
compression, so `Rack::Deflater` doesn't get in the way. If your app
|
|
216
|
+
compresses the page anyway, the gate reads gzip and deflate. A page compressed
|
|
217
|
+
another way, such as Brotli, still gets an offer, but the network sees only
|
|
218
|
+
its path, not its title or keywords.
|
|
219
|
+
|
|
220
|
+
## Support
|
|
221
|
+
|
|
222
|
+
Write to <hello@constile.app>, or see the [setup guides](https://constile.app/setup).
|
|
223
|
+
|
|
224
|
+
## License
|
|
225
|
+
|
|
226
|
+
MIT. See LICENSE.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
require_relative "lib/constile_gate/version"
|
|
2
|
+
|
|
3
|
+
Gem::Specification.new do |spec|
|
|
4
|
+
spec.name = "constile_gate"
|
|
5
|
+
spec.version = ConstileGate::VERSION
|
|
6
|
+
spec.authors = ["CCC2C Inc."]
|
|
7
|
+
spec.email = ["hello@constile.app"]
|
|
8
|
+
spec.summary = "The Constile gate as Rack middleware, for Rails, Sinatra, and any Rack app."
|
|
9
|
+
spec.description = "Serves the Constile offer to AI agents that fetch a page for a person, " \
|
|
10
|
+
"forwards the offer's callbacks to the Constile network, and passes everyone else through untouched."
|
|
11
|
+
spec.license = "MIT"
|
|
12
|
+
spec.homepage = "https://constile.app"
|
|
13
|
+
spec.metadata = {
|
|
14
|
+
"documentation_uri" => "https://constile.app/setup/rack",
|
|
15
|
+
"rubygems_mfa_required" => "true",
|
|
16
|
+
}
|
|
17
|
+
spec.required_ruby_version = ">= 2.7"
|
|
18
|
+
# The gemspec ships too, so an unpacked copy works as a Bundler path gem.
|
|
19
|
+
spec.files = Dir.chdir(__dir__) { Dir["lib/**/*.rb", "README.md", "LICENSE", "constile_gate.gemspec"].sort }
|
|
20
|
+
spec.require_paths = ["lib"]
|
|
21
|
+
end
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
require "constile_gate/js"
|
|
2
|
+
require "constile_gate/url"
|
|
3
|
+
|
|
4
|
+
module ConstileGate
|
|
5
|
+
# Who is asking for a page. A port of classifyClient and signatureAgentOrigin in
|
|
6
|
+
# packages/core/src/agents.ts. Registry entries are the network's config agents:
|
|
7
|
+
# Hashes with string keys (id, token, kind, tier, signature_agent).
|
|
8
|
+
module Agents
|
|
9
|
+
# The reference's /bot\b|crawler|spider|headless|scraper/, with an ASCII word boundary.
|
|
10
|
+
GENERIC_BOT = /bot(?![A-Za-z0-9_])|crawler|spider|headless|scraper/.freeze
|
|
11
|
+
QUOTED_HTTPS = %r{"(https://[^"]+)"}.freeze
|
|
12
|
+
|
|
13
|
+
module_function
|
|
14
|
+
|
|
15
|
+
# The origin named in a Signature-Agent header, in either the dictionary or the quoted-string form.
|
|
16
|
+
def signature_agent_origin(header)
|
|
17
|
+
m = QUOTED_HTTPS.match(header || "")
|
|
18
|
+
return nil unless m
|
|
19
|
+
|
|
20
|
+
URL.parse(m[1]).origin
|
|
21
|
+
rescue ArgumentError, RangeError
|
|
22
|
+
nil
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Returns { kind: "agent" | "crawler" | "search" | "browser" | "unknown_client", agent: entry or nil }.
|
|
26
|
+
def classify_client(headers, registry)
|
|
27
|
+
ua = JS.downcase(headers.get("user-agent") || "")
|
|
28
|
+
# Longest token first, so "claude-searchbot" wins over a shorter token it contains. Ties keep registry order.
|
|
29
|
+
by_length = registry.each_with_index.sort_by { |a, i| [-JS.utf16_length(a["token"].to_s), i] }.map(&:first)
|
|
30
|
+
hit = by_length.find { |a| JS.truthy?(a["token"]) && ua.include?(JS.downcase(a["token"])) }
|
|
31
|
+
if hit
|
|
32
|
+
return { kind: "agent", agent: hit } if hit["kind"] == "user_agent"
|
|
33
|
+
|
|
34
|
+
return { kind: hit["kind"], agent: hit }
|
|
35
|
+
end
|
|
36
|
+
# A signed request names its agent in Signature-Agent. The network checks the signature;
|
|
37
|
+
# an agent recognized only this way gets no offer unless the signature verifies.
|
|
38
|
+
origin = headers.has?("signature") ? signature_agent_origin(headers.get("signature-agent")) : nil
|
|
39
|
+
signer = origin && registry.find do |a|
|
|
40
|
+
a["kind"] == "user_agent" && JS.truthy?(a["signature_agent"]) && signature_agent_origin(%("#{a["signature_agent"]}")) == origin
|
|
41
|
+
end
|
|
42
|
+
return { kind: "agent", agent: signer } if signer
|
|
43
|
+
return { kind: "unknown_client", agent: nil } if ua.empty?
|
|
44
|
+
# Never show an offer to a person: any browser-looking client without a bot marker is a browser.
|
|
45
|
+
return { kind: "browser", agent: nil } if ua.start_with?("mozilla/") && !ua.match?(GENERIC_BOT)
|
|
46
|
+
|
|
47
|
+
{ kind: "unknown_client", agent: nil }
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|