drive 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +34 -0
- data/README.md +15 -8
- data/app/controllers/concerns/rate_limited.rb +34 -4
- data/app/controllers/drive/errors_controller.rb +33 -0
- data/app/models/crawler.rb +21 -5
- data/app/models/designation.rb +15 -0
- data/app/models/n_number.rb +29 -0
- data/app/models/serial.rb +11 -0
- data/app/views/drive/errors/show.html.erb +15 -0
- data/app/views/layouts/drive/errors.html.erb +26 -0
- data/lib/drive/cloudflare.rb +67 -0
- data/lib/drive/engine.rb +23 -2
- data/lib/drive/inflections.rb +9 -0
- data/lib/drive/version.rb +1 -1
- data/lib/drive.rb +5 -2
- metadata +13 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d7864c84eb75ad2394534293c1a9f3232eaff9b7a69352d84d8e64605aba56ca
|
|
4
|
+
data.tar.gz: 6545371dbd8af274b39307dc8139b1cb21ad76f49f4ef15a0ff7478b791eb8fe
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fc337a89d407e9b9ac35ae10febdffb7d202a056bf6403f6533e0cc289965917b74543fa625ee56a15718a9c0ddd818f03fc1fd7cca9cd02e139cf077033aaf9
|
|
7
|
+
data.tar.gz: 3d3b30536d3427158fd118908f8b9ab98f82e3a876d835cb7f62846c1c250bbda0e9a4c680f890de1c79d8ec30f66eb01a08f5ee13b30afbe1d88498c14af057
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,40 @@ For more information about changelogs, check [Keep a Changelog](http://keepachan
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## 0.6.0 - 2026-10-09
|
|
10
|
+
|
|
11
|
+
* [BREAKING] `include RateLimited` sets no limit by itself: a controller calls `rate_limited`
|
|
12
|
+
|
|
13
|
+
`rate_limited` keeps 20 a minute and 200 a day; `rate_limited minute: 120, day: nil,
|
|
14
|
+
scope: 'public'` sets an app's own, counted across every controller of the scope. A client over
|
|
15
|
+
the limit is told so in words, with status 429, instead of an error the app would report. A host
|
|
16
|
+
may say it its own way by defining `refuse_flood(minute:, day:)`, and `crawler?` is a helper.
|
|
17
|
+
|
|
18
|
+
* [FEATURE] `Crawler` also believes Cloudflare's verified bots, where `EDGE_KEY` is set
|
|
19
|
+
|
|
20
|
+
A request whose `X-Edge-Key` is `EDGE_KEY` and whose `X-Verified-Bot` is `true` is a crawler.
|
|
21
|
+
While a search engine's ranges cannot be fetched, no crawler naming it is believed, and the
|
|
22
|
+
request is served rather than failing.
|
|
23
|
+
|
|
24
|
+
* [FEATURE] The same error pages on every app
|
|
25
|
+
|
|
26
|
+
`Drive::ErrorsController` answers 400, 404, 422 and 500 at the paths Rails keeps for them, and
|
|
27
|
+
`/down.html`, as `config.exceptions_app` unless the app sets its own. Delete the app's
|
|
28
|
+
`public/400.html`, `404.html`, `422.html` and `500.html`: a static page wins over the route.
|
|
29
|
+
|
|
30
|
+
* [FEATURE] What every app behind Cloudflare sets alike
|
|
31
|
+
|
|
32
|
+
Cloudflare's addresses (`Drive::Cloudflare::RANGES`) are trusted proxies, unless the app sets
|
|
33
|
+
its own. `Drive::Cloudflare` purges pages by tag or everything, and warms pages, with
|
|
34
|
+
`CLOUDFLARE_ZONE_ID` and `CLOUDFLARE_API_TOKEN`. "aircraft" is its own plural, and FAA, DRS,
|
|
35
|
+
NTSB and XML are acronyms (a constant spelled `Drs` is now `DRS`).
|
|
36
|
+
|
|
37
|
+
* [FEATURE] `NNumber`, `Designation` and `Serial`
|
|
38
|
+
|
|
39
|
+
An N-number as the FAA writes it (`normalize`, `parse`, `valid?`, `display`, `to_param`); a
|
|
40
|
+
designation's and a part number's key (`Designation.key`, `Designation.part_key`), the same as
|
|
41
|
+
the SQL the refinery keys its tables with; a serial number's prefix and number (`Serial.parts`).
|
|
42
|
+
|
|
9
43
|
## 0.5.0 - 2026-10-07
|
|
10
44
|
|
|
11
45
|
* [FEATURE] `RateLimited` no longer limits Claude's, ChatGPT's or Perplexity's search crawler
|
data/README.md
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
# Drive
|
|
2
2
|
|
|
3
|
-
A Rails engine for
|
|
4
|
-
|
|
3
|
+
A Rails engine for the easyFAA apps. Each keeps a copy of what the FAA or the NTSB publishes,
|
|
4
|
+
changing only the rows the source did and counting them, and each sits behind Cloudflare and
|
|
5
|
+
shows the same error pages.
|
|
5
6
|
|
|
6
7
|
- **`Mirrored`**: a table copying one of the source's tables field for field, as text. A refresh
|
|
7
8
|
stages the source's table, then inserts, updates and deletes only the rows the source did.
|
|
@@ -9,10 +10,16 @@ files, a monthly database — changing only the rows the source did, and countin
|
|
|
9
10
|
- **`Run`**: one refresh, in one transaction, the only one running, with how many rows it
|
|
10
11
|
inserted, updated and deleted.
|
|
11
12
|
- **`AccessFile`**: stages a table of a Microsoft Access file, read with `mdb-export`.
|
|
12
|
-
- **`RateLimited`**: 20
|
|
13
|
-
engine's own crawler has no limit.
|
|
14
|
-
- **`Crawler`**: whether a request comes from
|
|
15
|
-
or
|
|
13
|
+
- **`RateLimited`**: `rate_limited` holds an address to 20 pages a minute and 200 a day, or the
|
|
14
|
+
app's own limits, and tells it so in words. A search engine's own crawler has no limit.
|
|
15
|
+
- **`Crawler`**: whether a request comes from a search engine's crawler: Cloudflare vouches for
|
|
16
|
+
it (where `EDGE_KEY` is set), or its address is in the ranges Google, Bing, Apple, Claude,
|
|
17
|
+
ChatGPT or Perplexity publish, fetched once a day.
|
|
18
|
+
- **`Drive::Cloudflare`**: purges pages by tag or everything, and warms pages. Its addresses are
|
|
19
|
+
the app's trusted proxies.
|
|
20
|
+
- **Error pages**: the same page for 400, 404, 422 and 500 on every app, and `/down.html`.
|
|
21
|
+
- **`NNumber`**, **`Designation`**, **`Serial`**: an N-number, a designation's key and a serial
|
|
22
|
+
number, as every app reads them.
|
|
16
23
|
|
|
17
24
|
It needs PostgreSQL.
|
|
18
25
|
|
|
@@ -24,10 +31,10 @@ gem install drive
|
|
|
24
31
|
|
|
25
32
|
```ruby
|
|
26
33
|
# Gemfile
|
|
27
|
-
gem 'drive', '~> 0.
|
|
34
|
+
gem 'drive', '~> 0.6.0'
|
|
28
35
|
```
|
|
29
36
|
|
|
30
|
-
`~> 0.
|
|
37
|
+
`~> 0.6.0` takes every 0.6.x and stops short of 0.7, which may break what 0.6 does.
|
|
31
38
|
|
|
32
39
|
Then copy its migrations, of runs, and run them:
|
|
33
40
|
|
|
@@ -1,10 +1,40 @@
|
|
|
1
|
-
# The limits of a page anybody may ask for, by the address asking
|
|
2
|
-
#
|
|
1
|
+
# The limits of a page anybody may ask for, by the address asking, which a search engine's own
|
|
2
|
+
# crawler (Crawler) is never held to, so it can list every page. Refused, a client is told why in
|
|
3
|
+
# words, with status 429, rather than with an error the app would report.
|
|
4
|
+
#
|
|
5
|
+
# rate_limited # 20 a minute and 200 a day
|
|
6
|
+
# rate_limited minute: 120, day: nil, scope: 'public'
|
|
3
7
|
module RateLimited
|
|
4
8
|
extend ActiveSupport::Concern
|
|
5
9
|
|
|
6
10
|
included do
|
|
7
|
-
|
|
8
|
-
|
|
11
|
+
helper_method :crawler? if respond_to? :helper_method
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
class_methods do
|
|
15
|
+
# @param minute [Integer, nil] pages an address may ask for a minute.
|
|
16
|
+
# @param day [Integer, nil] pages an address may ask for a day.
|
|
17
|
+
# @param scope [String, nil] the controllers counted together; each its own if nil.
|
|
18
|
+
def rate_limited(minute: 20, day: 200, scope: nil)
|
|
19
|
+
{ minute: [minute, 1.minute], day: [day, 1.day] }.each do |name, (to, within)|
|
|
20
|
+
next unless to
|
|
21
|
+
|
|
22
|
+
rate_limit(to:, within:, name: name.to_s, scope:, unless: -> { crawler? },
|
|
23
|
+
with: -> { refuse_flood minute:, day: })
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
# Whether a search engine vouches for this client, asked once a request.
|
|
31
|
+
def crawler? = defined?(@crawler) ? @crawler : @crawler = Crawler.verified?(request)
|
|
32
|
+
|
|
33
|
+
# What a client asking too often is told. A host may say it its own way.
|
|
34
|
+
def refuse_flood(minute:, day:)
|
|
35
|
+
limits = { minute:, day: }.compact.map { |per, count| "#{count} a #{per}" }
|
|
36
|
+
render plain: 'Too many requests. Each address may ask for so many pages: ' \
|
|
37
|
+
"#{limits.join ' and '}.\n",
|
|
38
|
+
status: :too_many_requests
|
|
9
39
|
end
|
|
10
40
|
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
module Drive
|
|
2
|
+
# The pages every app shows for 400, 404, 422 and 500 (config.exceptions_app, set by the
|
|
3
|
+
# engine). Plain ActionController::Base, not the host's ApplicationController: a 500 is exactly
|
|
4
|
+
# the moment the database or a session might be what broke, so nothing here may depend on either.
|
|
5
|
+
class ErrorsController < ActionController::Base # rubocop:disable Rails/ApplicationController
|
|
6
|
+
layout 'drive/errors'
|
|
7
|
+
|
|
8
|
+
# Heading and sentence by status.
|
|
9
|
+
MESSAGES = {
|
|
10
|
+
400 => ['Bad request', 'The request could not be understood.'],
|
|
11
|
+
404 => ['Page not found', 'You may have mistyped the address, or the page may have moved.'],
|
|
12
|
+
422 => ['Can’t process that', 'The change was rejected, often because a page had been ' \
|
|
13
|
+
'left open too long. Going back and trying again usually ' \
|
|
14
|
+
'fixes it.',],
|
|
15
|
+
500 => ['Something went wrong', 'We’ve been notified and are looking into it. Try again ' \
|
|
16
|
+
'in a moment.',],
|
|
17
|
+
}.freeze
|
|
18
|
+
|
|
19
|
+
# The page for the status the route names, whatever format was asked for.
|
|
20
|
+
def show
|
|
21
|
+
@status = params[:status].to_i
|
|
22
|
+
@heading, @message = MESSAGES.fetch(@status, MESSAGES.fetch(500))
|
|
23
|
+
render status: @status, formats: :html
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
# The page shown while the app is down for maintenance.
|
|
27
|
+
def down
|
|
28
|
+
@heading = 'Down for maintenance'
|
|
29
|
+
@message = 'easyFAA is being updated and will be back in a few minutes.'
|
|
30
|
+
render :show, formats: :html
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
data/app/models/crawler.rb
CHANGED
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
require 'ipaddr'
|
|
2
2
|
require 'net/http'
|
|
3
3
|
|
|
4
|
-
# A search engine's own crawler, told apart from a client that only names one in its user agent
|
|
5
|
-
#
|
|
4
|
+
# A search engine's own crawler, told apart from a client that only names one in its user agent.
|
|
5
|
+
# Either Cloudflare vouches for it (a verified bot, in `X-Verified-Bot`, beside the `X-Edge-Key`
|
|
6
|
+
# only Cloudflare and the app know, so a request that skips Cloudflare cannot forge it), or its
|
|
7
|
+
# address is in the ranges its search engine publishes for it.
|
|
6
8
|
module Crawler
|
|
7
9
|
# The ranges each search engine publishes, by the user agents of its crawlers.
|
|
8
10
|
RANGES = {
|
|
@@ -16,13 +18,24 @@ module Crawler
|
|
|
16
18
|
}.freeze
|
|
17
19
|
|
|
18
20
|
# @param request [ActionDispatch::Request] request a page was asked with.
|
|
19
|
-
# @return [Boolean] whether it came from a crawler
|
|
20
|
-
def self.verified?(request)
|
|
21
|
+
# @return [Boolean] whether it came from a crawler.
|
|
22
|
+
def self.verified?(request) = vouched?(request) || published?(request)
|
|
23
|
+
|
|
24
|
+
# @return [Boolean] whether Cloudflare says it verified the bot, where EDGE_KEY is set.
|
|
25
|
+
def self.vouched?(request)
|
|
26
|
+
key = ENV['EDGE_KEY'].presence or return false
|
|
27
|
+
ActiveSupport::SecurityUtils.secure_compare(request.headers['X-Edge-Key'].to_s, key) &&
|
|
28
|
+
request.headers['X-Verified-Bot'] == 'true'
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# @return [Boolean] whether the user agent names a crawler whose ranges hold the address.
|
|
32
|
+
def self.published?(request)
|
|
21
33
|
url = RANGES.find { |agent, _| request.user_agent.to_s.match? agent }&.last
|
|
22
34
|
url.present? && ranges(url).any? { it.include? request.remote_ip }
|
|
23
35
|
end
|
|
24
36
|
|
|
25
|
-
|
|
37
|
+
# The ranges at that address, fetched once a day; none while it cannot be reached.
|
|
38
|
+
def self.ranges(url)
|
|
26
39
|
Rails.cache.fetch(['crawler', url], expires_in: 1.day) do
|
|
27
40
|
uri = URI url
|
|
28
41
|
body = Net::HTTP.start(uri.host, use_ssl: true, open_timeout: 2, read_timeout: 2) do |http|
|
|
@@ -30,5 +43,8 @@ module Crawler
|
|
|
30
43
|
end
|
|
31
44
|
JSON.parse(body)['prefixes'].map { IPAddr.new it.values.first }
|
|
32
45
|
end
|
|
46
|
+
rescue Net::OpenTimeout, Net::ReadTimeout, SocketError, SystemCallError, JSON::ParserError
|
|
47
|
+
[]
|
|
33
48
|
end
|
|
49
|
+
private_class_method :vouched?, :published?, :ranges
|
|
34
50
|
end
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# A model, engine or part number as every app keys it, so one written two ways is found as one:
|
|
2
|
+
# what the refinery writes (models.designation_key, by the same rule in SQL) and the site reads.
|
|
3
|
+
module Designation
|
|
4
|
+
# @param designation [String, nil] as an AD, a data sheet or the registry writes it.
|
|
5
|
+
# @return [String] upcased, nothing after a parenthesis, letters, digits, slashes and pluses
|
|
6
|
+
# only, and a zero after the leading letters read as the letter O: "O320" for "0-320 (A)".
|
|
7
|
+
def self.key(designation)
|
|
8
|
+
designation.to_s.upcase.sub(/\s*\(.*\z/, '').gsub(%r{[^A-Z0-9/+]}, '')
|
|
9
|
+
.sub(/\A([A-Z]*)0(?=\d)/, '\1O')
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
# @param part [String, nil] a part number as a shop or an AD writes it.
|
|
13
|
+
# @return [String] upcased, without dashes or spaces: "ST07" for "st-07".
|
|
14
|
+
def self.part_key(part) = part.to_s.upcase.delete('- ')
|
|
15
|
+
end
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# A U.S. registration (an N-number) as text: as the FAA writes one, however it was typed.
|
|
2
|
+
module NNumber
|
|
3
|
+
# One the FAA could assign: N, then one to five characters, a digit 1 to 9 first, up to four
|
|
4
|
+
# more digits, and at most two letters last, never I or O.
|
|
5
|
+
ASSIGNABLE = /\AN(?=.{1,5}\z)[1-9]\d{0,4}[A-HJ-NP-Z]{0,2}\z/
|
|
6
|
+
# One as older records write it: a digit 1 to 9 after the N, then any four letters or digits.
|
|
7
|
+
WRITTEN = /\AN[1-9][A-Z0-9]{0,4}\z/
|
|
8
|
+
|
|
9
|
+
# @param typed [String, nil] as somebody typed it: "n-714ad", "714AD".
|
|
10
|
+
# @return [String] with its N, upcased, letters and digits only: "N714AD".
|
|
11
|
+
def self.normalize(typed) = typed.to_s.upcase.delete('^A-Z0-9').sub(/\A(?!N)/, 'N')
|
|
12
|
+
|
|
13
|
+
# @param typed [String, nil]
|
|
14
|
+
# @return [String, nil] normalized, where it reads as a U.S. registration at all ("A6FKF", another
|
|
15
|
+
# country's, does not).
|
|
16
|
+
def self.parse(typed) = normalize(typed).then { it if it.match? WRITTEN }
|
|
17
|
+
|
|
18
|
+
# @param n_number [String] normalized.
|
|
19
|
+
# @return [Boolean] whether the FAA could assign it.
|
|
20
|
+
def self.valid?(n_number) = n_number.to_s.match?(ASSIGNABLE)
|
|
21
|
+
|
|
22
|
+
# @param n_number [String] as stored: "N714AD" or "N-714AD".
|
|
23
|
+
# @return [String] as a page shows it, the N joined to the rest: "N714AD".
|
|
24
|
+
def self.display(n_number) = n_number.to_s.sub(/\AN-?/, 'N')
|
|
25
|
+
|
|
26
|
+
# @param n_number [String] normalized.
|
|
27
|
+
# @return [String] in an address, without its N: "714AD".
|
|
28
|
+
def self.to_param(n_number) = n_number.to_s.delete_prefix('N')
|
|
29
|
+
end
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# A serial number as the registry or a data sheet writes it, cut into what comes before its last
|
|
2
|
+
# run of digits and that number, so that two are compared as the sheets mean them: "15279406" is
|
|
3
|
+
# within "S/N 15279406 thru 15285594", and "A-100" within "S/N A-100 and up".
|
|
4
|
+
module Serial
|
|
5
|
+
# @param serial [String, nil]
|
|
6
|
+
# @return [Array(String, Integer), Array(nil, nil)] the prefix, dashes dropped, and the number.
|
|
7
|
+
def self.parts(serial)
|
|
8
|
+
match = serial.to_s.upcase.delete(' ').match(/\A(.*?)(\d+)\z/) or return [nil, nil]
|
|
9
|
+
[match[1].delete('-'), match[2].to_i]
|
|
10
|
+
end
|
|
11
|
+
end
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
<% content_for :title, "#{@heading} | easyFAA" %>
|
|
2
|
+
<h1>
|
|
3
|
+
<svg viewBox="0 0 64 64" aria-hidden="true">
|
|
4
|
+
<g stroke="currentColor" stroke-width="2.5" stroke-linejoin="round">
|
|
5
|
+
<polygon points="60,8 4,30 25,37" fill="#ffffff"/>
|
|
6
|
+
<polygon points="60,8 25,37 31,57" fill="#cfe0f3"/>
|
|
7
|
+
</g>
|
|
8
|
+
<g stroke="#7a93b3" stroke-width="1.6" stroke-linecap="round">
|
|
9
|
+
<line x1="16" y1="28.5" x2="36" y2="21"/><line x1="21" y1="31.5" x2="44" y2="22.5"/><line x1="27" y1="33" x2="46" y2="25.5"/>
|
|
10
|
+
</g>
|
|
11
|
+
<path d="M10 44 C14 50 18 53 24 55" fill="none" stroke="currentColor" stroke-width="2" stroke-dasharray="3 3" stroke-linecap="round"/>
|
|
12
|
+
</svg>
|
|
13
|
+
<%= @heading %>
|
|
14
|
+
</h1>
|
|
15
|
+
<p><%= @message %></p>
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
<!doctype html>
|
|
2
|
+
<html lang="en">
|
|
3
|
+
<head>
|
|
4
|
+
<meta charset="utf-8">
|
|
5
|
+
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
6
|
+
<meta name="color-scheme" content="light dark">
|
|
7
|
+
<meta name="robots" content="noindex">
|
|
8
|
+
<title><%= content_for(:title) || "easyFAA" %></title>
|
|
9
|
+
<style>
|
|
10
|
+
:root{color-scheme:light;--bg:#ffffff;--ink:#15202e;--muted:#50575d;--primary:#1a70d0}
|
|
11
|
+
@media (prefers-color-scheme: dark){:root{color-scheme:dark;--bg:#131415;--ink:#f3f5f8;--muted:#abb2b8;--primary:#689dff}}
|
|
12
|
+
*{box-sizing:border-box}
|
|
13
|
+
body{margin:0;min-height:100svh;display:flex;flex-direction:column;align-items:center;justify-content:center;background:var(--bg);color:var(--ink);font:500 17px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;text-align:center;padding:40px 24px}
|
|
14
|
+
a{color:var(--primary)}
|
|
15
|
+
h1{display:flex;align-items:center;justify-content:center;font-size:clamp(1.5rem,5.5vw,2.25rem);margin:0}
|
|
16
|
+
h1 svg{width:1.1em;height:1.1em;margin-right:0.3em;flex-shrink:0}
|
|
17
|
+
p{color:var(--muted);margin:18px 0 0;max-width:32rem}
|
|
18
|
+
footer{margin-top:40px;font-size:15px;color:var(--muted)}
|
|
19
|
+
footer a{color:inherit;text-decoration:none}
|
|
20
|
+
</style>
|
|
21
|
+
</head>
|
|
22
|
+
<body>
|
|
23
|
+
<%= yield %>
|
|
24
|
+
<footer><a href="https://easyfaa.com/">easyFAA</a></footer>
|
|
25
|
+
</body>
|
|
26
|
+
</html>
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
require 'ipaddr'
|
|
2
|
+
require 'net/http'
|
|
3
|
+
|
|
4
|
+
module Drive
|
|
5
|
+
# The Cloudflare zone in front of every easyFAA app: its addresses, and what it keeps of the
|
|
6
|
+
# pages, forgotten once a night's data has changed them. Nothing is purged where the zone is
|
|
7
|
+
# not named (CLOUDFLARE_ZONE_ID and CLOUDFLARE_API_TOKEN), as in development and test.
|
|
8
|
+
module Cloudflare
|
|
9
|
+
# The addresses Cloudflare reaches an app from, as https://www.cloudflare.com/ips lists them.
|
|
10
|
+
RANGES = %w[
|
|
11
|
+
173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18
|
|
12
|
+
108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17
|
|
13
|
+
162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
|
|
14
|
+
2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32 2405:8100::/32
|
|
15
|
+
2a06:98c0::/29 2c0f:f248::/32
|
|
16
|
+
].map { IPAddr.new it }.freeze
|
|
17
|
+
# Cloudflare takes up to a hundred tags a request.
|
|
18
|
+
BATCH = 100
|
|
19
|
+
|
|
20
|
+
# @return [Boolean] whether a zone and a token were given.
|
|
21
|
+
def self.configured? = zone.present? && token.present?
|
|
22
|
+
|
|
23
|
+
# Forgets the pages carrying any of these tags, a hundred tags a request.
|
|
24
|
+
# @param tags [Array<String>]
|
|
25
|
+
# @return [void]
|
|
26
|
+
def self.purge(tags) = tags.each_slice(BATCH) { post tags: it }
|
|
27
|
+
|
|
28
|
+
# Forgets every page the zone holds, of every app in it.
|
|
29
|
+
# @return [void]
|
|
30
|
+
def self.purge_everything = post(purge_everything: true)
|
|
31
|
+
|
|
32
|
+
# Asks the zone for each address, which is what makes it keep the answer, so the first reader
|
|
33
|
+
# of the morning is not the one who waits.
|
|
34
|
+
# @param urls [Array<String>]
|
|
35
|
+
# @return [Integer] how many were answered.
|
|
36
|
+
def self.warm(urls)
|
|
37
|
+
urls.count do |url|
|
|
38
|
+
uri = URI url
|
|
39
|
+
get = Net::HTTP::Get.new(uri, 'User-Agent' => 'easyfaa-cache-warmer')
|
|
40
|
+
options = { use_ssl: uri.scheme == 'https', read_timeout: 60 }
|
|
41
|
+
Net::HTTP.start(uri.host, uri.port, **options) { it.request(get).is_a? Net::HTTPSuccess }
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# Asks Cloudflare to purge, raising when it says no.
|
|
46
|
+
def self.post(body)
|
|
47
|
+
uri = URI "https://api.cloudflare.com/client/v4/zones/#{zone}/purge_cache"
|
|
48
|
+
response = Net::HTTP.post uri, body.to_json, 'Authorization' => "Bearer #{token}",
|
|
49
|
+
'Content-Type' => 'application/json'
|
|
50
|
+
return if response.is_a? Net::HTTPSuccess
|
|
51
|
+
|
|
52
|
+
raise "Cloudflare purge failed: #{response.code} #{response.body}"
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# The zone, from the environment or the credentials.
|
|
56
|
+
def self.zone = setting('CLOUDFLARE_ZONE_ID', :zone_id)
|
|
57
|
+
|
|
58
|
+
# The token, which may purge the zone's cache and nothing else.
|
|
59
|
+
def self.token = setting('CLOUDFLARE_API_TOKEN', :api_token)
|
|
60
|
+
|
|
61
|
+
# A setting from the environment, else from the credentials' `cloudflare:`.
|
|
62
|
+
def self.setting(name, key)
|
|
63
|
+
ENV[name].presence || Rails.application.credentials.dig(:cloudflare, key)
|
|
64
|
+
end
|
|
65
|
+
private_class_method :post, :zone, :token, :setting
|
|
66
|
+
end
|
|
67
|
+
end
|
data/lib/drive/engine.rb
CHANGED
|
@@ -1,7 +1,28 @@
|
|
|
1
1
|
module Drive
|
|
2
|
-
# Lends the host its models (Run, Mirrored, Counted, AccessFile, Crawler
|
|
3
|
-
# migrations of runs
|
|
2
|
+
# Lends the host its models (Run, Mirrored, Counted, AccessFile, Crawler, NNumber, Designation,
|
|
3
|
+
# Serial), RateLimited, the error pages and the migrations of runs
|
|
4
|
+
# (`bin/rails drive:install:migrations`).
|
|
4
5
|
class Engine < ::Rails::Engine
|
|
5
6
|
engine_name 'drive'
|
|
7
|
+
|
|
8
|
+
# Every request reaches the app through Cloudflare: trusted as proxies beside the private
|
|
9
|
+
# ones, its addresses are skipped, and `request.remote_ip` is the visitor's own, for every
|
|
10
|
+
# rate limit and log line.
|
|
11
|
+
initializer 'drive.trusted_proxies', before: :build_middleware_stack do |app|
|
|
12
|
+
app.config.action_dispatch.trusted_proxies ||=
|
|
13
|
+
ActionDispatch::RemoteIp::TRUSTED_PROXIES + Drive::Cloudflare::RANGES
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
# The same pages for 400, 404, 422 and 500 on every app (Drive::ErrorsController), at the
|
|
17
|
+
# paths Rails keeps for them, and the page shown while an app is down for maintenance.
|
|
18
|
+
initializer 'drive.errors' do |app|
|
|
19
|
+
app.config.exceptions_app ||= app.routes
|
|
20
|
+
app.routes.prepend do
|
|
21
|
+
%w[400 404 422 500].each do |status|
|
|
22
|
+
get status, to: 'drive/errors#show', defaults: { status: status.to_i }
|
|
23
|
+
end
|
|
24
|
+
get 'down.html', to: 'drive/errors#down', format: false
|
|
25
|
+
end
|
|
26
|
+
end
|
|
6
27
|
end
|
|
7
28
|
end
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# The words every easyFAA app names alike: more than one aircraft is still "aircraft", whatever
|
|
2
|
+
# precedes it ("NTSB::Aircraft", "accident_aircraft"), and the agencies and formats are the acronyms
|
|
3
|
+
# they are. Set when the gem loads, before any of the app's constants is named.
|
|
4
|
+
ActiveSupport::Inflector.inflections(:en) do |inflect|
|
|
5
|
+
inflect.uncountable 'aircraft'
|
|
6
|
+
inflect.plural(/(aircraft)\z/i, '\1')
|
|
7
|
+
inflect.singular(/(aircraft)\z/i, '\1')
|
|
8
|
+
%w[FAA DRS NTSB XML].each { inflect.acronym it }
|
|
9
|
+
end
|
data/lib/drive/version.rb
CHANGED
data/lib/drive.rb
CHANGED
|
@@ -1,7 +1,10 @@
|
|
|
1
1
|
require 'drive/version'
|
|
2
|
+
require 'drive/inflections'
|
|
3
|
+
require 'drive/cloudflare'
|
|
2
4
|
require 'drive/engine'
|
|
3
5
|
|
|
4
|
-
#
|
|
5
|
-
#
|
|
6
|
+
# What the easyFAA apps share: a copy of the tables another source publishes, changing only the
|
|
7
|
+
# rows the source did (Mirrored, Counted, Run), and what every app in front of Cloudflare needs
|
|
8
|
+
# alike (Drive::Cloudflare, Crawler, RateLimited, the error pages, NNumber, Designation, Serial).
|
|
6
9
|
module Drive
|
|
7
10
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: drive
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- claudiob
|
|
@@ -51,9 +51,9 @@ dependencies:
|
|
|
51
51
|
- - ">="
|
|
52
52
|
- !ruby/object:Gem::Version
|
|
53
53
|
version: '8.1'
|
|
54
|
-
description: 'A Rails engine for apps
|
|
55
|
-
|
|
56
|
-
|
|
54
|
+
description: 'A Rails engine for the easyFAA apps: each mirrors a public source (the
|
|
55
|
+
FAA, the NTSB) table for table, changing only the rows the source did, and each
|
|
56
|
+
shows the same error pages, limits, crawlers and N-numbers.'
|
|
57
57
|
email:
|
|
58
58
|
- claudiob@users.noreply.github.com
|
|
59
59
|
executables: []
|
|
@@ -64,16 +64,24 @@ files:
|
|
|
64
64
|
- MIT-LICENSE
|
|
65
65
|
- README.md
|
|
66
66
|
- app/controllers/concerns/rate_limited.rb
|
|
67
|
+
- app/controllers/drive/errors_controller.rb
|
|
67
68
|
- app/models/access_file.rb
|
|
68
69
|
- app/models/concerns/counted.rb
|
|
69
70
|
- app/models/concerns/mirrored.rb
|
|
70
71
|
- app/models/crawler.rb
|
|
72
|
+
- app/models/designation.rb
|
|
71
73
|
- app/models/drive/current.rb
|
|
74
|
+
- app/models/n_number.rb
|
|
72
75
|
- app/models/run.rb
|
|
76
|
+
- app/models/serial.rb
|
|
77
|
+
- app/views/drive/errors/show.html.erb
|
|
78
|
+
- app/views/layouts/drive/errors.html.erb
|
|
73
79
|
- db/migrate/20261005100000_create_runs_and_changes.rb
|
|
74
80
|
- db/migrate/20261006100000_drop_changes.rb
|
|
75
81
|
- lib/drive.rb
|
|
82
|
+
- lib/drive/cloudflare.rb
|
|
76
83
|
- lib/drive/engine.rb
|
|
84
|
+
- lib/drive/inflections.rb
|
|
77
85
|
- lib/drive/test_helper.rb
|
|
78
86
|
- lib/drive/version.rb
|
|
79
87
|
homepage: https://github.com/easyfaa/drive
|
|
@@ -99,5 +107,5 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
99
107
|
requirements: []
|
|
100
108
|
rubygems_version: 4.0.3
|
|
101
109
|
specification_version: 4
|
|
102
|
-
summary:
|
|
110
|
+
summary: 'What the easyFAA apps share: mirrored FAA and NTSB tables, and more.'
|
|
103
111
|
test_files: []
|