pubhost 0.2.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pubhost/__init__.py +24 -0
- pubhost/cli.py +65 -0
- pubhost/dns.py +318 -0
- pubhost/site.py +348 -0
- pubhost-0.2.0.dist-info/METADATA +574 -0
- pubhost-0.2.0.dist-info/RECORD +10 -0
- pubhost-0.2.0.dist-info/WHEEL +5 -0
- pubhost-0.2.0.dist-info/entry_points.txt +2 -0
- pubhost-0.2.0.dist-info/licenses/LICENSE +202 -0
- pubhost-0.2.0.dist-info/top_level.txt +1 -0
|
@@ -0,0 +1,574 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pubhost
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: A repository's hosting: Terraform modules for its zones and sites, the live DNS check, the apply workflow, and the image-build and static-deploy actions.
|
|
5
|
+
Author-email: David Schryer <schryer@gmail.com>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Repository, https://github.com/Sustenaut/pubhost
|
|
8
|
+
Requires-Python: >=3.11
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Requires-Dist: pubrules>=0.43.0
|
|
12
|
+
Dynamic: license-file
|
|
13
|
+
|
|
14
|
+
# pubhost
|
|
15
|
+
|
|
16
|
+
What makes a repository's sites and zones exist and stay right at the
|
|
17
|
+
outside systems that serve them: Terraform modules that turn `pubrules.json`'s
|
|
18
|
+
hosting block into DNS, mail, DNSSEC, Cloud Run and bucket resources at
|
|
19
|
+
Hetzner, Cloudflare, deSEC, Google Cloud and mailbox.org (`terraform/`); a
|
|
20
|
+
live check across a zone's nameservers (`pubhost dns check`); a reusable
|
|
21
|
+
workflow that applies a stack from CI (`terraform.yml`); and the actions that
|
|
22
|
+
build an image (`actions/docker-build`) and deploy a static site
|
|
23
|
+
(`actions/deploy-static`), which first runs `pubhost site check`.
|
|
24
|
+
|
|
25
|
+
What runs inside one repository about itself stays in
|
|
26
|
+
[pubrules](https://github.com/Sustenaut/pubrules): `pubrules sync`, which checks the
|
|
27
|
+
hosting block's shape, and the generated CI. Its `ci / site` job runs the
|
|
28
|
+
deploy-static the repository's own deploy workflow pins, in check-only mode,
|
|
29
|
+
and `job-service-terraform.yml` checks a stack without a backend.
|
|
30
|
+
|
|
31
|
+
pubhost dns check ZONE [--record NAME/TYPE...]
|
|
32
|
+
pubhost site check DIR [--url URL]
|
|
33
|
+
pubhost --version
|
|
34
|
+
|
|
35
|
+
## Installing
|
|
36
|
+
|
|
37
|
+
The modules, the workflow and the actions are used from this repository by
|
|
38
|
+
commit, with the version in a comment:
|
|
39
|
+
|
|
40
|
+
uses: Sustenaut/pubhost/actions/docker-build@<commit> # vX.Y.Z
|
|
41
|
+
source = "github.com/Sustenaut/pubhost//terraform/hosting?ref=<commit>" # vX.Y.Z
|
|
42
|
+
|
|
43
|
+
The commit is the release tag's
|
|
44
|
+
(`git ls-remote https://github.com/Sustenaut/pubhost refs/tags/vX.Y.Z^{}`).
|
|
45
|
+
|
|
46
|
+
The command is a wheel. A repository that runs `pubhost dns check` lists it
|
|
47
|
+
beside pubrules in its `tests/requirements.txt`, and `pubrules lock` pins both by
|
|
48
|
+
sha256:
|
|
49
|
+
|
|
50
|
+
pubhost @ https://github.com/Sustenaut/pubhost/releases/download/vX.Y.Z/pubhost-X.Y.Z-py3-none-any.whl
|
|
51
|
+
|
|
52
|
+
pubhost declares the oldest pubrules it works with (`pubrules>=` in
|
|
53
|
+
pyproject.toml); its own `tests/requirements.lock` pins the pubrules it is
|
|
54
|
+
tested with.
|
|
55
|
+
|
|
56
|
+
## What pubhost takes from pubrules
|
|
57
|
+
|
|
58
|
+
pubhost uses pubrules as a library for one thing: the hosting block's shape,
|
|
59
|
+
which `pubrules sync` checks and `pubhost dns check` reads. pubrules never
|
|
60
|
+
imports pubhost and names nothing of it; pubhost offers no `sustenaut.sync`
|
|
61
|
+
or `sustenaut.bump` hook, so pubrules never loads it into its own process.
|
|
62
|
+
|
|
63
|
+
These are all the names it takes, which `tests/features/pubrules.feature`
|
|
64
|
+
holds it to, and which the installed pubrules must have:
|
|
65
|
+
|
|
66
|
+
- `pubrules.hosting.settings`: the hosting block, checked.
|
|
67
|
+
- `pubrules.hosting.SEP` and `pubrules.hosting.dnskey`: a DNSKEY's key-signing
|
|
68
|
+
flag and its parsing, for the pinned keys.
|
|
69
|
+
|
|
70
|
+
The modules' variable types are pubhost's, and the block's shape is pubrules'.
|
|
71
|
+
Both read `terraform/testdata/pubrules.json`: the modules' tests run on it, and
|
|
72
|
+
`tests/features/pubrules.feature` checks that pubrules accepts it and that every
|
|
73
|
+
setting it names is one `terraform/hosting`'s `config` declares.
|
|
74
|
+
|
|
75
|
+
`src/pubhost/site.py` stays standard-library only, because deploy-static runs
|
|
76
|
+
it from its own source with nothing installed. It carries a copy of pubrules'
|
|
77
|
+
`conform.exceptions`, marked as one, to read the site rules' exceptions.
|
|
78
|
+
|
|
79
|
+
## Hosting modules
|
|
80
|
+
|
|
81
|
+
Terraform modules for a repository's domains: their DNS, their email and
|
|
82
|
+
their sites, each at the provider the repository chooses, and the actions
|
|
83
|
+
that build an image and deploy a static site. They are taken from this
|
|
84
|
+
repository by commit, with the version in a comment, and they hold no provider's names, hosts or accounts: those are in the
|
|
85
|
+
repository's own `pubrules.json`, under `hosting`, so each repository keeps
|
|
86
|
+
its own setup.
|
|
87
|
+
|
|
88
|
+
### The hosting block
|
|
89
|
+
|
|
90
|
+
`providers` are the providers the repository uses, each under a name it
|
|
91
|
+
chooses, with a `type` and that provider's facts. `zones` are its domains,
|
|
92
|
+
each naming the providers that serve its DNS, its email and its sites.
|
|
93
|
+
This is [`terraform/testdata/pubrules.json`](terraform/testdata/pubrules.json),
|
|
94
|
+
which the modules are tested with (secrets never go here; tokens stay with
|
|
95
|
+
`./with-secrets`):
|
|
96
|
+
|
|
97
|
+
```json
|
|
98
|
+
"hosting": {
|
|
99
|
+
"providers": {
|
|
100
|
+
"hetzner": {
|
|
101
|
+
"type": "hetzner",
|
|
102
|
+
"dns": { "external_nameservers": ["helium.ns.hetzner.de.", "hydrogen.ns.hetzner.com.", "oxygen.ns.hetzner.com."] }
|
|
103
|
+
},
|
|
104
|
+
"cloudflare": { "type": "cloudflare", "account_id": "fedcba9876543210fedcba9876543210" },
|
|
105
|
+
"desec": {
|
|
106
|
+
"type": "desec",
|
|
107
|
+
"dns": { "external_nameservers": ["ns1.desec.io.", "ns2.desec.org."] }
|
|
108
|
+
},
|
|
109
|
+
"gcp": {
|
|
110
|
+
"type": "google-cloud",
|
|
111
|
+
"project": "example-project",
|
|
112
|
+
"region": "europe-north1",
|
|
113
|
+
"cloud_run": { "domain_target": "ghs.googlehosted.com." }
|
|
114
|
+
},
|
|
115
|
+
"mailbox": {
|
|
116
|
+
"type": "mailbox-org",
|
|
117
|
+
"mx": ["10 mxext1.mailbox.org.", "10 mxext2.mailbox.org.", "10 mxext3.mailbox.org.", "10 mxext4.mailbox.org."],
|
|
118
|
+
"spf": "include:mailbox.org",
|
|
119
|
+
"dkim": { "mbo0001": "mbo0001._domainkey.mailbox.org.", "...": "..." }
|
|
120
|
+
}
|
|
121
|
+
},
|
|
122
|
+
"zones": {
|
|
123
|
+
"example.eu": {
|
|
124
|
+
"dns": ["hetzner", "cloudflare"],
|
|
125
|
+
"email": {
|
|
126
|
+
"provider": "mailbox",
|
|
127
|
+
"verification": { "f67e29eb00a65673481c0f9731bc90e0c6f94fd3": "<the key mailbox.org shows>" },
|
|
128
|
+
"dmarc": { "policy": "none", "rua": "alerts@example.eu" }
|
|
129
|
+
},
|
|
130
|
+
"apex_txt": ["google-site-verification=abc"],
|
|
131
|
+
"sites": {
|
|
132
|
+
"@": { "provider": "hetzner", "kind": "server", "server": "web" },
|
|
133
|
+
"blog": { "provider": "cloudflare", "kind": "pages", "project": "example-blog", "standby": "web" },
|
|
134
|
+
"pub": { "provider": "gcp", "kind": "cloud-run", "service": "pub-site" }
|
|
135
|
+
}
|
|
136
|
+
},
|
|
137
|
+
"example.net": {
|
|
138
|
+
"dns": ["desec", "cloudflare"],
|
|
139
|
+
"dnssec": {
|
|
140
|
+
"keys": {
|
|
141
|
+
"desec": ["257 3 13 Wegp...cvA=="],
|
|
142
|
+
"cloudflare": ["257 3 13 o+F6...vBA==", "256 3 13 vowd...MAiQ=="]
|
|
143
|
+
}
|
|
144
|
+
},
|
|
145
|
+
"sites": { "@": { "provider": "hetzner", "kind": "server", "server": "small" } }
|
|
146
|
+
}
|
|
147
|
+
}
|
|
148
|
+
}
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
The provider types pubrules supports, and what each holds:
|
|
152
|
+
|
|
153
|
+
| type | settings | serves DNS | sites |
|
|
154
|
+
| --- | --- | --- | --- |
|
|
155
|
+
| `hetzner` | `dns.external_nameservers`: the names Hetzner also serves a zone from, which a registrar such as Hetzner's own (konsoleH) lists next to another provider's; without them the registrar gets the assigned ones | yes; the zone must already exist at Hetzner | `server`: A (and AAAA) records to a server's addresses |
|
|
156
|
+
| `cloudflare` | `account_id` | yes; with another provider too, in multi-provider mode; signs with `dnssec` | `pages`: a Pages project (`project`; `subdomain` when Cloudflare gave it a pages.dev name other than `<project>.pages.dev`), served from Pages or, with `served_by: "server"`, from its `standby` server |
|
|
157
|
+
| `desec` | `dns.external_nameservers`: the names deSEC serves every zone from (`ns1.desec.io.`, `ns2.desec.org.`), which its API does not report | yes, through `terraform/desec-zones`; the zone must already exist at deSEC; always signs | no |
|
|
158
|
+
| `google-cloud` | `project`, `region`, `cloud_run.domain_target` (the host a custom domain's CNAME names) | no | `cloud-run`: a CNAME to `domain_target`, and a domain mapping to the Cloud Run `service` (`terraform/cloud-run-sites`) |
|
|
159
|
+
| `mailbox-org` | `mx` (`"priority host."`), `spf` (its mechanisms, such as `include:mailbox.org`), `dkim` (selector, in lowercase, to host) | no | no; a zone's `email.provider` |
|
|
160
|
+
|
|
161
|
+
A zone's `dns` lists the providers that serve its DNS, at most one of
|
|
162
|
+
each type; with more than one, every provider's nameservers are in the
|
|
163
|
+
apex NS rrset at each. Its `email` names a mail provider, with the domain's own
|
|
164
|
+
`verification` TXT records and `dmarc` (`policy` none, quarantine or
|
|
165
|
+
reject, and `rua`, where reports go). `apex_txt` holds other apex TXT
|
|
166
|
+
values, such as a Google site verification; they share the SPF record's
|
|
167
|
+
rrset, since Hetzner keeps one rrset per name and type. `sites` are by name
|
|
168
|
+
relative to the zone (`"@"` for the apex, which only a server site can
|
|
169
|
+
have, since the others are CNAMEs), each with its `provider` and that
|
|
170
|
+
provider's `kind`.
|
|
171
|
+
|
|
172
|
+
**DNSSEC.** A zone's `dnssec.keys` pins each of its DNS providers' public
|
|
173
|
+
keys, as DNSKEY values (`"flags 3 algorithm key"`, the key in base64
|
|
174
|
+
without spaces), by provider name. Every provider signs the zone with its
|
|
175
|
+
own keys and publishes the others' next to them (RFC 8901's multi-signer
|
|
176
|
+
model 2), so an answer from any of them validates against either DS
|
|
177
|
+
record at the registrar. Cloudflare's signing is turned on, with
|
|
178
|
+
multi-signer; deSEC signs every zone it serves. Each provider needs a
|
|
179
|
+
key-signing key (flags 257), and all keys one algorithm. Hetzner cannot
|
|
180
|
+
sign, so a zone it serves cannot have `dnssec`. The keys are pinned rather
|
|
181
|
+
than read from the providers: neither rotates them on its own, so a key
|
|
182
|
+
change is a reviewed change to `pubrules.json`. Cloudflare's are on its DNS
|
|
183
|
+
settings page, or `GET /zones/<id>/dnssec` (the KSK) and its DNSKEY rrset;
|
|
184
|
+
deSEC's are in `terraform/desec-zones`' output `keys`. Without `dnssec`,
|
|
185
|
+
Cloudflare does not sign.
|
|
186
|
+
|
|
187
|
+
**TTLs.** Records get a day for the apex NS rrset (written when two or
|
|
188
|
+
more providers serve a zone), an hour for the email records and the apex TXT
|
|
189
|
+
rrset, and five minutes for the sites, unless `ttl` says otherwise:
|
|
190
|
+
`hosting.ttl` for every zone, a zone's `ttl` for its records, or a site's
|
|
191
|
+
`ttl` (a number) for that site's; the nearest wins. Each is in seconds,
|
|
192
|
+
from 60 to a week. Lower them a TTL or more ahead of a change, so
|
|
193
|
+
resolvers pick it up sooner, and raise them again after.
|
|
194
|
+
|
|
195
|
+
```json
|
|
196
|
+
"hosting": {
|
|
197
|
+
"ttl": { "email": 1800 },
|
|
198
|
+
"zones": {
|
|
199
|
+
"example.eu": {
|
|
200
|
+
"ttl": { "sites": 60, "nameservers": 3600 },
|
|
201
|
+
"sites": { "www": { "provider": "hetzner", "kind": "server", "server": "web", "ttl": 120 } }
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
A new provider of a known type is a new entry in `providers`; a repository
|
|
208
|
+
on two Hetzner projects names both. A new type, such as another mail
|
|
209
|
+
provider, needs pubrules to learn it once, in `src/pubrules/hosting.py`, and then
|
|
210
|
+
pubhost, in `terraform/hosting` (or a module beside it): a pubrules release,
|
|
211
|
+
then a pubhost release. Terraform ignores a setting its types do not name,
|
|
212
|
+
so `pubrules sync` (and `sync --check`, in CI) checks the block first and
|
|
213
|
+
stops on anything it does not know.
|
|
214
|
+
|
|
215
|
+
### Using it
|
|
216
|
+
|
|
217
|
+
The root module reads the block, gives the servers' addresses (known only
|
|
218
|
+
once the servers exist), and configures the Terraform providers, each with
|
|
219
|
+
the one account it deploys to. A repository with Cloud Run sites adds
|
|
220
|
+
`cloud-run-sites`:
|
|
221
|
+
|
|
222
|
+
```hcl
|
|
223
|
+
module "hosting" {
|
|
224
|
+
source = "github.com/Sustenaut/pubhost//terraform/hosting?ref=<commit>" # vX.Y.Z
|
|
225
|
+
config = jsondecode(file("${path.root}/pubrules.json")).hosting
|
|
226
|
+
|
|
227
|
+
servers = {
|
|
228
|
+
web = { ipv4 = module.web.ipv4, ipv6 = module.web.ipv6 }
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
module "cloud_run_sites" {
|
|
233
|
+
source = "github.com/Sustenaut/pubhost//terraform/cloud-run-sites?ref=<commit>" # vX.Y.Z
|
|
234
|
+
config = jsondecode(file("${path.root}/pubrules.json")).hosting
|
|
235
|
+
|
|
236
|
+
# When the services are created in the same apply.
|
|
237
|
+
depends_on = [google_cloud_run_v2_service.pub_site]
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
A repository with zones at deSEC adds `desec-zones`, which writes the
|
|
242
|
+
same records there, and configures the deSEC provider with its token
|
|
243
|
+
(with `./with-secrets` and `"terraform.secrets": "declared"`, a
|
|
244
|
+
`DESEC_TOKEN` secret reaches it as `TF_VAR_desec_token`):
|
|
245
|
+
|
|
246
|
+
```hcl
|
|
247
|
+
variable "desec_token" {
|
|
248
|
+
type = string
|
|
249
|
+
sensitive = true
|
|
250
|
+
}
|
|
251
|
+
|
|
252
|
+
provider "desec" {
|
|
253
|
+
api_token = var.desec_token
|
|
254
|
+
retry_max = 30
|
|
255
|
+
}
|
|
256
|
+
|
|
257
|
+
module "desec_zones" {
|
|
258
|
+
source = "github.com/Sustenaut/pubhost//terraform/desec-zones?ref=<commit>" # vX.Y.Z
|
|
259
|
+
config = jsondecode(file("${path.root}/pubrules.json")).hosting
|
|
260
|
+
records = module.hosting.records
|
|
261
|
+
apex_nameservers = module.hosting.apex_nameservers
|
|
262
|
+
}
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
deSEC allows 15 rrset writes a minute (and 100 an hour) per domain, and
|
|
266
|
+
Terraform writes many at once, so a zone's first apply hits the limit. deSEC
|
|
267
|
+
then says how long to wait, and the provider waits that long before trying
|
|
268
|
+
again, up to `retry_max` times (5 by default, too few for a whole zone; 30
|
|
269
|
+
covers one of up to about 100 rrsets).
|
|
270
|
+
|
|
271
|
+
Its outputs are `nameservers` (by zone, for the registrar),
|
|
272
|
+
`cloudflare_zone_ids`, `records` (every rrset, by zone),
|
|
273
|
+
`apex_nameservers` (the apex NS rrset of each zone two or more providers
|
|
274
|
+
serve), `cloudflare_ds` (a signed zone's DS record at Cloudflare, for the
|
|
275
|
+
registrar) and `sites` (by full name: provider, kind, address, and a Pages
|
|
276
|
+
site's project and pages.dev name). `records` takes other rrsets, by zone,
|
|
277
|
+
for anything the block does not describe.
|
|
278
|
+
|
|
279
|
+
The modules need Terraform 1.11 or newer, `hcloud` 1.69, `cloudflare` 5.29,
|
|
280
|
+
`Valodim/desec` 0.6 and `google` 7. `terraform/hosting` needs no Cloudflare
|
|
281
|
+
token or provider block when no zone or site is at Cloudflare. The Google
|
|
282
|
+
and deSEC providers will not start without credentials, even with nothing
|
|
283
|
+
to do, so the Cloud Run domain mappings and the deSEC zones are in their
|
|
284
|
+
own modules, which only a repository on Google Cloud or deSEC takes.
|
|
285
|
+
|
|
286
|
+
- **`terraform/hosting`:** the block above, zone by zone: the records,
|
|
287
|
+
the Pages projects and their domains, and each zone's DNS.
|
|
288
|
+
- **`terraform/cloud-run-sites`:** the Cloud Run sites' domain mappings,
|
|
289
|
+
each to its `service` in its provider's `project` and `region`; output
|
|
290
|
+
`domains`, with the records Google asks for.
|
|
291
|
+
- **`terraform/cloud-run-service`:** what every Cloud Run site repeats: the
|
|
292
|
+
account it runs as, its registry (the ten newest images kept, older ones
|
|
293
|
+
deleted after 30 days), and the service, behind IAP by default, on a
|
|
294
|
+
placeholder image that Terraform ignores after the first deploy. Given
|
|
295
|
+
`deploy_repository`, an account only that GitHub repository can act as,
|
|
296
|
+
which can push to this registry and deploy this service and nothing else.
|
|
297
|
+
- **`terraform/backup-bucket`:** a private bucket with an age rule, and an
|
|
298
|
+
account that can only create objects in it, with a key by default.
|
|
299
|
+
- **`terraform/cloud-run-access`:** who may pass IAP to each Cloud Run
|
|
300
|
+
service, named rather than referred to, so it can be in another stack.
|
|
301
|
+
- **`terraform/desec-zones`:** each zone a `desec` provider serves: the
|
|
302
|
+
domain, every rrset `terraform/hosting` writes (a TTL below the
|
|
303
|
+
domain's `minimum_ttl` raised to it, as deSEC requires), the apex NS
|
|
304
|
+
rrset when other providers serve the zone too, and with `dnssec` the
|
|
305
|
+
other providers' keys; outputs `ds` (deSEC's DS records, by zone, for
|
|
306
|
+
the registrar) and `keys` (its DNSKEY values, for `dnssec.keys`).
|
|
307
|
+
- **`terraform/dns-zone`:** one zone's rrsets in zone-file form (`name`
|
|
308
|
+
relative, `"@"` for the apex; trailing dots, MX `"10 host."`, TXT
|
|
309
|
+
quoted), at Hetzner, Cloudflare or both, with deSEC's nameservers when
|
|
310
|
+
it serves the zone too. With more than one provider, Cloudflare runs in
|
|
311
|
+
multi-provider mode: one unproxied record per value (MX split into
|
|
312
|
+
priority and host), and every provider's nameservers in the apex NS
|
|
313
|
+
rrset at each. Each record is keyed by its rrset and its value
|
|
314
|
+
(`"nameservers ns1.desec.io"`, `"email_mx 10 mxext1.mailbox.org"`), so a
|
|
315
|
+
value that comes or goes changes only its own record; A, AAAA and CNAME
|
|
316
|
+
records, whose values may be known only after apply, keep their position
|
|
317
|
+
(`"www 0"`). With `dnssec`, Cloudflare signs in multi-signer mode and
|
|
318
|
+
serves the other providers' keys as DNSKEY records; denial of existence
|
|
319
|
+
is left as Cloudflare's plan has it (NSEC on Free; RFC 8901 lets each
|
|
320
|
+
provider choose). `terraform/hosting` calls it for each zone; it can be
|
|
321
|
+
used on its own with `hetzner`, `cloudflare`, `desec` and `dnssec` set
|
|
322
|
+
as in the block (`dnssec.keys` by provider type).
|
|
323
|
+
- **`actions/docker-build`:** builds a Docker image with buildx and a
|
|
324
|
+
GitHub Actions layer cache, so an unchanged stage is restored instead of
|
|
325
|
+
built again, and loads it into the runner's Docker for the job's checks
|
|
326
|
+
and `docker push`. Give each image its own `scope`. A cached layer keeps
|
|
327
|
+
the packages it was built with until something before it changes;
|
|
328
|
+
`no-cache: true` takes the current ones.
|
|
329
|
+
- **`actions/deploy-static`:** deploys a directory to a server over rsync
|
|
330
|
+
(`deploy-key`, `deploy-target`, and `known-hosts`, the server's pinned
|
|
331
|
+
host key; without it the key is scanned on every run), to Cloudflare
|
|
332
|
+
Pages (`cloudflare-api-token`, `cloudflare-account-id`,
|
|
333
|
+
`cloudflare-project`, with wrangler pinned), or to both. It fails rather
|
|
334
|
+
than deploy nowhere or to half a target. It runs `pubhost site check`
|
|
335
|
+
(below) on the directory first and deploys nothing if that fails; give
|
|
336
|
+
`site-url` so the check finds the og:image files, or `check: false` for
|
|
337
|
+
a site that is not public.
|
|
338
|
+
|
|
339
|
+
```yaml
|
|
340
|
+
- uses: Sustenaut/pubhost/actions/deploy-static@<commit> # vX.Y.Z
|
|
341
|
+
with:
|
|
342
|
+
dir: site
|
|
343
|
+
cloudflare-api-token: ${{ secrets.CLOUDFLARE_PAGES_TOKEN }}
|
|
344
|
+
cloudflare-account-id: ${{ vars.CLOUDFLARE_ACCOUNT_ID }}
|
|
345
|
+
cloudflare-project: example-blog
|
|
346
|
+
deploy-key: ${{ secrets.DEPLOY_KEY }}
|
|
347
|
+
deploy-target: deploy@example.eu:./
|
|
348
|
+
known-hosts: example.eu ssh-ed25519 AAAA...
|
|
349
|
+
site-url: https://example.eu/
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
Pin it as above, to the commit of a pubhost release, with the version in
|
|
353
|
+
a comment; the site check it runs is that release's. pubrules' nightly
|
|
354
|
+
moves a pin of this form to pubhost's latest release.
|
|
355
|
+
|
|
356
|
+
**Moving a zone that is already in use.** Hetzner made the apex NS rrset
|
|
357
|
+
with the zone, so the first apply with Cloudflare joining it must take it
|
|
358
|
+
over rather than create it, with an import block in the root module:
|
|
359
|
+
|
|
360
|
+
```hcl
|
|
361
|
+
import {
|
|
362
|
+
to = module.hosting.module.zone["example.eu"].hcloud_zone_rrset.records["nameservers"]
|
|
363
|
+
id = "example.eu/@/NS"
|
|
364
|
+
}
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
deSEC makes the zone's apex NS rrset too, so a zone it serves beside
|
|
368
|
+
another provider takes that over the same way, with the domain itself,
|
|
369
|
+
which must already exist at deSEC (created on desec.io):
|
|
370
|
+
|
|
371
|
+
```hcl
|
|
372
|
+
import {
|
|
373
|
+
to = module.desec_zones.desec_domain.zone["example.eu"]
|
|
374
|
+
id = "example.eu"
|
|
375
|
+
}
|
|
376
|
+
|
|
377
|
+
import {
|
|
378
|
+
to = module.desec_zones.desec_rrset.records["example.eu nameservers"]
|
|
379
|
+
id = "example.eu/@/NS"
|
|
380
|
+
}
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
Records a root module already manages, outside these modules, move with `moved` blocks, or with
|
|
384
|
+
import blocks like the one above when they were separate resources. In a
|
|
385
|
+
zone, the email records are `email_mx`, `email_dmarc`,
|
|
386
|
+
`email_dkim_<selector>` and `email_verification_<name>`, the apex TXT
|
|
387
|
+
rrset is `apex_txt`, and a site's are its name (`apex` for `"@"`, dots as
|
|
388
|
+
underscores), with `_a` and `_aaaa` for a server's addresses.
|
|
389
|
+
|
|
390
|
+
**Checking a zone.** `pubhost dns check ZONE` asks every nameserver of a
|
|
391
|
+
zone in the hosting block (its providers' from `pubrules.json`, the ones
|
|
392
|
+
resolvers find, and the ones each lists in its apex NS rrset), without
|
|
393
|
+
recursion, for every rrset the block describes (`--record NAME/TYPE` adds
|
|
394
|
+
others, such as a root module's own) and fails when their answers differ,
|
|
395
|
+
TTLs aside, or a nameserver does not serve the zone. With `dnssec`, each
|
|
396
|
+
nameserver must also serve exactly the pinned keys, and validate alone:
|
|
397
|
+
`delv` asks it for each rrset and for a name that does not exist, with
|
|
398
|
+
the pinned key-signing keys as trust anchors, as a resolver will once the
|
|
399
|
+
registrar has the DS records. When answers differ and one nameserver serves an older SOA serial than its provider's others (deSEC copies a change to ns2.desec.org a moment after ns1.desec.io), it says so: run it again in a minute. Run it after each step of a move, and
|
|
400
|
+
before any DS record goes to the registrar; it only reads, and needs
|
|
401
|
+
`dig` and `delv` (BIND 9; Debian's `bind9-dnsutils`).
|
|
402
|
+
|
|
403
|
+
Each module's tests (`terraform test` in its directory) mock the providers
|
|
404
|
+
and need no credentials; CI runs them on Terraform 1.11, with `fmt`,
|
|
405
|
+
`validate`, and the action's scripts under shellcheck and stubs.
|
|
406
|
+
|
|
407
|
+
|
|
408
|
+
### One stack per repository
|
|
409
|
+
|
|
410
|
+
A site's Terraform lives in the repository that owns its code, in `infra/`, with
|
|
411
|
+
its own state: one bucket, one prefix per repository (`prefix = "<repo>"`). What
|
|
412
|
+
every site shares (the project's APIs, the DNS zone, the Workload Identity
|
|
413
|
+
Federation pool) is in a base stack of its own. A site's stack never reads the
|
|
414
|
+
base's state, which holds keys: it reads the project and region from its own
|
|
415
|
+
`pubrules.json`, and finds shared things by their fixed names.
|
|
416
|
+
|
|
417
|
+
```hcl
|
|
418
|
+
locals {
|
|
419
|
+
gcp = jsondecode(file("${path.module}/../pubrules.json")).hosting.providers["gcp"]
|
|
420
|
+
}
|
|
421
|
+
|
|
422
|
+
module "site" {
|
|
423
|
+
source = "github.com/Sustenaut/pubhost//terraform/cloud-run-service?ref=<commit>" # vX.Y.Z
|
|
424
|
+
name = "pub-site"
|
|
425
|
+
project = local.gcp.project
|
|
426
|
+
project_number = "123456789"
|
|
427
|
+
region = local.gcp.region
|
|
428
|
+
|
|
429
|
+
account_id = "example-pub-site"
|
|
430
|
+
account_display_name = "Example pub-site"
|
|
431
|
+
|
|
432
|
+
deploy_repository = "example-org/pub-site"
|
|
433
|
+
}
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
`hosting` may list only `providers` here; the zones are the base's. The base's
|
|
437
|
+
pool must admit the repository, in its provider's `attribute_condition`, as this
|
|
438
|
+
module only makes the account the repository acts as.
|
|
439
|
+
|
|
440
|
+
**Moving a site out of a shared stack changes nothing in the cloud.** The site's
|
|
441
|
+
repository adds `infra/` with an `import` block for each existing resource, and
|
|
442
|
+
the shared stack replaces its file with `removed` blocks:
|
|
443
|
+
|
|
444
|
+
```hcl
|
|
445
|
+
removed {
|
|
446
|
+
from = google_cloud_run_v2_service.pub_site
|
|
447
|
+
|
|
448
|
+
lifecycle {
|
|
449
|
+
destroy = false
|
|
450
|
+
}
|
|
451
|
+
}
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
Both plans must show no change before either is applied: imports and removals,
|
|
455
|
+
with 0 to add, 0 to change, 0 to destroy. Apply the site's repository first, then
|
|
456
|
+
the shared stack, then delete the `import` and `removed` blocks. A write-only
|
|
457
|
+
secret version and a service account key do not survive an import unchanged: a
|
|
458
|
+
key's private half is not readable again, so leave those where they are, or move
|
|
459
|
+
the state itself.
|
|
460
|
+
|
|
461
|
+
## Terraform from CI
|
|
462
|
+
|
|
463
|
+
`terraform.yml` applies a Terraform stack from the default branch, as a
|
|
464
|
+
Google account it gets through Workload Identity Federation, so no key is
|
|
465
|
+
kept anywhere:
|
|
466
|
+
|
|
467
|
+
```yaml
|
|
468
|
+
name: terraform
|
|
469
|
+
on:
|
|
470
|
+
push: { branches: [main] }
|
|
471
|
+
workflow_dispatch:
|
|
472
|
+
permissions: { contents: read, id-token: write, pull-requests: read }
|
|
473
|
+
jobs:
|
|
474
|
+
terraform:
|
|
475
|
+
uses: Sustenaut/pubhost/.github/workflows/terraform.yml@<commit> # vX.Y.Z
|
|
476
|
+
with:
|
|
477
|
+
terraform-dir: infra
|
|
478
|
+
workload-identity-provider: projects/123456789/locations/global/workloadIdentityPools/github/providers/github-terraform
|
|
479
|
+
service-account: terraform-example@example-project.iam.gserviceaccount.com
|
|
480
|
+
secrets: inherit
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
- `runs-on` (optional) names the runners the jobs run on, such as a self-hosted runner scale set; `ubuntu-latest` by default. The runner needs what GitHub's does here: git, curl, unzip and Node for the actions.
|
|
484
|
+
- Nothing runs until the repository variable `TERRAFORM_CI` is `on`, nor
|
|
485
|
+
for any ref but the default branch. A first job, outside any environment,
|
|
486
|
+
checks both.
|
|
487
|
+
- The `terraform` job runs in the environment `environment` names
|
|
488
|
+
(`terraform`). Make it before `TERRAFORM_CI` is on, with its deployment
|
|
489
|
+
branches limited to the default branch: GitHub makes an environment it
|
|
490
|
+
does not find, open to every branch. GitHub Team allows that limit on a
|
|
491
|
+
private repository; required reviewers there need Enterprise, so merging
|
|
492
|
+
is what approves a change. Nothing runs for a pull request, so a pull
|
|
493
|
+
request's code never runs with the account's credentials: plan it from a
|
|
494
|
+
checkout first.
|
|
495
|
+
- Google trusts the job by its OIDC token's subject,
|
|
496
|
+
`repo:OWNER/REPO:environment:terraform`, or
|
|
497
|
+
`repo:OWNER@OWNER_ID/REPO@REPO_ID:environment:terraform` where the
|
|
498
|
+
repository uses GitHub's immutable subjects (`gh api
|
|
499
|
+
repos/OWNER/REPO/actions/oidc/customization/sub` shows which): the
|
|
500
|
+
provider's condition and the account's `roles/iam.workloadIdentityUser`
|
|
501
|
+
binding need nothing else, and a workflow that runs outside the
|
|
502
|
+
environment gets no credentials.
|
|
503
|
+
- It plans to a file and applies exactly that file, one run at a time
|
|
504
|
+
(concurrency group `terraform-<repository>`, never cancelled). The plan is
|
|
505
|
+
in the job's summary; the file never leaves the runner. A plan that
|
|
506
|
+
deletes anything, replacements included, is applied only when the pull
|
|
507
|
+
request merged as that commit has the label `terraform-destroy`; either
|
|
508
|
+
way the job lists what it deletes.
|
|
509
|
+
- Every run starts with a fresh `terraform init` against the backend the
|
|
510
|
+
stack names. A provider download can fail (GitHub's release downloads,
|
|
511
|
+
releases.hashicorp.com's SHA256SUMS), so init is tried three times, 15 s
|
|
512
|
+
apart. With a provider mirror in pubrules.json, a fourth try installs every
|
|
513
|
+
provider from it, through a CLI configuration with only its
|
|
514
|
+
`network_mirror`; the module's `.terraform.lock.hcl` still checks each
|
|
515
|
+
archive. The service CI's backend-less init does the same.
|
|
516
|
+
|
|
517
|
+
```json
|
|
518
|
+
"mirror": {"terraform": "https://storage.sustenaut.eu/terraform"}
|
|
519
|
+
```
|
|
520
|
+
- With `secrets` (the default), each terraform command runs as
|
|
521
|
+
`./pubsec run -- terraform -chdir=DIR ...`, with the environment's
|
|
522
|
+
secret `BWS_ACCESS_TOKEN`, so the stack gets what `./with-secrets` gives it
|
|
523
|
+
on a laptop. bws is the release's binary, checked against its sha256. With
|
|
524
|
+
`secrets: false` terraform runs plainly and no token is read.
|
|
525
|
+
|
|
526
|
+
## Checking a public site
|
|
527
|
+
|
|
528
|
+
`pubhost site check DIR [--url URL]` reads a built site the way robots and
|
|
529
|
+
other sites do, so that search results, screen readers and link previews
|
|
530
|
+
(LinkedIn, Facebook, Slack, WhatsApp) get what they need. It reads DIR
|
|
531
|
+
only, never the network, and `deploy-static` runs it before every deploy.
|
|
532
|
+
|
|
533
|
+
| Rule | Checked |
|
|
534
|
+
|---|---|
|
|
535
|
+
| WEB-PAGE-01 | every page names its language, `<html lang>` |
|
|
536
|
+
| WEB-PAGE-02 | every page has a `<title>` and a `<meta name="description">` |
|
|
537
|
+
| WEB-OG-01 | every page has og:title, og:description, og:type, og:url (https) and og:image |
|
|
538
|
+
| WEB-OG-02 | og:image is an https address; one under `--url` is in DIR, a PNG or JPEG of at least 1200×627 (LinkedIn's large card) and under 5 MB, and matches og:image:width and og:image:height where given |
|
|
539
|
+
| WEB-OG-03 | every page has `<meta name="twitter:card" content="summary_large_image">` |
|
|
540
|
+
| WEB-ROBOTS-01 | robots.txt lets LinkedInBot, facebookexternalhit, Twitterbot and Slackbot read every page and its og:image: they obey it, so a `Disallow: /` meant for search engines hides the card too (keep a preview out of search with `<meta name="robots" content="noindex">` instead) |
|
|
541
|
+
|
|
542
|
+
Every `*.html` file is a page, except `404.html` and redirects (`<meta
|
|
543
|
+
http-equiv="refresh">`). A page that asks not to be indexed (`<meta
|
|
544
|
+
name="robots" content="noindex">`, in any case and alongside other values
|
|
545
|
+
such as `noindex, nofollow`, or `none`) is taken as not meant for search or
|
|
546
|
+
sharing, such as a service's bare landing page: it still needs its language
|
|
547
|
+
and title (WEB-PAGE-01, and WEB-PAGE-02's title), but not a description, a
|
|
548
|
+
card or an image, and robots.txt may keep the preview bots from it. A rule that a site cannot meet yet takes an
|
|
549
|
+
exception in `pubrules.json`, as the Rust crate standard's do (a rule, a
|
|
550
|
+
reason, an until date); `pubrules conform` reads only its own rules'.
|
|
551
|
+
The rules live in `src/pubhost/site.py` as `RULES`, one function per
|
|
552
|
+
family, so accessibility and other compliance checks join as new families.
|
|
553
|
+
|
|
554
|
+
LinkedIn keeps a link's card for about a week: after fixing a page, paste
|
|
555
|
+
its address into the [Post Inspector](https://www.linkedin.com/post-inspector/).
|
|
556
|
+
|
|
557
|
+
## Developing
|
|
558
|
+
|
|
559
|
+
make venv # the hash-locked environment, this checkout installed editable
|
|
560
|
+
make check # what CI runs: ./pubrules sync --check and the suite
|
|
561
|
+
|
|
562
|
+
`.github/workflows/hosting.yml` runs what the Python suite does not: each
|
|
563
|
+
module's `fmt`, `validate` and `terraform test`, the deploy-static scripts
|
|
564
|
+
under shellcheck with their stubs, actionlint on terraform.yml, and a
|
|
565
|
+
docker-build of a test image. `hosting / passed` is required on main beside
|
|
566
|
+
`ci / passed`.
|
|
567
|
+
|
|
568
|
+
Releases are pubrel's: a code change records what it changes (`pubrel add`),
|
|
569
|
+
and `pubrel prepare` cuts the release pull request. `wheel.yml` attaches the
|
|
570
|
+
wheel once main has tagged it.
|
|
571
|
+
|
|
572
|
+
## Licence
|
|
573
|
+
|
|
574
|
+
Apache-2.0.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
pubhost/__init__.py,sha256=sHJkhZnyZ4zbqwxvZkY29N64W4UFngKJAhmc2Tw827k,874
|
|
2
|
+
pubhost/cli.py,sha256=HXvi6dG2ElguYonYw3vSKoU6UoCT1Jud9Gpd8MFw_vM,2872
|
|
3
|
+
pubhost/dns.py,sha256=Ye16g94v0gp6CFvsYyj1agqWMAfBXehbHhEl2r2l4mc,14659
|
|
4
|
+
pubhost/site.py,sha256=pRyf8bL5Y1H0VwQ-eMKoT0DNsI_Q2kmoIUlDKwI6uzM,15225
|
|
5
|
+
pubhost-0.2.0.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
|
|
6
|
+
pubhost-0.2.0.dist-info/METADATA,sha256=yY8TXcxpRbu2aGkVBhlU-YNCefT2DWB4THdpjZXbxgo,28303
|
|
7
|
+
pubhost-0.2.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
|
|
8
|
+
pubhost-0.2.0.dist-info/entry_points.txt,sha256=lPuGZHr9W29wQY6tze8JKwfDu0JNdrwlbY3TsAUbR9Q,45
|
|
9
|
+
pubhost-0.2.0.dist-info/top_level.txt,sha256=7LVzPfYSwwyv1zfEWUXmGRG3teImi_Pc1BZSrxovP4g,8
|
|
10
|
+
pubhost-0.2.0.dist-info/RECORD,,
|