angarium 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +29 -0
- data/README.md +21 -3
- data/SECURITY.md +10 -6
- data/app/controllers/angarium/api/base_controller.rb +34 -5
- data/app/jobs/angarium/deliver_job.rb +5 -2
- data/app/models/angarium/delivery.rb +52 -16
- data/app/models/angarium/endpoint.rb +44 -7
- data/app/policies/angarium/api/policy.rb +10 -0
- data/db/angarium_migrate/20260704000001_create_angarium_endpoints.rb +7 -2
- data/db/angarium_migrate/20260704000003_create_angarium_deliveries.rb +12 -1
- data/db/angarium_migrate/20260704000004_create_angarium_delivery_attempts.rb +7 -1
- data/lib/angarium/address_policy.rb +35 -2
- data/lib/angarium/configuration.rb +20 -1
- data/lib/angarium/dispatch.rb +7 -3
- data/lib/angarium/version.rb +1 -1
- data/lib/generators/angarium/policy/templates/policy.rb +4 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 98d1d438861a4ff090327c932205c0e6345c8f079562a99ca6ab344fff93da10
|
|
4
|
+
data.tar.gz: 6b37af363e8a253db8f9e66feb549a56a2152fb4fc7536c76551f4daf61688de
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1675398932024d506e7338faa7b3585df46caf3f6f80090347ee029bb608dd859333eff198259eee769e466d9fbc4bee91be9994e971055659e44dd1d2f87d65
|
|
7
|
+
data.tar.gz: 44481ffed27e96af446720d475124536eb27e384ad2fe28c777bf4b9cadb9033bec49ec345dbb4abc1376deab3d9d7d24917cb283c4aa4d9f6742f34533239e8
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,34 @@ All notable changes to this project are documented here. The format is based on
|
|
|
5
5
|
follow [Semantic Versioning](https://semver.org). While on 0.x, a minor release
|
|
6
6
|
may include breaking changes; patch releases stay backward compatible.
|
|
7
7
|
|
|
8
|
+
## [0.2.0] - 2026-07-07
|
|
9
|
+
|
|
10
|
+
### Bug Fixes
|
|
11
|
+
|
|
12
|
+
- Harden delivery concurrency and endpoint validation
|
|
13
|
+
- Block special-use ranges and bound DNS resolution
|
|
14
|
+
- Robust pagination, JSON error contract, event preload
|
|
15
|
+
|
|
16
|
+
### Documentation
|
|
17
|
+
|
|
18
|
+
- Add email reporting channel and auto-bump SECURITY.md version
|
|
19
|
+
- Document new config options and SSRF special-use hardening
|
|
20
|
+
|
|
21
|
+
### Features
|
|
22
|
+
|
|
23
|
+
- Expose endpoint owner in the API behind a policy gate
|
|
24
|
+
- Add dns_timeout, hosts-file toggle, and input limits
|
|
25
|
+
- Add hot-path indexes, persisted forced flag, text url column
|
|
26
|
+
|
|
27
|
+
### Performance
|
|
28
|
+
|
|
29
|
+
- Memoize the record-less API policy per request
|
|
30
|
+
- Load only the columns subscription matching needs
|
|
31
|
+
|
|
32
|
+
### Refactoring
|
|
33
|
+
|
|
34
|
+
- Rename ping event and message to plain "ping"
|
|
35
|
+
|
|
8
36
|
## [0.1.0] - 2026-07-07
|
|
9
37
|
|
|
10
38
|
### Bug Fixes
|
|
@@ -26,6 +54,7 @@ may include breaking changes; patch releases stay backward compatible.
|
|
|
26
54
|
- Close the last README review nits
|
|
27
55
|
- Include unverified in the ping status lists
|
|
28
56
|
- Document ActiveSupport::Notifications instrumentation
|
|
57
|
+
- Add RubyGems version badge to README
|
|
29
58
|
|
|
30
59
|
### Features
|
|
31
60
|
|
data/README.md
CHANGED
|
@@ -351,7 +351,7 @@ redelivery is at-least-once-safe either way. Set it to `nil` to disable reaping.
|
|
|
351
351
|
|
|
352
352
|
### Pinging an endpoint
|
|
353
353
|
|
|
354
|
-
Verify an endpoint end-to-end by delivering a synthetic `
|
|
354
|
+
Verify an endpoint end-to-end by delivering a synthetic `ping` event
|
|
355
355
|
(subscription matching is bypassed, so a ping is always sent). Returns the
|
|
356
356
|
`Angarium::Delivery`, so you can reload it to inspect the outcome:
|
|
357
357
|
|
|
@@ -443,6 +443,7 @@ methods:
|
|
|
443
443
|
| `create_unverified?` | `false` | Whether endpoints created through the API start `unverified` (no deliveries until a successful ping verifies them) instead of live. |
|
|
444
444
|
| `permit_allow_private_network?` | `false` | Whether `allow_private_network` (relax the private-IP block) is API-writable. Dangerous; trusted operators only. |
|
|
445
445
|
| `permit_allowed_networks?` | `false` | Whether `allowed_networks` (a restrictive CIDR allowlist) is API-writable. |
|
|
446
|
+
| `expose_owner?` | `false` | Whether serialized endpoints include `owner_type`/`owner_id`. Off by default (the owner is the tenancy boundary, resolved server-side). Turn on for a cross-owner console that must tell endpoints apart. |
|
|
446
447
|
| `index?` `show?` `create?` `update?` `destroy?` | `true` | Whether each action is allowed. |
|
|
447
448
|
| `rotate_secret?` `pause?` `enable?` `verify?` `ping?` `redeliver?` | `update?` | Member actions; default to the `update?` capability. |
|
|
448
449
|
|
|
@@ -455,6 +456,11 @@ class WebhookEndpointPolicy < Angarium::Api::Policy
|
|
|
455
456
|
# Multi-tenant visibility: compose on top of the relation you're given.
|
|
456
457
|
def scope(relation) = relation.where(owner_id: current_user.account.owner_ids)
|
|
457
458
|
|
|
459
|
+
# A console spanning owners needs owner_type/owner_id on each endpoint to tell
|
|
460
|
+
# them apart. Pair this with a scope (above) that actually spans owners —
|
|
461
|
+
# expose_owner? only affects serialization, not what the caller can see.
|
|
462
|
+
def expose_owner? = current_user.admin?
|
|
463
|
+
|
|
458
464
|
# Admins may create for any owner in their account (via an owner_id param);
|
|
459
465
|
# everyone else creates for themselves.
|
|
460
466
|
def owner
|
|
@@ -514,6 +520,9 @@ included (see the note below):
|
|
|
514
520
|
|
|
515
521
|
- **Secrets are never echoed.** `signing_secret` is returned only by `create` and
|
|
516
522
|
`rotate_secret`; `custom_headers` (which may hold a credential) is write-only.
|
|
523
|
+
- **Owner is opt-in.** The endpoint object omits `owner_type`/`owner_id` unless the
|
|
524
|
+
policy's `expose_owner?` returns true; `owner_id` is a string (a polymorphic
|
|
525
|
+
owner may have any primary-key type).
|
|
517
526
|
- **Pagination.** List endpoints take `?limit=` (default 50, max 200) and
|
|
518
527
|
`?offset=`, and each list response carries a `pagination` object (`limit`,
|
|
519
528
|
`offset`, `count` in this page, `total` overall); there are more when
|
|
@@ -582,7 +591,12 @@ at delivery time:
|
|
|
582
591
|
- **`config.block_private_ips`** (default `true`) blocks delivery to
|
|
583
592
|
private, loopback, and link-local addresses (e.g. `127.0.0.1`, `10.0.0.0/8`,
|
|
584
593
|
`169.254.169.254`), including IPv4-mapped IPv6 forms (e.g. `::ffff:127.0.0.1`)
|
|
585
|
-
and the unspecified address (`0.0.0.0` / `::`).
|
|
594
|
+
and the unspecified address (`0.0.0.0` / `::`). It also blocks IANA special-use
|
|
595
|
+
ranges that aren't flagged as "private" but are still unsafe targets —
|
|
596
|
+
`0.0.0.0/8` (routes to localhost on Linux), CGNAT (`100.64.0.0/10`), reserved
|
|
597
|
+
space (`240.0.0.0/4`), and the NAT64 prefix (`64:ff9b::/96`, which can embed the
|
|
598
|
+
metadata IP). Resolution is re-checked at delivery time and the connection is
|
|
599
|
+
pinned to the validated IP, so DNS rebinding can't slip past.
|
|
586
600
|
- **`endpoint.allow_private_network`** (default `false`) is the per-endpoint opt-in
|
|
587
601
|
required to deliver to a private address. An allowlist entry alone does **not**
|
|
588
602
|
unlock a private address.
|
|
@@ -687,7 +701,11 @@ which documents every option inline. The delivery and retry settings:
|
|
|
687
701
|
| `max_retry_after` | `3600` | Cap (seconds) on a honored `Retry-After`; `nil` is uncapped. |
|
|
688
702
|
| `auto_disable_endpoint_after` | `nil` | Deactivate an endpoint after N consecutive failures; `nil` never does. |
|
|
689
703
|
| `signing_secret_grace_period` | `24.hours` | How long a rotated endpoint's previous secret stays valid. |
|
|
690
|
-
| `block_private_ips` | `true` | Reject endpoint URLs resolving to private/loopback addresses (SSRF). |
|
|
704
|
+
| `block_private_ips` | `true` | Reject endpoint URLs resolving to private/loopback/special-use addresses (SSRF). |
|
|
705
|
+
| `dns_timeout` | `[1, 3]` | Per-try timeout(s) in seconds for resolving an endpoint host; `nil` uses the resolver default. |
|
|
706
|
+
| `resolve_dns_with_hosts_file` | `true` | Also resolve endpoint hosts via the system hosts file (e.g. `/etc/hosts`), not DNS only. Set `false` to harden to DNS-only. |
|
|
707
|
+
| `max_url_length` | `2048` | Maximum length of an endpoint URL. |
|
|
708
|
+
| `max_subscribed_events` | `100` | Maximum number of event patterns an endpoint may subscribe to. |
|
|
691
709
|
| `max_response_body_bytes` | `65_536` | Truncate the stored response body; `nil` stores it whole. |
|
|
692
710
|
| `delivery_attempt_retention` | `nil` | Age past which `angarium:prune` deletes attempts; `nil` keeps all. |
|
|
693
711
|
| `delivering_timeout` | `15.minutes` | Age after which `angarium:reap` requeues a stuck `delivering` delivery. |
|
data/SECURITY.md
CHANGED
|
@@ -2,23 +2,27 @@
|
|
|
2
2
|
|
|
3
3
|
## Supported versions
|
|
4
4
|
|
|
5
|
-
Angarium is pre-release. Security fixes are provided for the `0.
|
|
5
|
+
Angarium is pre-release. Security fixes are provided for the `0.2.x` series
|
|
6
6
|
during this phase. Once a stable line is released, this policy will be updated to
|
|
7
7
|
list the versions that receive security updates.
|
|
8
8
|
|
|
9
9
|
| Version | Supported |
|
|
10
10
|
| ------- | --------- |
|
|
11
|
-
| 0.
|
|
11
|
+
| 0.2.x | ✅ |
|
|
12
12
|
|
|
13
13
|
## Reporting a vulnerability
|
|
14
14
|
|
|
15
15
|
Please report security vulnerabilities **privately**. Do **not** open a public
|
|
16
16
|
issue, pull request, or discussion for a suspected vulnerability.
|
|
17
17
|
|
|
18
|
-
Use
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
(
|
|
18
|
+
Use one of the following private channels:
|
|
19
|
+
|
|
20
|
+
- **Preferred:** GitHub's private vulnerability reporting for the
|
|
21
|
+
[`radioactive-labs/angarium`](https://github.com/radioactive-labs/angarium)
|
|
22
|
+
repository — go to the **Security** tab and choose **Report a vulnerability**
|
|
23
|
+
(this opens a private draft advisory visible only to the maintainers).
|
|
24
|
+
- **Email:** [sfroelich01@gmail.com](mailto:sfroelich01@gmail.com) with the
|
|
25
|
+
subject line `[SECURITY] Angarium`.
|
|
22
26
|
|
|
23
27
|
We aim to acknowledge new reports within a few business days, and we'll keep you
|
|
24
28
|
informed as we investigate and prepare a fix. Please give us a reasonable window
|
|
@@ -15,6 +15,7 @@ module Angarium
|
|
|
15
15
|
rescue_from ActiveRecord::RecordInvalid, with: :angarium_render_invalid
|
|
16
16
|
rescue_from Angarium::Api::NotAuthorized, with: :angarium_render_forbidden
|
|
17
17
|
rescue_from Angarium::Api::UnpermittedParameter, with: :angarium_render_unpermitted
|
|
18
|
+
rescue_from ActionController::ParameterMissing, with: :angarium_render_bad_request
|
|
18
19
|
|
|
19
20
|
# Resolved current user (via config.current_user). Public so policies can
|
|
20
21
|
# read it as `controller.angarium_current_user`.
|
|
@@ -31,8 +32,19 @@ module Angarium
|
|
|
31
32
|
end
|
|
32
33
|
|
|
33
34
|
# The policy for this request (config.policy_class), holding scope,
|
|
34
|
-
# create-owner, and per-action permissions.
|
|
35
|
+
# create-owner, and per-action permissions. The record-less policy is
|
|
36
|
+
# memoized: scope/owner/expose_owner? and friends are request-global and
|
|
37
|
+
# get called repeatedly (expose_owner? runs once per serialized row), so a
|
|
38
|
+
# list response would otherwise allocate a fresh policy per endpoint. A
|
|
39
|
+
# record-bound policy (member authorize!) still constructs fresh, since it
|
|
40
|
+
# varies by record.
|
|
35
41
|
def angarium_policy(record = nil)
|
|
42
|
+
return build_angarium_policy(record) if record
|
|
43
|
+
|
|
44
|
+
@angarium_policy ||= build_angarium_policy
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def build_angarium_policy(record = nil)
|
|
36
48
|
Angarium.config.policy_class.to_s.constantize.new(self, record)
|
|
37
49
|
end
|
|
38
50
|
|
|
@@ -41,9 +53,10 @@ module Angarium
|
|
|
41
53
|
angarium_policy.scope(Angarium::Endpoint.all)
|
|
42
54
|
end
|
|
43
55
|
|
|
44
|
-
# A delivery whose endpoint is within the caller's scope, or 404.
|
|
56
|
+
# A delivery whose endpoint is within the caller's scope, or 404. Eager-load
|
|
57
|
+
# :event since delivery_json serializes event.name (avoids a follow-up query).
|
|
45
58
|
def scoped_delivery(id)
|
|
46
|
-
Angarium::Delivery.where(endpoint_id: endpoint_scope.select(:id)).find(id)
|
|
59
|
+
Angarium::Delivery.includes(:event).where(endpoint_id: endpoint_scope.select(:id)).find(id)
|
|
47
60
|
end
|
|
48
61
|
|
|
49
62
|
# Guard the current action with the policy's `<action>?` predicate.
|
|
@@ -55,8 +68,8 @@ module Angarium
|
|
|
55
68
|
# and advertises the window with a `pagination` object so clients can page
|
|
56
69
|
# (there's more when offset + count < total).
|
|
57
70
|
def render_collection(key, relation, &serializer)
|
|
58
|
-
limit =
|
|
59
|
-
offset = [
|
|
71
|
+
limit = pagination_param(:limit, 50).clamp(1, 200)
|
|
72
|
+
offset = [pagination_param(:offset, 0), 0].max
|
|
60
73
|
records = relation.limit(limit).offset(offset).to_a
|
|
61
74
|
render json: {
|
|
62
75
|
key => records.map(&serializer),
|
|
@@ -64,6 +77,15 @@ module Angarium
|
|
|
64
77
|
}
|
|
65
78
|
end
|
|
66
79
|
|
|
80
|
+
# Coerce a pagination query param to an integer, ignoring non-scalar input.
|
|
81
|
+
# A hostile `?limit[]=1` makes params[:limit] an Array/Parameters, which
|
|
82
|
+
# has no #to_i and would otherwise raise an uncaught 500; fall back to the
|
|
83
|
+
# default for anything that isn't a plain string.
|
|
84
|
+
def pagination_param(key, default)
|
|
85
|
+
raw = params[key]
|
|
86
|
+
raw.is_a?(String) ? raw.to_i : default
|
|
87
|
+
end
|
|
88
|
+
|
|
67
89
|
def render_error(status, message, **extra)
|
|
68
90
|
render json: {error: message, **extra}, status: status
|
|
69
91
|
end
|
|
@@ -71,6 +93,7 @@ module Angarium
|
|
|
71
93
|
def angarium_render_unauthorized = render_error(:unauthorized, "authentication required")
|
|
72
94
|
def angarium_render_forbidden = render_error(:forbidden, "not authorized")
|
|
73
95
|
def angarium_render_not_found = render_error(:not_found, "not found")
|
|
96
|
+
def angarium_render_bad_request(error) = render_error(:bad_request, "bad request", details: [error.message])
|
|
74
97
|
|
|
75
98
|
def angarium_render_invalid(error)
|
|
76
99
|
render_error(:unprocessable_entity, "validation failed", details: error.record.errors.full_messages)
|
|
@@ -99,6 +122,12 @@ module Angarium
|
|
|
99
122
|
created_at: endpoint.created_at.iso8601,
|
|
100
123
|
updated_at: endpoint.updated_at.iso8601
|
|
101
124
|
}
|
|
125
|
+
if angarium_policy.expose_owner?
|
|
126
|
+
# Raw columns off the row (no owner fetch, so no cross-database hit);
|
|
127
|
+
# the caller reassembles the owner via its own type/id convention.
|
|
128
|
+
json[:owner_type] = endpoint.owner_type
|
|
129
|
+
json[:owner_id] = endpoint.owner_id
|
|
130
|
+
end
|
|
102
131
|
json[:signing_secret] = endpoint.signing_secret if include_secret
|
|
103
132
|
json
|
|
104
133
|
end
|
|
@@ -1,11 +1,14 @@
|
|
|
1
1
|
module Angarium
|
|
2
2
|
class DeliverJob < ApplicationJob
|
|
3
|
-
|
|
3
|
+
# The second positional arg is accepted only for backward compatibility with
|
|
4
|
+
# jobs enqueued by an older version (force is now persisted on the delivery as
|
|
5
|
+
# `forced` and read by #deliver!); new enqueues pass just the id.
|
|
6
|
+
def perform(delivery_id, _legacy_force = nil)
|
|
4
7
|
delivery = Delivery.find_by(id: delivery_id)
|
|
5
8
|
return unless delivery
|
|
6
9
|
return unless delivery.pending?
|
|
7
10
|
|
|
8
|
-
delivery.deliver!
|
|
11
|
+
delivery.deliver!
|
|
9
12
|
end
|
|
10
13
|
end
|
|
11
14
|
end
|
|
@@ -13,12 +13,7 @@ module Angarium
|
|
|
13
13
|
belongs_to :endpoint, class_name: "Angarium::Endpoint"
|
|
14
14
|
has_many :delivery_attempts, class_name: "Angarium::DeliveryAttempt", dependent: :destroy
|
|
15
15
|
|
|
16
|
-
|
|
17
|
-
# endpoint.ping!(force: true)) so its first attempt bypasses the endpoint
|
|
18
|
-
# status guard. Not persisted; only the enqueued job carries it forward.
|
|
19
|
-
attr_accessor :force_send
|
|
20
|
-
|
|
21
|
-
after_create_commit { force_send ? DeliverJob.perform_later(id, true) : DeliverJob.perform_later(id) }
|
|
16
|
+
after_create_commit { DeliverJob.perform_later(id) }
|
|
22
17
|
|
|
23
18
|
# Recover deliveries stranded in "delivering": a worker set the state to
|
|
24
19
|
# "delivering" (in #deliver!) but died (crash, deploy, OOM) before
|
|
@@ -34,14 +29,30 @@ module Angarium
|
|
|
34
29
|
ids = where(state: "delivering").where(last_attempt_at: ..older_than.ago).pluck(:id)
|
|
35
30
|
return 0 if ids.empty?
|
|
36
31
|
|
|
37
|
-
|
|
38
|
-
ids
|
|
39
|
-
|
|
32
|
+
# Reset each id with a state-scoped compare-and-swap, not a blanket
|
|
33
|
+
# `where(id: ids)`: a delivery can leave `delivering` (succeed, be marked
|
|
34
|
+
# gone, etc.) between the pluck above and this reset. Re-asserting
|
|
35
|
+
# `state: "delivering"` means we never drag a completed delivery back to
|
|
36
|
+
# pending, and — since only the reaper that actually flipped the row
|
|
37
|
+
# enqueues — two reapers racing the same snapshot can't double-enqueue.
|
|
38
|
+
requeued = 0
|
|
39
|
+
ids.each do |id|
|
|
40
|
+
changed = where(id: id, state: "delivering")
|
|
41
|
+
.update_all(state: "pending", next_attempt_at: Time.current, updated_at: Time.current)
|
|
42
|
+
next if changed.zero?
|
|
43
|
+
|
|
44
|
+
DeliverJob.perform_later(id)
|
|
45
|
+
requeued += 1
|
|
46
|
+
end
|
|
47
|
+
requeued
|
|
40
48
|
end
|
|
41
49
|
|
|
42
50
|
# Performs one attempt. Records a DeliveryAttempt, then transitions to
|
|
43
|
-
# succeeded, blocked (SSRF), schedules a retry, or exhausts. Returns the
|
|
44
|
-
|
|
51
|
+
# succeeded, blocked (SSRF), schedules a retry, or exhausts. Returns the
|
|
52
|
+
# attempt. `force` defaults to the persisted `forced` flag so a reaped or
|
|
53
|
+
# redelivered forced attempt keeps bypassing the guard; pass it explicitly
|
|
54
|
+
# only to override in tests.
|
|
55
|
+
def deliver!(client: Client.new, force: forced)
|
|
45
56
|
payload = {delivery_id: id, endpoint_id: endpoint_id, event: event.name, force: force}
|
|
46
57
|
ActiveSupport::Notifications.instrument("deliver.angarium", payload) do
|
|
47
58
|
# An endpoint's status can change after a delivery is queued: auto-disable
|
|
@@ -62,7 +73,12 @@ module Angarium
|
|
|
62
73
|
end
|
|
63
74
|
end
|
|
64
75
|
|
|
65
|
-
|
|
76
|
+
# Atomically claim this delivery (pending -> delivering) so only one
|
|
77
|
+
# worker ever attempts it. Without the compare-and-swap, two jobs for the
|
|
78
|
+
# same id (a reaper requeue racing a stale enqueue, an adapter retry, etc.)
|
|
79
|
+
# would both pass the guard above, both POST, and both read the same stale
|
|
80
|
+
# attempt_count. If the CAS changes no row, another worker owns it: bail.
|
|
81
|
+
return nil unless claim_for_attempt!
|
|
66
82
|
payload[:attempt] = attempt_count
|
|
67
83
|
|
|
68
84
|
# Re-resolve at delivery time (rather than trusting the save-time check)
|
|
@@ -144,15 +160,33 @@ module Angarium
|
|
|
144
160
|
|
|
145
161
|
# Reset the retry cycle and re-enqueue immediately. Keeps prior
|
|
146
162
|
# DeliveryAttempt history. `force: true` sends even if the endpoint is no
|
|
147
|
-
# longer enabled (for the re-enqueued attempt)
|
|
163
|
+
# longer enabled (for the re-enqueued attempt); it is persisted so the reaper
|
|
164
|
+
# honors it too. Returns self.
|
|
148
165
|
def redeliver!(force: false)
|
|
149
|
-
update!(state: "pending", next_attempt_at: nil, attempt_count: 0)
|
|
150
|
-
|
|
166
|
+
update!(state: "pending", next_attempt_at: nil, attempt_count: 0, forced: force)
|
|
167
|
+
DeliverJob.perform_later(id)
|
|
151
168
|
self
|
|
152
169
|
end
|
|
153
170
|
|
|
154
171
|
private
|
|
155
172
|
|
|
173
|
+
# Atomic pending -> delivering claim: the single writer gate for an attempt.
|
|
174
|
+
# Returns true if THIS caller won the row (and bumps attempt_count in the same
|
|
175
|
+
# statement so concurrent claimers can't both read a stale count), false if
|
|
176
|
+
# another worker already claimed it. Reload so in-memory state matches the
|
|
177
|
+
# CAS (the rest of #deliver! reads attempt_count, and later update!s need a
|
|
178
|
+
# clean dirty baseline to persist the delivering -> succeeded/pending move).
|
|
179
|
+
def claim_for_attempt!
|
|
180
|
+
now = Time.current
|
|
181
|
+
claimed = self.class.where(id: id, state: "pending").update_all(
|
|
182
|
+
["state = 'delivering', attempt_count = attempt_count + 1, last_attempt_at = ?, updated_at = ?", now, now]
|
|
183
|
+
)
|
|
184
|
+
return false if claimed.zero?
|
|
185
|
+
|
|
186
|
+
reload
|
|
187
|
+
true
|
|
188
|
+
end
|
|
189
|
+
|
|
156
190
|
# Endpoint was paused after this delivery was queued. Park it (no attempt
|
|
157
191
|
# consumed, no failure recorded) by clearing the schedule and leaving it
|
|
158
192
|
# pending; Endpoint#enable! re-enqueues parked deliveries so a pause/resume
|
|
@@ -203,7 +237,9 @@ module Angarium
|
|
|
203
237
|
# misconfigured receiver could send a tiny Retry-After to defeat our
|
|
204
238
|
# backoff and make us hammer it. Retry-After can delay, never expedite.
|
|
205
239
|
wait = [wait, retry_after].max if retry_after
|
|
206
|
-
|
|
240
|
+
# Drop force: a recorded failure means the forced first attempt is spent,
|
|
241
|
+
# and per the deliver! contract any scheduled retry follows normal rules.
|
|
242
|
+
update!(state: "pending", next_attempt_at: Time.current + wait, forced: false)
|
|
207
243
|
DeliverJob.set(wait: wait).perform_later(id)
|
|
208
244
|
end
|
|
209
245
|
end
|
|
@@ -39,11 +39,13 @@ module Angarium
|
|
|
39
39
|
|
|
40
40
|
before_validation :ensure_signing_secret, on: :create
|
|
41
41
|
|
|
42
|
-
validates :name, presence: true
|
|
42
|
+
validates :name, presence: true, length: {maximum: 255}
|
|
43
43
|
validates :url, presence: true
|
|
44
44
|
validates :url, "angarium/endpoint_url": true, if: :verify_url_address?
|
|
45
|
+
validate :url_within_length_limit
|
|
45
46
|
validate :allowed_networks_are_valid_cidrs
|
|
46
47
|
validate :custom_headers_are_strings
|
|
48
|
+
validate :subscribed_events_are_bounded
|
|
47
49
|
|
|
48
50
|
def self.generate_signing_secret
|
|
49
51
|
"whsec_#{Base64.strict_encode64(SecureRandom.bytes(32))}"
|
|
@@ -120,8 +122,10 @@ module Angarium
|
|
|
120
122
|
def enable!
|
|
121
123
|
return false unless transition_status!(:enabled, consecutive_failures: 0)
|
|
122
124
|
# Resume deliveries parked while this endpoint was paused (pending with no
|
|
123
|
-
# scheduled attempt).
|
|
124
|
-
#
|
|
125
|
+
# scheduled attempt). This predicate can also match a delivery that already
|
|
126
|
+
# has an in-flight job (a forced ping, a just-redelivered row), but a stray
|
|
127
|
+
# re-enqueue is harmless: the atomic pending->delivering claim in
|
|
128
|
+
# Delivery#deliver! guarantees only one worker ever sends a given delivery.
|
|
125
129
|
deliveries.where(state: "pending", next_attempt_at: nil).find_each { |d| DeliverJob.perform_later(d.id) }
|
|
126
130
|
true
|
|
127
131
|
end
|
|
@@ -135,15 +139,15 @@ module Angarium
|
|
|
135
139
|
transition_status!(:enabled, from: :unverified) { Angarium.notify(:on_endpoint_verified, self) }
|
|
136
140
|
end
|
|
137
141
|
|
|
138
|
-
# Deliver a synthetic `
|
|
142
|
+
# Deliver a synthetic `ping` event to this endpoint, bypassing
|
|
139
143
|
# subscription matching (a ping is always sent). By default a ping also
|
|
140
144
|
# ignores the endpoint's status, so you can test an endpoint that is paused,
|
|
141
145
|
# disabled, or not yet enabled; pass `force: false` to respect the status
|
|
142
146
|
# guard instead. Returns the created Angarium::Delivery, whose
|
|
143
147
|
# after_create_commit enqueues the DeliverJob; reload it to inspect the outcome.
|
|
144
|
-
def ping!(payload = {message: "
|
|
145
|
-
event = Angarium::Event.create!(name: "
|
|
146
|
-
deliveries.create!(event: event
|
|
148
|
+
def ping!(payload = {message: "ping"}, force: true)
|
|
149
|
+
event = Angarium::Event.create!(name: "ping", payload: payload)
|
|
150
|
+
deliveries.create!(event: event, forced: force)
|
|
147
151
|
end
|
|
148
152
|
|
|
149
153
|
private
|
|
@@ -200,6 +204,31 @@ module Angarium
|
|
|
200
204
|
end
|
|
201
205
|
end
|
|
202
206
|
|
|
207
|
+
def url_within_length_limit
|
|
208
|
+
max = Angarium.config.max_url_length
|
|
209
|
+
return if url.blank? || max.nil? || url.length <= max
|
|
210
|
+
|
|
211
|
+
errors.add(:url, "is too long (maximum is #{max} characters)")
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
def subscribed_events_are_bounded
|
|
215
|
+
return if subscribed_events.blank?
|
|
216
|
+
|
|
217
|
+
unless subscribed_events.is_a?(Array)
|
|
218
|
+
errors.add(:subscribed_events, "must be an array of event patterns")
|
|
219
|
+
return
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
max = Angarium.config.max_subscribed_events
|
|
223
|
+
if max && subscribed_events.length > max
|
|
224
|
+
errors.add(:subscribed_events, "cannot have more than #{max} subscribed events")
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
unless subscribed_events.all? { |e| e.is_a?(String) && !e.empty? && e.length <= 255 }
|
|
228
|
+
errors.add(:subscribed_events, "must be non-empty strings of at most 255 characters")
|
|
229
|
+
end
|
|
230
|
+
end
|
|
231
|
+
|
|
203
232
|
def custom_headers_are_strings
|
|
204
233
|
return if custom_headers.blank?
|
|
205
234
|
|
|
@@ -212,6 +241,14 @@ module Angarium
|
|
|
212
241
|
if custom_headers.keys.any? { |k| RESERVED_HEADERS.include?(k.downcase) }
|
|
213
242
|
errors.add(:custom_headers, "cannot override reserved or transport headers (#{RESERVED_HEADERS.join(", ")})")
|
|
214
243
|
end
|
|
244
|
+
|
|
245
|
+
# Reject CR/LF/NUL in any key or value. The outbound HTTP client writes
|
|
246
|
+
# headers verbatim, so a CRLF in a value would inject an extra header line
|
|
247
|
+
# — smuggling e.g. a forged webhook-signature past the RESERVED_HEADERS
|
|
248
|
+
# denylist (which only guards whole keys). Fail closed instead.
|
|
249
|
+
if custom_headers.any? { |k, v| k.match?(/[\r\n\0]/) || v.match?(/[\r\n\0]/) }
|
|
250
|
+
errors.add(:custom_headers, "must not contain control characters (CR, LF, or NUL)")
|
|
251
|
+
end
|
|
215
252
|
end
|
|
216
253
|
end
|
|
217
254
|
end
|
|
@@ -61,6 +61,16 @@ module Angarium
|
|
|
61
61
|
false
|
|
62
62
|
end
|
|
63
63
|
|
|
64
|
+
# Should serialized endpoints include their owner (owner_type/owner_id)?
|
|
65
|
+
# Default: no. The owner is the tenancy boundary, resolved server-side, so
|
|
66
|
+
# the single-owner default has no reason to echo it back. Override for an
|
|
67
|
+
# admin/multi-tenant console that lists endpoints across owners and needs
|
|
68
|
+
# to tell them apart. The raw columns are exposed (already loaded on the
|
|
69
|
+
# row), not the owner record, so this adds no per-row database fetch.
|
|
70
|
+
def expose_owner?
|
|
71
|
+
false
|
|
72
|
+
end
|
|
73
|
+
|
|
64
74
|
def index? = true
|
|
65
75
|
def show? = true
|
|
66
76
|
def create? = true
|
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
class CreateAngariumEndpoints < ActiveRecord::Migration[7.1]
|
|
2
2
|
def change
|
|
3
3
|
create_table :angarium_endpoints, id: Angarium.primary_key_type do |t|
|
|
4
|
-
|
|
4
|
+
# index: false — the owner scope also orders by created_at, so the
|
|
5
|
+
# composite index below covers both the tenancy filter and the list order.
|
|
6
|
+
t.references :owner, polymorphic: true, null: false, type: :string, index: false
|
|
5
7
|
t.string :name, null: false
|
|
6
|
-
t.
|
|
8
|
+
t.text :url, null: false
|
|
7
9
|
t.string :status, null: false, default: "enabled"
|
|
8
10
|
t.text :signing_secret, null: false
|
|
9
11
|
t.json :subscribed_events, null: false, default: []
|
|
@@ -17,6 +19,9 @@ class CreateAngariumEndpoints < ActiveRecord::Migration[7.1]
|
|
|
17
19
|
t.datetime :secret_rotated_at
|
|
18
20
|
t.datetime :status_changed_at
|
|
19
21
|
t.timestamps
|
|
22
|
+
# Serves the owner-scoped list endpoint (WHERE owner_type/owner_id ORDER BY
|
|
23
|
+
# created_at DESC) in a single index scan.
|
|
24
|
+
t.index [:owner_type, :owner_id, :created_at], name: "idx_angarium_endpoints_on_owner_created_at"
|
|
20
25
|
end
|
|
21
26
|
end
|
|
22
27
|
end
|
|
@@ -2,12 +2,23 @@ class CreateAngariumDeliveries < ActiveRecord::Migration[7.1]
|
|
|
2
2
|
def change
|
|
3
3
|
create_table :angarium_deliveries, id: Angarium.primary_key_type do |t|
|
|
4
4
|
t.references :event, null: false, type: Angarium.primary_key_type, foreign_key: {to_table: :angarium_events}
|
|
5
|
-
|
|
5
|
+
# index: false — the endpoint-scoped delivery list orders by created_at, so
|
|
6
|
+
# the composite index below covers both the FK lookup and the list order.
|
|
7
|
+
t.references :endpoint, index: false, null: false, type: Angarium.primary_key_type, foreign_key: {to_table: :angarium_endpoints}
|
|
6
8
|
t.string :state, null: false, default: "pending"
|
|
7
9
|
t.integer :attempt_count, null: false, default: 0
|
|
10
|
+
# A manual ping!/redeliver! delivery whose next attempt bypasses the
|
|
11
|
+
# endpoint status guard. Persisted (not just a job arg) so the reaper can
|
|
12
|
+
# honor it when re-running an attempt that was stranded before completing;
|
|
13
|
+
# cleared once a recorded failure schedules a retry.
|
|
14
|
+
t.boolean :forced, null: false, default: false
|
|
8
15
|
t.datetime :last_attempt_at
|
|
9
16
|
t.datetime :next_attempt_at
|
|
10
17
|
t.timestamps
|
|
18
|
+
# Endpoint-scoped list endpoint (WHERE endpoint_id ORDER BY created_at DESC).
|
|
19
|
+
t.index [:endpoint_id, :created_at], name: "idx_angarium_deliveries_on_endpoint_created_at"
|
|
20
|
+
# Stalled-delivery reaper: WHERE state = 'delivering' AND last_attempt_at < ?.
|
|
21
|
+
t.index [:state, :last_attempt_at], name: "idx_angarium_deliveries_on_state_last_attempt"
|
|
11
22
|
end
|
|
12
23
|
end
|
|
13
24
|
end
|
|
@@ -1,12 +1,18 @@
|
|
|
1
1
|
class CreateAngariumDeliveryAttempts < ActiveRecord::Migration[7.1]
|
|
2
2
|
def change
|
|
3
3
|
create_table :angarium_delivery_attempts, id: Angarium.primary_key_type do |t|
|
|
4
|
-
|
|
4
|
+
# index: false — the delivery-scoped attempt list orders by created_at, so
|
|
5
|
+
# the composite index below covers both the FK lookup and the list order.
|
|
6
|
+
t.references :delivery, index: false, null: false, type: Angarium.primary_key_type, foreign_key: {to_table: :angarium_deliveries}
|
|
5
7
|
t.integer :response_code
|
|
6
8
|
t.text :response_body
|
|
7
9
|
t.string :error
|
|
8
10
|
t.float :duration
|
|
9
11
|
t.timestamps
|
|
12
|
+
# Delivery-scoped attempt list (WHERE delivery_id ORDER BY created_at DESC).
|
|
13
|
+
t.index [:delivery_id, :created_at], name: "idx_angarium_attempts_on_delivery_created_at"
|
|
14
|
+
# Retention prune: DELETE WHERE created_at < cutoff over the largest table.
|
|
15
|
+
t.index :created_at, name: "idx_angarium_attempts_on_created_at"
|
|
10
16
|
end
|
|
11
17
|
end
|
|
12
18
|
end
|
|
@@ -45,17 +45,50 @@ module Angarium
|
|
|
45
45
|
literal = to_ipaddr(host)
|
|
46
46
|
return [literal] if literal
|
|
47
47
|
|
|
48
|
-
|
|
48
|
+
# Bound DNS with config.dns_timeout via an explicit Resolv::DNS — an
|
|
49
|
+
# unbounded lookup lets a host with a deliberately slow authoritative
|
|
50
|
+
# nameserver tie up a delivery worker well past our HTTP timeouts. With the
|
|
51
|
+
# hosts file enabled (default), wrap it in a Resolv that consults
|
|
52
|
+
# /etc/hosts first (so an internally-pinned endpoint resolves, and a hosts
|
|
53
|
+
# hit short-circuits DNS); otherwise resolve DNS-only.
|
|
54
|
+
dns = Resolv::DNS.new
|
|
55
|
+
dns.timeouts = Angarium.config.dns_timeout if Angarium.config.dns_timeout
|
|
56
|
+
resolvers = Angarium.config.resolve_dns_with_hosts_file ? [Resolv::Hosts.new, dns] : [dns]
|
|
57
|
+
Resolv.new(resolvers).getaddresses(host).filter_map { |a| to_ipaddr(a.to_s) }
|
|
49
58
|
rescue
|
|
50
59
|
[]
|
|
60
|
+
ensure
|
|
61
|
+
dns&.close
|
|
51
62
|
end
|
|
52
63
|
|
|
64
|
+
# IANA special-use ranges that are unsafe webhook destinations but that
|
|
65
|
+
# Ruby's IPAddr predicates (loopback?/private?/link_local?/unique_local?) do
|
|
66
|
+
# NOT flag. Without these, e.g. `https://0.0.0.1/` (routes to localhost on
|
|
67
|
+
# Linux) or `https://[64:ff9b::a9fe:a9fe]/` (NAT64 of the cloud metadata IP)
|
|
68
|
+
# would pass validation and be delivered to. Documentation ranges
|
|
69
|
+
# (192.0.2.0/24, 2001:db8::/32, etc.) are intentionally omitted: they aren't
|
|
70
|
+
# an SSRF vector and blocking them would reject the TEST-NET fixtures.
|
|
71
|
+
SPECIAL_USE_RANGES = [
|
|
72
|
+
"0.0.0.0/8", # "this host" — routes to localhost on Linux
|
|
73
|
+
"100.64.0.0/10", # CGNAT (RFC 6598)
|
|
74
|
+
"192.0.0.0/24", # IETF protocol assignments (RFC 6890)
|
|
75
|
+
"198.18.0.0/15", # benchmarking (RFC 2544)
|
|
76
|
+
"240.0.0.0/4", # reserved / future use, incl. 255.255.255.255 broadcast
|
|
77
|
+
"64:ff9b::/96", # NAT64 well-known prefix (RFC 6052)
|
|
78
|
+
"64:ff9b:1::/48" # NAT64 local-use prefix (RFC 8215)
|
|
79
|
+
].map { |cidr| IPAddr.new(cidr) }.freeze
|
|
80
|
+
|
|
53
81
|
def private?(ip)
|
|
54
82
|
ip = normalize(ip)
|
|
55
83
|
return true if ip.to_i.zero? # 0.0.0.0 / :: (unspecified -> localhost on Linux)
|
|
56
84
|
|
|
57
85
|
ip.loopback? || ip.private? || ip.link_local? ||
|
|
58
|
-
(ip.respond_to?(:unique_local?) && ip.unique_local?)
|
|
86
|
+
(ip.respond_to?(:unique_local?) && ip.unique_local?) ||
|
|
87
|
+
special_use?(ip)
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def special_use?(ip)
|
|
91
|
+
SPECIAL_USE_RANGES.any? { |range| range.family == ip.family && range.include?(ip) }
|
|
59
92
|
end
|
|
60
93
|
|
|
61
94
|
# Collapse IPv4-mapped IPv6 (::ffff:x.x.x.x) to native IPv4 so the predicates
|
|
@@ -5,7 +5,8 @@ module Angarium
|
|
|
5
5
|
:primary_key_type, :database, :connects_to, :max_response_body_bytes,
|
|
6
6
|
:auto_disable_endpoint_after, :respect_retry_after,
|
|
7
7
|
:max_retry_after, :retry_jitter, :signing_secret_grace_period,
|
|
8
|
-
:delivery_attempt_retention, :delivering_timeout,
|
|
8
|
+
:delivery_attempt_retention, :delivering_timeout, :dns_timeout,
|
|
9
|
+
:resolve_dns_with_hosts_file, :max_url_length, :max_subscribed_events,
|
|
9
10
|
:on_delivery_exhausted, :on_endpoint_deactivated, :on_endpoint_verified,
|
|
10
11
|
:parent_controller, :current_user, :policy_class
|
|
11
12
|
|
|
@@ -34,6 +35,10 @@ module Angarium
|
|
|
34
35
|
# Takes precedence over @database for the connection.
|
|
35
36
|
@connects_to = nil
|
|
36
37
|
@max_response_body_bytes = 65_536
|
|
38
|
+
# Disable an endpoint after this many consecutive *failed deliveries* (a
|
|
39
|
+
# delivery that exhausts its whole retry schedule, or is blocked by the SSRF
|
|
40
|
+
# guard) — NOT individual failed HTTP attempts. A single delivery that
|
|
41
|
+
# retries and eventually gives up counts as one. nil disables auto-disable.
|
|
37
42
|
@auto_disable_endpoint_after = nil
|
|
38
43
|
@respect_retry_after = true
|
|
39
44
|
@max_retry_after = 3600
|
|
@@ -41,6 +46,20 @@ module Angarium
|
|
|
41
46
|
@signing_secret_grace_period = 24.hours
|
|
42
47
|
@delivery_attempt_retention = nil
|
|
43
48
|
@delivering_timeout = 15.minutes
|
|
49
|
+
# Per-try timeout(s), in seconds, for resolving an endpoint host before
|
|
50
|
+
# delivery. Bounds how long a hostile or misconfigured slow-resolving host
|
|
51
|
+
# can stall a delivery worker (Resolv::DNS retries across the array). Set to
|
|
52
|
+
# nil to use the resolver's own defaults.
|
|
53
|
+
@dns_timeout = [1, 3]
|
|
54
|
+
# Whether host resolution also consults the system hosts file (/etc/hosts),
|
|
55
|
+
# not just DNS. Default true, so an internal endpoint pinned via /etc/hosts
|
|
56
|
+
# resolves as expected. Set false to harden a deployment to DNS-only.
|
|
57
|
+
@resolve_dns_with_hosts_file = true
|
|
58
|
+
# Max length of an endpoint URL and the cap on how many event patterns an
|
|
59
|
+
# endpoint may subscribe to (each pattern is replayed by EventMatcher on
|
|
60
|
+
# every dispatch). Both bound user-supplied input.
|
|
61
|
+
@max_url_length = 2048
|
|
62
|
+
@max_subscribed_events = 100
|
|
44
63
|
@on_delivery_exhausted = nil # ->(delivery) { ... }
|
|
45
64
|
@on_endpoint_deactivated = nil # ->(endpoint, reason) { ... } reason: :consecutive_failures | :gone
|
|
46
65
|
@on_endpoint_verified = nil # ->(endpoint) { ... } fired when an unverified endpoint is verified
|
data/lib/angarium/dispatch.rb
CHANGED
|
@@ -5,9 +5,13 @@ module Angarium
|
|
|
5
5
|
def call(event_name, payload, owner:)
|
|
6
6
|
notify_payload = {event: event_name, event_id: nil, deliveries: 0}
|
|
7
7
|
ActiveSupport::Notifications.instrument("dispatch.angarium", notify_payload) do
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
# Load only the columns subscription matching needs. Endpoints carry
|
|
9
|
+
# large encrypted blobs (signing_secret, previous_signing_secret,
|
|
10
|
+
# custom_headers); pulling them for every candidate — most of which may
|
|
11
|
+
# not even match — is wasted I/O. The matched endpoints' secrets are
|
|
12
|
+
# loaded later, at delivery time, from the delivery's own association.
|
|
13
|
+
candidates = Endpoint.enabled.where(owner: owner).select(:id, :subscribed_events)
|
|
14
|
+
endpoints = candidates.to_a.select { |endpoint| endpoint.subscribed_to?(event_name) }
|
|
11
15
|
next nil if endpoints.empty?
|
|
12
16
|
|
|
13
17
|
Event.transaction do
|
data/lib/angarium/version.rb
CHANGED
|
@@ -23,6 +23,10 @@ class <%= class_name %> < Angarium::Api::Policy
|
|
|
23
23
|
# def permit_allow_private_network? = current_user.admin?
|
|
24
24
|
# def permit_allowed_networks? = true
|
|
25
25
|
|
|
26
|
+
# Include owner_type/owner_id in serialized endpoints (default off). Turn on
|
|
27
|
+
# for an admin/multi-tenant console that lists endpoints across owners.
|
|
28
|
+
# def expose_owner? = current_user.admin?
|
|
29
|
+
|
|
26
30
|
# Per-action permissions. rotate_secret?/pause?/enable?/ping?/redeliver? all
|
|
27
31
|
# default to update?.
|
|
28
32
|
# def index? = true
|