active_sanction 1.0.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +54 -1
- data/docs/api_stability.md +47 -2
- data/lib/active_sanction/client.rb +11 -2
- data/lib/active_sanction/configuration.rb +39 -0
- data/lib/active_sanction/fetcher.rb +38 -12
- data/lib/active_sanction/index.rb +61 -0
- data/lib/active_sanction/instrumentation/event.rb +122 -0
- data/lib/active_sanction/instrumentation/notifications.rb +122 -0
- data/lib/active_sanction/instrumentation.rb +245 -0
- data/lib/active_sanction/matcher.rb +82 -17
- data/lib/active_sanction/sources/base.rb +48 -6
- data/lib/active_sanction/sync.rb +54 -6
- data/lib/active_sanction/version.rb +1 -1
- data/lib/active_sanction.rb +1 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ffb2d77bf3207e3ef97a16269dcf91737e20d48871cc2aec9e747d4185793e52
|
|
4
|
+
data.tar.gz: eeca2cf2bb69972d83e15e982c2daea80969a636e4c88e86cbd29b670420098c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 748d59d2d06bb502eefd684a71c1abc74beb6b03d677eaf9a85c4563945c03fb7c9d69474988319d0ef1f03d4a68c328efd7d63ad14a62d33fa74ef4989e361b
|
|
7
|
+
data.tar.gz: dc9c4247f9e3ce24772a8045c4e7e0ea992ae7fbeb17c8f2f123f8a9ccf39fd741d7617a98cd67eef5968a16811c2cfc1b42a0c433df3da94d3b8e4f3f624f40
|
data/CHANGELOG.md
CHANGED
|
@@ -17,6 +17,58 @@ screening decision would come out as today.
|
|
|
17
17
|
|
|
18
18
|
Nothing yet.
|
|
19
19
|
|
|
20
|
+
## [1.1.0] - 2026-09-14
|
|
21
|
+
|
|
22
|
+
**A minor, because it adds.** Nothing that existed changed: no behaviour under `lib/`
|
|
23
|
+
moved, `MATCHER_VERSION` stays at `1`, and a name scores today exactly what it scored
|
|
24
|
+
under 1.0.0. Instrumentation is additive and off unless a host asks for it.
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
|
|
28
|
+
- **Instrumentation: six structured events, so a host can measure this library without
|
|
29
|
+
monkeypatching it** (#59). `ActiveSanction.configure { |c| c.instrumenter = ... }` takes
|
|
30
|
+
anything answering `#call(event)` and is handed a finished
|
|
31
|
+
`ActiveSanction::Instrumentation::Event` for each of `:fetch`, `:parse`, `:store`,
|
|
32
|
+
`:"index.build"`, `:screen` and `:sync` — every one carrying a duration and the ids
|
|
33
|
+
needed to correlate it, with no anonymous timings. The event names and every payload key
|
|
34
|
+
are public API, enumerated in [`docs/api_stability.md`](docs/api_stability.md) and on the
|
|
35
|
+
site's [instrumentation reference](https://babystep.tech/active_sanction/reference/instrumentation/),
|
|
36
|
+
and covered by the deprecation path from here.
|
|
37
|
+
|
|
38
|
+
**It is a measurement of the work and never part of it.** A subscriber that raises has its
|
|
39
|
+
exception caught, reported once through the configured logger, and dropped; the sync it
|
|
40
|
+
was watching finishes and returns the report it was going to return. The converse holds
|
|
41
|
+
too: a stage that raises emits its event with `error:` set and then the exception
|
|
42
|
+
continues exactly as if nothing were listening.
|
|
43
|
+
|
|
44
|
+
**Nothing is listening by default, and that costs nothing.** `nil` is a branch taken
|
|
45
|
+
before anything is allocated rather than a no-op object that gets called, so an
|
|
46
|
+
uninstrumented screening call builds no event and allocates no payload — which is the
|
|
47
|
+
only way a per-query event could be affordable at all. Measured rather than asserted:
|
|
48
|
+
building a matcher over 47,051 names allocates 262 more objects than before, out of
|
|
49
|
+
5.79 million, and `rake benchmark:latency` reports the same p50 either side of the change
|
|
50
|
+
(12.1–12.8 ms against a run-to-run spread that was already that wide).
|
|
51
|
+
|
|
52
|
+
Rails hosts get `ActiveSanction::Instrumentation::Notifications`, which republishes every
|
|
53
|
+
event into `ActiveSupport::Notifications` under `<event>.active_sanction`. It is an
|
|
54
|
+
adapter and not a dependency: **nothing in this gem requires ActiveSupport**, and building
|
|
55
|
+
one in a process that has not loaded it raises `ConfigurationError` rather than quietly
|
|
56
|
+
instrumenting nothing.
|
|
57
|
+
|
|
58
|
+
The issue asked for `ActiveSanction.instrumenter = ...`, and this is a configuration
|
|
59
|
+
setting instead. #55 ended process-global configuration deliberately, and a module-level
|
|
60
|
+
writer would have rebuilt the default client — dropping the matcher it had indexed every
|
|
61
|
+
stored list into — as a side effect of naming a subscriber. A `Matcher` takes its
|
|
62
|
+
instrumenter at build and freezes it with its weights, so a subscriber swapped halfway
|
|
63
|
+
through a batch cannot make half of it instrumented; that is the rule every other setting
|
|
64
|
+
on the query path already follows.
|
|
65
|
+
|
|
66
|
+
- **`Sources::Base#warnings`**, defaulting to none. Every shipped adapter already exposed
|
|
67
|
+
it and the adapter rules already required it of a new one, but the base class never said
|
|
68
|
+
so — and the `:parse` event counts warnings for every source, which a count that is
|
|
69
|
+
sometimes a `NoMethodError` cannot do. An optional hook with a default implementation, so
|
|
70
|
+
no adapter outside this repository has to change.
|
|
71
|
+
|
|
20
72
|
## [1.0.1] - 2026-09-14
|
|
21
73
|
|
|
22
74
|
A packaging and release-tooling release. **Nothing about screening changes**: no behaviour
|
|
@@ -600,6 +652,7 @@ summarized here because they are what a reader of a first release most needs:
|
|
|
600
652
|
- Recall at the default threshold is 0.939 overall on the labeled set, and every record this
|
|
601
653
|
version misses is named in the committed accuracy report.
|
|
602
654
|
|
|
603
|
-
[Unreleased]: https://github.com/Babystep-Technologies/active_sanction/compare/v1.0
|
|
655
|
+
[Unreleased]: https://github.com/Babystep-Technologies/active_sanction/compare/v1.1.0...main
|
|
656
|
+
[1.1.0]: https://github.com/Babystep-Technologies/active_sanction/compare/v1.0.1...v1.1.0
|
|
604
657
|
[1.0.1]: https://github.com/Babystep-Technologies/active_sanction/compare/v1.0.0...v1.0.1
|
|
605
658
|
[1.0.0]: https://github.com/Babystep-Technologies/active_sanction/releases/tag/v1.0.0
|
data/docs/api_stability.md
CHANGED
|
@@ -4,7 +4,7 @@ What this gem promises not to break, and what it reserves the right to change
|
|
|
4
4
|
in any release.
|
|
5
5
|
|
|
6
6
|
**The public surface is enumerated below, not inferred.** A constant being
|
|
7
|
-
reachable does not make it public;
|
|
7
|
+
reachable does not make it public; over 500 of them are reachable and 142 are
|
|
8
8
|
promised. Everything else is marked `@api private` in the source, is hidden
|
|
9
9
|
from the rendered documentation, and may be renamed, moved or deleted in a
|
|
10
10
|
patch release without a note anywhere. If you need something that is not on
|
|
@@ -107,7 +107,7 @@ on working.
|
|
|
107
107
|
|
|
108
108
|
## Contracts that are not constants
|
|
109
109
|
|
|
110
|
-
|
|
110
|
+
Five promises here are about behaviour rather than about a name, and none of
|
|
111
111
|
them is enforceable by the surface spec.
|
|
112
112
|
|
|
113
113
|
**The error hierarchy.** Within a major version an error does not move to a
|
|
@@ -128,6 +128,35 @@ anyone may produce or consume one, in any language.
|
|
|
128
128
|
store written against one schema version keeps being readable; the file layout
|
|
129
129
|
underneath a shipped store is not public and may change.
|
|
130
130
|
|
|
131
|
+
**The instrumentation events.** `ActiveSanction::Instrumentation::EVENTS`
|
|
132
|
+
names the six, and each one's payload keys are promised the same way a method
|
|
133
|
+
signature is: within a major version a key is not removed, renamed, or made to
|
|
134
|
+
mean something else, and a dashboard written against one keeps working. Keys
|
|
135
|
+
may be **added** to an event — that is how a new measurement ships without a
|
|
136
|
+
major version — so a subscriber reads the keys it knows and ignores the rest,
|
|
137
|
+
and must not assume the set is closed.
|
|
138
|
+
|
|
139
|
+
Every event carries `name`, `started_at` and `duration`, plus the keys below.
|
|
140
|
+
`error` is present only when the stage raised, in which case the keys it had
|
|
141
|
+
not reached yet are **absent rather than zero**.
|
|
142
|
+
|
|
143
|
+
| Event | Payload keys |
|
|
144
|
+
|---|---|
|
|
145
|
+
| `:fetch` | `source`, `key`, `url`, `forced`, `conditional`, `status`, `not_modified`, `bytes` |
|
|
146
|
+
| `:parse` | `source`, `bytes`, `records`, `warnings` |
|
|
147
|
+
| `:store` | `source`, `snapshot_id`, `entities`, `store`, `imported` (on an import only) |
|
|
148
|
+
| `:"index.build"` | `store`, `sources`, `snapshots`, `entities`, `names`, `keys`, `postings`, `bytes` |
|
|
149
|
+
| `:screen` | `candidates`, `scored`, `results`, `threshold`, `limit`, `sources`, `snapshots` |
|
|
150
|
+
| `:sync` | `sources`, `forced`, `concurrency`, `outcomes`, `updated`, `unchanged`, `failed`, `records` |
|
|
151
|
+
|
|
152
|
+
Two things about them are contracts rather than incidental. **A `:fetch` event
|
|
153
|
+
is one HTTP round trip** — a file served out of the payload cache after a 304
|
|
154
|
+
costs no request and emits nothing, and a source re-fetching one because its
|
|
155
|
+
cache was empty emits a second event rather than amending the first. And
|
|
156
|
+
**`bytes` on `:"index.build"` is an estimate**, documented as one on
|
|
157
|
+
`Index#profile`; it is good to within a factor a dashboard cares about and is
|
|
158
|
+
not a heap measurement.
|
|
159
|
+
|
|
131
160
|
**A `MatchResult` is reproducible.** The snapshot checksum, matcher version,
|
|
132
161
|
weights and query it stamps are what let a screening decision be re-derived
|
|
133
162
|
years later. Fields may be added to that stamp; the meaning of an existing one
|
|
@@ -138,6 +167,12 @@ does not change under it.
|
|
|
138
167
|
Named here because their absence from the list is a decision rather than an
|
|
139
168
|
oversight:
|
|
140
169
|
|
|
170
|
+
- **The instrumenter contract's other half.** A subscriber is anything
|
|
171
|
+
answering `#call(event)`, and that is public. `Instrumentation.instrument`
|
|
172
|
+
and `.emit`, which the library's own stages call, are not: where an event is
|
|
173
|
+
emitted from is an implementation detail of the stage, and a host that
|
|
174
|
+
wanted to emit one of these names itself would be publishing a measurement
|
|
175
|
+
of something this library did not do.
|
|
141
176
|
- **The matching internals** — `Index`, `Similarity`, `Phonetics`, and the
|
|
142
177
|
scorer's `Adjustments` and `NameScore`. These are where accuracy work
|
|
143
178
|
happens, and accuracy work that had to preserve a signature would stop.
|
|
@@ -193,6 +228,16 @@ ActiveSanction::Configuration::DEFAULT_USER_AGENT
|
|
|
193
228
|
ActiveSanction::Configuration::DEFAULT_XML_BACKEND
|
|
194
229
|
```
|
|
195
230
|
|
|
231
|
+
### Instrumentation
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
ActiveSanction::Instrumentation
|
|
235
|
+
ActiveSanction::Instrumentation::EVENTS
|
|
236
|
+
ActiveSanction::Instrumentation::Event
|
|
237
|
+
ActiveSanction::Instrumentation::Notifications
|
|
238
|
+
ActiveSanction::Instrumentation::Notifications::NAMESPACE
|
|
239
|
+
```
|
|
240
|
+
|
|
196
241
|
### The canonical record
|
|
197
242
|
|
|
198
243
|
```
|
|
@@ -5,6 +5,7 @@ require "sorbet-runtime"
|
|
|
5
5
|
|
|
6
6
|
require "active_sanction/configuration"
|
|
7
7
|
require "active_sanction/error"
|
|
8
|
+
require "active_sanction/instrumentation"
|
|
8
9
|
require "active_sanction/match_result"
|
|
9
10
|
|
|
10
11
|
module ActiveSanction
|
|
@@ -157,7 +158,8 @@ module ActiveSanction
|
|
|
157
158
|
@lock.synchronize do
|
|
158
159
|
@matcher ||= with_configuration do
|
|
159
160
|
Matcher.build(storage, sources: configuration.sources, weights: configuration.scorer_weights,
|
|
160
|
-
candidate_limit: configuration.candidate_limit, backend: backend
|
|
161
|
+
candidate_limit: configuration.candidate_limit, backend: backend,
|
|
162
|
+
instrumenter: configuration.instrumenter)
|
|
161
163
|
end
|
|
162
164
|
end
|
|
163
165
|
end
|
|
@@ -286,7 +288,14 @@ module ActiveSanction
|
|
|
286
288
|
sig { params(path: T.untyped, verify_with: T.untyped).returns(Snapshot) }
|
|
287
289
|
def import(path, verify_with: nil)
|
|
288
290
|
snapshot = ::File.open(path.to_s, "rb") { |io| Snapshot::Bundle.read(io, verify_with: verify_with) }
|
|
289
|
-
|
|
291
|
+
# The same `:store` event a sync emits, because it is the same fact: a
|
|
292
|
+
# list version was written to this store, and a host watching data
|
|
293
|
+
# freshness should not have to know which of the two ways it arrived.
|
|
294
|
+
fields = { source: snapshot.source, snapshot_id: snapshot.checksum, entities: snapshot.record_count,
|
|
295
|
+
store: storage.class.name, imported: true }
|
|
296
|
+
Instrumentation.instrument(configuration.instrumenter, :store, fields) do
|
|
297
|
+
with_configuration { storage.write_snapshot(snapshot) }
|
|
298
|
+
end
|
|
290
299
|
reload!
|
|
291
300
|
snapshot
|
|
292
301
|
end
|
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
require "sorbet-runtime"
|
|
5
5
|
|
|
6
6
|
require "active_sanction/error"
|
|
7
|
+
require "active_sanction/instrumentation"
|
|
7
8
|
require "active_sanction/normalizer"
|
|
8
9
|
require "active_sanction/scorer/weights"
|
|
9
10
|
require "active_sanction/version"
|
|
@@ -245,6 +246,11 @@ module ActiveSanction
|
|
|
245
246
|
sig { returns(T.untyped) }
|
|
246
247
|
attr_reader :logger
|
|
247
248
|
|
|
249
|
+
# Anything answering `#call(event)`, or nil for the default, which is that
|
|
250
|
+
# nothing is listening. See #instrumenter= and Instrumentation.
|
|
251
|
+
sig { returns(T.untyped) }
|
|
252
|
+
attr_reader :instrumenter
|
|
253
|
+
|
|
248
254
|
sig { void }
|
|
249
255
|
def initialize
|
|
250
256
|
@user_agent = T.let(DEFAULT_USER_AGENT, String)
|
|
@@ -267,6 +273,7 @@ module ActiveSanction
|
|
|
267
273
|
@normalizer_dictionary = T.let(Normalizer::Dictionary.default, Normalizer::Dictionary)
|
|
268
274
|
@scorer_weights = T.let(Scorer::Weights.default, Scorer::Weights)
|
|
269
275
|
@logger = T.let(nil, T.untyped)
|
|
276
|
+
@instrumenter = T.let(nil, T.untyped)
|
|
270
277
|
@storage = T.let(nil, T.nilable(Storage::Base))
|
|
271
278
|
@default_storage = T.let(nil, T.nilable(Storage::Base))
|
|
272
279
|
end
|
|
@@ -569,6 +576,38 @@ module ActiveSanction
|
|
|
569
576
|
@logger = value
|
|
570
577
|
end
|
|
571
578
|
|
|
579
|
+
# Where this library's structured events go: a lambda, a Method, or any
|
|
580
|
+
# object answering `#call(event)`.
|
|
581
|
+
#
|
|
582
|
+
# c.instrumenter = ->(event) { StatsD.timing("sanctions.#{event.name}", event.duration_ms) }
|
|
583
|
+
# c.instrumenter = ActiveSanction::Instrumentation::Notifications.new # a Rails host
|
|
584
|
+
#
|
|
585
|
+
# Six events -- `:fetch`, `:parse`, `:store`, `:"index.build"`, `:screen`
|
|
586
|
+
# and `:sync` -- each carrying a duration and the ids needed to correlate
|
|
587
|
+
# it. Their payload keys are public API; see Instrumentation, which is
|
|
588
|
+
# where all of it is documented, and docs/api_stability.md, which
|
|
589
|
+
# enumerates the keys.
|
|
590
|
+
#
|
|
591
|
+
# The default is nil, and nil is a branch rather than a no-op object: an
|
|
592
|
+
# installation that instruments nothing pays nothing, which is the only
|
|
593
|
+
# way a per-query event could be affordable at all.
|
|
594
|
+
#
|
|
595
|
+
# This is read where a stage is *built* rather than where it runs -- a
|
|
596
|
+
# Matcher takes its instrumenter at `build` and freezes it, the way it
|
|
597
|
+
# freezes its weights -- so changing it here affects the next matcher,
|
|
598
|
+
# sync or fetcher and never one already running. That is the same rule
|
|
599
|
+
# every other setting on the query path follows, and it is what keeps a
|
|
600
|
+
# batch from being half one set of numbers and half another.
|
|
601
|
+
sig { params(value: T.untyped).void }
|
|
602
|
+
def instrumenter=(value)
|
|
603
|
+
unless value.nil? || value.respond_to?(:call)
|
|
604
|
+
raise ConfigurationError,
|
|
605
|
+
"instrumenter must respond to #call(event), got #{value.class}. See ActiveSanction::Instrumentation."
|
|
606
|
+
end
|
|
607
|
+
|
|
608
|
+
@instrumenter = value
|
|
609
|
+
end
|
|
610
|
+
|
|
572
611
|
sig { returns(String) }
|
|
573
612
|
def self.default_storage_dir
|
|
574
613
|
-File.expand_path(File.join(Dir.home, DEFAULT_STORAGE_DIRNAME))
|
|
@@ -5,6 +5,7 @@ require "sorbet-runtime"
|
|
|
5
5
|
|
|
6
6
|
require "active_sanction/error"
|
|
7
7
|
require "active_sanction/http_client"
|
|
8
|
+
require "active_sanction/instrumentation"
|
|
8
9
|
require "active_sanction/validators"
|
|
9
10
|
require "active_sanction/validator_store"
|
|
10
11
|
require "active_sanction/fetcher/result"
|
|
@@ -64,20 +65,28 @@ module ActiveSanction
|
|
|
64
65
|
sig { returns(T.untyped) }
|
|
65
66
|
attr_reader :logger
|
|
66
67
|
|
|
68
|
+
# Where the `:fetch` event goes, or nil for nothing listening. See
|
|
69
|
+
# Instrumentation.
|
|
70
|
+
sig { returns(T.untyped) }
|
|
71
|
+
attr_reader :instrumenter
|
|
72
|
+
|
|
67
73
|
# The store defaults to disk, so the second run of a cron job benefits and
|
|
68
74
|
# not merely the second call in one process. A caller that would rather
|
|
69
75
|
# keep nothing between runs passes ValidatorStore::Memory.new.
|
|
70
76
|
sig do
|
|
71
|
-
params(client: HttpClient, store: T.untyped, stale_after: T.nilable(Numeric), logger: T.untyped
|
|
77
|
+
params(client: HttpClient, store: T.untyped, stale_after: T.nilable(Numeric), logger: T.untyped,
|
|
78
|
+
instrumenter: T.untyped).void
|
|
72
79
|
end
|
|
73
80
|
def initialize(client: HttpClient.new,
|
|
74
81
|
store: ValidatorStore::FileSystem.new,
|
|
75
82
|
stale_after: ActiveSanction.config.stale_after,
|
|
76
|
-
logger: ActiveSanction.config.logger
|
|
83
|
+
logger: ActiveSanction.config.logger,
|
|
84
|
+
instrumenter: ActiveSanction.config.instrumenter)
|
|
77
85
|
@client = T.let(client, HttpClient)
|
|
78
86
|
@store = T.let(store, T.untyped)
|
|
79
87
|
@stale_after = T.let(stale_after, T.nilable(Numeric))
|
|
80
88
|
@logger = T.let(logger, T.untyped)
|
|
89
|
+
@instrumenter = T.let(instrumenter, T.untyped)
|
|
81
90
|
end
|
|
82
91
|
|
|
83
92
|
# Fetches conditionally and buffers the body, like HttpClient#get.
|
|
@@ -91,12 +100,17 @@ module ActiveSanction
|
|
|
91
100
|
# `force: true` sends no validators, so the publisher has no way to answer
|
|
92
101
|
# 304. For the operator who suspects the cached copy is wrong and wants the
|
|
93
102
|
# bytes regardless of what the ETag says.
|
|
103
|
+
# `source:` is which list this file belongs to, and is only ever read by
|
|
104
|
+
# instrumentation: a multi-file source files its validators under
|
|
105
|
+
# `:"ofac_sdn-sdn"` and the rest, and a dashboard asking which *list* is
|
|
106
|
+
# degrading wants `:ofac_sdn`. Defaults to the fetch key, which is right
|
|
107
|
+
# for every single-file source and for a caller fetching a bare URL.
|
|
94
108
|
sig do
|
|
95
|
-
params(url: T.untyped, key: T.untyped, force: T::Boolean, headers: T::Hash[T.untyped, T.untyped]
|
|
96
|
-
|
|
109
|
+
params(url: T.untyped, key: T.untyped, force: T::Boolean, headers: T::Hash[T.untyped, T.untyped],
|
|
110
|
+
source: T.untyped).returns(Result)
|
|
97
111
|
end
|
|
98
|
-
def fetch(url, key: url, force: false, headers: {})
|
|
99
|
-
conditional(url, key, force, headers) { |request| client.get(url, headers: request) }
|
|
112
|
+
def fetch(url, key: url, force: false, headers: {}, source: nil)
|
|
113
|
+
conditional(url, key, force, headers, source) { |request| client.get(url, headers: request) }
|
|
100
114
|
end
|
|
101
115
|
|
|
102
116
|
# Streams conditionally to disk, like HttpClient#download. A 304 writes
|
|
@@ -104,10 +118,10 @@ module ActiveSanction
|
|
|
104
118
|
# is left exactly as the last download left it.
|
|
105
119
|
sig do
|
|
106
120
|
params(url: T.untyped, to: T.untyped, key: T.untyped, force: T::Boolean,
|
|
107
|
-
headers: T::Hash[T.untyped, T.untyped]).returns(Result)
|
|
121
|
+
headers: T::Hash[T.untyped, T.untyped], source: T.untyped).returns(Result)
|
|
108
122
|
end
|
|
109
|
-
def download(url, to:, key: url, force: false, headers: {})
|
|
110
|
-
conditional(url, key, force, headers) { |request| client.download(url, to: to, headers: request) }
|
|
123
|
+
def download(url, to:, key: url, force: false, headers: {}, source: nil)
|
|
124
|
+
conditional(url, key, force, headers, source) { |request| client.download(url, to: to, headers: request) }
|
|
111
125
|
end
|
|
112
126
|
|
|
113
127
|
# Whether a sync is due, answered locally and without a request.
|
|
@@ -149,14 +163,26 @@ module ActiveSanction
|
|
|
149
163
|
|
|
150
164
|
sig do
|
|
151
165
|
params(url: T.untyped, key: T.untyped, force: T::Boolean, headers: T::Hash[T.untyped, T.untyped],
|
|
166
|
+
source: T.untyped,
|
|
152
167
|
block: T.proc.params(request: T::Hash[T.untyped, T.untyped]).returns(HttpClient::Response))
|
|
153
168
|
.returns(Result)
|
|
154
169
|
end
|
|
155
|
-
def conditional(url, key, force, headers, &block)
|
|
170
|
+
def conditional(url, key, force, headers, source, &block)
|
|
156
171
|
stored = force ? nil : usable(key, url)
|
|
157
172
|
log_request(key, url, stored, force)
|
|
158
|
-
|
|
159
|
-
|
|
173
|
+
# One event per HTTP round trip, which is why `forced` is on it: a file
|
|
174
|
+
# served out of the payload cache after a 304 costs no request and emits
|
|
175
|
+
# nothing, and a source re-fetching one because its cache was empty
|
|
176
|
+
# emits a second event rather than amending the first.
|
|
177
|
+
fields = { source: source || key, key: key, url: url.to_s, forced: force }
|
|
178
|
+
Instrumentation.instrument(instrumenter, :fetch, fields) do |event|
|
|
179
|
+
response = block.call(merge(headers, stored))
|
|
180
|
+
event[:status] = response.status
|
|
181
|
+
event[:not_modified] = response.not_modified?
|
|
182
|
+
event[:bytes] = response.body.to_s.bytesize
|
|
183
|
+
event[:conditional] = !stored.nil?
|
|
184
|
+
record(key, url, stored, response)
|
|
185
|
+
end
|
|
160
186
|
end
|
|
161
187
|
|
|
162
188
|
# Validators stored against a different URL are not merely useless, they
|
|
@@ -144,6 +144,28 @@ module ActiveSanction
|
|
|
144
144
|
# numbers again on another machine.
|
|
145
145
|
POSTINGS_BUDGET = T.let(5_000, Integer)
|
|
146
146
|
|
|
147
|
+
# One posting: an Integer in an Array slot, which on a 64-bit CRuby is a
|
|
148
|
+
# tagged immediate and costs the slot and nothing else. See #profile,
|
|
149
|
+
# which is where these three are used and where their crudeness is
|
|
150
|
+
# admitted.
|
|
151
|
+
#
|
|
152
|
+
# @api private
|
|
153
|
+
POSTING_BYTES = T.let(8, Integer)
|
|
154
|
+
|
|
155
|
+
# One distinct feature: a short String -- a token, a trigram, a metaphone
|
|
156
|
+
# key -- plus its slot in a posting Hash and the empty Array header the
|
|
157
|
+
# slot points at. Measured as an order of magnitude rather than counted.
|
|
158
|
+
#
|
|
159
|
+
# @api private
|
|
160
|
+
FEATURE_BYTES = T.let(120, Integer)
|
|
161
|
+
|
|
162
|
+
# One indexed name: an Entry and the Normalizer::Form it holds, both small
|
|
163
|
+
# objects of references. The Entity and Name they point at belong to the
|
|
164
|
+
# snapshot and are not counted here.
|
|
165
|
+
#
|
|
166
|
+
# @api private
|
|
167
|
+
ENTRY_BYTES = T.let(200, Integer)
|
|
168
|
+
|
|
147
169
|
# Entries, in the order they were indexed. The posting lists hold
|
|
148
170
|
# positions in this array.
|
|
149
171
|
sig { returns(T::Array[Entry]).checked(:tests) }
|
|
@@ -247,6 +269,45 @@ module ActiveSanction
|
|
|
247
269
|
}
|
|
248
270
|
end
|
|
249
271
|
|
|
272
|
+
# How big this index is, as the `index.build` event reports it: names,
|
|
273
|
+
# distinct features, postings, and an estimate of what it all weighs.
|
|
274
|
+
#
|
|
275
|
+
# Deliberately not #stats, and the difference is the one number #stats
|
|
276
|
+
# carries that this does not. Counting distinct entities means walking
|
|
277
|
+
# every entry and uniquing 47,000 ids, which is a fine thing to do in a
|
|
278
|
+
# benchmark and not a thing to do at the end of every index build -- it
|
|
279
|
+
# allocates at exactly the moment the heap is already at its peak, with
|
|
280
|
+
# the snapshot, the builder and the finished index all still alive. The
|
|
281
|
+
# event reports entity count from the snapshots it read instead, where it
|
|
282
|
+
# is already known.
|
|
283
|
+
#
|
|
284
|
+
# **`bytes` is an estimate, and deliberately a crude one.** Ruby offers no
|
|
285
|
+
# way to ask what an object graph weighs that does not either walk every
|
|
286
|
+
# object in the heap or lie, and an index is tens of millions of small
|
|
287
|
+
# objects. What it counts is the three things that dominate: a posting is
|
|
288
|
+
# an Integer in an Array slot, a distinct feature is a String plus its
|
|
289
|
+
# slot in a Hash, and an entry is a small object holding references. What
|
|
290
|
+
# it deliberately does not count is the Entities and Names themselves --
|
|
291
|
+
# those are the snapshot's, shared with it rather than owned here, and
|
|
292
|
+
# counting them would report the same megabytes twice to a host watching
|
|
293
|
+
# both.
|
|
294
|
+
#
|
|
295
|
+
# Good to within a factor a dashboard cares about, which is what "is the
|
|
296
|
+
# index growing?" and "will another list fit?" actually need. Anything
|
|
297
|
+
# finer wants a real heap profiler.
|
|
298
|
+
sig { returns(T::Hash[Symbol, Integer]).checked(:tests) }
|
|
299
|
+
def profile
|
|
300
|
+
keys = @tokens.size + @trigrams.size + @phonetics.size
|
|
301
|
+
# `each_value` rather than a two-parameter block over the Hash: iterating
|
|
302
|
+
# a Hash as pairs allocates a two-element Array per distinct feature,
|
|
303
|
+
# which is twenty thousand short-lived objects at the exact moment the
|
|
304
|
+
# heap is at its peak -- the snapshot, the builder and the finished
|
|
305
|
+
# index all still alive. #stats can afford that and a build cannot.
|
|
306
|
+
postings = [@tokens, @trigrams, @phonetics].sum { |space| space.each_value.sum(&:size) }
|
|
307
|
+
{ names: entries.size, keys: keys, postings: postings,
|
|
308
|
+
bytes: (postings * POSTING_BYTES) + (keys * FEATURE_BYTES) + (entries.size * ENTRY_BYTES) }
|
|
309
|
+
end
|
|
310
|
+
|
|
250
311
|
sig { returns(String) }
|
|
251
312
|
def inspect = "#<#{self.class} #{size} names>"
|
|
252
313
|
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
# typed: strict
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
require "sorbet-runtime"
|
|
5
|
+
|
|
6
|
+
module ActiveSanction
|
|
7
|
+
module Instrumentation
|
|
8
|
+
# One thing this library did, after it finished doing it.
|
|
9
|
+
#
|
|
10
|
+
# ActiveSanction.configure do |c|
|
|
11
|
+
# c.instrumenter = ->(event) { StatsD.timing("sanctions.#{event.name}", event.duration_ms) }
|
|
12
|
+
# end
|
|
13
|
+
#
|
|
14
|
+
# event.name # => :fetch
|
|
15
|
+
# event.duration # => 2.418, seconds
|
|
16
|
+
# event[:source] # => :ofac_sdn
|
|
17
|
+
# event[:bytes] # => 12_845_056
|
|
18
|
+
#
|
|
19
|
+
# A subscriber is handed a finished event and never a running one. That is
|
|
20
|
+
# the difference between this and a block-based instrumenter, and it is
|
|
21
|
+
# deliberate: a subscriber that cannot wrap the work cannot retry it,
|
|
22
|
+
# cannot swallow its exception, and cannot leave a `begin` half-entered if
|
|
23
|
+
# it raises. The cost is that nothing here can time a stage from the
|
|
24
|
+
# outside -- which nothing needs to, since every event already carries its
|
|
25
|
+
# own duration.
|
|
26
|
+
#
|
|
27
|
+
# ### Every event carries a duration and enough to correlate it
|
|
28
|
+
#
|
|
29
|
+
# There are no anonymous timings. `duration` is monotonic seconds, taken
|
|
30
|
+
# across the stage this event describes, and `started_at` is the wall
|
|
31
|
+
# clock at its start -- both, because one answers "how long did it take"
|
|
32
|
+
# and the other answers "when, in the log I am reading beside this". Every
|
|
33
|
+
# event but `screen` names a `source`; `screen` names the snapshot
|
|
34
|
+
# checksums it consulted, which is the same question asked of a query.
|
|
35
|
+
#
|
|
36
|
+
# ### An event is emitted for work that raised
|
|
37
|
+
#
|
|
38
|
+
# With `error` set to the exception, and with whichever payload keys the
|
|
39
|
+
# stage had filled in before it failed. A publisher that starts timing out
|
|
40
|
+
# is exactly what a host is watching for, and an instrumentation layer
|
|
41
|
+
# that only reports successes cannot see it. See Instrumentation.
|
|
42
|
+
#
|
|
43
|
+
# Frozen, and its payload with it, so a subscriber cannot edit what the
|
|
44
|
+
# next one is handed.
|
|
45
|
+
class Event
|
|
46
|
+
extend T::Sig
|
|
47
|
+
|
|
48
|
+
# The stage this event describes -- one of Instrumentation::EVENTS.
|
|
49
|
+
sig { returns(Symbol) }
|
|
50
|
+
attr_reader :name
|
|
51
|
+
|
|
52
|
+
# What the stage measured, keyed as documented for each event name in
|
|
53
|
+
# docs/api_stability.md. Frozen.
|
|
54
|
+
sig { returns(T::Hash[Symbol, T.untyped]) }
|
|
55
|
+
attr_reader :payload
|
|
56
|
+
|
|
57
|
+
# The wall clock when the stage started, UTC. For lining an event up
|
|
58
|
+
# against a log; `duration` is what to measure with.
|
|
59
|
+
sig { returns(Time) }
|
|
60
|
+
attr_reader :started_at
|
|
61
|
+
|
|
62
|
+
# Seconds the stage took, from a monotonic clock, so it is not moved by
|
|
63
|
+
# a clock adjustment mid-stage.
|
|
64
|
+
sig { returns(Float) }
|
|
65
|
+
attr_reader :duration
|
|
66
|
+
|
|
67
|
+
sig do
|
|
68
|
+
params(name: Symbol, payload: T::Hash[Symbol, T.untyped], started_at: Time, duration: Float).void
|
|
69
|
+
end
|
|
70
|
+
def initialize(name:, payload:, started_at:, duration:)
|
|
71
|
+
@name = T.let(name, Symbol)
|
|
72
|
+
@payload = T.let(payload.freeze, T::Hash[Symbol, T.untyped])
|
|
73
|
+
@started_at = T.let(started_at, Time)
|
|
74
|
+
@duration = T.let(duration, Float)
|
|
75
|
+
freeze
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# One payload key, or nil for one this event does not carry.
|
|
79
|
+
sig { params(key: Symbol).returns(T.untyped) }
|
|
80
|
+
def [](key) = payload[key]
|
|
81
|
+
|
|
82
|
+
# Which list this was about, or nil for an event that is not about one
|
|
83
|
+
# -- `screen`, which is about a query, and `sync`, which is about a run.
|
|
84
|
+
sig { returns(T.nilable(Symbol)) }
|
|
85
|
+
def source = payload[:source]
|
|
86
|
+
|
|
87
|
+
# The exception the stage raised, or nil for one that finished. An event
|
|
88
|
+
# carrying one is a partial measurement: the keys the stage had not
|
|
89
|
+
# reached are absent rather than zero.
|
|
90
|
+
sig { returns(T.nilable(StandardError)) }
|
|
91
|
+
def error = payload[:error]
|
|
92
|
+
|
|
93
|
+
sig { returns(T::Boolean) }
|
|
94
|
+
def failed? = !error.nil?
|
|
95
|
+
|
|
96
|
+
# The wall clock when the stage finished. Derived from `started_at` and
|
|
97
|
+
# the monotonic duration rather than read again, so the two cannot
|
|
98
|
+
# disagree about how long this took.
|
|
99
|
+
sig { returns(Time) }
|
|
100
|
+
def finished_at = started_at + duration
|
|
101
|
+
|
|
102
|
+
# Milliseconds, which is what a metrics backend usually wants.
|
|
103
|
+
sig { returns(Float) }
|
|
104
|
+
def duration_ms = (duration * 1_000).round(3).to_f
|
|
105
|
+
|
|
106
|
+
# The whole event as one Hash, for a subscriber that forwards it
|
|
107
|
+
# somewhere structured. The payload is spread in at the top level, and
|
|
108
|
+
# its keys win over nothing -- `name`, `started_at` and `duration` are
|
|
109
|
+
# not payload keys on any event this library emits.
|
|
110
|
+
sig { returns(T::Hash[Symbol, T.untyped]) }
|
|
111
|
+
def to_h
|
|
112
|
+
{ name: name, started_at: started_at, duration: duration }.merge(payload)
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
sig { returns(String) }
|
|
116
|
+
def to_s = "#{name} #{format("%.3f", duration)}s#{" #{source}" if source}#{" failed" if failed?}"
|
|
117
|
+
|
|
118
|
+
sig { returns(String) }
|
|
119
|
+
def inspect = "#<#{self.class} #{name} #{payload.inspect} #{format("%.3f", duration)}s>"
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
end
|