patient_http 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +20 -0
- data/README.md +67 -1
- data/VERSION +1 -1
- data/lib/patient_http/inline_task_handler.rb +30 -0
- data/lib/patient_http/processor.rb +121 -32
- data/lib/patient_http/processor_observer.rb +35 -0
- data/lib/patient_http/request_helper.rb +16 -6
- data/lib/patient_http.rb +166 -0
- data/patient_http.gemspec +0 -2
- metadata +3 -16
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d0888d7e92f43f2031bf7f0eb5b0f4dd528ac7b79957b506f1df61825309990a
|
|
4
|
+
data.tar.gz: 87b5817b8205a0b605edb7cb41facf0f6b5690122ff63cb669dbcc6c6b090cd0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4bd3586a382436ac11872d10380afb28326497c9538383ae0e884ecaf901dd952df0202e9ed740047b6336c60960ebff20b83abcdc86a585c67e0915097dbe92
|
|
7
|
+
data.tar.gz: d86c67c48c1aa03d0aa47f4675626696ab9012b526add67a4d8ebf611aca40b8229e722bd2e07c2ef6de6caa6110a01507d3317f029668fa89a768d145fea299
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,26 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## 1.4.0
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- `ProcessorObserver` events for the full task pipeline: `request_enqueued` is sent when a task is announced to the processor, before the task is visible to the reactor, so observers can set up durable tracking (e.g. a crash-recovery registry entry) before `Processor#enqueue` returns or raises. `request_rejected` is sent when an announced task is not accepted (not running or at capacity). `request_requeued` is sent when an incomplete task is re-enqueued through its task handler, so observers can tear down tracking for tasks the job system owns again. Redirect tasks are announced with `request_enqueued` as well.
|
|
12
|
+
- `Processor#tracked_request_ids` returns the IDs of all tasks in the pipeline (queued, pending, and in-flight), so durable tracking can keep heartbeats alive for tasks that have not started yet.
|
|
13
|
+
|
|
14
|
+
### Fixed
|
|
15
|
+
|
|
16
|
+
- A task is now marked as started before the `request_start` notification is sent. Before this fix, a shutdown that snapshotted the task between the two steps re-enqueued the task without a `request_end` notification, which could leak observer tracking for the task and cause a duplicate execution through crash recovery.
|
|
17
|
+
|
|
18
|
+
## 1.3.0
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- `PatientHttp.inline!` registers a request handler that executes requests inline (synchronously, in-process) through `SynchronousExecutor`, for consoles, tests, and development environments with no job-system integration. `PatientHttp.inline?` checks whether the inline handler is currently registered, and `PatientHttp.execute_inline` executes a single request inline without registering a handler.
|
|
23
|
+
- `PatientHttp.register_secret` registers named secrets at the module level, independent of any `Configuration`. Module-level secrets are applied to the new `PatientHttp.default_configuration` (immediately if set, or when it is set later), making registration order between application code and integration gem configuration irrelevant. `PatientHttp.secret_registered?` checks whether a secret is registered at the module level or on the default configuration.
|
|
24
|
+
- `PatientHttp::RequestHelper`'s `request_template` and `async_request` now accept `preprocessors:`, matching `PatientHttp.request` and `RequestTemplate`.
|
|
25
|
+
- `PatientHttp.handler_registered?` checks whether a request handler is registered.
|
|
26
|
+
|
|
7
27
|
## 1.2.0
|
|
8
28
|
|
|
9
29
|
### Added
|
data/README.md
CHANGED
|
@@ -194,6 +194,30 @@ PatientHttp.get(
|
|
|
194
194
|
|
|
195
195
|
If you are using the [patient_http-sidekiq](https://github.com/bdurand/patient_http-sidekiq) gem or the [patient_http-solid_queue](https://github.com/bdurand/patient_http-solid_queue) gem, the appropriate handler will automatically be registered for you.
|
|
196
196
|
|
|
197
|
+
### Inline Execution
|
|
198
|
+
|
|
199
|
+
For consoles, tests, and development environments where no job system is configured, you can register a handler that executes requests inline — synchronously, in-process — instead of dispatching them to a queue:
|
|
200
|
+
|
|
201
|
+
```ruby
|
|
202
|
+
PatientHttp.inline!
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Now every request made through the `PatientHttp` interface (or the `RequestHelper` mixin) runs immediately through the full request lifecycle (timeouts, redirects, error wrapping) and invokes its callback on the calling thread before returning. Callbacks can make further requests; those execute inline as well.
|
|
206
|
+
|
|
207
|
+
```ruby
|
|
208
|
+
PatientHttp.inline!
|
|
209
|
+
PatientHttp.get("https://api.example.com/users/123", callback: FetchUserCallback)
|
|
210
|
+
# FetchUserCallback#on_complete has already been invoked by this point
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Inline requests run against `PatientHttp.default_configuration` by default (or a lazily created default configuration that includes any secrets registered with `PatientHttp.register_secret` — see [Secrets](#secrets)). You can also pass an explicit configuration:
|
|
214
|
+
|
|
215
|
+
```ruby
|
|
216
|
+
PatientHttp.inline!(config: PatientHttp::Configuration.new(raise_error_responses: true))
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Use `PatientHttp.inline?` to check whether the inline handler is the currently registered handler. To execute a single request inline without registering a handler, use `PatientHttp.execute_inline(request:, callback:)`.
|
|
220
|
+
|
|
197
221
|
### RequestHelper Mixin
|
|
198
222
|
|
|
199
223
|
Use `PatientHttp::RequestHelper` when you want a simple API for creating and dispatching async HTTP requests directly from your class.
|
|
@@ -505,6 +529,23 @@ config.register_secret(:api_key) { ENV["MY_API_KEY"] } # lazy block
|
|
|
505
529
|
|
|
506
530
|
If a secret is not found when resolving a request, a `PatientHttp::SecretManager::SecretNotFoundError` is raised, which surfaces through the normal request error path.
|
|
507
531
|
|
|
532
|
+
#### Module-level registration
|
|
533
|
+
|
|
534
|
+
If the `Configuration` is owned by an integration gem (patient_http-sidekiq, patient_http-solid_queue), your application code may not have a convenient reference to it — or may load before it exists. In that case, register secrets at the module level instead:
|
|
535
|
+
|
|
536
|
+
```ruby
|
|
537
|
+
PatientHttp.register_secret(:authorization, "Bearer #{ENV['API_TOKEN']}")
|
|
538
|
+
PatientHttp.register_secret(:api_key) { ENV["MY_API_KEY"] }
|
|
539
|
+
```
|
|
540
|
+
|
|
541
|
+
Module-level secrets are applied to `PatientHttp.default_configuration` — immediately if one is already set, or as soon as one is set later — so registration order between your application code and the integration gem's configuration does not matter. Integration gems set the default configuration at the end of their configure step; you can also set it yourself:
|
|
542
|
+
|
|
543
|
+
```ruby
|
|
544
|
+
PatientHttp.default_configuration = config
|
|
545
|
+
```
|
|
546
|
+
|
|
547
|
+
Use `PatientHttp.secret_registered?(name)` to check whether a secret is available, either at the module level or on the default configuration.
|
|
548
|
+
|
|
508
549
|
### Referencing secrets when building a request
|
|
509
550
|
|
|
510
551
|
Use `PatientHttp.secret(name)` anywhere you would put a sensitive header or query parameter value. No value is needed (or available) at build time:
|
|
@@ -567,12 +608,29 @@ PatientHttp.post(
|
|
|
567
608
|
)
|
|
568
609
|
```
|
|
569
610
|
|
|
570
|
-
Multiple preprocessors can be given as an array; they run in order, each seeing the changes made by the ones before it. `RequestTemplate` also
|
|
611
|
+
Multiple preprocessors can be given as an array; they run in order, each seeing the changes made by the ones before it. `RequestTemplate` and the `RequestHelper` mixin's `request_template` also accept `preprocessors:` as a template-wide default, and the mixin's `async_*` helpers accept `preprocessors:` per request.
|
|
571
612
|
|
|
572
613
|
If a request references a preprocessor name that is not registered, a `PatientHttp::RequestPreparer::PreprocessorNotFoundError` is raised, which surfaces through the normal request error path.
|
|
573
614
|
|
|
574
615
|
When redirects are followed, preprocessors are re-run against each redirect URL so signatures stay valid. On cross-origin redirects they are dropped entirely, consistent with the stripping of `Authorization` and `Cookie` headers, so signed credentials are never sent to an unexpected origin.
|
|
575
616
|
|
|
617
|
+
## Troubleshooting
|
|
618
|
+
|
|
619
|
+
### Warning: `ThreadError: Attempt to unlock a mutex which is not locked`
|
|
620
|
+
|
|
621
|
+
On some Ruby versions you may see a warning like this in your logs:
|
|
622
|
+
|
|
623
|
+
```
|
|
624
|
+
warn: Async::Task: Async::Pool::Controller Gardener [...]
|
|
625
|
+
| Task may have ended with unhandled exception.
|
|
626
|
+
| ThreadError: Attempt to unlock a mutex which is not locked
|
|
627
|
+
| → .../async-pool-x.y.z/lib/async/pool/controller.rb:132 in `synchronize'
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
This is caused by [Ruby bug #20907](https://bugs.ruby-lang.org/issues/20907) (see also [socketry/async#424](https://github.com/socketry/async/issues/424)): under the fiber scheduler, a fiber interrupted while waiting on a `ConditionVariable` fails to re-acquire its mutex before unwinding, raising a spurious `ThreadError`. It appears whenever a pooled HTTP client is closed while its connection pool's background "gardener" task is idle — for example when a connection is evicted after a connection error, when the least recently used client is evicted because the pool is full, or when the processor shuts down.
|
|
631
|
+
|
|
632
|
+
The warning is harmless — connections are still closed correctly; only the log noise is wrong. The fix is to upgrade Ruby: the bug is fixed in Ruby 3.2.7+, 3.3.7+, and 3.4+.
|
|
633
|
+
|
|
576
634
|
## Configuration
|
|
577
635
|
|
|
578
636
|
```ruby
|
|
@@ -686,6 +744,14 @@ end
|
|
|
686
744
|
processor.observe(MetricsObserver.new)
|
|
687
745
|
```
|
|
688
746
|
|
|
747
|
+
Observers can also track the full task pipeline:
|
|
748
|
+
|
|
749
|
+
- `request_enqueued(request_task)` is called when a task is announced to the processor (before the task is visible to the reactor). It is guaranteed to arrive before `request_start`, so observers can set up durable tracking (e.g. a crash-recovery registry entry) before `Processor#enqueue` returns or raises.
|
|
750
|
+
- `request_rejected(request_task)` is called when an announced task is not accepted (not running or at capacity), so observers can tear down anything they set up in `request_enqueued`.
|
|
751
|
+
- `request_requeued(request_task)` is called when an incomplete task is re-enqueued through its task handler (processor shutdown or reactor failure). The job system owns the request again once this is sent.
|
|
752
|
+
|
|
753
|
+
Use `Processor#tracked_request_ids` to get the IDs of all tasks in the pipeline (queued, pending, and in-flight), for example to keep heartbeats alive for tasks that have not started yet.
|
|
754
|
+
|
|
689
755
|
## Testing
|
|
690
756
|
|
|
691
757
|
Use `SynchronousExecutor` to execute requests synchronously in tests. This class can be used in place of the async processor for testing your request handling logic without needing to start the full async infrastructure.
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.
|
|
1
|
+
1.4.0
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module PatientHttp
|
|
4
|
+
# No-op task handler used for inline request execution.
|
|
5
|
+
#
|
|
6
|
+
# The {SynchronousExecutor} invokes the user callback directly, so the
|
|
7
|
+
# completion and error hooks here are never exercised in practice; they are
|
|
8
|
+
# defined as no-ops to satisfy the {TaskHandler} contract. Inline requests
|
|
9
|
+
# have no job queue, so retrying is not supported.
|
|
10
|
+
#
|
|
11
|
+
# @api private
|
|
12
|
+
class InlineTaskHandler < TaskHandler
|
|
13
|
+
# @param response [Response] the HTTP response object
|
|
14
|
+
# @param callback [String] callback class name
|
|
15
|
+
# @return [void]
|
|
16
|
+
def on_complete(response, callback)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# @param error [Error] the error object
|
|
20
|
+
# @param callback [String] callback class name
|
|
21
|
+
# @return [void]
|
|
22
|
+
def on_error(error, callback)
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# @raise [NotImplementedError] inline requests cannot be retried
|
|
26
|
+
def retry
|
|
27
|
+
raise NotImplementedError, "Inline requests cannot be retried"
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
@@ -33,6 +33,10 @@ module PatientHttp
|
|
|
33
33
|
@reactor_generation = 0
|
|
34
34
|
@inflight_requests = Concurrent::Hash.new
|
|
35
35
|
@pending_tasks = Concurrent::Hash.new
|
|
36
|
+
# Tasks pushed onto @queue but not yet popped by the reactor. Kept in a
|
|
37
|
+
# hash because Thread::Queue cannot be enumerated; used to report all
|
|
38
|
+
# tracked task ids (e.g. for heartbeat updates on queued tasks).
|
|
39
|
+
@queued_tasks = Concurrent::Hash.new
|
|
36
40
|
@tasks_lock = Mutex.new
|
|
37
41
|
@idle_condition = ConditionVariable.new
|
|
38
42
|
@testing_callback = nil
|
|
@@ -76,14 +80,21 @@ module PatientHttp
|
|
|
76
80
|
# lock; re-enqueueing runs outside it. This is idempotent with stop()'s
|
|
77
81
|
# reenqueue_pending_requests: whichever runs second snapshots an empty
|
|
78
82
|
# set.
|
|
83
|
+
current_generation = false
|
|
79
84
|
orphaned_tasks = @tasks_lock.synchronize do
|
|
80
85
|
if @reactor_generation == generation
|
|
86
|
+
current_generation = true
|
|
81
87
|
drain_tracked_tasks_locked
|
|
82
88
|
else
|
|
83
89
|
[]
|
|
84
90
|
end
|
|
85
91
|
end
|
|
86
92
|
reenqueue_tasks(orphaned_tasks)
|
|
93
|
+
# Hand back tasks still sitting in the queue as well; a reactor that
|
|
94
|
+
# exits without a stop() call is the last owner of those tasks.
|
|
95
|
+
# stop() performs the same drain after reaping the reactor, and
|
|
96
|
+
# whichever drain runs second finds nothing left.
|
|
97
|
+
reenqueue_remaining_queue_items if current_generation
|
|
87
98
|
end
|
|
88
99
|
|
|
89
100
|
# The transition can fail if the reactor thread already failed and
|
|
@@ -198,22 +209,18 @@ module PatientHttp
|
|
|
198
209
|
# @raise [MaxCapacityError] if at max capacity
|
|
199
210
|
# @return [void]
|
|
200
211
|
def enqueue(task)
|
|
201
|
-
|
|
212
|
+
raise NotRunningError.new("Cannot enqueue request: processor is #{state}") unless running?
|
|
202
213
|
|
|
203
|
-
|
|
214
|
+
accepted = announce_and_enqueue(task) do
|
|
215
|
+
# The pre-check above is advisory; re-check the running state under
|
|
216
|
+
# the lock since observers were notified outside of it.
|
|
204
217
|
raise NotRunningError.new("Cannot enqueue request: processor is #{state}") unless running?
|
|
205
218
|
|
|
206
|
-
# Check capacity -
|
|
207
|
-
|
|
208
|
-
if total >= @config.max_connections
|
|
209
|
-
at_capacity = true
|
|
210
|
-
else
|
|
211
|
-
task.enqueued!
|
|
212
|
-
@queue.push(task)
|
|
213
|
-
end
|
|
219
|
+
# Check capacity - the task is only accepted below max connections.
|
|
220
|
+
@queue.size + @pending_tasks.size + @inflight_requests.size < @config.max_connections
|
|
214
221
|
end
|
|
215
222
|
|
|
216
|
-
|
|
223
|
+
unless accepted
|
|
217
224
|
notify_observers { |observer| observer.capacity_exceeded }
|
|
218
225
|
raise MaxCapacityError.new("Cannot enqueue request: already at max capacity (#{@config.max_connections} connections)")
|
|
219
226
|
end
|
|
@@ -307,6 +314,17 @@ module PatientHttp
|
|
|
307
314
|
end
|
|
308
315
|
end
|
|
309
316
|
|
|
317
|
+
# Get the IDs of all tasks in the pipeline (queued, pending, and in-flight).
|
|
318
|
+
# Use this to keep durable tracking (e.g. heartbeats) alive for tasks the
|
|
319
|
+
# processor has accepted but not yet started.
|
|
320
|
+
#
|
|
321
|
+
# @return [Array<String>]
|
|
322
|
+
def tracked_request_ids
|
|
323
|
+
@tasks_lock.synchronize do
|
|
324
|
+
(@queued_tasks.keys + @pending_tasks.keys + @inflight_requests.keys).uniq
|
|
325
|
+
end
|
|
326
|
+
end
|
|
327
|
+
|
|
310
328
|
# Add an observer for processor events.
|
|
311
329
|
#
|
|
312
330
|
# @param observer [ProcessorObserver] the observer to add
|
|
@@ -393,6 +411,7 @@ module PatientHttp
|
|
|
393
411
|
|
|
394
412
|
# Track as pending immediately to avoid race condition with stop()
|
|
395
413
|
@tasks_lock.synchronize do
|
|
414
|
+
@queued_tasks.delete(request_task.id)
|
|
396
415
|
@pending_tasks[request_task.id] = request_task
|
|
397
416
|
end
|
|
398
417
|
|
|
@@ -432,10 +451,15 @@ module PatientHttp
|
|
|
432
451
|
@config.logger&.error("[PatientHttp] Reactor loop error: #{e.inspect}\n#{e.backtrace.join("\n")}")
|
|
433
452
|
ensure
|
|
434
453
|
# Close the HTTP connection pools while still inside the reactor so the
|
|
435
|
-
# pools
|
|
436
|
-
#
|
|
437
|
-
#
|
|
438
|
-
#
|
|
454
|
+
# pools shut down in an orderly fashion: in-flight responses have been
|
|
455
|
+
# delivered above, and each pool's background gardener task is stopped
|
|
456
|
+
# by the pool itself rather than force-cancelled by the dying reactor.
|
|
457
|
+
#
|
|
458
|
+
# Note: on Ruby < 3.2.7 / < 3.3.7, stopping a gardener still logs a
|
|
459
|
+
# spurious (harmless) ThreadError: "Attempt to unlock a mutex which is
|
|
460
|
+
# not locked" — a fiber interrupted in ConditionVariable#wait fails to
|
|
461
|
+
# re-acquire its mutex (https://bugs.ruby-lang.org/issues/20907, fixed
|
|
462
|
+
# in Ruby 3.2.7+, 3.3.7+, and 3.4+).
|
|
439
463
|
begin
|
|
440
464
|
@http_client.close
|
|
441
465
|
rescue => e
|
|
@@ -468,12 +492,15 @@ module PatientHttp
|
|
|
468
492
|
|
|
469
493
|
@pending_tasks.delete(task.id)
|
|
470
494
|
@inflight_requests[task.id] = task
|
|
495
|
+
# Mark the task started in the same locked section that tracks it so
|
|
496
|
+
# a shutdown snapshot always sees a consistent started state. The
|
|
497
|
+
# shutdown re-enqueue path uses started? to pair request_end with
|
|
498
|
+
# request_start.
|
|
499
|
+
task.started!
|
|
471
500
|
end
|
|
472
501
|
|
|
473
502
|
notify_observers { |observer| observer.request_start(task) }
|
|
474
503
|
|
|
475
|
-
# Mark task as started
|
|
476
|
-
task.started!
|
|
477
504
|
claimed = false
|
|
478
505
|
|
|
479
506
|
begin
|
|
@@ -606,13 +633,9 @@ module PatientHttp
|
|
|
606
633
|
# from in-flight) and enqueue the redirect. If the claim fails the
|
|
607
634
|
# shutdown sequence already re-enqueued the original, so drop the redirect.
|
|
608
635
|
redirect_task = task.redirect_task(location: location, status: status)
|
|
609
|
-
redirect_task.enqueued!
|
|
610
|
-
|
|
611
|
-
claimed = @tasks_lock.synchronize do
|
|
612
|
-
next false if @inflight_requests.delete(task.id).nil?
|
|
613
636
|
|
|
614
|
-
|
|
615
|
-
|
|
637
|
+
claimed = announce_and_enqueue(redirect_task) do
|
|
638
|
+
!@inflight_requests.delete(task.id).nil?
|
|
616
639
|
end
|
|
617
640
|
return false unless claimed
|
|
618
641
|
|
|
@@ -641,6 +664,47 @@ module PatientHttp
|
|
|
641
664
|
raise if PatientHttp.testing?
|
|
642
665
|
end
|
|
643
666
|
|
|
667
|
+
# Announce a task to observers and make it visible to the reactor. The
|
|
668
|
+
# task is announced before it can start, finish, or be re-enqueued, so
|
|
669
|
+
# observers can set up durable tracking first. Errors from the
|
|
670
|
+
# request_enqueued announcement propagate and reject the task, because a
|
|
671
|
+
# failed tracking setup must not let the task be accepted as if it were
|
|
672
|
+
# durable. The block runs while {@tasks_lock} is held and decides whether
|
|
673
|
+
# the task is accepted; when it returns false or raises, observers receive
|
|
674
|
+
# request_rejected so they can tear down anything they set up for the
|
|
675
|
+
# request_enqueued announcement. The rejection notification never replaces
|
|
676
|
+
# an exception that is already being raised.
|
|
677
|
+
#
|
|
678
|
+
# @param task [RequestTask] the request task to announce and enqueue
|
|
679
|
+
# @return [Boolean] true if the task was accepted
|
|
680
|
+
def announce_and_enqueue(task)
|
|
681
|
+
task.enqueued!
|
|
682
|
+
accepted = false
|
|
683
|
+
|
|
684
|
+
begin
|
|
685
|
+
notify_observers! { |observer| observer.request_enqueued(task) }
|
|
686
|
+
|
|
687
|
+
@tasks_lock.synchronize do
|
|
688
|
+
if yield
|
|
689
|
+
@queued_tasks[task.id] = task
|
|
690
|
+
@queue.push(task)
|
|
691
|
+
accepted = true
|
|
692
|
+
end
|
|
693
|
+
end
|
|
694
|
+
ensure
|
|
695
|
+
unless accepted
|
|
696
|
+
pending_error = $!
|
|
697
|
+
begin
|
|
698
|
+
notify_observers { |observer| observer.request_rejected(task) }
|
|
699
|
+
rescue
|
|
700
|
+
raise unless pending_error
|
|
701
|
+
end
|
|
702
|
+
end
|
|
703
|
+
end
|
|
704
|
+
|
|
705
|
+
accepted
|
|
706
|
+
end
|
|
707
|
+
|
|
644
708
|
# Notify all observers of an event. Observers are called outside of any
|
|
645
709
|
# internal lock so they can safely call back into the processor.
|
|
646
710
|
def notify_observers(&block)
|
|
@@ -650,6 +714,16 @@ module PatientHttp
|
|
|
650
714
|
end
|
|
651
715
|
end
|
|
652
716
|
|
|
717
|
+
# Notify all observers of an event and let observer errors propagate.
|
|
718
|
+
# Used for notifications the caller must be able to react to, such as
|
|
719
|
+
# durable tracking setup in request_enqueued.
|
|
720
|
+
def notify_observers!
|
|
721
|
+
observers = @tasks_lock.synchronize { @observers.dup }
|
|
722
|
+
observers.each do |observer|
|
|
723
|
+
yield(observer)
|
|
724
|
+
end
|
|
725
|
+
end
|
|
726
|
+
|
|
653
727
|
def notify_observer(observer)
|
|
654
728
|
yield(observer)
|
|
655
729
|
rescue => e
|
|
@@ -692,16 +766,27 @@ module PatientHttp
|
|
|
692
766
|
end
|
|
693
767
|
|
|
694
768
|
def reenqueue_remaining_queue_items
|
|
695
|
-
tasks_to_reenqueue =
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
769
|
+
tasks_to_reenqueue = @tasks_lock.synchronize do
|
|
770
|
+
tasks = []
|
|
771
|
+
|
|
772
|
+
# Drain remaining items from the queue (skip nil sentinels from stop)
|
|
773
|
+
until @queue.empty?
|
|
774
|
+
begin
|
|
775
|
+
task = @queue.pop(true)
|
|
776
|
+
tasks << task if task
|
|
777
|
+
rescue ThreadError
|
|
778
|
+
break
|
|
779
|
+
end
|
|
704
780
|
end
|
|
781
|
+
|
|
782
|
+
tasks.each { |task| @queued_tasks.delete(task.id) }
|
|
783
|
+
# The reactor has exited and no new tasks can be accepted, so any id
|
|
784
|
+
# still tracked as queued belongs to a task that left the queue
|
|
785
|
+
# without reaching pending or in-flight tracking. Reclaim those tasks
|
|
786
|
+
# as well so they are not tracked forever.
|
|
787
|
+
tasks.concat(@queued_tasks.values)
|
|
788
|
+
@queued_tasks.clear
|
|
789
|
+
tasks
|
|
705
790
|
end
|
|
706
791
|
|
|
707
792
|
reenqueue_tasks(tasks_to_reenqueue)
|
|
@@ -710,6 +795,10 @@ module PatientHttp
|
|
|
710
795
|
def reenqueue_tasks(tasks_to_reenqueue)
|
|
711
796
|
tasks_to_reenqueue.each do |task|
|
|
712
797
|
task.retry
|
|
798
|
+
# The task handler's job system owns the request again; let observers
|
|
799
|
+
# tear down any durable tracking for the task. Only sent after a
|
|
800
|
+
# successful retry so a failed retry leaves the tracking in place.
|
|
801
|
+
notify_observers { |observer| observer.request_requeued(task) }
|
|
713
802
|
# Only emit request_end for tasks that actually started, so observers
|
|
714
803
|
# that pair request_start/request_end (e.g. an in-flight gauge) stay
|
|
715
804
|
# balanced. Queued-but-never-started tasks emit neither.
|
|
@@ -24,6 +24,41 @@ module PatientHttp
|
|
|
24
24
|
def capacity_exceeded
|
|
25
25
|
end
|
|
26
26
|
|
|
27
|
+
# Called when a request task is handed to the processor, before the task is
|
|
28
|
+
# visible to the reactor. The notification is guaranteed to arrive before
|
|
29
|
+
# request_start for the task, so observers can set up durable tracking
|
|
30
|
+
# (e.g. a crash-recovery registry entry) with no risk that the task
|
|
31
|
+
# completes first. If the processor does not accept the task,
|
|
32
|
+
# request_rejected is sent afterward. Unlike other notifications, an error
|
|
33
|
+
# raised here propagates from Processor#enqueue and rejects the task, so a
|
|
34
|
+
# failed tracking setup does not let the task be accepted as if it were
|
|
35
|
+
# durable.
|
|
36
|
+
#
|
|
37
|
+
# @param request_task [RequestTask] the request task that was enqueued
|
|
38
|
+
# @return [void]
|
|
39
|
+
def request_enqueued(request_task)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Called when a request task announced with request_enqueued was not
|
|
43
|
+
# accepted by the processor (not running or at capacity). Observers should
|
|
44
|
+
# tear down anything they set up in request_enqueued; the caller owns the
|
|
45
|
+
# request again once this is sent.
|
|
46
|
+
#
|
|
47
|
+
# @param request_task [RequestTask] the request task that was rejected
|
|
48
|
+
# @return [void]
|
|
49
|
+
def request_rejected(request_task)
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# Called when an incomplete request task was re-enqueued through its task
|
|
53
|
+
# handler (processor shutdown or reactor failure). The task handler's job
|
|
54
|
+
# system owns the request again once this is sent, so observers should
|
|
55
|
+
# tear down any durable tracking for the task.
|
|
56
|
+
#
|
|
57
|
+
# @param request_task [RequestTask] the request task that was re-enqueued
|
|
58
|
+
# @return [void]
|
|
59
|
+
def request_requeued(request_task)
|
|
60
|
+
end
|
|
61
|
+
|
|
27
62
|
# Called when a request starts processing.
|
|
28
63
|
#
|
|
29
64
|
# @param request_task [RequestTask] the request task that started
|
|
@@ -114,13 +114,16 @@ module PatientHttp
|
|
|
114
114
|
# @param headers [Hash] default headers for requests
|
|
115
115
|
# @param params [Hash, nil] default query parameters for requests
|
|
116
116
|
# @param timeout [Float] default timeout in seconds
|
|
117
|
+
# @param preprocessors [String, Symbol, Array<String, Symbol>, nil] default names of
|
|
118
|
+
# preprocessors registered on the configuration to apply to requests
|
|
117
119
|
# @return [void]
|
|
118
|
-
def request_template(base_url: nil, headers: {}, params: nil, timeout: 30)
|
|
120
|
+
def request_template(base_url: nil, headers: {}, params: nil, timeout: 30, preprocessors: nil)
|
|
119
121
|
@patient_http_request_template = RequestTemplate.new(
|
|
120
122
|
base_url: base_url,
|
|
121
123
|
headers: headers,
|
|
122
124
|
params: params,
|
|
123
|
-
timeout: timeout
|
|
125
|
+
timeout: timeout,
|
|
126
|
+
preprocessors: preprocessors
|
|
124
127
|
)
|
|
125
128
|
end
|
|
126
129
|
|
|
@@ -140,6 +143,8 @@ module PatientHttp
|
|
|
140
143
|
# @param raise_error_responses [Boolean, nil] when true, non-success responses are
|
|
141
144
|
# reported as errors
|
|
142
145
|
# @param callback_args [Hash, nil] JSON-compatible callback arguments
|
|
146
|
+
# @param preprocessors [String, Symbol, Array<String, Symbol>, nil] names of preprocessors
|
|
147
|
+
# registered on the configuration to apply to the request when it is sent
|
|
143
148
|
# @return [Object] return value from the registered request handler
|
|
144
149
|
def async_request(
|
|
145
150
|
method,
|
|
@@ -151,10 +156,11 @@ module PatientHttp
|
|
|
151
156
|
params: nil,
|
|
152
157
|
timeout: nil,
|
|
153
158
|
raise_error_responses: nil,
|
|
154
|
-
callback_args: nil
|
|
159
|
+
callback_args: nil,
|
|
160
|
+
preprocessors: nil
|
|
155
161
|
)
|
|
156
162
|
template = async_request_template
|
|
157
|
-
kwargs = {body: body, json: json, headers: headers, params: params, timeout: timeout}
|
|
163
|
+
kwargs = {body: body, json: json, headers: headers, params: params, timeout: timeout, preprocessors: preprocessors}
|
|
158
164
|
request = if template
|
|
159
165
|
template.request(method, url, **kwargs)
|
|
160
166
|
else
|
|
@@ -198,6 +204,8 @@ module PatientHttp
|
|
|
198
204
|
# @param raise_error_responses [Boolean, nil] when true, non-success responses are
|
|
199
205
|
# reported as errors
|
|
200
206
|
# @param callback_args [Hash, nil] JSON-compatible callback arguments
|
|
207
|
+
# @param preprocessors [String, Symbol, Array<String, Symbol>, nil] names of preprocessors
|
|
208
|
+
# registered on the configuration to apply to the request when it is sent
|
|
201
209
|
# @return [Object] return value from the registered request handler
|
|
202
210
|
def async_request(
|
|
203
211
|
method,
|
|
@@ -209,7 +217,8 @@ module PatientHttp
|
|
|
209
217
|
params: nil,
|
|
210
218
|
timeout: nil,
|
|
211
219
|
raise_error_responses: nil,
|
|
212
|
-
callback_args: nil
|
|
220
|
+
callback_args: nil,
|
|
221
|
+
preprocessors: nil
|
|
213
222
|
)
|
|
214
223
|
self.class.async_request(
|
|
215
224
|
method,
|
|
@@ -221,7 +230,8 @@ module PatientHttp
|
|
|
221
230
|
params: params,
|
|
222
231
|
timeout: timeout,
|
|
223
232
|
raise_error_responses: raise_error_responses,
|
|
224
|
-
callback_args: callback_args
|
|
233
|
+
callback_args: callback_args,
|
|
234
|
+
preprocessors: preprocessors
|
|
225
235
|
)
|
|
226
236
|
end
|
|
227
237
|
|
data/lib/patient_http.rb
CHANGED
|
@@ -51,6 +51,7 @@ module PatientHttp
|
|
|
51
51
|
autoload :ExternalStorage, File.join(__dir__, "patient_http/external_storage")
|
|
52
52
|
autoload :HttpError, File.join(__dir__, "patient_http/http_error")
|
|
53
53
|
autoload :HttpHeaders, File.join(__dir__, "patient_http/http_headers")
|
|
54
|
+
autoload :InlineTaskHandler, File.join(__dir__, "patient_http/inline_task_handler")
|
|
54
55
|
autoload :LifecycleManager, File.join(__dir__, "patient_http/lifecycle_manager")
|
|
55
56
|
autoload :OutgoingRequest, File.join(__dir__, "patient_http/outgoing_request")
|
|
56
57
|
autoload :Payload, File.join(__dir__, "patient_http/payload")
|
|
@@ -78,6 +79,11 @@ module PatientHttp
|
|
|
78
79
|
@testing = %w[RAILS_ENV RACK_ENV APP_ENV].any? { |var| ENV[var] == "test" }
|
|
79
80
|
@handler = nil
|
|
80
81
|
@handler_mutex = Monitor.new
|
|
82
|
+
@inline_handler = nil
|
|
83
|
+
@default_configuration = nil
|
|
84
|
+
@inline_configuration = nil
|
|
85
|
+
@module_secrets = {}
|
|
86
|
+
@config_mutex = Monitor.new
|
|
81
87
|
|
|
82
88
|
class << self
|
|
83
89
|
# Check if running in testing mode.
|
|
@@ -153,6 +159,81 @@ module PatientHttp
|
|
|
153
159
|
end
|
|
154
160
|
end
|
|
155
161
|
|
|
162
|
+
# Registers a request handler that executes requests inline (synchronously,
|
|
163
|
+
# in-process) instead of dispatching them to a job system.
|
|
164
|
+
#
|
|
165
|
+
# This is intended for consoles, tests, and development environments where no
|
|
166
|
+
# job-system integration gem is configured. Each request runs through
|
|
167
|
+
# {SynchronousExecutor} and the callback is invoked on the calling thread
|
|
168
|
+
# before the handler returns.
|
|
169
|
+
#
|
|
170
|
+
# @param config [Configuration, nil] configuration to execute requests against.
|
|
171
|
+
# Defaults to {.default_configuration}, or a lazily created configuration that
|
|
172
|
+
# includes any secrets registered with {.register_secret}.
|
|
173
|
+
# @return [void]
|
|
174
|
+
def inline!(config: nil)
|
|
175
|
+
handler = lambda do |request:, callback:, callback_args: nil, raise_error_responses: nil|
|
|
176
|
+
execute_inline(
|
|
177
|
+
request: request,
|
|
178
|
+
callback: callback,
|
|
179
|
+
callback_args: callback_args,
|
|
180
|
+
raise_error_responses: raise_error_responses,
|
|
181
|
+
config: config
|
|
182
|
+
)
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
@handler_mutex.synchronize do
|
|
186
|
+
register_handler(handler)
|
|
187
|
+
@inline_handler = handler
|
|
188
|
+
end
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
# Check if the currently registered handler is the inline handler registered
|
|
192
|
+
# by {.inline!}.
|
|
193
|
+
#
|
|
194
|
+
# @return [Boolean]
|
|
195
|
+
def inline?
|
|
196
|
+
@handler_mutex.synchronize { !@handler.nil? && @handler.equal?(@inline_handler) }
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
# Check if a request handler is registered.
|
|
200
|
+
#
|
|
201
|
+
# @return [Boolean]
|
|
202
|
+
def handler_registered?
|
|
203
|
+
@handler_mutex.synchronize { !@handler.nil? }
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
# Executes a request inline (synchronously, in-process) through
|
|
207
|
+
# {SynchronousExecutor}, invoking the callback with the response or error
|
|
208
|
+
# before returning.
|
|
209
|
+
#
|
|
210
|
+
# @param request [Request] the HTTP request to execute
|
|
211
|
+
# @param callback [Class, String] the callback class or name
|
|
212
|
+
# @param callback_args [Hash, nil] JSON-compatible callback arguments
|
|
213
|
+
# @param raise_error_responses [Boolean, nil] when true, non-success responses are
|
|
214
|
+
# reported as errors; defaults to the configuration's setting
|
|
215
|
+
# @param config [Configuration, nil] configuration to execute the request against.
|
|
216
|
+
# Defaults to {.default_configuration}, or a lazily created configuration that
|
|
217
|
+
# includes any secrets registered with {.register_secret}.
|
|
218
|
+
# @return [String] the request id
|
|
219
|
+
def execute_inline(request:, callback:, callback_args: nil, raise_error_responses: nil, config: nil)
|
|
220
|
+
config ||= default_configuration || inline_configuration
|
|
221
|
+
raise_error_responses = config.raise_error_responses if raise_error_responses.nil?
|
|
222
|
+
|
|
223
|
+
task = RequestTask.new(
|
|
224
|
+
request: request,
|
|
225
|
+
task_handler: InlineTaskHandler.new,
|
|
226
|
+
callback: callback,
|
|
227
|
+
callback_args: callback_args,
|
|
228
|
+
raise_error_responses: raise_error_responses,
|
|
229
|
+
default_max_redirects: config.max_redirects
|
|
230
|
+
)
|
|
231
|
+
|
|
232
|
+
SynchronousExecutor.new(task, config: config).call
|
|
233
|
+
|
|
234
|
+
task.id
|
|
235
|
+
end
|
|
236
|
+
|
|
156
237
|
# Executes the registered request handler with the given request parameters.
|
|
157
238
|
#
|
|
158
239
|
# @param request [Request] the HTTP request to handle
|
|
@@ -287,8 +368,93 @@ module PatientHttp
|
|
|
287
368
|
SecretReference.new(name)
|
|
288
369
|
end
|
|
289
370
|
|
|
371
|
+
# Register a named secret at the module level, independent of any configuration.
|
|
372
|
+
#
|
|
373
|
+
# Secrets registered here are applied to the {.default_configuration} (immediately
|
|
374
|
+
# if one is already set, or when one is set later) and to the configuration used
|
|
375
|
+
# for inline execution. This makes boot order irrelevant: application code can
|
|
376
|
+
# register secrets before or after the job-system integration gem configures the
|
|
377
|
+
# processor.
|
|
378
|
+
#
|
|
379
|
+
# @param name [String, Symbol] the secret name
|
|
380
|
+
# @param value [Object, nil] the secret value (omit when providing a block)
|
|
381
|
+
# @yield [name] a block that returns the secret value (omit when providing a value)
|
|
382
|
+
# @raise [ArgumentError] if neither or both of value and block are provided
|
|
383
|
+
# @return [void]
|
|
384
|
+
# @see Configuration#register_secret
|
|
385
|
+
def register_secret(name, value = nil, &block)
|
|
386
|
+
if value.nil? && block.nil?
|
|
387
|
+
raise ArgumentError.new("register_secret requires a value or a block")
|
|
388
|
+
end
|
|
389
|
+
|
|
390
|
+
if !value.nil? && block
|
|
391
|
+
raise ArgumentError.new("register_secret accepts either a value or a block, not both")
|
|
392
|
+
end
|
|
393
|
+
|
|
394
|
+
@config_mutex.synchronize do
|
|
395
|
+
secret_value = block || value
|
|
396
|
+
@module_secrets[name.to_s] = secret_value
|
|
397
|
+
@default_configuration&.register_secret(name, secret_value)
|
|
398
|
+
@inline_configuration&.register_secret(name, secret_value)
|
|
399
|
+
end
|
|
400
|
+
end
|
|
401
|
+
|
|
402
|
+
# Check if a secret name is registered, either at the module level via
|
|
403
|
+
# {.register_secret} or on the {.default_configuration}.
|
|
404
|
+
#
|
|
405
|
+
# @param name [String, Symbol] the secret name
|
|
406
|
+
# @return [Boolean]
|
|
407
|
+
def secret_registered?(name)
|
|
408
|
+
@config_mutex.synchronize do
|
|
409
|
+
return true if @module_secrets.include?(name.to_s)
|
|
410
|
+
|
|
411
|
+
!@default_configuration.nil? && @default_configuration.secret_manager.include?(name)
|
|
412
|
+
end
|
|
413
|
+
end
|
|
414
|
+
|
|
415
|
+
# The default configuration used for inline execution when none is provided.
|
|
416
|
+
# Job-system integration gems should set this at the end of their configure
|
|
417
|
+
# step so that module-level secrets registered with {.register_secret} are
|
|
418
|
+
# applied to the configuration the processor runs with.
|
|
419
|
+
#
|
|
420
|
+
# @return [Configuration, nil] the default configuration
|
|
421
|
+
def default_configuration
|
|
422
|
+
@config_mutex.synchronize { @default_configuration }
|
|
423
|
+
end
|
|
424
|
+
|
|
425
|
+
# Set the default configuration. Any secrets registered with {.register_secret}
|
|
426
|
+
# are applied to it; the module-level registry is retained, so re-assigning a
|
|
427
|
+
# new configuration re-applies the same secrets.
|
|
428
|
+
#
|
|
429
|
+
# @param config [Configuration, nil] the configuration to use as the default
|
|
430
|
+
# @return [void]
|
|
431
|
+
def default_configuration=(config)
|
|
432
|
+
@config_mutex.synchronize do
|
|
433
|
+
@default_configuration = config
|
|
434
|
+
apply_module_secrets(config) if config
|
|
435
|
+
end
|
|
436
|
+
end
|
|
437
|
+
|
|
290
438
|
private
|
|
291
439
|
|
|
440
|
+
# The lazily created configuration used for inline execution when no explicit
|
|
441
|
+
# or default configuration is available. Module-level secrets are applied to it.
|
|
442
|
+
#
|
|
443
|
+
# @return [Configuration]
|
|
444
|
+
def inline_configuration
|
|
445
|
+
@config_mutex.synchronize do
|
|
446
|
+
@inline_configuration ||= Configuration.new.tap { |config| apply_module_secrets(config) }
|
|
447
|
+
end
|
|
448
|
+
end
|
|
449
|
+
|
|
450
|
+
# Apply all module-level secrets to the given configuration.
|
|
451
|
+
#
|
|
452
|
+
# @param config [Configuration] the configuration to apply secrets to
|
|
453
|
+
# @return [void]
|
|
454
|
+
def apply_module_secrets(config)
|
|
455
|
+
@module_secrets.each { |name, value| config.register_secret(name, value) }
|
|
456
|
+
end
|
|
457
|
+
|
|
292
458
|
# Validates that the handler accepts the required keyword arguments.
|
|
293
459
|
#
|
|
294
460
|
# @param handler [#call] the handler to validate
|
data/patient_http.gemspec
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: patient_http
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Brian Durand
|
|
@@ -65,20 +65,6 @@ dependencies:
|
|
|
65
65
|
- - ">="
|
|
66
66
|
- !ruby/object:Gem::Version
|
|
67
67
|
version: '0'
|
|
68
|
-
- !ruby/object:Gem::Dependency
|
|
69
|
-
name: bundler
|
|
70
|
-
requirement: !ruby/object:Gem::Requirement
|
|
71
|
-
requirements:
|
|
72
|
-
- - ">="
|
|
73
|
-
- !ruby/object:Gem::Version
|
|
74
|
-
version: '0'
|
|
75
|
-
type: :development
|
|
76
|
-
prerelease: false
|
|
77
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
78
|
-
requirements:
|
|
79
|
-
- - ">="
|
|
80
|
-
- !ruby/object:Gem::Version
|
|
81
|
-
version: '0'
|
|
82
68
|
description: This gem provides a dedicated async HTTP processor that uses Ruby's Fiber
|
|
83
69
|
scheduler for non-blocking I/O. Application threads hand off HTTP requests to the
|
|
84
70
|
processor and return immediately. The processor handles hundreds of concurrent HTTP
|
|
@@ -109,6 +95,7 @@ files:
|
|
|
109
95
|
- lib/patient_http/external_storage.rb
|
|
110
96
|
- lib/patient_http/http_error.rb
|
|
111
97
|
- lib/patient_http/http_headers.rb
|
|
98
|
+
- lib/patient_http/inline_task_handler.rb
|
|
112
99
|
- lib/patient_http/lifecycle_manager.rb
|
|
113
100
|
- lib/patient_http/outgoing_request.rb
|
|
114
101
|
- lib/patient_http/payload.rb
|
|
@@ -158,7 +145,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
158
145
|
- !ruby/object:Gem::Version
|
|
159
146
|
version: '0'
|
|
160
147
|
requirements: []
|
|
161
|
-
rubygems_version:
|
|
148
|
+
rubygems_version: 3.6.9
|
|
162
149
|
specification_version: 4
|
|
163
150
|
summary: Generic async HTTP connection pool for Ruby applications using Fiber-based
|
|
164
151
|
concurrency
|