obxcura 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +50 -0
- data/README.md +34 -23
- data/lib/obxcura/browser.rb +14 -0
- data/lib/obxcura/client.rb +8 -5
- data/lib/obxcura/frame/dom.rb +3 -4
- data/lib/obxcura/frame/runtime.rb +0 -26
- data/lib/obxcura/node.rb +40 -16
- data/lib/obxcura/page.rb +82 -36
- data/lib/obxcura/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ecefba7c3040b8f7659fceb7de9f9b3c36903ee3b370dbc43193362047edecb5
|
|
4
|
+
data.tar.gz: 84de6d7107dc9d147ba3f24ba2dc2e5567cd507fb202919a672d1b1cf01a410a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 554a2a92cfc7e93dc9d3a15bd3e9bf5c04751a03e9c933c8c291811043dae050f63246c7aa27c5c319d6d536bc4f78a6ea9282bf11c8328c26674aa614817ba6
|
|
7
|
+
data.tar.gz: 897e5a2a9ce08f4c3dd4d2a9f8861022e8f333b940c89d30f5ccea9bd3a5610b8f3b601b3a3682f739a678b782886b21c413481dc293adf1e7f71dace295c449
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,55 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.3.0] - 2026-07-29
|
|
4
|
+
|
|
5
|
+
Realigns the client with Obscura 0.1.11, which lifted several of the browser
|
|
6
|
+
limits this gem was built around. Every change below was verified against the
|
|
7
|
+
0.1.11 binary rather than taken from the release notes — two claims in those
|
|
8
|
+
notes did not hold, and are documented as still-standing constraints.
|
|
9
|
+
|
|
10
|
+
### Changed
|
|
11
|
+
|
|
12
|
+
- **Breaking:** `Page#xhr_post` is now `Page#post` and uses `fetch`, which
|
|
13
|
+
Obscura routes as of 0.1.11. `xhr_post` remains as a deprecated alias.
|
|
14
|
+
`timeout:` is now enforced *in the page* by racing the request against a
|
|
15
|
+
timer, so a tarpitting server fails at roughly `timeout:` instead of waiting
|
|
16
|
+
out the CDP reply timeout. It races rather than using `AbortSignal.timeout`
|
|
17
|
+
because an abort's rejection is swallowed and surfaces as `undefined`.
|
|
18
|
+
- **Breaking:** `Node#type` dispatches real key events via
|
|
19
|
+
`Input.dispatchKeyEvent` instead of assigning `value` directly, so the browser
|
|
20
|
+
performs the insertion and `keydown`/`input`/`keyup` fire as they would from a
|
|
21
|
+
keyboard. `change` no longer fires per keystroke — real browsers only fire it
|
|
22
|
+
on blur, so listeners depending on the old behaviour should use `input`.
|
|
23
|
+
`Input.insertText` is still unimplemented, so insertion is per character.
|
|
24
|
+
- **Breaking:** `Frame#read_string` and `Runtime::EVALUATE_CHUNK` are removed,
|
|
25
|
+
along with the `window.__obxcura_read` global. Obscura's single-message ceiling
|
|
26
|
+
went from ~500–700 KB to 64 MiB, so `Page#html` reads `outerHTML` in one round
|
|
27
|
+
trip. Verified with 8 MiB and 16 MiB strings.
|
|
28
|
+
- `Node#submit` always uses `requestSubmit` and no longer falls back to
|
|
29
|
+
`submit()`. In 0.1.11 the two genuinely differ — `submit()` bypasses the
|
|
30
|
+
cancelable submit event — so the fallback would have quietly skipped both
|
|
31
|
+
constraint validation and any registered listener.
|
|
32
|
+
|
|
33
|
+
### Added
|
|
34
|
+
|
|
35
|
+
- `Page#network_log` — the request log has existed as internal state since
|
|
36
|
+
0.1.0, but `Network.enable` was never sent, so no events ever arrived and
|
|
37
|
+
nothing could read it. The domain is now enabled per page and the log is
|
|
38
|
+
exposed. Scope is navigation-driven requests only: script-initiated requests
|
|
39
|
+
emit no CDP network events, so `#post` traffic does not appear. This
|
|
40
|
+
contradicts the 0.1.11 note for #415, and was confirmed by measurement.
|
|
41
|
+
- `Browser#clear_cookies` — listed in the 0.1.0 changelog and referenced in the
|
|
42
|
+
docs, but never actually implemented; the call site was commented out in
|
|
43
|
+
`Page#close`. Now a real method. Note cookies no longer leak between
|
|
44
|
+
connections in 0.1.11, so this only resets state within one connection.
|
|
45
|
+
|
|
46
|
+
### Performance
|
|
47
|
+
|
|
48
|
+
- `Client::READ_CHUNK` raised from 512 B to 64 KiB. With frames now reaching
|
|
49
|
+
64 MiB, a 512-byte read turned a single reply into tens of thousands of
|
|
50
|
+
syscalls. Measured on a 16 MiB reply: 0.188s → 0.105s. Past 64 KiB the curve
|
|
51
|
+
flattens.
|
|
52
|
+
|
|
3
53
|
## [0.2.0] - 2026-07-20
|
|
4
54
|
|
|
5
55
|
### Added
|
data/README.md
CHANGED
|
@@ -70,11 +70,11 @@ JS
|
|
|
70
70
|
|
|
71
71
|
### POSTing from inside the page
|
|
72
72
|
|
|
73
|
-
|
|
74
|
-
|
|
73
|
+
`#post` runs the POST from the page's context with `fetch`, reusing its cookies.
|
|
74
|
+
Values cross as arguments:
|
|
75
75
|
|
|
76
76
|
```ruby
|
|
77
|
-
result = page.
|
|
77
|
+
result = page.post(
|
|
78
78
|
"https://example.com/api/login",
|
|
79
79
|
URI.encode_www_form(user: "me", pass: "secret"), # payload
|
|
80
80
|
"application/x-www-form-urlencoded", # content type
|
|
@@ -91,24 +91,27 @@ A transport failure — the request never reached the server (CORS, the
|
|
|
91
91
|
private-network SSRF guard, mixed origin, a dead host) — raises
|
|
92
92
|
`Obxcura::ConnectionError`. If the server accepts the connection but never
|
|
93
93
|
answers, the wait ends with `Obxcura::TimeoutError`; pass `timeout:` (seconds)
|
|
94
|
-
to bound it
|
|
95
|
-
`obscura serve --stealth`.
|
|
94
|
+
to bound it, and the request is dropped in the page at roughly that mark. Some
|
|
95
|
+
anti-bot endpoints tarpit non-stealth clients — try `obscura serve --stealth`.
|
|
96
|
+
|
|
97
|
+
> Before Obscura 0.1.11 this had to use `XMLHttpRequest`, because `fetch` wasn't
|
|
98
|
+
> routed. The old name `#xhr_post` still works as a deprecated alias.
|
|
96
99
|
|
|
97
100
|
## API
|
|
98
101
|
|
|
99
102
|
`Obxcura.start(**opts)` is sugar for `Obxcura::Browser.new`.
|
|
100
103
|
|
|
101
104
|
- **`Obxcura::Browser`** — `.new(host:, port:, timeout:)`, `#create_page(url)`,
|
|
102
|
-
`#go_to`/`#goto`, `#targets`, `#version`, `#
|
|
103
|
-
Readers: `#client`, `#pages`, `#host`, `#port`.
|
|
105
|
+
`#go_to`/`#goto`, `#targets`, `#version`, `#clear_cookies`, `#command`,
|
|
106
|
+
`#close`/`#quit`. Readers: `#client`, `#pages`, `#host`, `#port`.
|
|
104
107
|
- **`Obxcura::Page`** — `#goto`/`#go_to`, `#evaluate`, `#evaluate_func`,
|
|
105
|
-
`#html`/`#body`, `#title`, `#current_url`, `#at_css`, `#css`, `#
|
|
106
|
-
`#cookies`, `#refresh`/`#reload`, `#command`, `#close`,
|
|
107
|
-
Readers: `#frame`, `#target_id`, `#session_id`, `#client`.
|
|
108
|
-
- **`Obxcura::Node`** (from `#at_css`/`#css`) — `#text`, `#[]`
|
|
109
|
-
`#
|
|
108
|
+
`#html`/`#body`, `#title`, `#current_url`, `#at_css`, `#css`, `#post`,
|
|
109
|
+
`#cookies`, `#network_log`, `#refresh`/`#reload`, `#command`, `#close`,
|
|
110
|
+
`#close_connection`. Readers: `#frame`, `#target_id`, `#session_id`, `#client`.
|
|
111
|
+
- **`Obxcura::Node`** (from `#at_css`/`#css`) — `#text`, `#value`, `#[]`
|
|
112
|
+
(attribute), `#at_css`, `#focus`, `#type`, `#submit`, `#outer_html`.
|
|
110
113
|
- **`Obxcura::Frame`** — the main frame behind a Page; carries the DOM/Runtime
|
|
111
|
-
methods Page delegates (`#evaluate`, `#at_css`, `#
|
|
114
|
+
methods Page delegates (`#evaluate`, `#at_css`, `#call_on`, …).
|
|
112
115
|
- **`Obxcura::Client`** — the CDP transport, if you need raw `#command`,
|
|
113
116
|
`#subscribe` / `#unsubscribe`, or `#close`.
|
|
114
117
|
|
|
@@ -122,22 +125,30 @@ API, so they're worth knowing:
|
|
|
122
125
|
|
|
123
126
|
- **No paint engine.** There is no screenshot API, and there never will be one
|
|
124
127
|
here. `Page.captureScreenshot` is unusable.
|
|
125
|
-
- **~500–700 KB message ceiling.** Obscura won't send a single CDP message
|
|
126
|
-
larger than that. `Page#html` works around it by snapshotting `outerHTML` into
|
|
127
|
-
a page global and pulling it back in 400 KB slices. Don't return giant values
|
|
128
|
-
from `#evaluate` directly.
|
|
129
128
|
- **DOM nodes don't serialize.** A node returned by value comes back as an
|
|
130
129
|
internal stub, so `#at_css` / `#css` resolve it (via `DOM.resolveNode`) to a
|
|
131
130
|
live handle and read it with `Runtime.callFunctionOn`.
|
|
132
|
-
- **
|
|
133
|
-
|
|
134
|
-
`
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
131
|
+
- **In-page throws vanish.** A JS `throw` comes back as `undefined` rather than
|
|
132
|
+
an exception, so the gem returns `{ error: ... }` sentinels and raises from
|
|
133
|
+
Ruby. It's also why `#post` bounds itself by racing a timer instead of using
|
|
134
|
+
`AbortSignal.timeout` — the abort works, but its rejection doesn't survive the
|
|
135
|
+
trip, so the reason has to come back as a resolved value.
|
|
136
|
+
- **The network log only sees navigation.** `#network_log` records the document
|
|
137
|
+
and its subresources. Script-initiated requests — including `#post` — emit no
|
|
138
|
+
CDP network events at all, so they never appear.
|
|
139
|
+
- **No bulk text insertion.** `Input.insertText` isn't implemented, so `#type`
|
|
140
|
+
dispatches one key event per character via `Input.dispatchKeyEvent`. Real
|
|
141
|
+
`keydown`/`input`/`keyup` fire; `change` does not, matching browsers, which
|
|
142
|
+
only fire it on blur.
|
|
138
143
|
- **Private networks are blocked by default.** To drive a local site, start the
|
|
139
144
|
browser with `obscura serve --allow-private-network`.
|
|
140
145
|
|
|
146
|
+
Two long-standing limits were lifted in Obscura 0.1.11, so if you've read older
|
|
147
|
+
notes: the **~500–700 KB message ceiling is gone** (64 MiB now, so `#html`
|
|
148
|
+
returns even a huge document in one round trip), and **cookies no longer leak
|
|
149
|
+
between connections** — each `Browser` gets its own browser context, and
|
|
150
|
+
`#clear_cookies` is now only about resetting state within one connection.
|
|
151
|
+
|
|
141
152
|
The transport is built directly on `websocket-driver` rather than
|
|
142
153
|
`websocket-client-simple`, which reads a byte at a time and wedges on large
|
|
143
154
|
frames.
|
data/lib/obxcura/browser.rb
CHANGED
|
@@ -82,6 +82,20 @@ module Obxcura
|
|
|
82
82
|
JSON.parse(Net::HTTP.get(uri))
|
|
83
83
|
end
|
|
84
84
|
|
|
85
|
+
# Drop every cookie held on this connection.
|
|
86
|
+
#
|
|
87
|
+
# Since Obscura 0.1.11 each connection owns its own browser context, so cookies
|
|
88
|
+
# no longer leak between `Browser` instances and this is only about resetting
|
|
89
|
+
# state *within* one connection — between logical sessions on the same socket,
|
|
90
|
+
# say. `obscura serve` is still long-lived and {#quit} only drops the socket, so
|
|
91
|
+
# a fresh `Browser` is the other way to get a clean jar.
|
|
92
|
+
#
|
|
93
|
+
# @return [void]
|
|
94
|
+
def clear_cookies
|
|
95
|
+
command("Network.clearBrowserCookies")
|
|
96
|
+
nil
|
|
97
|
+
end
|
|
98
|
+
|
|
85
99
|
# Stop tracking a page. Called by {Obxcura::Page#close}.
|
|
86
100
|
#
|
|
87
101
|
# @param page [Obxcura::Page] the page to forget.
|
data/lib/obxcura/client.rb
CHANGED
|
@@ -25,17 +25,20 @@ module Obxcura
|
|
|
25
25
|
# @return [Integer] default seconds to wait for a command reply.
|
|
26
26
|
DEFAULT_TIMEOUT = 30
|
|
27
27
|
|
|
28
|
-
# Obscura
|
|
29
|
-
#
|
|
28
|
+
# Matches Obscura's own 64 MiB frame ceiling, so anything the browser is
|
|
29
|
+
# willing to send, the driver is willing to assemble.
|
|
30
30
|
#
|
|
31
31
|
# @return [Integer] largest CDP frame (bytes) the driver will assemble.
|
|
32
32
|
MAX_MESSAGE_SIZE = 64 * 1024 * 1024
|
|
33
33
|
|
|
34
|
-
# Bytes pulled per read syscall.
|
|
35
|
-
#
|
|
34
|
+
# Bytes pulled per read syscall. This has to be large: since Obscura raised
|
|
35
|
+
# its frame ceiling to 64 MiB, multi-megabyte replies are routine, and a
|
|
36
|
+
# small value turns one reply into tens of thousands of syscalls. Measured
|
|
37
|
+
# on a 16 MiB reply: 512B → 0.188s, 64KiB → 0.105s. Past 64KiB the curve
|
|
38
|
+
# flattens, so the extra buffer buys nothing.
|
|
36
39
|
#
|
|
37
40
|
# @return [Integer]
|
|
38
|
-
READ_CHUNK =
|
|
41
|
+
READ_CHUNK = 64 * 1024
|
|
39
42
|
|
|
40
43
|
# @return [String] the WebSocket URL of the browser endpoint.
|
|
41
44
|
attr_reader :url
|
data/lib/obxcura/frame/dom.rb
CHANGED
|
@@ -19,13 +19,12 @@ module Obxcura
|
|
|
19
19
|
evaluate("document.title")
|
|
20
20
|
end
|
|
21
21
|
|
|
22
|
-
# The live, post-JS HTML.
|
|
23
|
-
#
|
|
24
|
-
# Aliased as `html`.
|
|
22
|
+
# The live, post-JS HTML, in a single round trip. Obscura's frame ceiling is
|
|
23
|
+
# 64 MiB, so even a very large document comes back whole. Aliased as `html`.
|
|
25
24
|
#
|
|
26
25
|
# @return [String] the rendered document's outer HTML.
|
|
27
26
|
def body
|
|
28
|
-
|
|
27
|
+
evaluate("document.documentElement.outerHTML")
|
|
29
28
|
end
|
|
30
29
|
alias_method :html, :body
|
|
31
30
|
|
|
@@ -10,13 +10,6 @@ module Obxcura
|
|
|
10
10
|
# being string-interpolated into source. Node-handle resolution, cyclic-node
|
|
11
11
|
# detection and a retry loop are deliberately left out.
|
|
12
12
|
module Runtime
|
|
13
|
-
# Obscura won't send a single CDP message larger than ~500-700KB, so we pull
|
|
14
|
-
# large strings back in slices this size and stitch them together. Well under
|
|
15
|
-
# the ceiling, and each slice is fast over the websocket-driver transport.
|
|
16
|
-
#
|
|
17
|
-
# @return [Integer] slice size (bytes) used by {#read_string}.
|
|
18
|
-
EVALUATE_CHUNK = 400_000
|
|
19
|
-
|
|
20
13
|
# Evaluate a JS expression and return its value (awaits promises).
|
|
21
14
|
#
|
|
22
15
|
# Extra args are passed to the page as real values, reachable in the
|
|
@@ -75,25 +68,6 @@ module Obxcura
|
|
|
75
68
|
))
|
|
76
69
|
end
|
|
77
70
|
|
|
78
|
-
# Pull a possibly-large JS string back in {EVALUATE_CHUNK}-sized slices. The
|
|
79
|
-
# expression is snapshotted into a page global once, so it's evaluated a
|
|
80
|
-
# single time no matter how big the result is; we then slice that global.
|
|
81
|
-
#
|
|
82
|
-
# @param js_expression [String] a JS expression producing a string.
|
|
83
|
-
# @return [String] the full string, reassembled from slices.
|
|
84
|
-
def read_string(js_expression)
|
|
85
|
-
length = evaluate("(window.__obxcura_read = String(#{js_expression})).length").to_i
|
|
86
|
-
return "" if length.zero?
|
|
87
|
-
|
|
88
|
-
buffer = String.new(capacity: length)
|
|
89
|
-
offset = 0
|
|
90
|
-
while offset < length
|
|
91
|
-
buffer << evaluate("window.__obxcura_read.slice(#{offset}, #{offset + EVALUATE_CHUNK})")
|
|
92
|
-
offset += EVALUATE_CHUNK
|
|
93
|
-
end
|
|
94
|
-
buffer
|
|
95
|
-
end
|
|
96
|
-
|
|
97
71
|
private
|
|
98
72
|
|
|
99
73
|
# objectId of the live global object, fetched fresh so it can't go stale
|
data/lib/obxcura/node.rb
CHANGED
|
@@ -51,28 +51,42 @@ module Obxcura
|
|
|
51
51
|
tap { @frame.page.command("DOM.focus", objectId: remote_object_id) }
|
|
52
52
|
end
|
|
53
53
|
|
|
54
|
-
# Type text into this node
|
|
55
|
-
#
|
|
56
|
-
# and fires
|
|
57
|
-
#
|
|
54
|
+
# Type text into this node with real key events, one character at a time —
|
|
55
|
+
# Obscura implements `Input.dispatchKeyEvent`, so the browser itself performs
|
|
56
|
+
# the insertion and fires `keydown`, `input` and `keyup` the way a keyboard
|
|
57
|
+
# would. Focuses the node first, since key events go to the active element.
|
|
58
|
+
# Appends, matching keyboard behaviour when the node already has text.
|
|
59
|
+
#
|
|
60
|
+
# Note `change` does *not* fire per keystroke — real browsers only fire it on
|
|
61
|
+
# blur. The previous implementation synthesized both by assigning `value`
|
|
62
|
+
# directly; listeners that relied on that `change` need to react to `input`.
|
|
63
|
+
#
|
|
64
|
+
# `Input.insertText` is still unimplemented in Obscura 0.1.11, so there is no
|
|
65
|
+
# bulk-insert fast path: this costs two CDP round trips per character. Fine for
|
|
66
|
+
# form fields (100 chars ≈ 40ms), but it is genuinely slower than the old
|
|
67
|
+
# single-assignment approach — 500 chars ≈ 0.11s — so don't use it to stuff
|
|
68
|
+
# large text through. Assign `value` via {Frame::Runtime#call_on} for that, and
|
|
69
|
+
# accept that no key events fire.
|
|
58
70
|
#
|
|
59
71
|
# @param keys [Array<String>] text fragments to type (joined).
|
|
60
72
|
# @return [self]
|
|
61
73
|
def type(*keys)
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
JS
|
|
69
|
-
}
|
|
74
|
+
focus
|
|
75
|
+
keys.join.each_char do |char|
|
|
76
|
+
dispatch_key("keyDown", char, text: char)
|
|
77
|
+
dispatch_key("keyUp", char)
|
|
78
|
+
end
|
|
79
|
+
self
|
|
70
80
|
end
|
|
71
81
|
|
|
72
82
|
# Submit this node's form. Works whether the node is the `<form>` itself or a
|
|
73
|
-
# control inside one (resolved via `.form` / closest `<form>`).
|
|
74
|
-
#
|
|
75
|
-
#
|
|
83
|
+
# control inside one (resolved via `.form` / closest `<form>`).
|
|
84
|
+
#
|
|
85
|
+
# Always goes through `requestSubmit`, which follows the interactive path:
|
|
86
|
+
# constraint validation runs and a cancelable `submit` event fires. There is no
|
|
87
|
+
# fallback to `submit()` — as of Obscura 0.1.11 the two genuinely differ, with
|
|
88
|
+
# `submit()` bypassing the submit event entirely, so falling back would quietly
|
|
89
|
+
# skip both validation and any listener a caller registered.
|
|
76
90
|
#
|
|
77
91
|
# @return [self]
|
|
78
92
|
# @raise [Obxcura::ProtocolError] if the node isn't a form or inside one.
|
|
@@ -81,7 +95,7 @@ module Obxcura
|
|
|
81
95
|
function() {
|
|
82
96
|
const form = this.tagName === "FORM" ? this : (this.form || this.closest("form"));
|
|
83
97
|
if (!form) return { error: "node is not a form and has no ancestor form" };
|
|
84
|
-
form.requestSubmit
|
|
98
|
+
form.requestSubmit();
|
|
85
99
|
}
|
|
86
100
|
JS
|
|
87
101
|
raise ProtocolError, result["error"] if result.is_a?(Hash) && result["error"]
|
|
@@ -93,5 +107,15 @@ module Obxcura
|
|
|
93
107
|
def outer_html
|
|
94
108
|
@frame.call_on(remote_object_id, "function() { return this.outerHTML; }")
|
|
95
109
|
end
|
|
110
|
+
|
|
111
|
+
private
|
|
112
|
+
|
|
113
|
+
# `code` is deliberately left off: it describes a physical key, which we can't
|
|
114
|
+
# infer from a character, and Obscura keys insertion off `text` anyway.
|
|
115
|
+
def dispatch_key(type, char, text: nil)
|
|
116
|
+
params = { type: type, key: char }
|
|
117
|
+
params[:text] = text if text
|
|
118
|
+
@frame.page.command("Input.dispatchKeyEvent", params)
|
|
119
|
+
end
|
|
96
120
|
end
|
|
97
121
|
end
|
data/lib/obxcura/page.rb
CHANGED
|
@@ -23,6 +23,12 @@ module Obxcura
|
|
|
23
23
|
class Page
|
|
24
24
|
extend Forwardable
|
|
25
25
|
|
|
26
|
+
# Seconds of slack given to the CDP reply beyond a {#post} timeout, so the
|
|
27
|
+
# in-page abort is what surfaces rather than the transport giving up first.
|
|
28
|
+
#
|
|
29
|
+
# @return [Integer]
|
|
30
|
+
TIMEOUT_HEADROOM = 5
|
|
31
|
+
|
|
26
32
|
# @return [String] the CDP target id backing this page.
|
|
27
33
|
# @return [String] the CDP session id attached to the target.
|
|
28
34
|
# @return [Obxcura::Client] the shared CDP transport.
|
|
@@ -47,6 +53,21 @@ module Obxcura
|
|
|
47
53
|
@network_mutex = Mutex.new
|
|
48
54
|
|
|
49
55
|
@client.subscribe(@session_id) { |method, params| dispatch_event(method, params) }
|
|
56
|
+
command("Network.enable")
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Requests this page issued, oldest first, as
|
|
60
|
+
# `{ url:, request_id:, finished: }`.
|
|
61
|
+
#
|
|
62
|
+
# Scope is deliberately narrow: Obscura emits Network events for requests the
|
|
63
|
+
# *navigation* drives (the document and its subresources), but not for ones
|
|
64
|
+
# started from script. A {#post} — or any in-page `fetch`/`XMLHttpRequest` —
|
|
65
|
+
# therefore never shows up here. Verified against Obscura 0.1.11: enabling the
|
|
66
|
+
# Network domain and issuing a scripted POST produces no events at all.
|
|
67
|
+
#
|
|
68
|
+
# @return [Array<Hash>] a snapshot of the log, safe to iterate.
|
|
69
|
+
def network_log
|
|
70
|
+
@network_mutex.synchronize { @network_log.map(&:dup) }
|
|
50
71
|
end
|
|
51
72
|
|
|
52
73
|
# Navigate to `url` and block until the page's load event fires. Aliased as
|
|
@@ -105,50 +126,70 @@ module Obxcura
|
|
|
105
126
|
@client.command(method, params, session_id: @session_id)
|
|
106
127
|
end
|
|
107
128
|
|
|
108
|
-
# POST
|
|
109
|
-
#
|
|
110
|
-
#
|
|
111
|
-
#
|
|
112
|
-
#
|
|
113
|
-
# a dead host) — raises ConnectionError instead of silently returning nil.
|
|
129
|
+
# POST from the page context via `fetch`. All values cross as arguments, never
|
|
130
|
+
# interpolated into the JS. Returns { status, ok, body } on any HTTP reply
|
|
131
|
+
# (including 4xx/5xx). A transport failure — the request never reached the
|
|
132
|
+
# server (blocked by CORS / private-network SSRF guard, mixed origin, or a
|
|
133
|
+
# dead host) — raises ConnectionError instead of silently returning nil.
|
|
114
134
|
#
|
|
115
|
-
# `timeout` (seconds)
|
|
116
|
-
# accepts the connection
|
|
117
|
-
# tarpit non-stealth clients)
|
|
118
|
-
#
|
|
119
|
-
#
|
|
135
|
+
# `timeout` (seconds) is enforced in the page by racing the fetch against a
|
|
136
|
+
# timer, so a server that accepts the connection and never answers (some
|
|
137
|
+
# anti-bot endpoints tarpit non-stealth clients) fails in roughly `timeout`
|
|
138
|
+
# seconds instead of {Client::DEFAULT_TIMEOUT}. The CDP reply gets a little
|
|
139
|
+
# headroom past that so the in-page result is what we observe.
|
|
140
|
+
#
|
|
141
|
+
# The race is deliberate, and not the obvious `AbortSignal.timeout`. Obscura
|
|
142
|
+
# does accept an abort signal, but when the abort actually fires, the fetch
|
|
143
|
+
# rejection is swallowed and the call returns `undefined` — the same
|
|
144
|
+
# in-page-throws-vanish behaviour that makes {Frame::Runtime} use `{ error: }`
|
|
145
|
+
# sentinels. A rejection can't carry the reason across, so a resolved value
|
|
146
|
+
# has to. We still abort the underlying request once the timer wins, purely so
|
|
147
|
+
# it stops occupying the connection.
|
|
148
|
+
#
|
|
149
|
+
# Requests made here do not appear in {#network_log}; see that method.
|
|
120
150
|
#
|
|
121
151
|
# @param url [String] the URL to POST to.
|
|
122
152
|
# @param payload [String] the raw request body.
|
|
123
153
|
# @param content_type [String] the Content-Type header value.
|
|
124
154
|
# @param headers [Hash{String=>String}] extra request headers.
|
|
125
|
-
# @param timeout [Integer, nil] seconds to
|
|
155
|
+
# @param timeout [Integer, nil] seconds to allow before giving up.
|
|
126
156
|
# @return [Hash] `{ "status" => Integer, "ok" => Boolean, "body" => String }`.
|
|
127
157
|
# @raise [Obxcura::ConnectionError] if the request never reached the server.
|
|
128
158
|
# @raise [Obxcura::TimeoutError] if the server accepts but never answers.
|
|
129
|
-
def
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
159
|
+
def post(url, payload, content_type, headers, timeout: nil)
|
|
160
|
+
timeout_ms = timeout && (timeout * 1000).to_i
|
|
161
|
+
result = evaluate_func(<<~JS, url, payload, content_type, headers, timeout_ms, timeout: timeout && timeout + TIMEOUT_HEADROOM)
|
|
162
|
+
function(url, payload, contentType, headers, timeoutMs) {
|
|
163
|
+
const controller = timeoutMs ? new AbortController() : null;
|
|
164
|
+
const init = {
|
|
165
|
+
method: "POST",
|
|
166
|
+
headers: Object.assign({ "Content-Type": contentType }, headers),
|
|
167
|
+
body: payload
|
|
168
|
+
};
|
|
169
|
+
if (controller) init.signal = controller.signal;
|
|
170
|
+
|
|
171
|
+
let timer = null;
|
|
172
|
+
const request = fetch(url, init)
|
|
173
|
+
.then((r) => r.text().then((body) => ({ status: r.status, ok: r.ok, body: body })))
|
|
174
|
+
.catch((e) => ({ error: String(e) }))
|
|
175
|
+
.then((outcome) => { if (timer) clearTimeout(timer); return outcome; });
|
|
176
|
+
|
|
177
|
+
if (!timeoutMs) return request;
|
|
178
|
+
|
|
179
|
+
return Promise.race([
|
|
180
|
+
request,
|
|
181
|
+
new Promise((resolve) => {
|
|
182
|
+
timer = setTimeout(() => {
|
|
183
|
+
if (controller) controller.abort();
|
|
184
|
+
resolve({ timeout: true });
|
|
185
|
+
}, timeoutMs);
|
|
186
|
+
})
|
|
187
|
+
]);
|
|
149
188
|
}
|
|
150
189
|
JS
|
|
151
190
|
|
|
191
|
+
raise TimeoutError, timeout_message(url) if result.is_a?(Hash) && result["timeout"]
|
|
192
|
+
|
|
152
193
|
if result.nil? || result["error"]
|
|
153
194
|
reason = result&.dig("error") || "no response (request blocked or never settled)"
|
|
154
195
|
raise ConnectionError, "POST #{url} failed: #{reason}"
|
|
@@ -156,14 +197,19 @@ module Obxcura
|
|
|
156
197
|
|
|
157
198
|
result
|
|
158
199
|
rescue TimeoutError
|
|
159
|
-
raise TimeoutError,
|
|
160
|
-
"POST #{url} did not complete in time. The server accepted the connection " \
|
|
161
|
-
"but never answered the XHR — likely anti-bot tarpitting. Try submitting the " \
|
|
162
|
-
"real form with #type/#submit, run `obscura serve --stealth`, or pass a larger timeout:."
|
|
200
|
+
raise TimeoutError, timeout_message(url)
|
|
163
201
|
end
|
|
202
|
+
# @deprecated Renamed to {#post} once Obscura started routing `fetch`.
|
|
203
|
+
alias_method :xhr_post, :post
|
|
164
204
|
|
|
165
205
|
private
|
|
166
206
|
|
|
207
|
+
def timeout_message(url)
|
|
208
|
+
"POST #{url} did not complete in time. The server accepted the connection " \
|
|
209
|
+
"but never answered — likely anti-bot tarpitting. Try submitting the real " \
|
|
210
|
+
"form with #type/#submit, run `obscura serve --stealth`, or pass a larger timeout:."
|
|
211
|
+
end
|
|
212
|
+
|
|
167
213
|
def dispatch_event(method, params)
|
|
168
214
|
case method
|
|
169
215
|
when "Page.loadEventFired"
|
data/lib/obxcura/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: obxcura
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- memoxmrdl
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-07-
|
|
11
|
+
date: 2026-07-31 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: websocket-driver
|