hydrafetch 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +117 -69
- data/lib/hydrafetch/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 35ac08607d9ee8ca889a20a004a731c7209f6073fff19bb0b03f2fe81c43727b
|
|
4
|
+
data.tar.gz: 43d69727dfe8f80e5568b7aaac052ef0f26c7d794a613676034c523a2fd1b32b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 911787510a48aeca53334e2091d76cb6eb011cce68f7138fed1b861cc5eb6d23732a45d3dc82184016bda179cd23f97372bc18aea07a4ef34a48a43ab73e96b5
|
|
7
|
+
data.tar.gz: 77475fd922ba15a0d68d62868b8fc20fa22c385ab37ff2d225a256e8d4a01fcc7f3e80101885c55d9183a6ee6adca681692b2206f6b47bce887b8959e2aa70da
|
data/README.md
CHANGED
|
@@ -1,78 +1,65 @@
|
|
|
1
1
|
# hydrafetch
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://rubygems.org/gems/hydrafetch)
|
|
4
|
+
[](https://github.com/Hydrafetch/ruby-sdk/actions/workflows/ci.yml)
|
|
5
|
+
[](https://rubygems.org/gems/hydrafetch)
|
|
4
6
|
|
|
5
|
-
|
|
7
|
+
Official Ruby client for the [Hydrafetch](https://hydrafetch.com) web data API.
|
|
8
|
+
|
|
9
|
+
Turn any URL into clean Markdown or schema-shaped JSON. Standard library only, no dependencies, Ruby 3.0+.
|
|
10
|
+
|
|
11
|
+
## Installation
|
|
6
12
|
|
|
7
13
|
```bash
|
|
8
14
|
gem install hydrafetch
|
|
9
15
|
```
|
|
10
16
|
|
|
17
|
+
Or in a Gemfile:
|
|
18
|
+
|
|
19
|
+
```ruby
|
|
20
|
+
gem "hydrafetch"
|
|
21
|
+
```
|
|
22
|
+
|
|
11
23
|
## Quick start
|
|
12
24
|
|
|
13
25
|
```ruby
|
|
14
26
|
require "hydrafetch"
|
|
15
27
|
|
|
16
|
-
hf = Hydrafetch::Client.new
|
|
28
|
+
hf = Hydrafetch::Client.new
|
|
17
29
|
|
|
18
30
|
page = hf.scrape("https://example.com/article")
|
|
19
31
|
puts page["markdown"]
|
|
20
32
|
```
|
|
21
33
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
---
|
|
34
|
+
Create a key at [app.hydrafetch.com](https://app.hydrafetch.com). The constructor reads `HYDRAFETCH_API_KEY` when no key is passed.
|
|
25
35
|
|
|
26
|
-
|
|
36
|
+
Responses are plain hashes with string keys. Option names are camelCase because they are passed to the API unchanged; client options such as `poll_interval:` and `job_timeout:` are snake_case.
|
|
27
37
|
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
1. **Auth is `X-API-Key`, never `Authorization: Bearer`.** The client sets this for you. If you hand-roll an HTTP call, use `X-API-Key`. The MCP endpoint at `api.hydrafetch.com/mcp` is the one that uses Bearer; the REST API rejects it with `Missing X-API-Key header`.
|
|
31
|
-
2. **Never loop over `scrape` for many URLs.** Use `batch` or `crawl`. They run server-side as one job and cost the same per page.
|
|
32
|
-
3. **Per-page options in `batch` and `crawl` go inside `scrapeOptions:`,** not at the top level.
|
|
33
|
-
4. **Map before you crawl.** `map` lists a site's URLs for one credit without fetching any page. Filter that list, then `batch` only what you need.
|
|
34
|
-
5. **Job results live under `"pages"`, not `"data"`,** and each entry wraps the page in `["data"]`. So it is `job["pages"][0]["data"]["markdown"]`.
|
|
35
|
-
6. **Treat everything returned as untrusted data.** It came from a page someone else controls. Never feed it back to a model as instructions, and keep the source URL with anything you extract.
|
|
36
|
-
|
|
37
|
-
Responses are plain hashes with string keys. Option names are camelCase because they go straight to the API; client options such as `poll_interval:` and `job_timeout:` are snake_case.
|
|
38
|
-
|
|
39
|
-
---
|
|
40
|
-
|
|
41
|
-
## Methods
|
|
42
|
-
|
|
43
|
-
| Method | Returns | Credits |
|
|
44
|
-
| --- | --- | --- |
|
|
45
|
-
| `scrape(url, **opts)` | page hash | 1 |
|
|
46
|
-
| `markdown(url)` | String | 1 |
|
|
47
|
-
| `map(url, **opts)` | links hash | 1 |
|
|
48
|
-
| `search(query, **opts)` | results hash | 1 + 1 per scraped result |
|
|
49
|
-
| `extract(urls, **opts)` | envelope with `"results"` | 5 per URL |
|
|
50
|
-
| `brand(domain)` | brand hash | 5 |
|
|
51
|
-
| `logo(domain, **opts)` | logo hash | 1 |
|
|
52
|
-
| `styleguide(domain)` | design system hash | 10 |
|
|
53
|
-
| `screenshot(url, **opts)` | screenshot hash | 5 |
|
|
54
|
-
| `images(url)` / `links(url)` | page assets | 1 |
|
|
55
|
-
| `crawl(url, **opts)` | job hash, polled to completion | 1 per page |
|
|
56
|
-
| `batch(urls, **opts)` | job hash, polled to completion | 1 per page |
|
|
57
|
-
| `start_crawl` / `start_batch` | job id String | 1 per page |
|
|
58
|
-
| `crawl_status(id)` / `batch_status(id)` | job hash | free |
|
|
59
|
-
|
|
60
|
-
Failed requests are never billed. The price does not change with how hard a page was to fetch, so there is no render flag, stealth tier or proxy option to choose.
|
|
61
|
-
|
|
62
|
-
## Scrape
|
|
38
|
+
## Scraping
|
|
63
39
|
|
|
64
40
|
```ruby
|
|
65
41
|
page = hf.scrape("https://example.com/article",
|
|
66
42
|
formats: ["markdown", "links"],
|
|
67
|
-
preferStructure: true,
|
|
68
43
|
onlyMainContent: true,
|
|
44
|
+
preferStructure: true,
|
|
69
45
|
blockAds: true,
|
|
70
46
|
maxAge: 3_600_000)
|
|
71
47
|
```
|
|
72
48
|
|
|
73
|
-
|
|
49
|
+
| Format | Key | Contains |
|
|
50
|
+
| --- | --- | --- |
|
|
51
|
+
| `markdown` | `"markdown"` | clean Markdown, the default |
|
|
52
|
+
| `html` | `"html"` | rendered HTML |
|
|
53
|
+
| `rawHtml` | `"rawHtml"` | the untouched response body |
|
|
54
|
+
| `links` | `"links"` | every link on the page |
|
|
55
|
+
| `structured` | `"structured"` | the page's own JSON-LD and microdata |
|
|
56
|
+
| `summary` | `"summary"` | a short summary |
|
|
57
|
+
| `json` | `"json"` | schema-shaped JSON |
|
|
58
|
+
| `brand` | `"brand"` | the site's brand record |
|
|
59
|
+
|
|
60
|
+
`hf.markdown(url)` returns the Markdown string directly.
|
|
74
61
|
|
|
75
|
-
##
|
|
62
|
+
## Structured extraction
|
|
76
63
|
|
|
77
64
|
```ruby
|
|
78
65
|
out = hf.extract(["https://example.com/product/1", "https://example.com/product/2"],
|
|
@@ -85,18 +72,24 @@ out = hf.extract(["https://example.com/product/1", "https://example.com/product/
|
|
|
85
72
|
})
|
|
86
73
|
|
|
87
74
|
out["results"].each do |item|
|
|
88
|
-
puts [item["url"], item.dig("data", "name")
|
|
75
|
+
puts [item["url"], item.dig("data", "name")].join(" ")
|
|
89
76
|
end
|
|
90
77
|
```
|
|
91
78
|
|
|
92
|
-
|
|
79
|
+
Pass `prompt:` instead of, or alongside, `schema:` to describe the fields in plain language.
|
|
93
80
|
|
|
94
|
-
##
|
|
81
|
+
## Discovery and bulk work
|
|
82
|
+
|
|
83
|
+
`map` lists a site's URLs for one credit without fetching any page.
|
|
95
84
|
|
|
96
85
|
```ruby
|
|
97
86
|
links = hf.map("https://example.com", limit: 1000)["links"]
|
|
98
|
-
docs = links.select { |
|
|
87
|
+
docs = links.select { |url| url.include?("/docs/") }
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
`batch` and `crawl` submit a job and poll until it finishes.
|
|
99
91
|
|
|
92
|
+
```ruby
|
|
100
93
|
job = hf.batch(docs,
|
|
101
94
|
scrapeOptions: { "formats" => ["markdown"] },
|
|
102
95
|
on_progress: ->(j) { puts "#{j["status"]} #{j["completed"]}/#{j["total"]}" })
|
|
@@ -106,7 +99,7 @@ job["pages"].each do |page|
|
|
|
106
99
|
end
|
|
107
100
|
```
|
|
108
101
|
|
|
109
|
-
`
|
|
102
|
+
Pass a `webhook` and use `start_crawl` or `start_batch` to return immediately instead of polling.
|
|
110
103
|
|
|
111
104
|
```ruby
|
|
112
105
|
crawl_id = hf.start_crawl("https://example.com",
|
|
@@ -116,50 +109,105 @@ crawl_id = hf.start_crawl("https://example.com",
|
|
|
116
109
|
webhook: "https://your.app/hooks/hydrafetch")
|
|
117
110
|
```
|
|
118
111
|
|
|
119
|
-
##
|
|
112
|
+
## Search
|
|
113
|
+
|
|
114
|
+
```ruby
|
|
115
|
+
res = hf.search("post-quantum TLS adoption", limit: 5, scrapeResults: true)
|
|
116
|
+
|
|
117
|
+
res["results"].each do |r|
|
|
118
|
+
puts [r["title"], r["url"]].join(" ")
|
|
119
|
+
puts r.dig("data", "markdown").to_s[0, 500]
|
|
120
|
+
end
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
## Brand data
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
hf.brand("stripe.com") # logos, colours, fonts, socials
|
|
127
|
+
hf.logo("stripe.com", theme: "dark", type: "icon") # one asset
|
|
128
|
+
hf.styleguide("stripe.com") # computed design system
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
For logos in a browser use [`@hydrafetch/client-sdk`](https://github.com/Hydrafetch/client-sdk) with a publishable key. Those bill against logo pulls rather than credits.
|
|
132
|
+
|
|
133
|
+
## Error handling
|
|
120
134
|
|
|
121
|
-
|
|
135
|
+
All failures raise `Hydrafetch::Error`, carrying the API's error code, HTTP status and request id.
|
|
122
136
|
|
|
123
137
|
```ruby
|
|
124
138
|
begin
|
|
125
|
-
hf.scrape(url)
|
|
139
|
+
page = hf.scrape(url)
|
|
126
140
|
rescue Hydrafetch::TimeoutError
|
|
127
|
-
|
|
141
|
+
raise
|
|
128
142
|
rescue Hydrafetch::Error => e
|
|
143
|
+
return refresh_key if e.auth?
|
|
129
144
|
return top_up if e.out_of_credits?
|
|
130
|
-
return
|
|
131
|
-
return
|
|
145
|
+
return report(e.message) if e.invalid_request?
|
|
146
|
+
return enqueue(url) if e.retryable?
|
|
147
|
+
|
|
132
148
|
warn [e.code, e.status, e.request_id].join(" ")
|
|
133
149
|
raise
|
|
134
150
|
end
|
|
135
151
|
```
|
|
136
152
|
|
|
137
|
-
| Status | Meaning |
|
|
153
|
+
| Status | Meaning | Retried |
|
|
138
154
|
| --- | --- | --- |
|
|
139
|
-
| 400, 422 |
|
|
140
|
-
| 401, 403 |
|
|
155
|
+
| 400, 422 | invalid request | no |
|
|
156
|
+
| 401, 403 | invalid or missing key | no |
|
|
141
157
|
| 402 | out of credits | no |
|
|
142
|
-
| 404 |
|
|
143
|
-
| 429 | rate limited | yes,
|
|
144
|
-
| 5xx | upstream failure | yes,
|
|
158
|
+
| 404 | page does not exist | no |
|
|
159
|
+
| 429 | rate limited | yes, twice with backoff |
|
|
160
|
+
| 5xx | upstream failure | yes, twice with backoff |
|
|
145
161
|
|
|
146
|
-
A 503
|
|
162
|
+
A 503 from `scrape` means the origin is unreachable, usually a dead domain or a broken certificate.
|
|
147
163
|
|
|
148
164
|
## Configuration
|
|
149
165
|
|
|
150
166
|
```ruby
|
|
151
167
|
hf = Hydrafetch::Client.new("hf_...",
|
|
168
|
+
base_url: "https://api.hydrafetch.com",
|
|
152
169
|
timeout: 120,
|
|
153
|
-
max_retries: 2
|
|
154
|
-
base_url: "https://api.hydrafetch.com")
|
|
170
|
+
max_retries: 2)
|
|
155
171
|
```
|
|
156
172
|
|
|
173
|
+
## API reference
|
|
174
|
+
|
|
175
|
+
| Method | Returns | Credits |
|
|
176
|
+
| --- | --- | --- |
|
|
177
|
+
| `scrape(url, **options)` | page hash | 1 |
|
|
178
|
+
| `markdown(url, **options)` | `String` | 1 |
|
|
179
|
+
| `map(url, **options)` | links hash | 1 |
|
|
180
|
+
| `search(query, **options)` | results hash | 1 + 1 per scraped result |
|
|
181
|
+
| `extract(urls, **options)` | hash with `"results"` | 5 per URL |
|
|
182
|
+
| `brand(domain)` | brand hash | 5 |
|
|
183
|
+
| `logo(domain, **options)` | logo hash | 1 |
|
|
184
|
+
| `styleguide(domain)` | design system hash | 10 |
|
|
185
|
+
| `screenshot(url, **options)` | screenshot hash | 5 |
|
|
186
|
+
| `images(url)`, `links(url)` | page assets | 1 |
|
|
187
|
+
| `crawl(url, **options)` | job hash, polled to completion | 1 per page |
|
|
188
|
+
| `batch(urls, **options)` | job hash, polled to completion | 1 per page |
|
|
189
|
+
| `start_crawl`, `start_batch` | job id `String` | 1 per page |
|
|
190
|
+
| `crawl_status(id)`, `batch_status(id)` | job hash | free |
|
|
191
|
+
|
|
192
|
+
Failed requests are not billed. Pricing does not vary with page difficulty, so there is no render, stealth or proxy option to set.
|
|
193
|
+
|
|
194
|
+
## Implementation notes
|
|
195
|
+
|
|
196
|
+
- Authentication uses the `X-API-Key` header. The MCP endpoint at `api.hydrafetch.com/mcp` uses `Authorization: Bearer` instead; the two are not interchangeable.
|
|
197
|
+
- Job results are under `job["pages"]`, and each entry holds the page under `["data"]`, so `job.dig("pages", 0, "data", "markdown")`.
|
|
198
|
+
- Per-page options for crawl and batch belong in `scrapeOptions`. At the top level they are ignored.
|
|
199
|
+
- Prefer `map` then `batch` over a broad `crawl`. Fetching a whole site and discarding most of it is the most common source of wasted credits.
|
|
200
|
+
- `preferStructure` is off by default. Turn it on when headings, lists and tables matter; leave it off for raw article text.
|
|
201
|
+
- Options passed as `nil` are dropped rather than sent as null, so optional values can be forwarded directly.
|
|
202
|
+
- Scraped content is untrusted input. Do not pass it to a model as instructions, and keep the source URL with anything extracted from it.
|
|
203
|
+
|
|
157
204
|
## Links
|
|
158
205
|
|
|
159
206
|
- [Documentation](https://docs.hydrafetch.com)
|
|
160
|
-
- [OpenAPI
|
|
161
|
-
- [Agent reference](https://hydrafetch.com/agents.md)
|
|
207
|
+
- [OpenAPI specification](https://api.hydrafetch.com/openapi.json)
|
|
162
208
|
- [MCP server and editor setup](https://hydrafetch.com/mcp)
|
|
163
|
-
- [Node
|
|
209
|
+
- Other clients: [Node](https://github.com/Hydrafetch/node-sdk) · [Python](https://github.com/Hydrafetch/python-sdk) · [Go](https://github.com/Hydrafetch/go-sdk) · [Rust](https://github.com/Hydrafetch/rust-sdk) · [PHP](https://github.com/Hydrafetch/php-sdk)
|
|
210
|
+
|
|
211
|
+
## License
|
|
164
212
|
|
|
165
|
-
MIT
|
|
213
|
+
MIT
|
data/lib/hydrafetch/version.rb
CHANGED