synthient 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +43 -22
- data/lib/synthient/client.rb +19 -7
- data/lib/synthient/dataset.rb +99 -0
- data/lib/synthient/export_requests.rb +82 -0
- data/lib/synthient/feed.rb +46 -53
- data/lib/synthient/grpcschema.rb +3 -2
- data/lib/synthient/model.rb +7 -4
- data/lib/synthient/models/account.rb +17 -0
- data/lib/synthient/models/domain.rb +14 -0
- data/lib/synthient/models/events.rb +30 -3
- data/lib/synthient/models/export.rb +93 -15
- data/lib/synthient/models/lookup.rb +3 -2
- data/lib/synthient/transport.rb +4 -3
- data/lib/synthient/version.rb +1 -1
- data/lib/synthient.rb +4 -3
- data/sig/synthient/models.rbs +112 -18
- data/sig/synthient.rbs +24 -7
- metadata +5 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 39b72f6745e18ddfbd2c1b973b2a4bb688c9d1da00578c97fecfa79c4f76c264
|
|
4
|
+
data.tar.gz: 421b4e1a3eb11f76bd430fa7f51d0a1297538f16072715bcb6d51822d366dcb9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fcbcc98ad754e07eaaf31ba24091b842f197bfd5df6c1adb4cb0cb073123af080d8f0773a33f29886b271ac7877f83e7950f037b3bcff00704477111554a86a5
|
|
7
|
+
data.tar.gz: 2aba90acabd702b8b7d651bb8bdacaae3782ff4ad9e314e736922f53a308b06149139b82a9129ba3333432ff1022951302e023714564910899f0fc16603a9d52
|
data/README.md
CHANGED
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
|
|
14
14
|
# synthient-ruby
|
|
15
15
|
|
|
16
|
-
The `synthient` gem is the official Ruby SDK for the [Synthient API](https://docs.synthient.com): IP
|
|
16
|
+
The `synthient` gem is the official Ruby SDK for the [Synthient API](https://docs.synthient.com): IP intelligence, account details, and real-time and bulk data feeds.
|
|
17
17
|
|
|
18
18
|
[](https://rubygems.org/gems/synthient)
|
|
19
19
|
[](https://github.com/synthient/synthient-ruby/actions/workflows/test.yml)
|
|
@@ -72,7 +72,7 @@ Ruby 3.3 or later. The gem uses only Ruby's standard library. It ships RBS signa
|
|
|
72
72
|
|
|
73
73
|
## API Keys
|
|
74
74
|
|
|
75
|
-
Every request needs a Synthient API key. Keys are issued from the Synthient dashboard; if you don't have one, [contact us](mailto:contact@synthient.com). A key's scopes decide what it can call: `BASIC` covers the lookups and `account`, and each feed has its own scopes, listed under [Feeds](#feeds).
|
|
75
|
+
Every request needs a Synthient API key. Keys are issued from the Synthient dashboard; if you don't have one, [contact us](mailto:contact@synthient.com). A key's scopes decide what it can call: `BASIC` covers the lookups and `account`, and each feed and dataset has its own scopes, listed under [Feeds](#feeds) and [Datasets](#datasets).
|
|
76
76
|
|
|
77
77
|
If a request fails even though you are sending a key:
|
|
78
78
|
|
|
@@ -122,7 +122,7 @@ A client keeps its connections open and is safe to share between threads, so cre
|
|
|
122
122
|
|
|
123
123
|
### Look up many IP addresses
|
|
124
124
|
|
|
125
|
-
Look up
|
|
125
|
+
Look up a batch of addresses in one request for `ceil(n * 0.9)` credits. The batch cap depends on your plan: 100 on Free, 500 on Starter, and 1000 on Premium and Enterprise, and the API rejects a larger batch with a 422. Private and reserved addresses are skipped and duplicates are merged, so match results by `ip`.
|
|
126
126
|
|
|
127
127
|
```ruby
|
|
128
128
|
results = client.lookup_ips(["8.8.8.8", "1.1.1.1"])
|
|
@@ -131,18 +131,6 @@ results.each do |result|
|
|
|
131
131
|
end
|
|
132
132
|
```
|
|
133
133
|
|
|
134
|
-
### Look up a domain
|
|
135
|
-
|
|
136
|
-
Get the honeypot traffic observed for a domain and its most recent events. Each lookup costs one credit.
|
|
137
|
-
|
|
138
|
-
```ruby
|
|
139
|
-
domain = client.lookup_domain("example.com", limit: 10)
|
|
140
|
-
puts "#{domain.status} #{domain.stats.events_24h}"
|
|
141
|
-
domain.recent_events.each do |event|
|
|
142
|
-
puts "#{event.timestamp.iso8601} #{event.target_subdomain} #{event.port}"
|
|
143
|
-
end
|
|
144
|
-
```
|
|
145
|
-
|
|
146
134
|
### Check your account
|
|
147
135
|
|
|
148
136
|
Get the key's account, scopes, and remaining lookup credits, at no cost.
|
|
@@ -160,8 +148,7 @@ puts "#{account.organization.name} #{account.scopes.join(", ")} #{account.lookup
|
|
|
160
148
|
### [Client](https://rubydoc.info/gems/synthient/Synthient/Client)
|
|
161
149
|
|
|
162
150
|
- [`lookup_ip`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_ip-instance_method) - Look up an IPv4 or IPv6 address
|
|
163
|
-
- [`lookup_ips`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_ips-instance_method) - Look up
|
|
164
|
-
- [`lookup_domain`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_domain-instance_method) - Look up the honeypot traffic observed for a domain
|
|
151
|
+
- [`lookup_ips`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_ips-instance_method) - Look up a batch of addresses in one request
|
|
165
152
|
- [`account`](https://rubydoc.info/gems/synthient/Synthient/Client#account-instance_method) - Get the key's account, scopes, and remaining lookup credits
|
|
166
153
|
- [`health`](https://rubydoc.info/gems/synthient/Synthient/Client#health-instance_method) - Check that the API and its dependencies are healthy
|
|
167
154
|
|
|
@@ -172,6 +159,7 @@ Each feed is a method of the client, such as `client.proxies`, and has the same
|
|
|
172
159
|
- [`stream`](https://rubydoc.info/gems/synthient/Synthient/Feed#stream-instance_method) - Yield the feed's real-time events
|
|
173
160
|
- [`list_exports`](https://rubydoc.info/gems/synthient/Synthient/Feed#list_exports-instance_method) - List one page of the feed's Parquet exports, newest first
|
|
174
161
|
- [`each_export`](https://rubydoc.info/gems/synthient/Synthient/Feed#each_export-instance_method) - Yield every export, fetching pages as needed
|
|
162
|
+
- [`export_volume`](https://rubydoc.info/gems/synthient/Synthient/Feed#export_volume-instance_method) - Get the rows and bytes the feed exported per UTC day
|
|
175
163
|
- [`export_meta`](https://rubydoc.info/gems/synthient/Synthient/Feed#export_meta-instance_method) - Get an export's checksum, size, row count, and Parquet schema
|
|
176
164
|
- [`download`](https://rubydoc.info/gems/synthient/Synthient/Feed#download-instance_method) - Download an export's Parquet file in chunks
|
|
177
165
|
- [`download_url`](https://rubydoc.info/gems/synthient/Synthient/Feed#download_url-instance_method) - Get a presigned URL for an export's Parquet file, valid for 24 hours
|
|
@@ -181,12 +169,27 @@ Each feed is a method of the client, such as `client.proxies`, and has the same
|
|
|
181
169
|
| `proxies` | Residential, mobile, and datacenter proxies | `ProxyEvent` | `PROXY_FIREHOSE` | `PROXY_FEEDS` |
|
|
182
170
|
| `anonymizers` | VPN and anonymizer IP ranges | `AnonymizerEvent` | `ANONYMIZERS_STREAM` | `ANONYMIZERS_FEED` |
|
|
183
171
|
| `torrents` | BitTorrent DHT and tracker peers | `TorrentEvent` | `TORRENTS_STREAM` | `TORRENTS_FEED` |
|
|
184
|
-
| `ja4t` | JA4T TCP fingerprints | `JA4TEvent` | `JA4T_STREAM` | `JA4T_FEED` |
|
|
185
172
|
| `helios_http` | Helios honeypot HTTP requests | `HeliosHTTPEvent` | `HONEYPOT_HTTP_STREAM` | `HONEYPOT_HTTP_FEED` |
|
|
186
173
|
| `helios_https` | Helios honeypot TLS ClientHellos | `HeliosHTTPSEvent` | `HONEYPOT_HTTPS_STREAM` | `HONEYPOT_HTTPS_FEED` |
|
|
187
174
|
| `helios_dns` | Helios honeypot tunnel destinations | `HeliosDNSEvent` | `HONEYPOT_DNS_STREAM` | `HONEYPOT_DNS_FEED` |
|
|
188
175
|
| `helios_adb` | Helios honeypot Android Debug Bridge commands | `HeliosADBEvent` | `HONEYPOT_ADB_STREAM` | `HONEYPOT_ADB_FEED` |
|
|
189
176
|
|
|
177
|
+
### [Datasets](https://rubydoc.info/gems/synthient/Synthient/Dataset)
|
|
178
|
+
|
|
179
|
+
Datasets have one export per UTC day and no stream. Each dataset is a method of the client, such as `client.geolocation`, and has the same methods:
|
|
180
|
+
|
|
181
|
+
- [`list_exports`](https://rubydoc.info/gems/synthient/Synthient/Dataset#list_exports-instance_method) - List one page of the dataset's daily exports, newest first
|
|
182
|
+
- [`each_export`](https://rubydoc.info/gems/synthient/Synthient/Dataset#each_export-instance_method) - Yield every export, fetching pages as needed
|
|
183
|
+
- [`export_volume`](https://rubydoc.info/gems/synthient/Synthient/Dataset#export_volume-instance_method) - Get the rows and bytes the dataset exported per UTC day
|
|
184
|
+
- [`export_meta`](https://rubydoc.info/gems/synthient/Synthient/Dataset#export_meta-instance_method) - Get a file's checksum, size, and row count, and list every file of the day
|
|
185
|
+
- [`download`](https://rubydoc.info/gems/synthient/Synthient/Dataset#download-instance_method) - Download a day's file in the format you choose, in chunks
|
|
186
|
+
- [`download_url`](https://rubydoc.info/gems/synthient/Synthient/Dataset#download_url-instance_method) - Get a presigned URL for a day's file, valid for 24 hours
|
|
187
|
+
|
|
188
|
+
| Method | Data | Scope | Formats |
|
|
189
|
+
| ------------- | --------------------------------------- | ------------------ | ----------------------------------------- |
|
|
190
|
+
| `geolocation` | IP geolocation, network, and risk data | `GEOLOCATION_FEED` | `parquet` (default), `csv` (gzip), `mmdb` |
|
|
191
|
+
| `whois` | WHOIS registry records as Parquet files | `WHOIS_FEED` | `tar.gz` |
|
|
192
|
+
|
|
190
193
|
</details>
|
|
191
194
|
|
|
192
195
|
## Streaming
|
|
@@ -210,7 +213,7 @@ rescue Synthient::Error => e
|
|
|
210
213
|
end
|
|
211
214
|
```
|
|
212
215
|
|
|
213
|
-
- The server closes each stream after up to 30 minutes. `stream` then returns
|
|
216
|
+
- The server closes each stream after up to 30 minutes. `stream` then returns without an error, and the loop reconnects at once.
|
|
214
217
|
- A 401 or 403 stops the loop, because reconnecting can't fix a rejected key or a missing scope.
|
|
215
218
|
- Any other failure raises a `Synthient::Error`, and the loop waits 2 seconds before reconnecting.
|
|
216
219
|
- Events sent while disconnected are missed. The API has no resume cursor.
|
|
@@ -238,11 +241,19 @@ end
|
|
|
238
241
|
|
|
239
242
|
Without a block, it returns an `Enumerator`, so `client.proxies.each_export.first(10)` fetches only the pages it needs.
|
|
240
243
|
|
|
244
|
+
Pass `day:`, a `Date` or a `"YYYY-MM-DD"` string, to list only one UTC day's exports:
|
|
245
|
+
|
|
246
|
+
```ruby
|
|
247
|
+
client.proxies.each_export(day: Date.new(2026, 11, 17)) do |export|
|
|
248
|
+
puts "#{export.id} #{export.row_count}"
|
|
249
|
+
end
|
|
250
|
+
```
|
|
251
|
+
|
|
241
252
|
On the first page, the newest hourly export has the ID `"latest"`, a moving alias for the freshest export. Its `date` and `hour` give its stable ID, such as `"2026-10-14/21"`.
|
|
242
253
|
|
|
243
254
|
## Downloads
|
|
244
255
|
|
|
245
|
-
|
|
256
|
+
Feed exports are Parquet files. `download` yields a file's bytes in chunks as they arrive:
|
|
246
257
|
|
|
247
258
|
```ruby
|
|
248
259
|
File.open("proxies.parquet", "wb") do |file|
|
|
@@ -250,10 +261,20 @@ File.open("proxies.parquet", "wb") do |file|
|
|
|
250
261
|
end
|
|
251
262
|
```
|
|
252
263
|
|
|
253
|
-
|
|
264
|
+
Datasets offer more formats. This downloads the newest geolocation database as a MaxMind DB:
|
|
265
|
+
|
|
266
|
+
```ruby
|
|
267
|
+
File.open("geolocation.mmdb", "wb") do |file|
|
|
268
|
+
client.geolocation.download("latest", format: Synthient::ExportFormat::MMDB) { |chunk| file.write(chunk) }
|
|
269
|
+
end
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
- Name an export by its `id` from a listing, by `"latest"`, or by a `Date` or `Time` for a UTC day or hour. Datasets have no hourly exports.
|
|
254
273
|
- `export_meta` returns the file's SHA-256 `checksum`. Use a fixed ID for both calls, because `"latest"` can move between them.
|
|
255
274
|
- `download_url` returns a presigned URL that works without a key for 24 hours, so treat it as a secret. Your API key is never sent to the storage host.
|
|
256
275
|
- Hourly exports roll into the daily export shortly after 00:30 UTC the next day. After that, a request for the hour raises a 400.
|
|
276
|
+
- `proxies` and `anonymizers` also publish lite exports, deduplicated to the latest row per address or range and provider. Pass `variant: Synthient::ExportVariant::LITE` to `list_exports` to list them. A lite export has the same ID as the full one, so pass `variant:` again to `export_meta`, `download_url`, and `download`.
|
|
277
|
+
- Files come back exactly as served: a `csv` download is a `.csv.gz` file, and a WHOIS download is a `.tar.gz` archive of Parquet files.
|
|
257
278
|
|
|
258
279
|
## Retries
|
|
259
280
|
|
|
@@ -336,7 +357,7 @@ end
|
|
|
336
357
|
| 402 | The team is out of lookup credits |
|
|
337
358
|
| 403 | The key lacks the scope for the endpoint or feed |
|
|
338
359
|
| 404 | The export doesn't exist, or a daily export is still rolling up (`retry_after` is set) |
|
|
339
|
-
| 422 | The input is invalid, such as a private address
|
|
360
|
+
| 422 | The input is invalid, such as a private address |
|
|
340
361
|
| 429 | Rate limited if `retry_after` is set; otherwise the budget is spent or too many streams are open |
|
|
341
362
|
| 5xx | The server failed |
|
|
342
363
|
|
data/lib/synthient/client.rb
CHANGED
|
@@ -38,6 +38,10 @@ module Synthient
|
|
|
38
38
|
attr_reader :helios_dns
|
|
39
39
|
# @return [Feed<HeliosADBEvent>] ADB shell commands captured by Helios honeypots
|
|
40
40
|
attr_reader :helios_adb
|
|
41
|
+
# @return [Dataset] daily IP geolocation and network data, needs GEOLOCATION_FEED
|
|
42
|
+
attr_reader :geolocation
|
|
43
|
+
# @return [Dataset] daily registry WHOIS data, needs WHOIS_FEED
|
|
44
|
+
attr_reader :whois
|
|
41
45
|
|
|
42
46
|
# Makes no network requests.
|
|
43
47
|
#
|
|
@@ -83,10 +87,12 @@ module Synthient
|
|
|
83
87
|
@anonymizers = Feed.new(@transport, "/feeds/anonymizers", AnonymizerEvent)
|
|
84
88
|
@torrents = Feed.new(@transport, "/feeds/torrents", TorrentEvent)
|
|
85
89
|
@ja4t = Feed.new(@transport, "/feeds/ja4t", JA4TEvent)
|
|
86
|
-
@helios_http = Feed.new(@transport, "/feeds/
|
|
87
|
-
@helios_https = Feed.new(@transport, "/feeds/
|
|
88
|
-
@helios_dns = Feed.new(@transport, "/feeds/
|
|
89
|
-
@helios_adb = Feed.new(@transport, "/feeds/
|
|
90
|
+
@helios_http = Feed.new(@transport, "/feeds/helios/http", HeliosHTTPEvent)
|
|
91
|
+
@helios_https = Feed.new(@transport, "/feeds/helios/https", HeliosHTTPSEvent)
|
|
92
|
+
@helios_dns = Feed.new(@transport, "/feeds/helios/dns", HeliosDNSEvent)
|
|
93
|
+
@helios_adb = Feed.new(@transport, "/feeds/helios/adb", HeliosADBEvent)
|
|
94
|
+
@geolocation = Dataset.new(@transport, "/feeds/geolocation")
|
|
95
|
+
@whois = Dataset.new(@transport, "/feeds/whois")
|
|
90
96
|
end
|
|
91
97
|
|
|
92
98
|
# Costs one lookup credit; failed lookups are free.
|
|
@@ -101,9 +107,12 @@ module Synthient
|
|
|
101
107
|
|
|
102
108
|
# Looks up to 1000 addresses in one request. The API drops duplicate,
|
|
103
109
|
# private, and reserved addresses, so match results by +ip+, not position.
|
|
104
|
-
#
|
|
105
|
-
#
|
|
106
|
-
#
|
|
110
|
+
# The plan sets the batch limit: 100 on Free, 500 on Starter, and 1000 on
|
|
111
|
+
# Premium and Enterprise, and a larger batch raises a 422 APIError. More
|
|
112
|
+
# than 1000 addresses, or an invalid one, raises ArgumentError before the
|
|
113
|
+
# request. Costs ceil(n * 0.9) credits for n distinct usable addresses,
|
|
114
|
+
# and nothing if the request fails. An empty list returns [] without a
|
|
115
|
+
# request.
|
|
107
116
|
#
|
|
108
117
|
# @param ips [Enumerable<String, IPAddr>]
|
|
109
118
|
# @return [Array<IPInfo>]
|
|
@@ -126,6 +135,9 @@ module Synthient
|
|
|
126
135
|
# Looks up the honeypot traffic Synthient has observed for a domain. Costs
|
|
127
136
|
# one lookup credit.
|
|
128
137
|
#
|
|
138
|
+
# @deprecated Domain lookup was removed from the Synthient API and will
|
|
139
|
+
# stop working when the endpoint is turned off. It will be removed in the
|
|
140
|
+
# next major version.
|
|
129
141
|
# @param limit [Integer, nil] how many recent events to return; the API defaults to 50 and clamps values above 1000
|
|
130
142
|
# @return [DomainInfo]
|
|
131
143
|
def lookup_domain(domain, limit: nil)
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Synthient
|
|
4
|
+
# A Synthient dataset: daily exports with no stream and no hourly exports.
|
|
5
|
+
# Access datasets through the client, such as Client#geolocation.
|
|
6
|
+
#
|
|
7
|
+
# An export is identified by "latest", the newest day, or a UTC day as a
|
|
8
|
+
# Date or "YYYY-MM-DD". Export#id values can be passed as they are.
|
|
9
|
+
#
|
|
10
|
+
# Each day has one file per format. Geolocation has ExportFormat::PARQUET,
|
|
11
|
+
# the default, ExportFormat::CSV, gzip-compressed, and ExportFormat::MMDB, a
|
|
12
|
+
# MaxMind DB. WHOIS has only ExportFormat::TAR_GZ, an archive of parquet
|
|
13
|
+
# files that ExportMeta#entities describes. Requests for a day that isn't
|
|
14
|
+
# published yet, or has no file in the requested format, raise a 404
|
|
15
|
+
# APIError.
|
|
16
|
+
class Dataset
|
|
17
|
+
include ExportRequests
|
|
18
|
+
|
|
19
|
+
# @api private
|
|
20
|
+
def initialize(transport, path)
|
|
21
|
+
@transport = transport
|
|
22
|
+
@path = path
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Returns one page of the dataset's exports, newest first.
|
|
26
|
+
#
|
|
27
|
+
# @param limit [Integer, nil] exports per page; the API defaults to 100 and clamps values above 500
|
|
28
|
+
# @param cursor [String, nil] the +next_cursor+ of the previous page
|
|
29
|
+
# @param day [String, Date, nil] lists only this UTC day's export
|
|
30
|
+
# @return [ExportPage]
|
|
31
|
+
def list_exports(limit: nil, cursor: nil, day: nil)
|
|
32
|
+
list_page(limit, cursor, day, {})
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# Yields every export, newest first, fetching pages as they are needed.
|
|
36
|
+
# Breaking out of the block stops fetching. Without a block, returns an
|
|
37
|
+
# Enumerator.
|
|
38
|
+
#
|
|
39
|
+
# @param limit [Integer, nil] exports per page
|
|
40
|
+
# @param cursor [String, nil] a +next_cursor+ to resume from
|
|
41
|
+
# @param day [String, Date, nil] yields only this UTC day's export
|
|
42
|
+
# @yieldparam export [Export]
|
|
43
|
+
# @return [nil]
|
|
44
|
+
# @raise [Error] if the API returns a cursor it already returned
|
|
45
|
+
def each_export(limit: nil, cursor: nil, day: nil, &block)
|
|
46
|
+
return enum_for(:each_export, limit:, cursor:, day:) unless block
|
|
47
|
+
|
|
48
|
+
each_listed_export(cursor, block) { |next_cursor| list_exports(limit:, cursor: next_cursor, day:) }
|
|
49
|
+
nil
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# Returns the rows and bytes exported on each UTC day of a window that
|
|
53
|
+
# ends today, oldest first. Today's totals are still growing.
|
|
54
|
+
#
|
|
55
|
+
# @param days [Integer, nil] the window's length; the API defaults to 90 and allows up to 366
|
|
56
|
+
# @return [ExportVolume]
|
|
57
|
+
def export_volume(days: nil)
|
|
58
|
+
volume(days)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# Returns an export's metadata. Its +checksum+ and +size+ describe the
|
|
62
|
+
# file in +format+, and +formats+ lists every file of the day.
|
|
63
|
+
#
|
|
64
|
+
# @param export [String, Date]
|
|
65
|
+
# @param format [String, nil] an ExportFormat; by default parquet for
|
|
66
|
+
# geolocation and tar.gz for WHOIS
|
|
67
|
+
# @return [ExportMeta]
|
|
68
|
+
def export_meta(export, format: nil)
|
|
69
|
+
meta(export_path(export, hourly: false), { format: })
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# Returns a presigned URL for downloading an export. The URL is valid for
|
|
73
|
+
# 24 hours and grants access without an API key, so treat it as a secret.
|
|
74
|
+
#
|
|
75
|
+
# @param export [String, Date]
|
|
76
|
+
# @param format [String, nil] an ExportFormat
|
|
77
|
+
# @return [URI::HTTP]
|
|
78
|
+
def download_url(export, format: nil)
|
|
79
|
+
location(export_path(export, hourly: false), { format: })
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Downloads an export, yielding the file's bytes in chunks as they arrive,
|
|
83
|
+
# exactly as stored: a csv file stays gzip-compressed and a tar.gz stays
|
|
84
|
+
# packed. The SDK does not verify them: compare a SHA-256 of the bytes
|
|
85
|
+
# with the matching ExportFile#checksum. Breaking out of the block closes
|
|
86
|
+
# the connection. Without a block, returns an Enumerator.
|
|
87
|
+
#
|
|
88
|
+
# @param export [String, Date]
|
|
89
|
+
# @param format [String, nil] an ExportFormat
|
|
90
|
+
# @yieldparam chunk [String] binary bytes
|
|
91
|
+
# @return [nil]
|
|
92
|
+
def download(export, format: nil, &)
|
|
93
|
+
return enum_for(:download, export, format:) unless block_given?
|
|
94
|
+
|
|
95
|
+
@transport.download(download_url(export, format:), &)
|
|
96
|
+
nil
|
|
97
|
+
end
|
|
98
|
+
end
|
|
99
|
+
end
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Synthient
|
|
4
|
+
# The export requests that Feed and Dataset share. +options+ holds Feed's
|
|
5
|
+
# +variant+ or Dataset's +format+.
|
|
6
|
+
module ExportRequests
|
|
7
|
+
private
|
|
8
|
+
|
|
9
|
+
def list_page(limit, cursor, day, options)
|
|
10
|
+
raise ArgumentError, "limit must be a positive Integer" unless limit.nil? || (limit.is_a?(Integer) && limit.positive?)
|
|
11
|
+
raise TypeError, "cursor must be a String" unless cursor.nil? || cursor.is_a?(String)
|
|
12
|
+
|
|
13
|
+
day = day.strftime("%Y-%m-%d") if day.instance_of?(Date)
|
|
14
|
+
raise TypeError, "day must be a String or Date" unless day.nil? || day.is_a?(String)
|
|
15
|
+
unless day.nil? || (day.match?(/\A\d{4}-\d{2}-\d{2}\z/) && Date.valid_date?(*day.split("-").map(&:to_i)))
|
|
16
|
+
raise ArgumentError, "invalid day #{day.inspect}"
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
query = { limit:, cursor:, day: }.reject { |_, value| value.nil? || value == "" }.merge(option_query(options))
|
|
20
|
+
Model.decode(ExportPage, @transport.json("GET", "#{@path}/export", policy: :transient, query:))
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def each_listed_export(cursor, block)
|
|
24
|
+
seen = Set.new
|
|
25
|
+
while seen.add?(cursor)
|
|
26
|
+
page = yield cursor
|
|
27
|
+
page.exports.each(&block)
|
|
28
|
+
cursor = page.next_cursor
|
|
29
|
+
return if cursor.nil? || cursor.empty?
|
|
30
|
+
end
|
|
31
|
+
raise Error, "export listing returned a repeated cursor"
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def volume(days)
|
|
35
|
+
raise ArgumentError, "days must be a positive Integer" unless days.nil? || (days.is_a?(Integer) && days.positive?)
|
|
36
|
+
|
|
37
|
+
json = @transport.json("GET", "#{@path}/export/daily", policy: :transient, query: { days: }.compact)
|
|
38
|
+
Model.decode(ExportVolume, json)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def meta(id, options)
|
|
42
|
+
json = @transport.json("GET", "#{@path}/export/#{id}/meta", policy: :transient, query: option_query(options))
|
|
43
|
+
Model.decode(ExportMeta, json)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def location(id, options)
|
|
47
|
+
@transport.redirect_location("#{@path}/export/#{id}", query: option_query(options))
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def option_query(options)
|
|
51
|
+
options.each { |name, value| raise TypeError, "#{name} must be a String" unless value.nil? || value.is_a?(String) }
|
|
52
|
+
options.reject { |_, value| value.nil? || value == "" }
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def export_path(export, hourly:)
|
|
56
|
+
case export
|
|
57
|
+
when Time
|
|
58
|
+
raise TypeError, "export must be a String or Date, because datasets have no hourly exports" unless hourly
|
|
59
|
+
|
|
60
|
+
time = export.getutc
|
|
61
|
+
"#{time.strftime("%Y-%m-%d")}/#{time.hour}"
|
|
62
|
+
when DateTime
|
|
63
|
+
raise TypeError, "export must be a #{hourly ? "Time" : "String"} or Date, not a DateTime"
|
|
64
|
+
when Date
|
|
65
|
+
export.strftime("%Y-%m-%d")
|
|
66
|
+
when String
|
|
67
|
+
return export if export == "latest"
|
|
68
|
+
|
|
69
|
+
match = %r{\A(\d{4})-(\d{2})-(\d{2})(?:/(\d{1,2}))?\z}.match(export)
|
|
70
|
+
unless match && Date.valid_date?(match[1].to_i, match[2].to_i, match[3].to_i) && match[4].to_i <= 23
|
|
71
|
+
raise ArgumentError, "invalid export ID #{export.inspect}"
|
|
72
|
+
end
|
|
73
|
+
raise ArgumentError, "invalid export ID #{export.inspect}: datasets have no hourly exports" if match[4] && !hourly
|
|
74
|
+
|
|
75
|
+
match[4] ? "#{match[1]}-#{match[2]}-#{match[3]}/#{match[4].to_i}" : export
|
|
76
|
+
else
|
|
77
|
+
raise TypeError, hourly ? "export must be a String, Date, or Time" : "export must be a String or Date"
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
private_constant :ExportRequests
|
|
82
|
+
end
|
data/lib/synthient/feed.rb
CHANGED
|
@@ -15,7 +15,18 @@ module Synthient
|
|
|
15
15
|
# shortly after 00:30 UTC the next day. After that, requests for the hour
|
|
16
16
|
# raise a 400 APIError. While a daily export is being rolled up, requests
|
|
17
17
|
# for it raise a 404 APIError with +retry_after+ set.
|
|
18
|
+
#
|
|
19
|
+
# The proxies, anonymizers, and JA4T feeds also publish a lite variant of
|
|
20
|
+
# each export, deduplicated to the latest row in its hour or day: one row
|
|
21
|
+
# per address and provider for proxies, per address, fingerprint, and
|
|
22
|
+
# provider for JA4T, and per range and provider for anonymizers. Pass
|
|
23
|
+
# +variant: ExportVariant::LITE+ to reach it. A lite file is published
|
|
24
|
+
# shortly after its export, and until then requests for it raise a 404
|
|
25
|
+
# APIError. Other feeds have no lite variant, so their lite listings are
|
|
26
|
+
# empty.
|
|
18
27
|
class Feed
|
|
28
|
+
include ExportRequests
|
|
29
|
+
|
|
19
30
|
# @api private
|
|
20
31
|
def initialize(transport, path, event_type)
|
|
21
32
|
@transport = transport
|
|
@@ -25,11 +36,11 @@ module Synthient
|
|
|
25
36
|
|
|
26
37
|
# Needs the feed's stream scope, such as PROXY_FIREHOSE.
|
|
27
38
|
#
|
|
28
|
-
# The server
|
|
29
|
-
# returns normally. Events sent while disconnected are
|
|
30
|
-
# reconnect in a loop. Breaking out of the block closes the
|
|
31
|
-
# Without a block, returns an Enumerator that opens the
|
|
32
|
-
# iterated.
|
|
39
|
+
# The server can end a stream at any time, such as during a deploy, and
|
|
40
|
+
# the call then returns normally. Events sent while disconnected are
|
|
41
|
+
# missed, so reconnect in a loop. Breaking out of the block closes the
|
|
42
|
+
# connection. Without a block, returns an Enumerator that opens the
|
|
43
|
+
# connection when iterated.
|
|
33
44
|
#
|
|
34
45
|
# @yieldparam event [Object] an event of this feed's type
|
|
35
46
|
# @return [nil]
|
|
@@ -44,13 +55,12 @@ module Synthient
|
|
|
44
55
|
#
|
|
45
56
|
# @param limit [Integer, nil] exports per page; the API defaults to 100 and clamps values above 500
|
|
46
57
|
# @param cursor [String, nil] the +next_cursor+ of the previous page
|
|
58
|
+
# @param day [String, Date, nil] lists only this UTC day's exports
|
|
59
|
+
# @param variant [String, nil] an ExportVariant. A lite export's ID is the
|
|
60
|
+
# same as the full one's, so pass the variant again to reach its file.
|
|
47
61
|
# @return [ExportPage]
|
|
48
|
-
def list_exports(limit: nil, cursor: nil)
|
|
49
|
-
|
|
50
|
-
raise TypeError, "cursor must be a String" unless cursor.nil? || cursor.is_a?(String)
|
|
51
|
-
|
|
52
|
-
query = { limit:, cursor: }.reject { |_, value| value.nil? || value == "" }
|
|
53
|
-
Model.decode(ExportPage, @transport.json("GET", "#{@path}/export", policy: :transient, query:))
|
|
62
|
+
def list_exports(limit: nil, cursor: nil, day: nil, variant: nil)
|
|
63
|
+
list_page(limit, cursor, day, { variant: })
|
|
54
64
|
end
|
|
55
65
|
|
|
56
66
|
# Yields every export, newest first, fetching pages as they are needed.
|
|
@@ -59,37 +69,44 @@ module Synthient
|
|
|
59
69
|
#
|
|
60
70
|
# @param limit [Integer, nil] exports per page
|
|
61
71
|
# @param cursor [String, nil] a +next_cursor+ to resume from
|
|
72
|
+
# @param day [String, Date, nil] yields only this UTC day's exports
|
|
73
|
+
# @param variant [String, nil] an ExportVariant
|
|
62
74
|
# @yieldparam export [Export]
|
|
63
75
|
# @return [nil]
|
|
64
76
|
# @raise [Error] if the API returns a cursor it already returned
|
|
65
|
-
def each_export(limit: nil, cursor: nil, &block)
|
|
66
|
-
return enum_for(:each_export, limit:, cursor:) unless block
|
|
77
|
+
def each_export(limit: nil, cursor: nil, day: nil, variant: nil, &block)
|
|
78
|
+
return enum_for(:each_export, limit:, cursor:, day:, variant:) unless block
|
|
79
|
+
|
|
80
|
+
each_listed_export(cursor, block) { |next_cursor| list_exports(limit:, cursor: next_cursor, day:, variant:) }
|
|
81
|
+
nil
|
|
82
|
+
end
|
|
67
83
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
raise Error, "export listing returned a repeated cursor"
|
|
84
|
+
# Returns the rows and bytes exported on each UTC day of a window that
|
|
85
|
+
# ends today, oldest first. Today's totals are still growing.
|
|
86
|
+
#
|
|
87
|
+
# @param days [Integer, nil] the window's length; the API defaults to 90 and allows up to 366
|
|
88
|
+
# @return [ExportVolume]
|
|
89
|
+
def export_volume(days: nil)
|
|
90
|
+
volume(days)
|
|
76
91
|
end
|
|
77
92
|
|
|
78
93
|
# Returns an export's metadata, including its parquet schema.
|
|
79
94
|
#
|
|
80
95
|
# @param export [String, Date, Time]
|
|
96
|
+
# @param variant [String, nil] an ExportVariant
|
|
81
97
|
# @return [ExportMeta]
|
|
82
|
-
def export_meta(export)
|
|
83
|
-
|
|
98
|
+
def export_meta(export, variant: nil)
|
|
99
|
+
meta(export_path(export, hourly: true), { variant: })
|
|
84
100
|
end
|
|
85
101
|
|
|
86
102
|
# Returns a presigned URL for downloading an export. The URL is valid for
|
|
87
103
|
# 24 hours and grants access without an API key, so treat it as a secret.
|
|
88
104
|
#
|
|
89
105
|
# @param export [String, Date, Time]
|
|
106
|
+
# @param variant [String, nil] an ExportVariant
|
|
90
107
|
# @return [URI::HTTP]
|
|
91
|
-
def download_url(export)
|
|
92
|
-
|
|
108
|
+
def download_url(export, variant: nil)
|
|
109
|
+
location(export_path(export, hourly: true), { variant: })
|
|
93
110
|
end
|
|
94
111
|
|
|
95
112
|
# Downloads an export, yielding the parquet file's bytes in chunks as they
|
|
@@ -98,38 +115,14 @@ module Synthient
|
|
|
98
115
|
# closes the connection. Without a block, returns an Enumerator.
|
|
99
116
|
#
|
|
100
117
|
# @param export [String, Date, Time]
|
|
118
|
+
# @param variant [String, nil] an ExportVariant
|
|
101
119
|
# @yieldparam chunk [String] binary bytes
|
|
102
120
|
# @return [nil]
|
|
103
|
-
def download(export, &)
|
|
104
|
-
return enum_for(:download, export) unless block_given?
|
|
121
|
+
def download(export, variant: nil, &)
|
|
122
|
+
return enum_for(:download, export, variant:) unless block_given?
|
|
105
123
|
|
|
106
|
-
@transport.download(download_url(export), &)
|
|
124
|
+
@transport.download(download_url(export, variant:), &)
|
|
107
125
|
nil
|
|
108
126
|
end
|
|
109
|
-
|
|
110
|
-
private
|
|
111
|
-
|
|
112
|
-
def export_path(export)
|
|
113
|
-
case export
|
|
114
|
-
when Time
|
|
115
|
-
time = export.getutc
|
|
116
|
-
"#{time.strftime("%Y-%m-%d")}/#{time.hour}"
|
|
117
|
-
when DateTime
|
|
118
|
-
raise TypeError, "export must be a Time or Date, not a DateTime"
|
|
119
|
-
when Date
|
|
120
|
-
export.strftime("%Y-%m-%d")
|
|
121
|
-
when String
|
|
122
|
-
return export if export == "latest"
|
|
123
|
-
|
|
124
|
-
match = %r{\A(\d{4})-(\d{2})-(\d{2})(?:/(\d{1,2}))?\z}.match(export)
|
|
125
|
-
unless match && Date.valid_date?(match[1].to_i, match[2].to_i, match[3].to_i) && match[4].to_i <= 23
|
|
126
|
-
raise ArgumentError, "invalid export ID #{export.inspect}"
|
|
127
|
-
end
|
|
128
|
-
|
|
129
|
-
match[4] ? "#{match[1]}-#{match[2]}-#{match[3]}/#{match[4].to_i}" : export
|
|
130
|
-
else
|
|
131
|
-
raise TypeError, "export must be a String, Date, or Time"
|
|
132
|
-
end
|
|
133
|
-
end
|
|
134
127
|
end
|
|
135
128
|
end
|
data/lib/synthient/grpcschema.rb
CHANGED
|
@@ -108,8 +108,9 @@ module Synthient
|
|
|
108
108
|
# Listing skips the grpc.reflection services; name one in +symbols+ to
|
|
109
109
|
# fetch it.
|
|
110
110
|
#
|
|
111
|
-
# @param api_key [String, nil] sent as x-api-key metadata
|
|
112
|
-
# by default from SYNTHIENT_API_KEY
|
|
111
|
+
# @param api_key [String, nil] sent as x-api-key metadata, which
|
|
112
|
+
# reflection requires; by default from SYNTHIENT_API_KEY. A blank key
|
|
113
|
+
# is not sent, and the server answers UNAUTHENTICATED.
|
|
113
114
|
# @param endpoint [String] host:port, a bare host (port 443 is assumed), or
|
|
114
115
|
# a grpc:// or https:// URL
|
|
115
116
|
# @param plaintext [Boolean] disables TLS, for local development servers
|
data/lib/synthient/model.rb
CHANGED
|
@@ -8,7 +8,7 @@ module Synthient
|
|
|
8
8
|
#
|
|
9
9
|
# @api private
|
|
10
10
|
module Model
|
|
11
|
-
Field = Data.define(:name, :wire, :type, :required)
|
|
11
|
+
Field = Data.define(:name, :wire, :type, :required, :null_as)
|
|
12
12
|
|
|
13
13
|
class Builder
|
|
14
14
|
attr_reader :fields
|
|
@@ -17,12 +17,14 @@ module Synthient
|
|
|
17
17
|
@fields = []
|
|
18
18
|
end
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
20
|
+
# +null_as+ is decoded in place of a null or missing value, for a
|
|
21
|
+
# required field the API has been seen to send as null.
|
|
22
|
+
def required(name, type, wire: name.to_s, null_as: nil)
|
|
23
|
+
@fields << Field.new(name:, wire:, type:, required: true, null_as:)
|
|
22
24
|
end
|
|
23
25
|
|
|
24
26
|
def optional(name, type, wire: name.to_s)
|
|
25
|
-
@fields << Field.new(name:, wire:, type:, required: false)
|
|
27
|
+
@fields << Field.new(name:, wire:, type:, required: false, null_as: nil)
|
|
26
28
|
end
|
|
27
29
|
end
|
|
28
30
|
|
|
@@ -42,6 +44,7 @@ module Synthient
|
|
|
42
44
|
|
|
43
45
|
attrs = model.fields.to_h do |field|
|
|
44
46
|
value = object[field.wire]
|
|
47
|
+
value = field.null_as if value.nil?
|
|
45
48
|
field_path = "#{path}.#{field.wire}"
|
|
46
49
|
raise DecodeError, "#{field_path} is required" if value.nil? && field.required
|
|
47
50
|
|
|
@@ -31,6 +31,22 @@ module Synthient
|
|
|
31
31
|
JA4_STREAM = "JA4_STREAM"
|
|
32
32
|
JA4T_FEED = "JA4T_FEED"
|
|
33
33
|
JA4T_STREAM = "JA4T_STREAM"
|
|
34
|
+
GEOLOCATION_FEED = "GEOLOCATION_FEED"
|
|
35
|
+
WHOIS_FEED = "WHOIS_FEED"
|
|
36
|
+
# Reserved: no operation accepts it yet.
|
|
37
|
+
WHOIS_STREAM = "WHOIS_STREAM"
|
|
38
|
+
# Reserved for an upcoming network IOC feed.
|
|
39
|
+
NETWORK_IOCS_FEED = "NETWORK_IOCS_FEED"
|
|
40
|
+
# Reserved for an upcoming network IOC feed.
|
|
41
|
+
NETWORK_IOCS_STREAM = "NETWORK_IOCS_STREAM"
|
|
42
|
+
# Reserved for an upcoming file IOC feed.
|
|
43
|
+
FILE_IOCS_FEED = "FILE_IOCS_FEED"
|
|
44
|
+
# Reserved for an upcoming file IOC feed.
|
|
45
|
+
FILE_IOCS_STREAM = "FILE_IOCS_STREAM"
|
|
46
|
+
# A dashboard permission that authorizes no API operation.
|
|
47
|
+
HELIOS = "HELIOS"
|
|
48
|
+
# @deprecated The API no longer lists this scope, and it never authorized
|
|
49
|
+
# an API operation. It will be removed in the next major version.
|
|
34
50
|
ADMIN_PROXIES = "ADMIN_PROXIES"
|
|
35
51
|
end
|
|
36
52
|
|
|
@@ -55,6 +71,7 @@ module Synthient
|
|
|
55
71
|
end
|
|
56
72
|
|
|
57
73
|
# The account that owns the API key. +scopes+ holds Scope values.
|
|
74
|
+
# +api_key+ is nil if the key was deleted while the request was in flight.
|
|
58
75
|
Account = Model.define do
|
|
59
76
|
required :email, :string
|
|
60
77
|
required :first_name, :string
|