synthient 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2696f98934be28fdf82de9ebc74ef028b180b804eaf314ae4e58503ffcb11ae9
4
- data.tar.gz: 5135ce3a26b57f2f3058ad23d271395fdb5386537e2f3766514bc2e0905dee78
3
+ metadata.gz: 39b72f6745e18ddfbd2c1b973b2a4bb688c9d1da00578c97fecfa79c4f76c264
4
+ data.tar.gz: 421b4e1a3eb11f76bd430fa7f51d0a1297538f16072715bcb6d51822d366dcb9
5
5
  SHA512:
6
- metadata.gz: 473ce8cd208288961a5aa3cef6e49b8de98981281cb3962d4cd089eb41217f6847484f912850f328c0df248e898c64462ad4a6f22d2049864b3ee8f7a96c1222
7
- data.tar.gz: f7cf2f3ec63a98f9c89adbf1fd4b2d799097bf18263934482a7bc4fc92bab15c7cd4b437b152bbf5c33f9011950cc2a838435f8011a40f609227088c58411641
6
+ metadata.gz: fcbcc98ad754e07eaaf31ba24091b842f197bfd5df6c1adb4cb0cb073123af080d8f0773a33f29886b271ac7877f83e7950f037b3bcff00704477111554a86a5
7
+ data.tar.gz: 2aba90acabd702b8b7d651bb8bdacaae3782ff4ad9e314e736922f53a308b06149139b82a9129ba3333432ff1022951302e023714564910899f0fc16603a9d52
data/README.md CHANGED
@@ -13,7 +13,7 @@
13
13
 
14
14
  # synthient-ruby
15
15
 
16
- The `synthient` gem is the official Ruby SDK for the [Synthient API](https://docs.synthient.com): IP and domain intelligence, account details, and real-time and bulk data feeds.
16
+ The `synthient` gem is the official Ruby SDK for the [Synthient API](https://docs.synthient.com): IP intelligence, account details, and real-time and bulk data feeds.
17
17
 
18
18
  [![Gem Version](https://img.shields.io/gem/v/synthient)](https://rubygems.org/gems/synthient)
19
19
  [![test](https://github.com/synthient/synthient-ruby/actions/workflows/test.yml/badge.svg)](https://github.com/synthient/synthient-ruby/actions/workflows/test.yml)
@@ -72,7 +72,7 @@ Ruby 3.3 or later. The gem uses only Ruby's standard library. It ships RBS signa
72
72
 
73
73
  ## API Keys
74
74
 
75
- Every request needs a Synthient API key. Keys are issued from the Synthient dashboard; if you don't have one, [contact us](mailto:contact@synthient.com). A key's scopes decide what it can call: `BASIC` covers the lookups and `account`, and each feed has its own scopes, listed under [Feeds](#feeds).
75
+ Every request needs a Synthient API key. Keys are issued from the Synthient dashboard; if you don't have one, [contact us](mailto:contact@synthient.com). A key's scopes decide what it can call: `BASIC` covers the lookups and `account`, and each feed and dataset has its own scopes, listed under [Feeds](#feeds) and [Datasets](#datasets).
76
76
 
77
77
  If a request fails even though you are sending a key:
78
78
 
@@ -122,7 +122,7 @@ A client keeps its connections open and is safe to share between threads, so cre
122
122
 
123
123
  ### Look up many IP addresses
124
124
 
125
- Look up as many as 1000 addresses in one request for `ceil(n * 0.9)` credits. Private and reserved addresses are skipped and duplicates are merged, so match results by `ip`.
125
+ Look up a batch of addresses in one request for `ceil(n * 0.9)` credits. The batch cap depends on your plan: 100 on Free, 500 on Starter, and 1000 on Premium and Enterprise, and the API rejects a larger batch with a 422. Private and reserved addresses are skipped and duplicates are merged, so match results by `ip`.
126
126
 
127
127
  ```ruby
128
128
  results = client.lookup_ips(["8.8.8.8", "1.1.1.1"])
@@ -131,18 +131,6 @@ results.each do |result|
131
131
  end
132
132
  ```
133
133
 
134
- ### Look up a domain
135
-
136
- Get the honeypot traffic observed for a domain and its most recent events. Each lookup costs one credit.
137
-
138
- ```ruby
139
- domain = client.lookup_domain("example.com", limit: 10)
140
- puts "#{domain.status} #{domain.stats.events_24h}"
141
- domain.recent_events.each do |event|
142
- puts "#{event.timestamp.iso8601} #{event.target_subdomain} #{event.port}"
143
- end
144
- ```
145
-
146
134
  ### Check your account
147
135
 
148
136
  Get the key's account, scopes, and remaining lookup credits, at no cost.
@@ -160,8 +148,7 @@ puts "#{account.organization.name} #{account.scopes.join(", ")} #{account.lookup
160
148
  ### [Client](https://rubydoc.info/gems/synthient/Synthient/Client)
161
149
 
162
150
  - [`lookup_ip`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_ip-instance_method) - Look up an IPv4 or IPv6 address
163
- - [`lookup_ips`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_ips-instance_method) - Look up as many as 1000 addresses in one request
164
- - [`lookup_domain`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_domain-instance_method) - Look up the honeypot traffic observed for a domain
151
+ - [`lookup_ips`](https://rubydoc.info/gems/synthient/Synthient/Client#lookup_ips-instance_method) - Look up a batch of addresses in one request
165
152
  - [`account`](https://rubydoc.info/gems/synthient/Synthient/Client#account-instance_method) - Get the key's account, scopes, and remaining lookup credits
166
153
  - [`health`](https://rubydoc.info/gems/synthient/Synthient/Client#health-instance_method) - Check that the API and its dependencies are healthy
167
154
 
@@ -172,6 +159,7 @@ Each feed is a method of the client, such as `client.proxies`, and has the same
172
159
  - [`stream`](https://rubydoc.info/gems/synthient/Synthient/Feed#stream-instance_method) - Yield the feed's real-time events
173
160
  - [`list_exports`](https://rubydoc.info/gems/synthient/Synthient/Feed#list_exports-instance_method) - List one page of the feed's Parquet exports, newest first
174
161
  - [`each_export`](https://rubydoc.info/gems/synthient/Synthient/Feed#each_export-instance_method) - Yield every export, fetching pages as needed
162
+ - [`export_volume`](https://rubydoc.info/gems/synthient/Synthient/Feed#export_volume-instance_method) - Get the rows and bytes the feed exported per UTC day
175
163
  - [`export_meta`](https://rubydoc.info/gems/synthient/Synthient/Feed#export_meta-instance_method) - Get an export's checksum, size, row count, and Parquet schema
176
164
  - [`download`](https://rubydoc.info/gems/synthient/Synthient/Feed#download-instance_method) - Download an export's Parquet file in chunks
177
165
  - [`download_url`](https://rubydoc.info/gems/synthient/Synthient/Feed#download_url-instance_method) - Get a presigned URL for an export's Parquet file, valid for 24 hours
@@ -181,12 +169,27 @@ Each feed is a method of the client, such as `client.proxies`, and has the same
181
169
  | `proxies` | Residential, mobile, and datacenter proxies | `ProxyEvent` | `PROXY_FIREHOSE` | `PROXY_FEEDS` |
182
170
  | `anonymizers` | VPN and anonymizer IP ranges | `AnonymizerEvent` | `ANONYMIZERS_STREAM` | `ANONYMIZERS_FEED` |
183
171
  | `torrents` | BitTorrent DHT and tracker peers | `TorrentEvent` | `TORRENTS_STREAM` | `TORRENTS_FEED` |
184
- | `ja4t` | JA4T TCP fingerprints | `JA4TEvent` | `JA4T_STREAM` | `JA4T_FEED` |
185
172
  | `helios_http` | Helios honeypot HTTP requests | `HeliosHTTPEvent` | `HONEYPOT_HTTP_STREAM` | `HONEYPOT_HTTP_FEED` |
186
173
  | `helios_https` | Helios honeypot TLS ClientHellos | `HeliosHTTPSEvent` | `HONEYPOT_HTTPS_STREAM` | `HONEYPOT_HTTPS_FEED` |
187
174
  | `helios_dns` | Helios honeypot tunnel destinations | `HeliosDNSEvent` | `HONEYPOT_DNS_STREAM` | `HONEYPOT_DNS_FEED` |
188
175
  | `helios_adb` | Helios honeypot Android Debug Bridge commands | `HeliosADBEvent` | `HONEYPOT_ADB_STREAM` | `HONEYPOT_ADB_FEED` |
189
176
 
177
+ ### [Datasets](https://rubydoc.info/gems/synthient/Synthient/Dataset)
178
+
179
+ Datasets have one export per UTC day and no stream. Each dataset is a method of the client, such as `client.geolocation`, and has the same methods:
180
+
181
+ - [`list_exports`](https://rubydoc.info/gems/synthient/Synthient/Dataset#list_exports-instance_method) - List one page of the dataset's daily exports, newest first
182
+ - [`each_export`](https://rubydoc.info/gems/synthient/Synthient/Dataset#each_export-instance_method) - Yield every export, fetching pages as needed
183
+ - [`export_volume`](https://rubydoc.info/gems/synthient/Synthient/Dataset#export_volume-instance_method) - Get the rows and bytes the dataset exported per UTC day
184
+ - [`export_meta`](https://rubydoc.info/gems/synthient/Synthient/Dataset#export_meta-instance_method) - Get a file's checksum, size, and row count, and list every file of the day
185
+ - [`download`](https://rubydoc.info/gems/synthient/Synthient/Dataset#download-instance_method) - Download a day's file in the format you choose, in chunks
186
+ - [`download_url`](https://rubydoc.info/gems/synthient/Synthient/Dataset#download_url-instance_method) - Get a presigned URL for a day's file, valid for 24 hours
187
+
188
+ | Method | Data | Scope | Formats |
189
+ | ------------- | --------------------------------------- | ------------------ | ----------------------------------------- |
190
+ | `geolocation` | IP geolocation, network, and risk data | `GEOLOCATION_FEED` | `parquet` (default), `csv` (gzip), `mmdb` |
191
+ | `whois` | WHOIS registry records as Parquet files | `WHOIS_FEED` | `tar.gz` |
192
+
190
193
  </details>
191
194
 
192
195
  ## Streaming
@@ -210,7 +213,7 @@ rescue Synthient::Error => e
210
213
  end
211
214
  ```
212
215
 
213
- - The server closes each stream after up to 30 minutes. `stream` then returns normally, and the loop reconnects at once.
216
+ - The server closes each stream after up to 30 minutes. `stream` then returns without an error, and the loop reconnects at once.
214
217
  - A 401 or 403 stops the loop, because reconnecting can't fix a rejected key or a missing scope.
215
218
  - Any other failure raises a `Synthient::Error`, and the loop waits 2 seconds before reconnecting.
216
219
  - Events sent while disconnected are missed. The API has no resume cursor.
@@ -238,11 +241,19 @@ end
238
241
 
239
242
  Without a block, it returns an `Enumerator`, so `client.proxies.each_export.first(10)` fetches only the pages it needs.
240
243
 
244
+ Pass `day:`, a `Date` or a `"YYYY-MM-DD"` string, to list only one UTC day's exports:
245
+
246
+ ```ruby
247
+ client.proxies.each_export(day: Date.new(2026, 11, 17)) do |export|
248
+ puts "#{export.id} #{export.row_count}"
249
+ end
250
+ ```
251
+
241
252
  On the first page, the newest hourly export has the ID `"latest"`, a moving alias for the freshest export. Its `date` and `hour` give its stable ID, such as `"2026-10-14/21"`.
242
253
 
243
254
  ## Downloads
244
255
 
245
- Exports are Parquet files. `download` yields a file's bytes in chunks as they arrive:
256
+ Feed exports are Parquet files. `download` yields a file's bytes in chunks as they arrive:
246
257
 
247
258
  ```ruby
248
259
  File.open("proxies.parquet", "wb") do |file|
@@ -250,10 +261,20 @@ File.open("proxies.parquet", "wb") do |file|
250
261
  end
251
262
  ```
252
263
 
253
- - Name an export by its `id` from a listing, by `"latest"`, or by a `Date` or `Time` for a UTC day or hour.
264
+ Datasets offer more formats. This downloads the newest geolocation database as a MaxMind DB:
265
+
266
+ ```ruby
267
+ File.open("geolocation.mmdb", "wb") do |file|
268
+ client.geolocation.download("latest", format: Synthient::ExportFormat::MMDB) { |chunk| file.write(chunk) }
269
+ end
270
+ ```
271
+
272
+ - Name an export by its `id` from a listing, by `"latest"`, or by a `Date` or `Time` for a UTC day or hour. Datasets have no hourly exports.
254
273
  - `export_meta` returns the file's SHA-256 `checksum`. Use a fixed ID for both calls, because `"latest"` can move between them.
255
274
  - `download_url` returns a presigned URL that works without a key for 24 hours, so treat it as a secret. Your API key is never sent to the storage host.
256
275
  - Hourly exports roll into the daily export shortly after 00:30 UTC the next day. After that, a request for the hour raises a 400.
276
+ - `proxies` and `anonymizers` also publish lite exports, deduplicated to the latest row per address or range and provider. Pass `variant: Synthient::ExportVariant::LITE` to `list_exports` to list them. A lite export has the same ID as the full one, so pass `variant:` again to `export_meta`, `download_url`, and `download`.
277
+ - Files come back exactly as served: a `csv` download is a `.csv.gz` file, and a WHOIS download is a `.tar.gz` archive of Parquet files.
257
278
 
258
279
  ## Retries
259
280
 
@@ -336,7 +357,7 @@ end
336
357
  | 402 | The team is out of lookup credits |
337
358
  | 403 | The key lacks the scope for the endpoint or feed |
338
359
  | 404 | The export doesn't exist, or a daily export is still rolling up (`retry_after` is set) |
339
- | 422 | The input is invalid, such as a private address or a malformed domain |
360
+ | 422 | The input is invalid, such as a private address |
340
361
  | 429 | Rate limited if `retry_after` is set; otherwise the budget is spent or too many streams are open |
341
362
  | 5xx | The server failed |
342
363
 
@@ -38,6 +38,10 @@ module Synthient
38
38
  attr_reader :helios_dns
39
39
  # @return [Feed<HeliosADBEvent>] ADB shell commands captured by Helios honeypots
40
40
  attr_reader :helios_adb
41
+ # @return [Dataset] daily IP geolocation and network data, needs GEOLOCATION_FEED
42
+ attr_reader :geolocation
43
+ # @return [Dataset] daily registry WHOIS data, needs WHOIS_FEED
44
+ attr_reader :whois
41
45
 
42
46
  # Makes no network requests.
43
47
  #
@@ -83,10 +87,12 @@ module Synthient
83
87
  @anonymizers = Feed.new(@transport, "/feeds/anonymizers", AnonymizerEvent)
84
88
  @torrents = Feed.new(@transport, "/feeds/torrents", TorrentEvent)
85
89
  @ja4t = Feed.new(@transport, "/feeds/ja4t", JA4TEvent)
86
- @helios_http = Feed.new(@transport, "/feeds/helio/http", HeliosHTTPEvent)
87
- @helios_https = Feed.new(@transport, "/feeds/helio/https", HeliosHTTPSEvent)
88
- @helios_dns = Feed.new(@transport, "/feeds/helio/dns", HeliosDNSEvent)
89
- @helios_adb = Feed.new(@transport, "/feeds/helio/adb", HeliosADBEvent)
90
+ @helios_http = Feed.new(@transport, "/feeds/helios/http", HeliosHTTPEvent)
91
+ @helios_https = Feed.new(@transport, "/feeds/helios/https", HeliosHTTPSEvent)
92
+ @helios_dns = Feed.new(@transport, "/feeds/helios/dns", HeliosDNSEvent)
93
+ @helios_adb = Feed.new(@transport, "/feeds/helios/adb", HeliosADBEvent)
94
+ @geolocation = Dataset.new(@transport, "/feeds/geolocation")
95
+ @whois = Dataset.new(@transport, "/feeds/whois")
90
96
  end
91
97
 
92
98
  # Costs one lookup credit; failed lookups are free.
@@ -101,9 +107,12 @@ module Synthient
101
107
 
102
108
  # Looks up to 1000 addresses in one request. The API drops duplicate,
103
109
  # private, and reserved addresses, so match results by +ip+, not position.
104
- # An invalid address raises ArgumentError before the request. Costs
105
- # ceil(n * 0.9) credits for n distinct usable addresses, and nothing if the
106
- # request fails. An empty list returns [] without a request.
110
+ # The plan sets the batch limit: 100 on Free, 500 on Starter, and 1000 on
111
+ # Premium and Enterprise, and a larger batch raises a 422 APIError. More
112
+ # than 1000 addresses, or an invalid one, raises ArgumentError before the
113
+ # request. Costs ceil(n * 0.9) credits for n distinct usable addresses,
114
+ # and nothing if the request fails. An empty list returns [] without a
115
+ # request.
107
116
  #
108
117
  # @param ips [Enumerable<String, IPAddr>]
109
118
  # @return [Array<IPInfo>]
@@ -126,6 +135,9 @@ module Synthient
126
135
  # Looks up the honeypot traffic Synthient has observed for a domain. Costs
127
136
  # one lookup credit.
128
137
  #
138
+ # @deprecated Domain lookup was removed from the Synthient API and will
139
+ # stop working when the endpoint is turned off. It will be removed in the
140
+ # next major version.
129
141
  # @param limit [Integer, nil] how many recent events to return; the API defaults to 50 and clamps values above 1000
130
142
  # @return [DomainInfo]
131
143
  def lookup_domain(domain, limit: nil)
@@ -0,0 +1,99 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Synthient
4
+ # A Synthient dataset: daily exports with no stream and no hourly exports.
5
+ # Access datasets through the client, such as Client#geolocation.
6
+ #
7
+ # An export is identified by "latest", the newest day, or a UTC day as a
8
+ # Date or "YYYY-MM-DD". Export#id values can be passed as they are.
9
+ #
10
+ # Each day has one file per format. Geolocation has ExportFormat::PARQUET,
11
+ # the default, ExportFormat::CSV, gzip-compressed, and ExportFormat::MMDB, a
12
+ # MaxMind DB. WHOIS has only ExportFormat::TAR_GZ, an archive of parquet
13
+ # files that ExportMeta#entities describes. Requests for a day that isn't
14
+ # published yet, or has no file in the requested format, raise a 404
15
+ # APIError.
16
+ class Dataset
17
+ include ExportRequests
18
+
19
+ # @api private
20
+ def initialize(transport, path)
21
+ @transport = transport
22
+ @path = path
23
+ end
24
+
25
+ # Returns one page of the dataset's exports, newest first.
26
+ #
27
+ # @param limit [Integer, nil] exports per page; the API defaults to 100 and clamps values above 500
28
+ # @param cursor [String, nil] the +next_cursor+ of the previous page
29
+ # @param day [String, Date, nil] lists only this UTC day's export
30
+ # @return [ExportPage]
31
+ def list_exports(limit: nil, cursor: nil, day: nil)
32
+ list_page(limit, cursor, day, {})
33
+ end
34
+
35
+ # Yields every export, newest first, fetching pages as they are needed.
36
+ # Breaking out of the block stops fetching. Without a block, returns an
37
+ # Enumerator.
38
+ #
39
+ # @param limit [Integer, nil] exports per page
40
+ # @param cursor [String, nil] a +next_cursor+ to resume from
41
+ # @param day [String, Date, nil] yields only this UTC day's export
42
+ # @yieldparam export [Export]
43
+ # @return [nil]
44
+ # @raise [Error] if the API returns a cursor it already returned
45
+ def each_export(limit: nil, cursor: nil, day: nil, &block)
46
+ return enum_for(:each_export, limit:, cursor:, day:) unless block
47
+
48
+ each_listed_export(cursor, block) { |next_cursor| list_exports(limit:, cursor: next_cursor, day:) }
49
+ nil
50
+ end
51
+
52
+ # Returns the rows and bytes exported on each UTC day of a window that
53
+ # ends today, oldest first. Today's totals are still growing.
54
+ #
55
+ # @param days [Integer, nil] the window's length; the API defaults to 90 and allows up to 366
56
+ # @return [ExportVolume]
57
+ def export_volume(days: nil)
58
+ volume(days)
59
+ end
60
+
61
+ # Returns an export's metadata. Its +checksum+ and +size+ describe the
62
+ # file in +format+, and +formats+ lists every file of the day.
63
+ #
64
+ # @param export [String, Date]
65
+ # @param format [String, nil] an ExportFormat; by default parquet for
66
+ # geolocation and tar.gz for WHOIS
67
+ # @return [ExportMeta]
68
+ def export_meta(export, format: nil)
69
+ meta(export_path(export, hourly: false), { format: })
70
+ end
71
+
72
+ # Returns a presigned URL for downloading an export. The URL is valid for
73
+ # 24 hours and grants access without an API key, so treat it as a secret.
74
+ #
75
+ # @param export [String, Date]
76
+ # @param format [String, nil] an ExportFormat
77
+ # @return [URI::HTTP]
78
+ def download_url(export, format: nil)
79
+ location(export_path(export, hourly: false), { format: })
80
+ end
81
+
82
+ # Downloads an export, yielding the file's bytes in chunks as they arrive,
83
+ # exactly as stored: a csv file stays gzip-compressed and a tar.gz stays
84
+ # packed. The SDK does not verify them: compare a SHA-256 of the bytes
85
+ # with the matching ExportFile#checksum. Breaking out of the block closes
86
+ # the connection. Without a block, returns an Enumerator.
87
+ #
88
+ # @param export [String, Date]
89
+ # @param format [String, nil] an ExportFormat
90
+ # @yieldparam chunk [String] binary bytes
91
+ # @return [nil]
92
+ def download(export, format: nil, &)
93
+ return enum_for(:download, export, format:) unless block_given?
94
+
95
+ @transport.download(download_url(export, format:), &)
96
+ nil
97
+ end
98
+ end
99
+ end
@@ -0,0 +1,82 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Synthient
4
+ # The export requests that Feed and Dataset share. +options+ holds Feed's
5
+ # +variant+ or Dataset's +format+.
6
+ module ExportRequests
7
+ private
8
+
9
+ def list_page(limit, cursor, day, options)
10
+ raise ArgumentError, "limit must be a positive Integer" unless limit.nil? || (limit.is_a?(Integer) && limit.positive?)
11
+ raise TypeError, "cursor must be a String" unless cursor.nil? || cursor.is_a?(String)
12
+
13
+ day = day.strftime("%Y-%m-%d") if day.instance_of?(Date)
14
+ raise TypeError, "day must be a String or Date" unless day.nil? || day.is_a?(String)
15
+ unless day.nil? || (day.match?(/\A\d{4}-\d{2}-\d{2}\z/) && Date.valid_date?(*day.split("-").map(&:to_i)))
16
+ raise ArgumentError, "invalid day #{day.inspect}"
17
+ end
18
+
19
+ query = { limit:, cursor:, day: }.reject { |_, value| value.nil? || value == "" }.merge(option_query(options))
20
+ Model.decode(ExportPage, @transport.json("GET", "#{@path}/export", policy: :transient, query:))
21
+ end
22
+
23
+ def each_listed_export(cursor, block)
24
+ seen = Set.new
25
+ while seen.add?(cursor)
26
+ page = yield cursor
27
+ page.exports.each(&block)
28
+ cursor = page.next_cursor
29
+ return if cursor.nil? || cursor.empty?
30
+ end
31
+ raise Error, "export listing returned a repeated cursor"
32
+ end
33
+
34
+ def volume(days)
35
+ raise ArgumentError, "days must be a positive Integer" unless days.nil? || (days.is_a?(Integer) && days.positive?)
36
+
37
+ json = @transport.json("GET", "#{@path}/export/daily", policy: :transient, query: { days: }.compact)
38
+ Model.decode(ExportVolume, json)
39
+ end
40
+
41
+ def meta(id, options)
42
+ json = @transport.json("GET", "#{@path}/export/#{id}/meta", policy: :transient, query: option_query(options))
43
+ Model.decode(ExportMeta, json)
44
+ end
45
+
46
+ def location(id, options)
47
+ @transport.redirect_location("#{@path}/export/#{id}", query: option_query(options))
48
+ end
49
+
50
+ def option_query(options)
51
+ options.each { |name, value| raise TypeError, "#{name} must be a String" unless value.nil? || value.is_a?(String) }
52
+ options.reject { |_, value| value.nil? || value == "" }
53
+ end
54
+
55
+ def export_path(export, hourly:)
56
+ case export
57
+ when Time
58
+ raise TypeError, "export must be a String or Date, because datasets have no hourly exports" unless hourly
59
+
60
+ time = export.getutc
61
+ "#{time.strftime("%Y-%m-%d")}/#{time.hour}"
62
+ when DateTime
63
+ raise TypeError, "export must be a #{hourly ? "Time" : "String"} or Date, not a DateTime"
64
+ when Date
65
+ export.strftime("%Y-%m-%d")
66
+ when String
67
+ return export if export == "latest"
68
+
69
+ match = %r{\A(\d{4})-(\d{2})-(\d{2})(?:/(\d{1,2}))?\z}.match(export)
70
+ unless match && Date.valid_date?(match[1].to_i, match[2].to_i, match[3].to_i) && match[4].to_i <= 23
71
+ raise ArgumentError, "invalid export ID #{export.inspect}"
72
+ end
73
+ raise ArgumentError, "invalid export ID #{export.inspect}: datasets have no hourly exports" if match[4] && !hourly
74
+
75
+ match[4] ? "#{match[1]}-#{match[2]}-#{match[3]}/#{match[4].to_i}" : export
76
+ else
77
+ raise TypeError, hourly ? "export must be a String, Date, or Time" : "export must be a String or Date"
78
+ end
79
+ end
80
+ end
81
+ private_constant :ExportRequests
82
+ end
@@ -15,7 +15,18 @@ module Synthient
15
15
  # shortly after 00:30 UTC the next day. After that, requests for the hour
16
16
  # raise a 400 APIError. While a daily export is being rolled up, requests
17
17
  # for it raise a 404 APIError with +retry_after+ set.
18
+ #
19
+ # The proxies, anonymizers, and JA4T feeds also publish a lite variant of
20
+ # each export, deduplicated to the latest row in its hour or day: one row
21
+ # per address and provider for proxies, per address, fingerprint, and
22
+ # provider for JA4T, and per range and provider for anonymizers. Pass
23
+ # +variant: ExportVariant::LITE+ to reach it. A lite file is published
24
+ # shortly after its export, and until then requests for it raise a 404
25
+ # APIError. Other feeds have no lite variant, so their lite listings are
26
+ # empty.
18
27
  class Feed
28
+ include ExportRequests
29
+
19
30
  # @api private
20
31
  def initialize(transport, path, event_type)
21
32
  @transport = transport
@@ -25,11 +36,11 @@ module Synthient
25
36
 
26
37
  # Needs the feed's stream scope, such as PROXY_FIREHOSE.
27
38
  #
28
- # The server ends each stream after up to 30 minutes, and the call then
29
- # returns normally. Events sent while disconnected are missed, so
30
- # reconnect in a loop. Breaking out of the block closes the connection.
31
- # Without a block, returns an Enumerator that opens the connection when
32
- # iterated.
39
+ # The server can end a stream at any time, such as during a deploy, and
40
+ # the call then returns normally. Events sent while disconnected are
41
+ # missed, so reconnect in a loop. Breaking out of the block closes the
42
+ # connection. Without a block, returns an Enumerator that opens the
43
+ # connection when iterated.
33
44
  #
34
45
  # @yieldparam event [Object] an event of this feed's type
35
46
  # @return [nil]
@@ -44,13 +55,12 @@ module Synthient
44
55
  #
45
56
  # @param limit [Integer, nil] exports per page; the API defaults to 100 and clamps values above 500
46
57
  # @param cursor [String, nil] the +next_cursor+ of the previous page
58
+ # @param day [String, Date, nil] lists only this UTC day's exports
59
+ # @param variant [String, nil] an ExportVariant. A lite export's ID is the
60
+ # same as the full one's, so pass the variant again to reach its file.
47
61
  # @return [ExportPage]
48
- def list_exports(limit: nil, cursor: nil)
49
- raise ArgumentError, "limit must be a positive Integer" unless limit.nil? || (limit.is_a?(Integer) && limit.positive?)
50
- raise TypeError, "cursor must be a String" unless cursor.nil? || cursor.is_a?(String)
51
-
52
- query = { limit:, cursor: }.reject { |_, value| value.nil? || value == "" }
53
- Model.decode(ExportPage, @transport.json("GET", "#{@path}/export", policy: :transient, query:))
62
+ def list_exports(limit: nil, cursor: nil, day: nil, variant: nil)
63
+ list_page(limit, cursor, day, { variant: })
54
64
  end
55
65
 
56
66
  # Yields every export, newest first, fetching pages as they are needed.
@@ -59,37 +69,44 @@ module Synthient
59
69
  #
60
70
  # @param limit [Integer, nil] exports per page
61
71
  # @param cursor [String, nil] a +next_cursor+ to resume from
72
+ # @param day [String, Date, nil] yields only this UTC day's exports
73
+ # @param variant [String, nil] an ExportVariant
62
74
  # @yieldparam export [Export]
63
75
  # @return [nil]
64
76
  # @raise [Error] if the API returns a cursor it already returned
65
- def each_export(limit: nil, cursor: nil, &block)
66
- return enum_for(:each_export, limit:, cursor:) unless block
77
+ def each_export(limit: nil, cursor: nil, day: nil, variant: nil, &block)
78
+ return enum_for(:each_export, limit:, cursor:, day:, variant:) unless block
79
+
80
+ each_listed_export(cursor, block) { |next_cursor| list_exports(limit:, cursor: next_cursor, day:, variant:) }
81
+ nil
82
+ end
67
83
 
68
- seen = Set.new
69
- while seen.add?(cursor)
70
- page = list_exports(limit:, cursor:)
71
- page.exports.each(&block)
72
- cursor = page.next_cursor
73
- return if cursor.nil? || cursor.empty?
74
- end
75
- raise Error, "export listing returned a repeated cursor"
84
+ # Returns the rows and bytes exported on each UTC day of a window that
85
+ # ends today, oldest first. Today's totals are still growing.
86
+ #
87
+ # @param days [Integer, nil] the window's length; the API defaults to 90 and allows up to 366
88
+ # @return [ExportVolume]
89
+ def export_volume(days: nil)
90
+ volume(days)
76
91
  end
77
92
 
78
93
  # Returns an export's metadata, including its parquet schema.
79
94
  #
80
95
  # @param export [String, Date, Time]
96
+ # @param variant [String, nil] an ExportVariant
81
97
  # @return [ExportMeta]
82
- def export_meta(export)
83
- Model.decode(ExportMeta, @transport.json("GET", "#{@path}/export/#{export_path(export)}/meta", policy: :transient))
98
+ def export_meta(export, variant: nil)
99
+ meta(export_path(export, hourly: true), { variant: })
84
100
  end
85
101
 
86
102
  # Returns a presigned URL for downloading an export. The URL is valid for
87
103
  # 24 hours and grants access without an API key, so treat it as a secret.
88
104
  #
89
105
  # @param export [String, Date, Time]
106
+ # @param variant [String, nil] an ExportVariant
90
107
  # @return [URI::HTTP]
91
- def download_url(export)
92
- @transport.redirect_location("#{@path}/export/#{export_path(export)}")
108
+ def download_url(export, variant: nil)
109
+ location(export_path(export, hourly: true), { variant: })
93
110
  end
94
111
 
95
112
  # Downloads an export, yielding the parquet file's bytes in chunks as they
@@ -98,38 +115,14 @@ module Synthient
98
115
  # closes the connection. Without a block, returns an Enumerator.
99
116
  #
100
117
  # @param export [String, Date, Time]
118
+ # @param variant [String, nil] an ExportVariant
101
119
  # @yieldparam chunk [String] binary bytes
102
120
  # @return [nil]
103
- def download(export, &)
104
- return enum_for(:download, export) unless block_given?
121
+ def download(export, variant: nil, &)
122
+ return enum_for(:download, export, variant:) unless block_given?
105
123
 
106
- @transport.download(download_url(export), &)
124
+ @transport.download(download_url(export, variant:), &)
107
125
  nil
108
126
  end
109
-
110
- private
111
-
112
- def export_path(export)
113
- case export
114
- when Time
115
- time = export.getutc
116
- "#{time.strftime("%Y-%m-%d")}/#{time.hour}"
117
- when DateTime
118
- raise TypeError, "export must be a Time or Date, not a DateTime"
119
- when Date
120
- export.strftime("%Y-%m-%d")
121
- when String
122
- return export if export == "latest"
123
-
124
- match = %r{\A(\d{4})-(\d{2})-(\d{2})(?:/(\d{1,2}))?\z}.match(export)
125
- unless match && Date.valid_date?(match[1].to_i, match[2].to_i, match[3].to_i) && match[4].to_i <= 23
126
- raise ArgumentError, "invalid export ID #{export.inspect}"
127
- end
128
-
129
- match[4] ? "#{match[1]}-#{match[2]}-#{match[3]}/#{match[4].to_i}" : export
130
- else
131
- raise TypeError, "export must be a String, Date, or Time"
132
- end
133
- end
134
127
  end
135
128
  end
@@ -108,8 +108,9 @@ module Synthient
108
108
  # Listing skips the grpc.reflection services; name one in +symbols+ to
109
109
  # fetch it.
110
110
  #
111
- # @param api_key [String, nil] sent as x-api-key metadata unless blank;
112
- # by default from SYNTHIENT_API_KEY
111
+ # @param api_key [String, nil] sent as x-api-key metadata, which
112
+ # reflection requires; by default from SYNTHIENT_API_KEY. A blank key
113
+ # is not sent, and the server answers UNAUTHENTICATED.
113
114
  # @param endpoint [String] host:port, a bare host (port 443 is assumed), or
114
115
  # a grpc:// or https:// URL
115
116
  # @param plaintext [Boolean] disables TLS, for local development servers
@@ -8,7 +8,7 @@ module Synthient
8
8
  #
9
9
  # @api private
10
10
  module Model
11
- Field = Data.define(:name, :wire, :type, :required)
11
+ Field = Data.define(:name, :wire, :type, :required, :null_as)
12
12
 
13
13
  class Builder
14
14
  attr_reader :fields
@@ -17,12 +17,14 @@ module Synthient
17
17
  @fields = []
18
18
  end
19
19
 
20
- def required(name, type, wire: name.to_s)
21
- @fields << Field.new(name:, wire:, type:, required: true)
20
+ # +null_as+ is decoded in place of a null or missing value, for a
21
+ # required field the API has been seen to send as null.
22
+ def required(name, type, wire: name.to_s, null_as: nil)
23
+ @fields << Field.new(name:, wire:, type:, required: true, null_as:)
22
24
  end
23
25
 
24
26
  def optional(name, type, wire: name.to_s)
25
- @fields << Field.new(name:, wire:, type:, required: false)
27
+ @fields << Field.new(name:, wire:, type:, required: false, null_as: nil)
26
28
  end
27
29
  end
28
30
 
@@ -42,6 +44,7 @@ module Synthient
42
44
 
43
45
  attrs = model.fields.to_h do |field|
44
46
  value = object[field.wire]
47
+ value = field.null_as if value.nil?
45
48
  field_path = "#{path}.#{field.wire}"
46
49
  raise DecodeError, "#{field_path} is required" if value.nil? && field.required
47
50
 
@@ -31,6 +31,22 @@ module Synthient
31
31
  JA4_STREAM = "JA4_STREAM"
32
32
  JA4T_FEED = "JA4T_FEED"
33
33
  JA4T_STREAM = "JA4T_STREAM"
34
+ GEOLOCATION_FEED = "GEOLOCATION_FEED"
35
+ WHOIS_FEED = "WHOIS_FEED"
36
+ # Reserved: no operation accepts it yet.
37
+ WHOIS_STREAM = "WHOIS_STREAM"
38
+ # Reserved for an upcoming network IOC feed.
39
+ NETWORK_IOCS_FEED = "NETWORK_IOCS_FEED"
40
+ # Reserved for an upcoming network IOC feed.
41
+ NETWORK_IOCS_STREAM = "NETWORK_IOCS_STREAM"
42
+ # Reserved for an upcoming file IOC feed.
43
+ FILE_IOCS_FEED = "FILE_IOCS_FEED"
44
+ # Reserved for an upcoming file IOC feed.
45
+ FILE_IOCS_STREAM = "FILE_IOCS_STREAM"
46
+ # A dashboard permission that authorizes no API operation.
47
+ HELIOS = "HELIOS"
48
+ # @deprecated The API no longer lists this scope, and it never authorized
49
+ # an API operation. It will be removed in the next major version.
34
50
  ADMIN_PROXIES = "ADMIN_PROXIES"
35
51
  end
36
52
 
@@ -55,6 +71,7 @@ module Synthient
55
71
  end
56
72
 
57
73
  # The account that owns the API key. +scopes+ holds Scope values.
74
+ # +api_key+ is nil if the key was deleted while the request was in flight.
58
75
  Account = Model.define do
59
76
  required :email, :string
60
77
  required :first_name, :string