ukemi 0.1.0 → 0.4.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (21) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/test.yaml +27 -0
  3. data/.overcommit.yml +9 -0
  4. data/.standard.yml +4 -0
  5. data/README.md +140 -364
  6. data/exe/ukemi +3 -1
  7. data/lib/ukemi.rb +4 -0
  8. data/lib/ukemi/cli.rb +16 -0
  9. data/lib/ukemi/configuration.rb +24 -0
  10. data/lib/ukemi/moderator.rb +39 -9
  11. data/lib/ukemi/record.rb +1 -4
  12. data/lib/ukemi/services/circl.rb +5 -5
  13. data/lib/ukemi/services/dnsdb.rb +53 -0
  14. data/lib/ukemi/services/otx.rb +72 -0
  15. data/lib/ukemi/services/passivetotal.rb +5 -5
  16. data/lib/ukemi/services/securitytrails.rb +21 -20
  17. data/lib/ukemi/services/virustotal.rb +5 -5
  18. data/lib/ukemi/version.rb +1 -1
  19. data/ukemi.gemspec +24 -20
  20. metadata +86 -25
  21. data/.travis.yml +0 -6
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a77579f014e97cc048e95dc5488e228ee19baba62d5074dbc9bfcb5c75f9d568
4
- data.tar.gz: e156b9c6de521ea49d7a69725c724176a5d1739deee7bba32aef4e662c64ddae
3
+ metadata.gz: a512ff8b644d793535d9beac64ab550b5628bbb86fd042e2cc3ab5b2cdad1ea1
4
+ data.tar.gz: d7433d7e80ff7723bf097db0bbda4a35f3feab7d2fbd4d3d7726c1ff3319c3a0
5
5
  SHA512:
6
- metadata.gz: f4e3eb6f8f4dd2223ba2eed1846ff8d95ee8d14913f5f6e682269489ee9348fea3af858c9bf849bac4d612a42a0a3c1c7bae3b27dde3461b1104c3e0e3752964
7
- data.tar.gz: 34467c49986dc11f2ac4fae646c1d96419ebe199838fe20a8f859c1619137ce2f3d0bcfc11e1f356545e14f63a6b5ff81b7ad9f39b206b70469959a8ca9a5986
6
+ metadata.gz: 9e88eefa02c8071b840e16c25ecdd52198abdd16ede41ff657692b9fc6ff2c9b21e0b8130d6562cc348eb8b101d91b0309a7feb4b5a48cb3283968518e649d1f
7
+ data.tar.gz: 6cca209ba2a055a195e862b0f10b2e431d8555d32985050b5debc6e4d20164c640af1afa30960ebde6842bc109b4e8176be206303ee0f170de5d441436339a45
data/.github/workflows/test.yaml ADDED
@@ -0,0 +1,27 @@
1
+ name: Ruby CI
2
+
3
+ on: [pull_request]
4
+
5
+ jobs:
6
+ build:
7
+
8
+ runs-on: ubuntu-latest
9
+
10
+ strategy:
11
+ fail-fast: false
12
+ matrix:
13
+ ruby: [2.7, '3.0']
14
+
15
+ steps:
16
+ - uses: actions/checkout@v2
17
+ - name: Set up Ruby
18
+ uses: ruby/setup-ruby@v1
19
+ with:
20
+ ruby-version: ${{ matrix.ruby }}
21
+ bundler-cache: true
22
+
23
+ - name: Build and test with Rake
24
+ run: |
25
+ gem install bundler
26
+ bundle install
27
+ bundle exec rake
data/.overcommit.yml ADDED
@@ -0,0 +1,9 @@
1
+ PreCommit:
2
+ BundleCheck:
3
+ enabled: true
4
+
5
+ RuboCop:
6
+ enabled: true
7
+ required_executable: bundle
8
+ command: ["bundle", "exec", "standardrb"]
9
+ on_warn: fail
data/.standard.yml ADDED
@@ -0,0 +1,4 @@
1
+ ignore:
2
+ - "**/*":
3
+ - Layout/SpaceInsideHashLiteralBraces
4
+ - Style/RescueStandardError
data/README.md CHANGED
@@ -1,6 +1,7 @@
1
1
  # ukemi
2
2
 
3
- [![Build Status](https://travis-ci.com/ninoseki/ukemi.svg?branch=master)](https://travis-ci.com/ninoseki/ukemi)
3
+ [![Gem Version](https://badge.fury.io/rb/ukemi.svg)](https://badge.fury.io/rb/ukemi)
4
+ [![Ruby CI](https://github.com/ninoseki/ukemi/actions/workflows/test.yaml/badge.svg)](https://github.com/ninoseki/ukemi/actions/workflows/test.yaml)
4
5
  [![Coverage Status](https://coveralls.io/repos/github/ninoseki/ukemi/badge.svg?branch=master)](https://coveralls.io/github/ninoseki/ukemi?branch=master)
5
6
  [![CodeFactor](https://www.codefactor.io/repository/github/ninoseki/ukemi/badge)](https://www.codefactor.io/repository/github/ninoseki/ukemi)
6
7
 
@@ -9,13 +10,19 @@ Ukemi is a CIL tool for querying passive DNS services.
9
10
  It supports the following services.
10
11
 
11
12
  - [CIRCL passive DNS](https://www.circl.lu/services/passive-dns/)
13
+ - [DNSDB](https://api.dnsdb.info/)
14
+ - [OTX](https://otx.alienvault.com)
12
15
  - [PassiveTotal](https://community.riskiq.com/)
13
16
  - [SecurityTrails](https://securitytrails.com/)
14
17
  - [VirusTotal](http://virustotal.com)
15
18
 
16
19
  It outputs passive DNS resolutions as JSON.
17
20
 
18
- ## Instalattion
21
+ ## Requirements
22
+
23
+ - Ruby 2.7+ or Ruby 3.x
24
+
25
+ ## Installation
19
26
 
20
27
  ```bash
21
28
  gem install ukemi
@@ -29,6 +36,8 @@ Configuration is done via environment variables.
29
36
  |------------------------|----------------------------|
30
37
  | CIRCL_PASSIVE_PASSWORD | CIRCL passive DNS password |
31
38
  | CIRCL_PASSIVE_USERNAME | CIRCL passive DNS username |
39
+ | DNSDB_API_KEY | DNSDB API key |
40
+ | OTX_API_KEY | OTX API key |
32
41
  | PASSIVETOTAL_API_KEY | PassiveTotal API key |
33
42
  | PASSIVETOTAL_USERNAME | PassiveTotal username |
34
43
  | SECURITYTRAILS_API_KEY | SecurityTrails API key |
@@ -42,379 +51,146 @@ Commands:
42
51
  ukemi help [COMMAND] # Describe available commands or one specific command
43
52
  ukemi lookup [IP|DOMAIN] # Lookup passive DNS services
44
53
 
45
- $ ukemi help looup
54
+ $ ukemi help lookup
46
55
  Usage:
47
56
  ukemi lookup [IP|DOMAIN]
48
57
 
49
- Lookup passive DNS services
58
+ Options:
59
+ [--order-by=ORDER_BY] # Ordering of the passve DNS resolutions (last_seen or first_seen)
60
+ # Default: -last_seen
61
+
62
+ Lookup passive DNS servicess
50
63
  ```
51
64
 
52
65
  ```bash
53
- $ ukemi lookup circl.lu
66
+ $ ukemi lookup example.com
54
67
  {
55
- "149.13.33.14": [
56
- {
57
- "firtst_seen": "2016-10-07",
58
- "last_seen": "2018-10-26",
59
- "source": "CIRCL"
60
- },
61
- {
62
- "firtst_seen": "2017-05-26",
63
- "last_seen": "2020-03-15",
64
- "source": "SecurityTrails"
65
- },
66
- {
67
- "firtst_seen": "2019-12-04",
68
- "last_seen": "2019-12-04",
69
- "source": "VirusTotal"
70
- }
71
- ],
72
- "149.13.33.4": [
73
- {
74
- "firtst_seen": "2011-03-08",
75
- "last_seen": "2012-02-13",
76
- "source": "CIRCL"
77
- },
78
- {
79
- "firtst_seen": "2013-07-30",
80
- "last_seen": "2013-07-30",
81
- "source": "VirusTotal"
82
- }
83
- ],
84
- "194.154.205.24": [
85
- {
86
- "firtst_seen": "2011-03-03",
87
- "last_seen": "2011-03-03",
88
- "source": "CIRCL"
89
- }
90
- ]
68
+ "93.184.216.34": {
69
+ "first_seen": "2016-03-01",
70
+ "last_seen": "2020-03-16",
71
+ "sources": [
72
+ {
73
+ "first_seen": "2016-10-07",
74
+ "last_seen": "2018-10-30",
75
+ "source": "CIRCL"
76
+ },
77
+ {
78
+ "first_seen": "2016-03-01",
79
+ "last_seen": "2020-03-16",
80
+ "source": "SecurityTrails"
81
+ },
82
+ {
83
+ "first_seen": "2020-03-03",
84
+ "last_seen": "2020-03-03",
85
+ "source": "VirusTotal"
86
+ }
87
+ ]
88
+ },
89
+ ...
91
90
  }
92
91
 
93
92
  $ ukemi lookup 195.123.226.243
94
93
  {
95
- "liankt.club": [
96
- {
97
- "firtst_seen": "2020-02-15",
98
- "last_seen": "2020-03-13",
99
- "source": "PassiveTotal"
100
- },
101
- {
102
- "firtst_seen": "2020-02-16",
103
- "last_seen": "2020-02-16",
104
- "source": "VirusTotal"
105
- }
106
- ],
107
- "weidt.club": [
108
- {
109
- "firtst_seen": "2020-03-12",
110
- "last_seen": "2020-03-12",
111
- "source": "PassiveTotal"
112
- }
113
- ],
114
- "jikt.club": [
115
- {
116
- "firtst_seen": "2020-03-04",
117
- "last_seen": "2020-03-12",
118
- "source": "PassiveTotal"
119
- },
120
- {
121
- "firtst_seen": "2020-03-05",
122
- "last_seen": "2020-03-05",
123
- "source": "VirusTotal"
124
- }
125
- ],
126
- "biesi.club": [
127
- {
128
- "firtst_seen": "2020-02-15",
129
- "last_seen": "2020-03-12",
130
- "source": "PassiveTotal"
131
- },
132
- {
133
- "firtst_seen": "2020-02-20",
134
- "last_seen": "2020-02-20",
135
- "source": "VirusTotal"
136
- }
137
- ],
138
- "kaikt.club": [
139
- {
140
- "firtst_seen": "2020-02-15",
141
- "last_seen": "2020-03-12",
142
- "source": "PassiveTotal"
143
- },
144
- {
145
- "firtst_seen": "2020-02-21",
146
- "last_seen": "2020-02-21",
147
- "source": "VirusTotal"
148
- }
149
- ],
150
- "zhaokt.club": [
151
- {
152
- "firtst_seen": "2020-02-15",
153
- "last_seen": "2020-03-11",
154
- "source": "PassiveTotal"
155
- },
156
- {
157
- "firtst_seen": "2020-02-18",
158
- "last_seen": "2020-02-18",
159
- "source": "VirusTotal"
160
- }
161
- ],
162
- "yangdt.club": [
163
- {
164
- "firtst_seen": "2020-02-26",
165
- "last_seen": "2020-03-10",
166
- "source": "PassiveTotal"
167
- },
168
- {
169
- "firtst_seen": "2020-02-27",
170
- "last_seen": "2020-02-27",
171
- "source": "VirusTotal"
172
- }
173
- ],
174
- "jinkt.club": [
175
- {
176
- "firtst_seen": "2020-02-21",
177
- "last_seen": "2020-03-10",
178
- "source": "PassiveTotal"
179
- },
180
- {
181
- "firtst_seen": "2020-02-22",
182
- "last_seen": "2020-02-22",
183
- "source": "VirusTotal"
184
- }
185
- ],
186
- "taokt.club": [
187
- {
188
- "firtst_seen": "2020-03-10",
189
- "last_seen": "2020-03-10",
190
- "source": "PassiveTotal"
191
- }
192
- ],
193
- "xinkt.club": [
194
- {
195
- "firtst_seen": "2020-02-17",
196
- "last_seen": "2020-03-09",
197
- "source": "PassiveTotal"
198
- },
199
- {
200
- "firtst_seen": "2020-02-19",
201
- "last_seen": "2020-02-19",
202
- "source": "VirusTotal"
203
- }
204
- ],
205
- "mail.realty-advertising.ru": [
206
- {
207
- "firtst_seen": "2019-11-08",
208
- "last_seen": "2020-03-09",
209
- "source": "PassiveTotal"
210
- }
211
- ],
212
- "realty-advertising.ru": [
213
- {
214
- "firtst_seen": "2019-11-08",
215
- "last_seen": "2020-03-06",
216
- "source": "PassiveTotal"
217
- }
218
- ],
219
- "ns1.realty-advertising.ru": [
220
- {
221
- "firtst_seen": "2019-12-02",
222
- "last_seen": "2020-03-04",
223
- "source": "PassiveTotal"
224
- }
225
- ],
226
- "ns2.realty-advertising.ru": [
227
- {
228
- "firtst_seen": "2019-12-04",
229
- "last_seen": "2020-03-04",
230
- "source": "PassiveTotal"
231
- }
232
- ],
233
- "xiankt.club": [
234
- {
235
- "firtst_seen": "2020-02-15",
236
- "last_seen": "2020-03-03",
237
- "source": "PassiveTotal"
238
- },
239
- {
240
- "firtst_seen": "2020-02-16",
241
- "last_seen": "2020-02-16",
242
- "source": "VirusTotal"
243
- }
244
- ],
245
- "nittsu-si.com": [
246
- {
247
- "firtst_seen": "2020-02-15",
248
- "last_seen": "2020-03-03",
249
- "source": "PassiveTotal"
250
- },
251
- {
252
- "firtst_seen": "2020-02-21",
253
- "last_seen": "2020-02-21",
254
- "source": "VirusTotal"
255
- }
256
- ],
257
- "mailer.realty-advertising.ru": [
258
- {
259
- "firtst_seen": "2020-02-23",
260
- "last_seen": "2020-02-23",
261
- "source": "PassiveTotal"
262
- }
263
- ],
264
- "mail7.realty-advertising.ru": [
265
- {
266
- "firtst_seen": "2020-02-23",
267
- "last_seen": "2020-02-23",
268
- "source": "PassiveTotal"
269
- }
270
- ],
271
- "zimbra.realty-advertising.ru": [
272
- {
273
- "firtst_seen": "2020-02-23",
274
- "last_seen": "2020-02-23",
275
- "source": "PassiveTotal"
276
- }
277
- ],
278
- "relay2.realty-advertising.ru": [
279
- {
280
- "firtst_seen": "2020-02-23",
281
- "last_seen": "2020-02-23",
282
- "source": "PassiveTotal"
283
- }
284
- ],
285
- "sniper.realty-advertising.ru": [
286
- {
287
- "firtst_seen": "2020-02-22",
288
- "last_seen": "2020-02-22",
289
- "source": "PassiveTotal"
290
- }
291
- ],
292
- "mailx.realty-advertising.ru": [
293
- {
294
- "firtst_seen": "2020-02-22",
295
- "last_seen": "2020-02-22",
296
- "source": "PassiveTotal"
297
- }
298
- ],
299
- "send.realty-advertising.ru": [
300
- {
301
- "firtst_seen": "2020-02-22",
302
- "last_seen": "2020-02-22",
303
- "source": "PassiveTotal"
304
- }
305
- ],
306
- "mta.realty-advertising.ru": [
307
- {
308
- "firtst_seen": "2020-02-22",
309
- "last_seen": "2020-02-22",
310
- "source": "PassiveTotal"
311
- }
312
- ],
313
- "home.realty-advertising.ru": [
314
- {
315
- "firtst_seen": "2020-02-22",
316
- "last_seen": "2020-02-22",
317
- "source": "PassiveTotal"
318
- }
319
- ],
320
- "pbrand.realty-advertising.ru": [
321
- {
322
- "firtst_seen": "2020-02-22",
323
- "last_seen": "2020-02-22",
324
- "source": "PassiveTotal"
325
- }
326
- ],
327
- "smtpauth.realty-advertising.ru": [
328
- {
329
- "firtst_seen": "2020-02-22",
330
- "last_seen": "2020-02-22",
331
- "source": "PassiveTotal"
332
- }
333
- ],
334
- "gate.realty-advertising.ru": [
335
- {
336
- "firtst_seen": "2020-02-21",
337
- "last_seen": "2020-02-21",
338
- "source": "PassiveTotal"
339
- }
340
- ],
341
- "mx02.realty-advertising.ru": [
342
- {
343
- "firtst_seen": "2020-02-21",
344
- "last_seen": "2020-02-21",
345
- "source": "PassiveTotal"
346
- }
347
- ],
348
- "outmail.realty-advertising.ru": [
349
- {
350
- "firtst_seen": "2020-02-21",
351
- "last_seen": "2020-02-21",
352
- "source": "PassiveTotal"
353
- }
354
- ],
355
- "exchange.realty-advertising.ru": [
356
- {
357
- "firtst_seen": "2020-02-21",
358
- "last_seen": "2020-02-21",
359
- "source": "PassiveTotal"
360
- }
361
- ],
362
- "ms.realty-advertising.ru": [
363
- {
364
- "firtst_seen": "2020-02-21",
365
- "last_seen": "2020-02-21",
366
- "source": "PassiveTotal"
367
- }
368
- ],
369
- "owa.realty-advertising.ru": [
370
- {
371
- "firtst_seen": "2020-02-20",
372
- "last_seen": "2020-02-20",
373
- "source": "PassiveTotal"
374
- }
375
- ],
376
- "mail8.realty-advertising.ru": [
377
- {
378
- "firtst_seen": "2020-02-20",
379
- "last_seen": "2020-02-20",
380
- "source": "PassiveTotal"
381
- }
382
- ],
383
- "mta-sts.realty-advertising.ru": [
384
- {
385
- "firtst_seen": "2019-11-11",
386
- "last_seen": "2020-02-08",
387
- "source": "PassiveTotal"
388
- }
389
- ],
390
- "mail02.realty-advertising.ru": [
391
- {
392
- "firtst_seen": "2020-01-18",
393
- "last_seen": "2020-01-18",
394
- "source": "PassiveTotal"
395
- }
396
- ],
397
- "www.realty-advertising.ru": [
398
- {
399
- "firtst_seen": "2019-11-08",
400
- "last_seen": "2019-11-12",
401
- "source": "PassiveTotal"
402
- }
403
- ],
404
- "ln-048.rd-00003024.id-11744955.v0.tun.vpnoverdns.com": [
405
- {
406
- "firtst_seen": "2017-04-06",
407
- "last_seen": "2017-04-06",
408
- "source": "PassiveTotal"
409
- }
410
- ],
411
- "mnen6k7g.info": [
412
- {
413
- "firtst_seen": "2010-10-28",
414
- "last_seen": "2010-10-28",
415
- "source": "PassiveTotal"
416
- }
417
- ]
94
+ "example.org": {
95
+ "first_seen": "2011-04-11",
96
+ "last_seen": "2020-03-16",
97
+ "sources": [
98
+ {
99
+ "first_seen": "2011-04-11",
100
+ "last_seen": "2011-04-11",
101
+ "source": "CIRCL"
102
+ },
103
+ {
104
+ "first_seen": "2016-10-09",
105
+ "last_seen": "2018-10-28",
106
+ "source": "CIRCL"
107
+ },
108
+ {
109
+ "first_seen": "2014-12-09",
110
+ "last_seen": "2020-03-16",
111
+ "source": "PassiveTotal"
112
+ },
113
+ {
114
+ "first_seen": null,
115
+ "last_seen": null,
116
+ "source": "SecurityTrails"
117
+ }
118
+ ]
119
+ },
120
+ ...
121
+ }
122
+
123
+ # or you scan skip "lookup"
124
+ $ ukemi example.com
125
+ $ ukemi 195.123.226.243
126
+
127
+ # You can specify the order of resolutions
128
+
129
+ # Order by last_seen DESC
130
+ $ ukemi lookup example.com --order-by -last_seen
131
+
132
+ # Order by last_seen ASC
133
+ $ ukemi lookup example.com --order-by last_seen
134
+
135
+ # Order by first_seen DESC
136
+ $ ukemi lookup example.com --order-by -first_seen
137
+
138
+ # Order by first_seen ASC
139
+ $ ukemi lookup example.com --order-by first_seen
140
+ ```
141
+
142
+ ### Using with jq
143
+
144
+ [jq](https://stedolan.github.io/jq/)'s powerful processor helps to interact with the output.
145
+
146
+ ```bash
147
+ # List up resolutions only
148
+ $ ukemi lookup example.com | jq "keys"
149
+ [
150
+ "192.0.32.10",
151
+ "192.0.43.10",
152
+ "208.77.188.166",
153
+ "209.67.208.202",
154
+ "221.121.159.162",
155
+ "93.184.216.119",
156
+ "93.184.216.34"
157
+ ]
158
+
159
+ # List up the first 2 objects
160
+ $ ukemi lookup example.com | jq "to_entries | .[:2] | from_entries"
161
+ {
162
+ "93.184.216.34": {
163
+ "first_seen": "2016-03-01",
164
+ "last_seen": "2020-03-16",
165
+ "sources": [
166
+ {
167
+ "first_seen": "2016-10-07",
168
+ "last_seen": "2018-10-30",
169
+ "source": "CIRCL"
170
+ },
171
+ {
172
+ "first_seen": "2016-03-01",
173
+ "last_seen": "2020-03-16",
174
+ "source": "SecurityTrails"
175
+ },
176
+ {
177
+ "first_seen": "2020-03-03",
178
+ "last_seen": "2020-03-03",
179
+ "source": "VirusTotal"
180
+ }
181
+ ]
182
+ },
183
+ "221.121.159.162": {
184
+ "first_seen": "2019-11-04",
185
+ "last_seen": "2019-11-04",
186
+ "sources": [
187
+ {
188
+ "first_seen": "2019-11-04",
189
+ "last_seen": "2019-11-04",
190
+ "source": "VirusTotal"
191
+ }
192
+ ]
193
+ }
418
194
  }
419
195
  ```
420
196