fluent-plugin-shodan 0.0.1 → 0.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +56 -7
- data/VERSION +1 -1
- data/lib/fluent/plugin/in_shodan_alert.rb +61 -0
- data/test/test_in_shodan_alert.rb +44 -0
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2fe44cee9c9cd48566c10962f21e41a17e533e64922ab9310279e4e04c0cc8f8
|
4
|
+
data.tar.gz: 54775456dd2156f19b1f77944438302f9ec4bce33040478e4521aad3028c4d2f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7a81fffe6a3a5266936f8372ca518ca8a67e36dc9d72290b0c6aea97ceb411d6d22eef7e4e67b4a45dfa1a5bf30e61db318ff0e056399fd0fde5c67884d038d2
|
7
|
+
data.tar.gz: c895e61fa305082e4b98c206bb9886352a8c23257a2fa58cb37326493dfa343094fd768dbb2fb492c8660977b793a50a34a683813842bcda46e8f90c101f57fa
|
data/README.md
CHANGED
@@ -1,6 +1,9 @@
|
|
1
1
|
# fluent-plugin-shodan
|
2
2
|
|
3
|
-
[![Unit tests](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml)
|
3
|
+
[![Unit tests](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml)
|
4
|
+
[![Ruby Gem](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml)
|
5
|
+
[![GPR](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml)
|
6
|
+
[![Gem Version](https://badge.fury.io/rb/fluent-plugin-shodan.svg)](https://badge.fury.io/rb/fluent-plugin-shodan)
|
4
7
|
|
5
8
|
The [Shodan](https://www.shodan.io/) plugin offers [Fluentd](https://fluentd.org/) capacities to gather data from shodan and send them to whatever system you want (on the condition Fluentd has an [output plugin](https://docs.fluentd.org/output) fitting your needs).
|
6
9
|
|
@@ -8,13 +11,11 @@ The Shodan plugin can adress three ways of gathering data
|
|
8
11
|
|
9
12
|
- by querying the [Search API](https://developer.shodan.io/api)
|
10
13
|
- by consuming the [Stream API](https://developer.shodan.io/api/stream) (WIP)
|
11
|
-
- or by consuming the [Alert API](https://developer.shodan.io/api/stream)
|
14
|
+
- or by consuming the [Alert API](https://developer.shodan.io/api/stream)
|
12
15
|
|
13
16
|
The outputed "logs" follow the Shodan [Banner specification](https://datapedia.shodan.io/).
|
14
17
|
|
15
|
-
A valid API key will be necessary for this plugin to work. The Shodan Search plugin will work with a _Free_ account with limited functionnalities, but the Shodans Stream and the Shodan Alert plugins will need a
|
16
|
-
|
17
|
-
This plugin relies on the
|
18
|
+
A valid API key will be necessary for this plugin to work. The Shodan Search plugin will work with a _Free_ account with limited functionnalities, but the Shodans Stream and the Shodan Alert plugins will need at least a membership to work.
|
18
19
|
|
19
20
|
## Installation
|
20
21
|
|
@@ -56,7 +57,7 @@ $ bundle
|
|
56
57
|
|
57
58
|
When Fluentd is started with `in_shodan_search`, it will create a Shodan client and passes to it the API key. It will then query the Shodan API to get the account information to check if the API key is valid. If it is not, an error will be logged and the plugin will stop.
|
58
59
|
|
59
|
-
Once the client is ready, a timer will be set to query the Shodan API a the interval set up in the configuration. One line of "log" will be generated per element contained in the `matches` array from the query result. An other query will be submitted to gather data
|
60
|
+
Once the client is ready, a timer will be set to query the Shodan API a the interval set up in the configuration. One line of "log" will be generated per element contained in the `matches` array from the query result. An other query will be submitted to gather data from the next page if
|
60
61
|
|
61
62
|
- the amount of read entries is lesser than the total available entries
|
62
63
|
- the current read page is not greater than the `max_pages` parameter
|
@@ -98,7 +99,51 @@ WIP
|
|
98
99
|
|
99
100
|
## Shodan Alert
|
100
101
|
|
101
|
-
|
102
|
+
### Example configuration
|
103
|
+
|
104
|
+
```
|
105
|
+
<source>
|
106
|
+
@type shodan_search
|
107
|
+
interval 15m
|
108
|
+
alert_id GA3FRJ1HJNDPORHV
|
109
|
+
api_key 1234567890AZERTYUIOP
|
110
|
+
</source>
|
111
|
+
```
|
112
|
+
|
113
|
+
### How it works
|
114
|
+
|
115
|
+
When Fluentd is started with `in_shodan_alert`, it will create a Shodan client and passes to it the API key. It will then query the Shodan API to get the account information to check if the API key is valid. If it is not, an error will be logged and the plugin will stop.
|
116
|
+
|
117
|
+
Once the client is ready, a timer will be set to query the Shodan Streaming API a the interval set up in the configuration. One line of log will be generated for each alert yield by the API.
|
118
|
+
|
119
|
+
### Plugin helpers
|
120
|
+
|
121
|
+
* [timer](https://docs.fluentd.org/v/1.0/plugin-helper-overview/api-plugin-helper-timer)
|
122
|
+
* See also: [Input Plugin Overview](https://docs.fluentd.org/v/1.0/input#overview)
|
123
|
+
|
124
|
+
### Configuration
|
125
|
+
|
126
|
+
#### api_key (string) (required)
|
127
|
+
|
128
|
+
The API key to connect to the Shodan API.
|
129
|
+
|
130
|
+
#### interval (time) (optional)
|
131
|
+
|
132
|
+
The interval time between running queries.
|
133
|
+
|
134
|
+
Default value: `3600`.
|
135
|
+
|
136
|
+
#### tag (string) (optional)
|
137
|
+
|
138
|
+
The tag to apply to each shodan entries. If none are given, the alert name will be used to tag each associated emitted log.
|
139
|
+
|
140
|
+
Default value: `nil`
|
141
|
+
|
142
|
+
#### alert_id (string) (optional)
|
143
|
+
|
144
|
+
The identifier of the alert to crawl. If none are given, all alerts are imported.
|
145
|
+
|
146
|
+
Default value: `nil`
|
102
147
|
|
103
148
|
## Testing
|
104
149
|
|
@@ -118,6 +163,10 @@ On a system where fluentd is installed
|
|
118
163
|
3. Install the built gem with `fluent-gem install fluent-plugin-shodan-<version>.gem`
|
119
164
|
4. Follow the [debugging guide from FluentD](https://docs.fluentd.org/plugin-development#debugging-plugins)
|
120
165
|
|
166
|
+
## Credits
|
167
|
+
|
168
|
+
This plugin heavily relies on the [shodanz](https://github.com/picatz/shodanz) gem by [Kent 'picat' Gruber](https://picatz.github.io/) which makes it really easy to query the Shodan API.
|
169
|
+
|
121
170
|
## Copyright
|
122
171
|
|
123
172
|
* Copyright(c) 2022 Marc-André Doll
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
0.0.2
|
@@ -0,0 +1,61 @@
|
|
1
|
+
require 'fluent/plugin/input'
|
2
|
+
require 'shodanz'
|
3
|
+
|
4
|
+
module Fluent::Plugin
|
5
|
+
class ShodanAlert < Input
|
6
|
+
Fluent::Plugin.register_input('shodan_alert', self)
|
7
|
+
|
8
|
+
helpers :timer
|
9
|
+
|
10
|
+
desc "The API key to connect to the Shodan API."
|
11
|
+
config_param :api_key, :string, secret: true
|
12
|
+
desc "The tag to apply to each shodan entries. If not defined, the alert name will be used."
|
13
|
+
config_param :tag, :string, default: nil
|
14
|
+
desc "The Shodan alert ID to follow. If not defined, all alerts will be followed"
|
15
|
+
config_param :alert_id, :string, default: nil
|
16
|
+
desc "The interval time between running queries."
|
17
|
+
config_param :interval, :time, default: 300
|
18
|
+
|
19
|
+
def multi_workers_ready?
|
20
|
+
false
|
21
|
+
end
|
22
|
+
|
23
|
+
def configure(conf)
|
24
|
+
super
|
25
|
+
|
26
|
+
@client = Shodanz::Client.new(key: @api_key)
|
27
|
+
begin
|
28
|
+
log.info "Shodan client properly registered", client_info: @client.info
|
29
|
+
rescue RuntimeError => exception
|
30
|
+
raise Fluent::ConfigError.new "Invalid Shodan API key"
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def start
|
35
|
+
super
|
36
|
+
|
37
|
+
timer_execute("shodan_#{self.class.name}#{@alert_id.nil? ? "_#{@alert_id}" : '' }".to_sym, @interval, repeat: true, &method(:run))
|
38
|
+
end
|
39
|
+
|
40
|
+
private
|
41
|
+
|
42
|
+
def run
|
43
|
+
log.debug "Start polling Shodan alerts", alert_id: @alert_id
|
44
|
+
|
45
|
+
if @alert_id.nil?
|
46
|
+
@client.streaming_api.alerts { |alert| process_alert(alert) }
|
47
|
+
else
|
48
|
+
@client.streaming_api.alert(@alert_id) { |alert| process_alert(alert) }
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
def process_alert(alert)
|
53
|
+
router.emit(
|
54
|
+
(@tag.nil? ? alert['shodan']['alert']['name'] : @tag),
|
55
|
+
Fluent::EventTime.parse(alert['timestamp']),
|
56
|
+
alert
|
57
|
+
)
|
58
|
+
end
|
59
|
+
|
60
|
+
end
|
61
|
+
end
|
@@ -0,0 +1,44 @@
|
|
1
|
+
require 'test/unit'
|
2
|
+
require 'fluent/env'
|
3
|
+
require 'fluent/test'
|
4
|
+
require 'fluent/test/driver/input'
|
5
|
+
require 'fluent/test/helpers'
|
6
|
+
require "fluent/plugin/in_shodan_alert"
|
7
|
+
|
8
|
+
class ShodanAlertInputTest < Test::Unit::TestCase
|
9
|
+
include Fluent::Test::Helpers
|
10
|
+
|
11
|
+
API_KEY = ENV['SHODAN_TEST_API_KEY']
|
12
|
+
|
13
|
+
def setup
|
14
|
+
Fluent::Test.setup
|
15
|
+
end
|
16
|
+
|
17
|
+
sub_test_case 'Configuration' do
|
18
|
+
test 'is invalid without API key' do
|
19
|
+
assert_raise Fluent::ConfigError do
|
20
|
+
create_driver(%[])
|
21
|
+
end
|
22
|
+
end
|
23
|
+
test 'is invalid if the API key is invalid' do
|
24
|
+
assert_raise Fluent::ConfigError do
|
25
|
+
create_driver(%[
|
26
|
+
api_key 1234567890AZERTYUIOP
|
27
|
+
])
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
32
|
+
sub_test_case 'Plugin emission' do
|
33
|
+
end
|
34
|
+
|
35
|
+
private
|
36
|
+
|
37
|
+
CONFIG = %[
|
38
|
+
api_key #{API_KEY}
|
39
|
+
]
|
40
|
+
|
41
|
+
def create_driver(conf = CONFIG)
|
42
|
+
Fluent::Test::Driver::Input.new(Fluent::Plugin::ShodanAlert).configure(conf)
|
43
|
+
end
|
44
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-shodan
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- srilumpa
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-
|
11
|
+
date: 2022-02-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: fluentd
|
@@ -110,7 +110,9 @@ files:
|
|
110
110
|
- AUTHORS
|
111
111
|
- README.md
|
112
112
|
- VERSION
|
113
|
+
- lib/fluent/plugin/in_shodan_alert.rb
|
113
114
|
- lib/fluent/plugin/in_shodan_search.rb
|
115
|
+
- test/test_in_shodan_alert.rb
|
114
116
|
- test/test_in_shodan_search.rb
|
115
117
|
homepage: https://github.com/srilumpa/fluent-plugin-shodan
|
116
118
|
licenses:
|
@@ -136,4 +138,5 @@ signing_key:
|
|
136
138
|
specification_version: 3
|
137
139
|
summary: Fluentd plugin to extract data from Shodan
|
138
140
|
test_files:
|
141
|
+
- test/test_in_shodan_alert.rb
|
139
142
|
- test/test_in_shodan_search.rb
|