fluent-plugin-shodan 0.0.1 → 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c5065dc8260c9015eebe3ab7e1095d050fdb64b3fa0d0168c414342e31f7960d
4
- data.tar.gz: b26a7a5b53c56cc6469433eb5867749ec26c7298958294dcf7c6c5b2f8a335e0
3
+ metadata.gz: 2fe44cee9c9cd48566c10962f21e41a17e533e64922ab9310279e4e04c0cc8f8
4
+ data.tar.gz: 54775456dd2156f19b1f77944438302f9ec4bce33040478e4521aad3028c4d2f
5
5
  SHA512:
6
- metadata.gz: 3244f0dd4e8c9d916a83220e2a534631f3c413a15b68ce985dc93a2632691a75b4147c838a2a871136a58c0e753b701304faf2cf86e6b18bd421302162836ac7
7
- data.tar.gz: 14b471034818688f951fdb36d0102dccd63bd8f3a6437e6ee8e44de9d8a0d799190241250aaacd7aa631e5a4ab467f36d11ad4e59e2615cbfccfb33e92db174a
6
+ metadata.gz: 7a81fffe6a3a5266936f8372ca518ca8a67e36dc9d72290b0c6aea97ceb411d6d22eef7e4e67b4a45dfa1a5bf30e61db318ff0e056399fd0fde5c67884d038d2
7
+ data.tar.gz: c895e61fa305082e4b98c206bb9886352a8c23257a2fa58cb37326493dfa343094fd768dbb2fb492c8660977b793a50a34a683813842bcda46e8f90c101f57fa
data/README.md CHANGED
@@ -1,6 +1,9 @@
1
1
  # fluent-plugin-shodan
2
2
 
3
- [![Unit tests](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml)[![Ruby Gem](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml)[![GPR](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml)
3
+ [![Unit tests](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/ruby.yml)
4
+ [![Ruby Gem](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/rubygem-push.yml)
5
+ [![GPR](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml/badge.svg)](https://github.com/srilumpa/fluent-plugin-shodan/actions/workflows/gpr-push.yml)
6
+ [![Gem Version](https://badge.fury.io/rb/fluent-plugin-shodan.svg)](https://badge.fury.io/rb/fluent-plugin-shodan)
4
7
 
5
8
  The [Shodan](https://www.shodan.io/) plugin offers [Fluentd](https://fluentd.org/) capacities to gather data from shodan and send them to whatever system you want (on the condition Fluentd has an [output plugin](https://docs.fluentd.org/output) fitting your needs).
6
9
 
@@ -8,13 +11,11 @@ The Shodan plugin can adress three ways of gathering data
8
11
 
9
12
  - by querying the [Search API](https://developer.shodan.io/api)
10
13
  - by consuming the [Stream API](https://developer.shodan.io/api/stream) (WIP)
11
- - or by consuming the [Alert API](https://developer.shodan.io/api/stream) (WIP)
14
+ - or by consuming the [Alert API](https://developer.shodan.io/api/stream)
12
15
 
13
16
  The outputed "logs" follow the Shodan [Banner specification](https://datapedia.shodan.io/).
14
17
 
15
- A valid API key will be necessary for this plugin to work. The Shodan Search plugin will work with a _Free_ account with limited functionnalities, but the Shodans Stream and the Shodan Alert plugins will need a subscription plan to work.
16
-
17
- This plugin relies on the
18
+ A valid API key will be necessary for this plugin to work. The Shodan Search plugin will work with a _Free_ account with limited functionnalities, but the Shodans Stream and the Shodan Alert plugins will need at least a membership to work.
18
19
 
19
20
  ## Installation
20
21
 
@@ -56,7 +57,7 @@ $ bundle
56
57
 
57
58
  When Fluentd is started with `in_shodan_search`, it will create a Shodan client and passes to it the API key. It will then query the Shodan API to get the account information to check if the API key is valid. If it is not, an error will be logged and the plugin will stop.
58
59
 
59
- Once the client is ready, a timer will be set to query the Shodan API a the interval set up in the configuration. One line of "log" will be generated per element contained in the `matches` array from the query result. An other query will be submitted to gather data fro mthe next page if
60
+ Once the client is ready, a timer will be set to query the Shodan API a the interval set up in the configuration. One line of "log" will be generated per element contained in the `matches` array from the query result. An other query will be submitted to gather data from the next page if
60
61
 
61
62
  - the amount of read entries is lesser than the total available entries
62
63
  - the current read page is not greater than the `max_pages` parameter
@@ -98,7 +99,51 @@ WIP
98
99
 
99
100
  ## Shodan Alert
100
101
 
101
- WIP
102
+ ### Example configuration
103
+
104
+ ```
105
+ <source>
106
+ @type shodan_search
107
+ interval 15m
108
+ alert_id GA3FRJ1HJNDPORHV
109
+ api_key 1234567890AZERTYUIOP
110
+ </source>
111
+ ```
112
+
113
+ ### How it works
114
+
115
+ When Fluentd is started with `in_shodan_alert`, it will create a Shodan client and passes to it the API key. It will then query the Shodan API to get the account information to check if the API key is valid. If it is not, an error will be logged and the plugin will stop.
116
+
117
+ Once the client is ready, a timer will be set to query the Shodan Streaming API a the interval set up in the configuration. One line of log will be generated for each alert yield by the API.
118
+
119
+ ### Plugin helpers
120
+
121
+ * [timer](https://docs.fluentd.org/v/1.0/plugin-helper-overview/api-plugin-helper-timer)
122
+ * See also: [Input Plugin Overview](https://docs.fluentd.org/v/1.0/input#overview)
123
+
124
+ ### Configuration
125
+
126
+ #### api_key (string) (required)
127
+
128
+ The API key to connect to the Shodan API.
129
+
130
+ #### interval (time) (optional)
131
+
132
+ The interval time between running queries.
133
+
134
+ Default value: `3600`.
135
+
136
+ #### tag (string) (optional)
137
+
138
+ The tag to apply to each shodan entries. If none are given, the alert name will be used to tag each associated emitted log.
139
+
140
+ Default value: `nil`
141
+
142
+ #### alert_id (string) (optional)
143
+
144
+ The identifier of the alert to crawl. If none are given, all alerts are imported.
145
+
146
+ Default value: `nil`
102
147
 
103
148
  ## Testing
104
149
 
@@ -118,6 +163,10 @@ On a system where fluentd is installed
118
163
  3. Install the built gem with `fluent-gem install fluent-plugin-shodan-<version>.gem`
119
164
  4. Follow the [debugging guide from FluentD](https://docs.fluentd.org/plugin-development#debugging-plugins)
120
165
 
166
+ ## Credits
167
+
168
+ This plugin heavily relies on the [shodanz](https://github.com/picatz/shodanz) gem by [Kent 'picat' Gruber](https://picatz.github.io/) which makes it really easy to query the Shodan API.
169
+
121
170
  ## Copyright
122
171
 
123
172
  * Copyright(c) 2022 Marc-André Doll
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.0.1
1
+ 0.0.2
@@ -0,0 +1,61 @@
1
+ require 'fluent/plugin/input'
2
+ require 'shodanz'
3
+
4
+ module Fluent::Plugin
5
+ class ShodanAlert < Input
6
+ Fluent::Plugin.register_input('shodan_alert', self)
7
+
8
+ helpers :timer
9
+
10
+ desc "The API key to connect to the Shodan API."
11
+ config_param :api_key, :string, secret: true
12
+ desc "The tag to apply to each shodan entries. If not defined, the alert name will be used."
13
+ config_param :tag, :string, default: nil
14
+ desc "The Shodan alert ID to follow. If not defined, all alerts will be followed"
15
+ config_param :alert_id, :string, default: nil
16
+ desc "The interval time between running queries."
17
+ config_param :interval, :time, default: 300
18
+
19
+ def multi_workers_ready?
20
+ false
21
+ end
22
+
23
+ def configure(conf)
24
+ super
25
+
26
+ @client = Shodanz::Client.new(key: @api_key)
27
+ begin
28
+ log.info "Shodan client properly registered", client_info: @client.info
29
+ rescue RuntimeError => exception
30
+ raise Fluent::ConfigError.new "Invalid Shodan API key"
31
+ end
32
+ end
33
+
34
+ def start
35
+ super
36
+
37
+ timer_execute("shodan_#{self.class.name}#{@alert_id.nil? ? "_#{@alert_id}" : '' }".to_sym, @interval, repeat: true, &method(:run))
38
+ end
39
+
40
+ private
41
+
42
+ def run
43
+ log.debug "Start polling Shodan alerts", alert_id: @alert_id
44
+
45
+ if @alert_id.nil?
46
+ @client.streaming_api.alerts { |alert| process_alert(alert) }
47
+ else
48
+ @client.streaming_api.alert(@alert_id) { |alert| process_alert(alert) }
49
+ end
50
+ end
51
+
52
+ def process_alert(alert)
53
+ router.emit(
54
+ (@tag.nil? ? alert['shodan']['alert']['name'] : @tag),
55
+ Fluent::EventTime.parse(alert['timestamp']),
56
+ alert
57
+ )
58
+ end
59
+
60
+ end
61
+ end
@@ -0,0 +1,44 @@
1
+ require 'test/unit'
2
+ require 'fluent/env'
3
+ require 'fluent/test'
4
+ require 'fluent/test/driver/input'
5
+ require 'fluent/test/helpers'
6
+ require "fluent/plugin/in_shodan_alert"
7
+
8
+ class ShodanAlertInputTest < Test::Unit::TestCase
9
+ include Fluent::Test::Helpers
10
+
11
+ API_KEY = ENV['SHODAN_TEST_API_KEY']
12
+
13
+ def setup
14
+ Fluent::Test.setup
15
+ end
16
+
17
+ sub_test_case 'Configuration' do
18
+ test 'is invalid without API key' do
19
+ assert_raise Fluent::ConfigError do
20
+ create_driver(%[])
21
+ end
22
+ end
23
+ test 'is invalid if the API key is invalid' do
24
+ assert_raise Fluent::ConfigError do
25
+ create_driver(%[
26
+ api_key 1234567890AZERTYUIOP
27
+ ])
28
+ end
29
+ end
30
+ end
31
+
32
+ sub_test_case 'Plugin emission' do
33
+ end
34
+
35
+ private
36
+
37
+ CONFIG = %[
38
+ api_key #{API_KEY}
39
+ ]
40
+
41
+ def create_driver(conf = CONFIG)
42
+ Fluent::Test::Driver::Input.new(Fluent::Plugin::ShodanAlert).configure(conf)
43
+ end
44
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-shodan
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - srilumpa
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-01-28 00:00:00.000000000 Z
11
+ date: 2022-02-07 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -110,7 +110,9 @@ files:
110
110
  - AUTHORS
111
111
  - README.md
112
112
  - VERSION
113
+ - lib/fluent/plugin/in_shodan_alert.rb
113
114
  - lib/fluent/plugin/in_shodan_search.rb
115
+ - test/test_in_shodan_alert.rb
114
116
  - test/test_in_shodan_search.rb
115
117
  homepage: https://github.com/srilumpa/fluent-plugin-shodan
116
118
  licenses:
@@ -136,4 +138,5 @@ signing_key:
136
138
  specification_version: 3
137
139
  summary: Fluentd plugin to extract data from Shodan
138
140
  test_files:
141
+ - test/test_in_shodan_alert.rb
139
142
  - test/test_in_shodan_search.rb