logstash-input-cloudflare-logs 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: a5922b918729a34328598df10a8af72fcc863173
4
+ data.tar.gz: 4887ca903d286cbf63a8d91a095ca2ad607d0537
5
+ SHA512:
6
+ metadata.gz: d3abe108bb779c54bbdf157ffb1f1905330160642d13da92cbb1a157242936c1ebd495cb56f95a211a6ef29bb37df84b921d415be298c640a15ff0805574e5fd
7
+ data.tar.gz: 9bcd7472a95ad6eb694d67df0ebde7240a1059ca89cd7d66819c3eb874ffb65d76d534170b4570f06e93a0407acc7516c690d2ad7fde1d98be20cab464e07916
@@ -0,0 +1,2 @@
1
+ ## 0.1.0
2
+ - Plugin created with the logstash plugin generator
@@ -0,0 +1,10 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Josh Moore - joshsmoore@gmail.com
6
+
7
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
8
+ Logstash, and you aren't on the list above and want to be, please let us know
9
+ and we'll make sure you're here. Contributions from folks like you are what make
10
+ open source awesome.
@@ -0,0 +1,2 @@
1
+ # logstash-input-cloudflare-logs
2
+ Example input plugin. This should help bootstrap your effort to write your own input plugin!
data/Gemfile ADDED
@@ -0,0 +1,2 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,11 @@
1
+ Licensed under the Apache License, Version 2.0 (the "License");
2
+ you may not use this file except in compliance with the License.
3
+ You may obtain a copy of the License at
4
+
5
+ http://www.apache.org/licenses/LICENSE-2.0
6
+
7
+ Unless required by applicable law or agreed to in writing, software
8
+ distributed under the License is distributed on an "AS IS" BASIS,
9
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
10
+ See the License for the specific language governing permissions and
11
+ limitations under the License.
@@ -0,0 +1,86 @@
1
+ # Logstash Plugin
2
+
3
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
4
+
5
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
6
+
7
+ ## Documentation
8
+
9
+ Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
10
+
11
+ - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
12
+ - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
13
+
14
+ ## Need Help?
15
+
16
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
17
+
18
+ ## Developing
19
+
20
+ ### 1. Plugin Developement and Testing
21
+
22
+ #### Code
23
+ - To get started, you'll need JRuby with the Bundler gem installed.
24
+
25
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
26
+
27
+ - Install dependencies
28
+ ```sh
29
+ bundle install
30
+ ```
31
+
32
+ #### Test
33
+
34
+ - Update your dependencies
35
+
36
+ ```sh
37
+ bundle install
38
+ ```
39
+
40
+ - Run tests
41
+
42
+ ```sh
43
+ bundle exec rspec
44
+ ```
45
+
46
+ ### 2. Running your unpublished Plugin in Logstash
47
+
48
+ #### 2.1 Run in a local Logstash clone
49
+
50
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
51
+ ```ruby
52
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
53
+ ```
54
+ - Install plugin
55
+ ```sh
56
+ bin/logstash-plugin install --no-verify
57
+ ```
58
+ - Run Logstash with your plugin
59
+ ```sh
60
+ bin/logstash -e 'filter {awesome {}}'
61
+ ```
62
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
63
+
64
+ #### 2.2 Run in an installed Logstash
65
+
66
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
67
+
68
+ - Build your plugin gem
69
+ ```sh
70
+ gem build logstash-filter-awesome.gemspec
71
+ ```
72
+ - Install the plugin from the Logstash home
73
+ ```sh
74
+ bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
75
+ ```
76
+ - Start Logstash and proceed to test the plugin
77
+
78
+ ## Contributing
79
+
80
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
81
+
82
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
83
+
84
+ It is more important to the community that you are able to contribute.
85
+
86
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,109 @@
1
+ # encoding: utf-8
2
+
3
+ require 'logstash/inputs/base'
4
+ require 'logstash/namespace'
5
+ require 'stud/interval'
6
+ require 'socket' # for Socket.gethostname
7
+ require 'rest-client'
8
+ require 'json'
9
+
10
+ class CloudflareAccess
11
+ DEFAULT_FIELDS = %w(CacheCacheStatus CacheResponseBytes CacheResponseStatus ClientASN ClientCountry ClientDeviceType ClientIP ClientIPClass ClientRequestBytes ClientRequestHost ClientRequestMethod ClientRequestProtocol ClientRequestReferer ClientRequestURI ClientRequestUserAgent ClientSSLCipher ClientSSLProtocol ClientSrcPort EdgeColoID EdgeEndTimestamp EdgePathingStatus EdgeResponseBytes EdgeResponseCompressionRatio EdgeResponseStatus EdgeStartTimestamp OriginIP OriginResponseBytes OriginResponseHTTPExpires OriginResponseHTTPLastModified OriginResponseStatus OriginResponseTime RayID WAFAction WAFRuleID ZoneID).freeze
12
+
13
+ attr_accessor :auth_email, :auth_key, :domain, :fields, :metadata_file
14
+ def initialize(arguments)
15
+ @auth_email = arguments[:auth_email]
16
+ @auth_key = arguments[:auth_key]
17
+ @domain = arguments[:domain]
18
+ @fields = arguments[:fields] || DEFAULT_FIELDS
19
+ @metadata_file = arguments[:metadata_file]
20
+ end
21
+
22
+ def start_time
23
+ if File.exist?(metadata_file)
24
+ start_time = Time.parse(JSON.parse(File.read(metadata_file))['start_time']).to_datetime.rfc3339
25
+ else
26
+ start_time = (Time.now - (15 * 60)).to_datetime.rfc3339
27
+ end
28
+ start_time
29
+ end
30
+
31
+ def end_time
32
+ @end_time ||= (Time.now - (5 * 60)).to_datetime.rfc3339
33
+ end
34
+
35
+ def update_metadata_file(key, value)
36
+ key = key.to_s
37
+ meta_data = {}
38
+
39
+ if File.exist?(metadata_file)
40
+ begin
41
+ meta_data = JSON.parse(File.read(metadata_file))
42
+ rescue
43
+ end
44
+ end
45
+
46
+ meta_data[key] = value
47
+
48
+ File.open(metadata_file, 'w+') { |file| file.write(meta_data.to_json) }
49
+ end
50
+
51
+ def logs
52
+ results = RestClient.get("https://api.cloudflare.com/client/v4/zones/#{domain}/logs/received?start=#{start_time}&end=#{end_time}&fields=#{fields.join(',')}", 'X-Auth-Email' => auth_email, 'X-Auth-Key' => auth_key)
53
+ results.body.split("\n").collect { |raw_log| JSON.parse(raw_log) }
54
+ end
55
+ end
56
+
57
+ class LogStash::Inputs::CloudflareLogs < LogStash::Inputs::Base
58
+ config_name 'cloudflare-logs'
59
+
60
+ # If undefined, Logstash will complain, even if codec is unused.
61
+ default :codec, 'json'
62
+
63
+ config :auth_email, validate: :string, required: true
64
+ config :auth_key, validates: :string, required: true
65
+ config :domain_key, validates: :string, required: true
66
+ config :metadata_file, validates: :string, default: '/etc/logstash/cf_metadata.json'
67
+
68
+ config :interval, validate: :number, default: 600
69
+
70
+ def register
71
+ @host = Socket.gethostname
72
+ end # def register
73
+
74
+ def cloudflare_access
75
+ CloudflareAccess.new(auth_key: @auth_key,
76
+ auth_email: @auth_email,
77
+ domain: @domain_key,
78
+ metadata_file: @metadata_file)
79
+ end
80
+
81
+ def process_logs(queue)
82
+ cloudflare_access.logs.each do |log|
83
+ event = LogStash::Event.new(log)
84
+ decorate(event)
85
+ queue << event
86
+ end
87
+ end
88
+
89
+ def run(queue)
90
+ # we can abort the loop if stop? becomes true
91
+ until stop?
92
+ process_logs(queue)
93
+
94
+ # because the sleep interval can be big, when shutdown happens
95
+ # we want to be able to abort the sleep
96
+ # Stud.stoppable_sleep will frequently evaluate the given block
97
+ # and abort the sleep(@interval) if the return value is true
98
+ Stud.stoppable_sleep(@interval) { stop? }
99
+ end # loop
100
+ end # def run
101
+
102
+ def stop
103
+ # nothing to do in this case so it is not necessary to define stop
104
+ # examples of common 'stop' tasks:
105
+ # * close sockets (unblocking blocking reads/accepts)
106
+ # * cleanup temporary files
107
+ # * terminate spawned threads
108
+ end
109
+ end # class LogStash::Inputs::CloudflareLogs
@@ -0,0 +1,41 @@
1
+ Gem::Specification.new do |s|
2
+ s.name = 'logstash-input-cloudflare-logs'
3
+ s.version = '0.1.0'
4
+ s.licenses = ['Apache License (2.0)']
5
+ s.summary = 'write cloudflare logs to logstash'
6
+ s.description = 'Write cloudflare lgos to logstash.' \
7
+ ' This requires an Enterprise account with cloudflare'
8
+ s.homepage = 'https://github.com/resumecompanion/logstash-input-cloudflare-logs'
9
+ s.authors = ['Josh Moore']
10
+ s.email = 'joshsmoore@gmail.com'
11
+ s.require_paths = ['lib']
12
+
13
+ # Files
14
+ s.files = Dir[
15
+ 'lib/**/*',
16
+ 'spec/**/*',
17
+ 'vendor/**/*',
18
+ '*.gemspec',
19
+ '*.md',
20
+ 'CONTRIBUTORS',
21
+ 'Gemfile',
22
+ 'LICENSE',
23
+ 'NOTICE.TXT'
24
+ ]
25
+
26
+ # Tests
27
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
28
+
29
+ # Special flag to let us know this is actually a logstash plugin
30
+ s.metadata = { 'logstash_plugin' => 'true', 'logstash_group' => 'input' }
31
+
32
+ # Gem dependencies
33
+ s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.0'
34
+ s.add_runtime_dependency 'logstash-codec-json'
35
+ s.add_runtime_dependency 'stud', '>= 0.0.22'
36
+ s.add_runtime_dependency 'rest-client', '>= 2.0.2'
37
+ s.add_development_dependency 'logstash-devutils', '>= 0.0.16'
38
+ s.add_development_dependency 'webmock'
39
+ s.add_development_dependency 'vcr', '> 3.0.0', '< 4.0.0'
40
+ s.add_development_dependency 'timecop'
41
+ end
@@ -0,0 +1,57 @@
1
+ ---
2
+ http_interactions:
3
+ - request:
4
+ method: get
5
+ uri: https://api.cloudflare.com/client/v4/zones/11/logs/received?end=2017-10-10T23:55:00Z&fields=CacheCacheStatus,CacheResponseBytes,CacheResponseStatus,ClientASN,ClientCountry,ClientDeviceType,ClientIP,ClientIPClass,ClientRequestBytes,ClientRequestHost,ClientRequestMethod,ClientRequestProtocol,ClientRequestReferer,ClientRequestURI,ClientRequestUserAgent,ClientSSLCipher,ClientSSLProtocol,ClientSrcPort,EdgeColoID,EdgeEndTimestamp,EdgePathingStatus,EdgeResponseBytes,EdgeResponseCompressionRatio,EdgeResponseStatus,EdgeStartTimestamp,OriginIP,OriginResponseBytes,OriginResponseHTTPExpires,OriginResponseHTTPLastModified,OriginResponseStatus,OriginResponseTime,RayID,WAFAction,WAFRuleID,ZoneID&start=2017-10-10T23:54:57Z
6
+ body:
7
+ encoding: US-ASCII
8
+ string: ''
9
+ headers:
10
+ Accept:
11
+ - "*/*"
12
+ Accept-Encoding:
13
+ - gzip, deflate
14
+ User-Agent:
15
+ - rest-client/2.0.2 (darwin x86_64) jruby/9.1.8.0 (2.3.1p0)
16
+ X-Auth-Email:
17
+ - test@test.com
18
+ X-Auth-Key:
19
+ - secret
20
+ Host:
21
+ - api.cloudflare.com
22
+ response:
23
+ status:
24
+ code: 200
25
+ message: OK
26
+ headers:
27
+ Date:
28
+ - Fri, 13 Oct 2017 20:56:45 GMT
29
+ Content-Type:
30
+ - application/json
31
+ Transfer-Encoding:
32
+ - chunked
33
+ Connection:
34
+ - keep-alive
35
+ Set-Cookie:
36
+ - __cfduid=d3a9abc4cc5f499fefc56e6a17d7ef1fe1507928203; expires=Sat, 13-Oct-18
37
+ 20:56:43 GMT; path=/; domain=.cloudflare.com; HttpOnly
38
+ Cf-Version:
39
+ - 2017.9.14
40
+ Content-Encoding:
41
+ - txt
42
+ Vary:
43
+ - Accept-Encoding
44
+ Strict-Transport-Security:
45
+ - max-age=31536000
46
+ Served-In-Seconds:
47
+ - '2.217'
48
+ Server:
49
+ - cloudflare-nginx
50
+ Cf-Ray:
51
+ - 3ad53504de493882-ATL
52
+ body:
53
+ encoding: UTF-8
54
+ string: "{\"CacheCacheStatus\":\"unknown\",\"CacheResponseBytes\":8361,\"CacheResponseStatus\":200,\"ClientASN\":7922,\"ClientCountry\":\"us\",\"ClientDeviceType\":\"desktop\",\"ClientIP\":\"2601:181:c380:7d77:69a8:cc6e:391a:5a23\",\"ClientIPClass\":\"noRecord\",\"ClientRequestBytes\":3155,\"ClientRequestHost\":\"app.resumegenius.com\",\"ClientRequestMethod\":\"GET\",\"ClientRequestProtocol\":\"HTTP/2\",\"ClientRequestReferer\":\"https://app.resumegenius.com/resumes/12709185/edit\",\"ClientRequestURI\":\"/api/v1/templates?template_type=resume\",\"ClientRequestUserAgent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\",\"ClientSSLCipher\":\"ECDHE-ECDSA-AES128-GCM-SHA256\",\"ClientSSLProtocol\":\"TLSv1.2\",\"ClientSrcPort\":54582,\"EdgeColoID\":106,\"EdgeEndTimestamp\":1507679695560000000,\"EdgePathingStatus\":\"nr\",\"EdgeResponseBytes\":8948,\"EdgeResponseCompressionRatio\":0,\"EdgeResponseStatus\":200,\"EdgeStartTimestamp\":1507679695346999808,\"OriginIP\":\"104.237.135.192\",\"OriginResponseBytes\":0,\"OriginResponseHTTPExpires\":\"\",\"OriginResponseHTTPLastModified\":\"Tue, 10 Oct 2017 23:54:55 UTC\",\"OriginResponseStatus\":200,\"OriginResponseTime\":0,\"RayID\":\"3abd81efef015a4a\",\"WAFAction\":\"unknown\",\"WAFRuleID\":\"\",\"ZoneID\":2963680}\n{\"CacheCacheStatus\":\"unknown\",\"CacheResponseBytes\":1483,\"CacheResponseStatus\":200,\"ClientASN\":7922,\"ClientCountry\":\"us\",\"ClientDeviceType\":\"desktop\",\"ClientIP\":\"2601:1c0:cc01:1f6:35de:7ca2:d7b2:8389\",\"ClientIPClass\":\"noRecord\",\"ClientRequestBytes\":2457,\"ClientRequestHost\":\"app.resumegenius.com\",\"ClientRequestMethod\":\"PUT\",\"ClientRequestProtocol\":\"HTTP/2\",\"ClientRequestReferer\":\"https://app.resumegenius.com/letter-v2/1633229/personalize-preview\",\"ClientRequestURI\":\"/api/v1/letters/1633229\",\"ClientRequestUserAgent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8\",\"ClientSSLCipher\":\"ECDHE-ECDSA-AES128-GCM-SHA256\",\"ClientSSLProtocol\":\"TLSv1.2\",\"ClientSrcPort\":52995,\"EdgeColoID\":119,\"EdgeEndTimestamp\":1507679695801000192,\"EdgePathingStatus\":\"nr\",\"EdgeResponseBytes\":1989,\"EdgeResponseCompressionRatio\":0,\"EdgeResponseStatus\":200,\"EdgeStartTimestamp\":1507679695681999872,\"OriginIP\":\"104.237.135.192\",\"OriginResponseBytes\":0,\"OriginResponseHTTPExpires\":\"\",\"OriginResponseHTTPLastModified\":\"Tue, 10 Oct 2017 23:54:55 UTC\",\"OriginResponseStatus\":200,\"OriginResponseTime\":0,\"RayID\":\"3abd81f208cf8d0b\",\"WAFAction\":\"unknown\",\"WAFRuleID\":\"\",\"ZoneID\":2963680}\n{\"CacheCacheStatus\":\"hit\",\"CacheResponseBytes\":3135,\"CacheResponseStatus\":200,\"ClientASN\":7018,\"ClientCountry\":\"us\",\"ClientDeviceType\":\"desktop\",\"ClientIP\":\"12.227.213.34\",\"ClientIPClass\":\"noRecord\",\"ClientRequestBytes\":1374,\"ClientRequestHost\":\"resumegenius.com\",\"ClientRequestMethod\":\"GET\",\"ClientRequestProtocol\":\"HTTP/2\",\"ClientRequestReferer\":\"https://resumegenius.com/\",\"ClientRequestURI\":\"/wp-content/themes/genesis-rg/images/favicons/favicon-192.png\",\"ClientRequestUserAgent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\",\"ClientSSLCipher\":\"ECDHE-ECDSA-AES128-GCM-SHA256\",\"ClientSSLProtocol\":\"TLSv1.2\",\"ClientSrcPort\":56207,\"EdgeColoID\":27,\"EdgeEndTimestamp\":1507679696020999936,\"EdgePathingStatus\":\"nr\",\"EdgeResponseBytes\":3587,\"EdgeResponseCompressionRatio\":0,\"EdgeResponseStatus\":200,\"EdgeStartTimestamp\":1507679696020000000,\"OriginIP\":\"\",\"OriginResponseBytes\":0,\"OriginResponseHTTPExpires\":\"\",\"OriginResponseHTTPLastModified\":\"\",\"OriginResponseStatus\":0,\"OriginResponseTime\":0,\"RayID\":\"3abd81f42f0f8291\",\"WAFAction\":\"unknown\",\"WAFRuleID\":\"\",\"ZoneID\":2963680}\n{\"CacheCacheStatus\":\"hit\",\"CacheResponseBytes\":9752,\"CacheResponseStatus\":200,\"ClientASN\":25640,\"ClientCountry\":\"us\",\"ClientDeviceType\":\"desktop\",\"ClientIP\":\"66.27.46.31\",\"ClientIPClass\":\"noRecord\",\"ClientRequestBytes\":1615,\"ClientRequestHost\":\"resumegenius.com\",\"ClientRequestMethod\":\"GET\",\"ClientRequestProtocol\":\"HTTP/2\",\"ClientRequestReferer\":\"https://resumegenius.com/\",\"ClientRequestURI\":\"/wp-content/uploads/2015/02/Dublin-Green-Template-e1437465375594-225x291.jpg\",\"ClientRequestUserAgent\":\"Mozilla/5.0 (X11; CrOS x86_64 9693.1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3144.0 Safari/537.36\",\"ClientSSLCipher\":\"ECDHE-ECDSA-CHACHA20-POLY1305\",\"ClientSSLProtocol\":\"TLSv1.2\",\"ClientSrcPort\":52826,\"EdgeColoID\":12,\"EdgeEndTimestamp\":1507679696856000000,\"EdgePathingStatus\":\"nr\",\"EdgeResponseBytes\":10264,\"EdgeResponseCompressionRatio\":0,\"EdgeResponseStatus\":200,\"EdgeStartTimestamp\":1507679696852000000,\"OriginIP\":\"\",\"OriginResponseBytes\":0,\"OriginResponseHTTPExpires\":\"\",\"OriginResponseHTTPLastModified\":\"\",\"OriginResponseStatus\":0,\"OriginResponseTime\":0,\"RayID\":\"3abd81f95caf7844\",\"WAFAction\":\"unknown\",\"WAFRuleID\":\"\",\"ZoneID\":2963680}"
55
+ http_version:
56
+ recorded_at: Fri, 13 Oct 2017 20:56:46 GMT
57
+ recorded_with: VCR 3.0.3
@@ -0,0 +1,40 @@
1
+ # encoding: utf-8
2
+
3
+ require "#{File.expand_path(File.dirname(__FILE__))}/../spec_helper"
4
+ require 'logstash/inputs/cloudflare-logs'
5
+
6
+ describe LogStash::Inputs::CloudflareLogs do
7
+ it_behaves_like 'an interruptible input plugin' do
8
+ let(:config) { { 'interval' => 100, 'auth_email' => 'test@test.com', 'auth_key' => 'test', 'domain_key' => 'asdf' } }
9
+ end
10
+
11
+ describe '#process_logs' do
12
+ let(:config) { { 'interval' => 100, 'auth_email' => 'test@test.com', 'auth_key' => 'test', 'domain_key' => 'asdf' } }
13
+ let(:queue) { double(:queue, :<< => 'test') }
14
+ before(:each) { allow_any_instance_of(CloudflareAccess).to receive(:logs).and_return([{ a: 1 }]) }
15
+ let(:plugin) { described_class.new(config) }
16
+ subject { plugin.process_logs(queue) }
17
+
18
+ it 'should create each log event' do
19
+ expect(LogStash::Event).to receive(:new).with(a: 1)
20
+ subject
21
+ end
22
+
23
+ it 'should create each log event' do
24
+ expect(LogStash::Event).to receive(:new).with(a: 1)
25
+ subject
26
+ end
27
+
28
+ it 'should decorate the event' do
29
+ expect(plugin).to receive(:decorate).with(instance_of(LogStash::Event))
30
+
31
+ subject
32
+ end
33
+
34
+ it 'should add the event to the queue' do
35
+ expect(queue).to receive(:<<).with(instance_of(LogStash::Event))
36
+
37
+ subject
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,96 @@
1
+ # encoding: utf-8
2
+ require 'json'
3
+ require "#{File.expand_path(File.dirname(__FILE__))}/../spec_helper"
4
+ require 'logstash/inputs/cloudflare-logs'
5
+
6
+ describe CloudflareAccess do
7
+ let(:meta_filename) { "#{File.expand_path(File.dirname(__FILE__))}/../tmp/metadata.json" }
8
+ let(:cloudflare_access) do
9
+ CloudflareAccess.new(auth_email: 'test@test.com',
10
+ auth_key: 'secret',
11
+ domain: '11',
12
+ metadata_file: meta_filename)
13
+ end
14
+
15
+ describe '#start_time' do
16
+ subject { cloudflare_access.start_time }
17
+ it 'should return current time - 15 min if metadata file is not there' do
18
+ `rm #{meta_filename}`
19
+ Timecop.freeze do
20
+ expect(subject).to eq((Time.now - (15 * 60)).to_datetime.rfc3339)
21
+ end
22
+ end
23
+
24
+ it 'should return the start time in the file if it is correct' do
25
+ Timecop.freeze do
26
+ File.open(meta_filename, 'w+') { |file| file.write({ start_time: Time.now - (24 * 60 * 60) }.to_json) }
27
+ expect(subject).to eq((Time.now - (24 * 60 * 60)).to_datetime.rfc3339)
28
+ end
29
+ end
30
+ end
31
+
32
+ describe '#end_time' do
33
+ subject { cloudflare_access.end_time }
34
+
35
+ it 'should return 5 min before current time' do
36
+ Timecop.freeze do
37
+ expect(subject).to eq((Time.now - (5 * 60)).to_datetime.rfc3339)
38
+ end
39
+ end
40
+ end
41
+
42
+ describe '#update_metadata_file' do
43
+ def read_meta_data
44
+ Time.parse(JSON.parse(File.read(meta_filename))['start_time']).to_datetime.rfc3339
45
+ end
46
+
47
+ before(:all) { Timecop.freeze }
48
+ after(:all) { Timecop.freeze }
49
+ subject { cloudflare_access.update_metadata_file(:start_time, Time.now) }
50
+
51
+ it 'should set time if the metadata file does not exist' do
52
+ `rm #{meta_filename}`
53
+
54
+ subject
55
+
56
+ expect(read_meta_data).to eq Time.now.to_datetime.rfc3339
57
+ end
58
+
59
+ it 'should set the value of the meta data if the file exists' do
60
+ subject
61
+ @value = Time.now - (14 * 60)
62
+ cloudflare_access.update_metadata_file(:start_time, @value)
63
+
64
+ expect(read_meta_data).to eq((Time.now - (14 * 60)).to_datetime.rfc3339)
65
+ end
66
+
67
+ it 'should work correctly if the meta data file is there and blank' do
68
+ `rm #{meta_filename}`
69
+ `touch #{meta_filename}`
70
+ subject
71
+
72
+ expect(read_meta_data).to eq Time.now.to_datetime.rfc3339
73
+ end
74
+ end
75
+
76
+ describe '#logs' do
77
+ before :each do
78
+ allow(cloudflare_access).to receive(:start_time).and_return('2017-10-10T23:54:57Z')
79
+ allow(cloudflare_access).to receive(:end_time).and_return('2017-10-10T23:55:00Z')
80
+ end
81
+
82
+ subject do
83
+ VCR.use_cassette(:logs) do
84
+ cloudflare_access.logs
85
+ end
86
+ end
87
+
88
+ it 'it should return the 4 log hashes' do
89
+ expect(subject.count).to eq(4)
90
+ end
91
+
92
+ it 'should return each item as a hash' do
93
+ expect(subject.first).to be_instance_of(Hash)
94
+ end
95
+ end
96
+ end
@@ -0,0 +1,62 @@
1
+ # require 'webmock/rspec'
2
+ require 'vcr'
3
+ require 'timecop'
4
+
5
+ VCR.configure do |config|
6
+ config.before_record do |i|
7
+ i.response.body.force_encoding('UTF-8')
8
+ end
9
+ config.cassette_serializers[:json]
10
+ config.cassette_library_dir = 'spec/fixtures/vcr_cassettes'
11
+ config.hook_into :webmock
12
+ end
13
+
14
+ if ENV['COVERAGE']
15
+ require 'simplecov'
16
+ require 'coveralls'
17
+
18
+ SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
19
+ SimpleCov::Formatter::HTMLFormatter,
20
+ Coveralls::SimpleCov::Formatter
21
+ ]
22
+ SimpleCov.start do
23
+ add_filter 'spec/'
24
+ add_filter 'vendor/'
25
+ end
26
+ end
27
+
28
+ require 'logstash-core'
29
+ require 'logstash/logging'
30
+ require 'logstash/environment'
31
+ require 'logstash/devutils/rspec/logstash_helpers'
32
+ require 'logstash/devutils/rspec/shared_examples'
33
+ require 'insist'
34
+
35
+ Thread.abort_on_exception = true
36
+
37
+ RSpec.configure do |config|
38
+ # for now both include and extend are required because the newly refactored
39
+ # 'input' helper method need to be visible in a 'it' block
40
+ # and this is only possible by calling include on LogStashHelper
41
+ config.include LogStashHelper
42
+ config.extend LogStashHelper
43
+
44
+ exclude_tags = {
45
+ redis: true,
46
+ socket: true,
47
+ performance: true,
48
+ couchdb: true,
49
+ elasticsearch: true,
50
+ elasticsearch_secure: true,
51
+ export_cypher: true,
52
+ integration: true
53
+ }
54
+
55
+ config.filter_run_excluding exclude_tags
56
+
57
+ # Run specs in random order to surface order dependencies. If you find an
58
+ # order dependency and want to debug it, you can fix the order by providing
59
+ # the seed, which is printed after each run.
60
+ # --seed 1234
61
+ config.order = :random
62
+ end
@@ -0,0 +1 @@
1
+ {"start_time":"2017-10-16 11:52:53 -0400"}
metadata ADDED
@@ -0,0 +1,182 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-input-cloudflare-logs
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Josh Moore
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2017-10-17 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - "~>"
17
+ - !ruby/object:Gem::Version
18
+ version: '2.0'
19
+ name: logstash-core-plugin-api
20
+ prerelease: false
21
+ type: :runtime
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '2.0'
27
+ - !ruby/object:Gem::Dependency
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - ">="
31
+ - !ruby/object:Gem::Version
32
+ version: '0'
33
+ name: logstash-codec-json
34
+ prerelease: false
35
+ type: :runtime
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - ">="
45
+ - !ruby/object:Gem::Version
46
+ version: 0.0.22
47
+ name: stud
48
+ prerelease: false
49
+ type: :runtime
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: 0.0.22
55
+ - !ruby/object:Gem::Dependency
56
+ requirement: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - ">="
59
+ - !ruby/object:Gem::Version
60
+ version: 2.0.2
61
+ name: rest-client
62
+ prerelease: false
63
+ type: :runtime
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: 2.0.2
69
+ - !ruby/object:Gem::Dependency
70
+ requirement: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - ">="
73
+ - !ruby/object:Gem::Version
74
+ version: 0.0.16
75
+ name: logstash-devutils
76
+ prerelease: false
77
+ type: :development
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - ">="
81
+ - !ruby/object:Gem::Version
82
+ version: 0.0.16
83
+ - !ruby/object:Gem::Dependency
84
+ requirement: !ruby/object:Gem::Requirement
85
+ requirements:
86
+ - - ">="
87
+ - !ruby/object:Gem::Version
88
+ version: '0'
89
+ name: webmock
90
+ prerelease: false
91
+ type: :development
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - ">="
95
+ - !ruby/object:Gem::Version
96
+ version: '0'
97
+ - !ruby/object:Gem::Dependency
98
+ requirement: !ruby/object:Gem::Requirement
99
+ requirements:
100
+ - - ">"
101
+ - !ruby/object:Gem::Version
102
+ version: 3.0.0
103
+ - - "<"
104
+ - !ruby/object:Gem::Version
105
+ version: 4.0.0
106
+ name: vcr
107
+ prerelease: false
108
+ type: :development
109
+ version_requirements: !ruby/object:Gem::Requirement
110
+ requirements:
111
+ - - ">"
112
+ - !ruby/object:Gem::Version
113
+ version: 3.0.0
114
+ - - "<"
115
+ - !ruby/object:Gem::Version
116
+ version: 4.0.0
117
+ - !ruby/object:Gem::Dependency
118
+ requirement: !ruby/object:Gem::Requirement
119
+ requirements:
120
+ - - ">="
121
+ - !ruby/object:Gem::Version
122
+ version: '0'
123
+ name: timecop
124
+ prerelease: false
125
+ type: :development
126
+ version_requirements: !ruby/object:Gem::Requirement
127
+ requirements:
128
+ - - ">="
129
+ - !ruby/object:Gem::Version
130
+ version: '0'
131
+ description: Write cloudflare lgos to logstash. This requires an Enterprise account
132
+ with cloudflare
133
+ email: joshsmoore@gmail.com
134
+ executables: []
135
+ extensions: []
136
+ extra_rdoc_files: []
137
+ files:
138
+ - CHANGELOG.md
139
+ - CONTRIBUTORS
140
+ - DEVELOPER.md
141
+ - Gemfile
142
+ - LICENSE
143
+ - README.md
144
+ - lib/logstash/inputs/cloudflare-logs.rb
145
+ - logstash-input-cloudflare-logs.gemspec
146
+ - spec/fixtures/vcr_cassettes/logs.yml
147
+ - spec/inputs/cloudflare-logs_spec.rb
148
+ - spec/inputs/cloudflare_access_spec.rb
149
+ - spec/spec_helper.rb
150
+ - spec/tmp/metadata.json
151
+ homepage: https://github.com/resumecompanion/logstash-input-cloudflare-logs
152
+ licenses:
153
+ - Apache License (2.0)
154
+ metadata:
155
+ logstash_plugin: 'true'
156
+ logstash_group: input
157
+ post_install_message:
158
+ rdoc_options: []
159
+ require_paths:
160
+ - lib
161
+ required_ruby_version: !ruby/object:Gem::Requirement
162
+ requirements:
163
+ - - ">="
164
+ - !ruby/object:Gem::Version
165
+ version: '0'
166
+ required_rubygems_version: !ruby/object:Gem::Requirement
167
+ requirements:
168
+ - - ">="
169
+ - !ruby/object:Gem::Version
170
+ version: '0'
171
+ requirements: []
172
+ rubyforge_project:
173
+ rubygems_version: 2.6.8
174
+ signing_key:
175
+ specification_version: 4
176
+ summary: write cloudflare logs to logstash
177
+ test_files:
178
+ - spec/fixtures/vcr_cassettes/logs.yml
179
+ - spec/inputs/cloudflare-logs_spec.rb
180
+ - spec/inputs/cloudflare_access_spec.rb
181
+ - spec/spec_helper.rb
182
+ - spec/tmp/metadata.json