logstash-input-cloudflare 0.9.10 → 0.9.11
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +3 -0
- data/README.md +45 -2
- data/lib/logstash/inputs/cloudflare.rb +14 -2
- data/logstash-input-cloudflare.gemspec +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 94f95e1f9294b1290f75f3746bc1388c4eeea9d2
|
4
|
+
data.tar.gz: b9cad63d34e29be682d81ae49029ebec572f6ab7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 94ba03576fa49b97f0b9e760ba3a1ef33d2e465cfc0baae77503d582fc13c2f7ea4891f0f2bfa9012c6afd98abc81b5a5b03249630996d3d1e251d3fb2bc960b
|
7
|
+
data.tar.gz: 3d4883a7ccae03efd7bc64935daf7e24cf8b09cbe18229e63c8111d48b7f8de3ea0abed4ffcdff718b0907df73ae549bb0175747b8944bfd597e241d22175c6b
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -2,10 +2,53 @@
|
|
2
2
|
|
3
3
|
[![Circle CI](https://circleci.com/gh/iserko/logstash-input-cloudflare/tree/master.svg?style=svg&circle-token=78044d92053ebb2ad4ca3b45cdf3cbd271d71ac1)](https://circleci.com/gh/iserko/logstash-input-cloudflare/tree/master)
|
4
4
|
|
5
|
-
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
5
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash) and it allows Logstash to read web request logs from the Cloudflare ELS API. Then Logstash can parse those logs and store them into your store (ElasticSearch if you use the ELK stack).
|
6
|
+
|
7
|
+
The repository provides a sample logstash configuration which you can use and should just work (as long as you fill in `CF_AUTH_EMAIL`, `CF_AUTH_KEY` and `CF_DOMAIN`). Take the [example config file](https://github.com/iserko/logstash-input-cloudflare/blob/master/logstash.conf.m4) and play around.
|
8
|
+
|
9
|
+
Take a look at [Cloudflare Support page for Enterprise Log Share REST API](https://support.cloudflare.com/hc/en-us/articles/216672448-Enterprise-Log-Share-REST-API) for more information about the feature. **You are required to be a Cloudflare's Enterprise customer in order to use this plugin**
|
10
|
+
|
11
|
+
## Configuration
|
12
|
+
|
13
|
+
```
|
14
|
+
input {
|
15
|
+
cloudflare {
|
16
|
+
auth_email => "CF_AUTH_EMAIL"
|
17
|
+
auth_key => "CF_AUTH_KEY"
|
18
|
+
domain => "CF_DOMAIN"
|
19
|
+
type => "cloudflare_logs"
|
20
|
+
poll_time => 15
|
21
|
+
poll_interval => 120
|
22
|
+
metadata_filepath => "/logstash-input-cloudflare/cf_metadata.json"
|
23
|
+
fields => [
|
24
|
+
'timestamp', 'zoneId', 'ownerId', 'zoneName', 'rayId', 'securityLevel',
|
25
|
+
'client.ip', 'client.country', 'client.sslProtocol', 'client.sslCipher',
|
26
|
+
'client.deviceType', 'client.asNum', 'clientRequest.bytes',
|
27
|
+
'clientRequest.httpHost', 'clientRequest.httpMethod', 'clientRequest.uri',
|
28
|
+
'clientRequest.httpProtocol', 'clientRequest.userAgent', 'cache.cacheStatus',
|
29
|
+
'edge.cacheResponseTime', 'edge.startTimestamp', 'edge.endTimestamp',
|
30
|
+
'edgeResponse.status', 'edgeResponse.bytes', 'edgeResponse.bodyBytes',
|
31
|
+
'originResponse.status', 'origin.responseTime'
|
32
|
+
]
|
33
|
+
}
|
34
|
+
}
|
35
|
+
```
|
36
|
+
|
37
|
+
Setting | Description | Default Value | Required
|
38
|
+
------- | ----------- | ------------- | --------
|
39
|
+
auth_email | Email used to login to Cloudflare (suggest creating a new user with only the permissions to access the ELS API | - | true
|
40
|
+
auth_key | API key user to login to Cloudflare | - | true
|
41
|
+
domain | The domain you watch to read logs for (since Cloudflare works on top level domains, that usually means something like `example.com`) | - | true
|
42
|
+
poll_time | The time in seconds between different API calls | 15 | false
|
43
|
+
poll_interval | The time in seconds which determines how many web request logs we pull down from the API (only used when there is no state) | 60 | false
|
44
|
+
start_from_secs_ago | The time in seconds which determines how far back in the past you want to start processing logs from | 1200 | false
|
45
|
+
batch_size | Number of events per API call to get. Helps reduce memory overhead | 1000 | false
|
46
|
+
fields | List of fields you want to process from the API (read the [ELS schema](https://support.cloudflare.com/hc/en-us/article_attachments/205413947/els_schema.json)) | See [fields](https://github.com/iserko/logstash-input-cloudflare/blob/master/lib/logstash/inputs/cloudflare.rb#L54-L60) | false
|
6
47
|
|
7
48
|
## Running in isolation (for testing)
|
8
49
|
|
50
|
+
**You need to be running Docker locally in order to use this!**
|
51
|
+
|
9
52
|
```
|
10
53
|
export CF_AUTH_EMAIL=<email>
|
11
54
|
export CF_AUTH_KEY=<api_key>
|
@@ -14,4 +57,4 @@ make
|
|
14
57
|
```
|
15
58
|
|
16
59
|
Logstash will run in verbose mode, so you will see some messages coming through. In order to verify you're getting results you can open up your browser to http://<IP>:5601 and check Kibana.
|
17
|
-
Value for the IP address is whatever `docker-machine ip default` says.
|
60
|
+
Value for the IP address is whatever `docker-machine ip default` says or if you use Docker For Mac ... it's just 127.0.0.1.
|
@@ -72,6 +72,8 @@ class LogStash::Inputs::Cloudflare < LogStash::Inputs::Base
|
|
72
72
|
validate: :string, default: '/var/lib/logstash/cf_metadata.json', required: false
|
73
73
|
config :poll_time, validate: :number, default: 15, required: false
|
74
74
|
config :poll_interval, validate: :number, default: 60, required: false
|
75
|
+
config :open_timeout, validate: :number, default: 60, required: false
|
76
|
+
config :read_timeout, validate: :number, default: 60, required: false
|
75
77
|
config :start_from_secs_ago, validate: :number, default: 1200, required: false
|
76
78
|
config :batch_size, validate: :number, default: 1000, required: false
|
77
79
|
config :fields, validate: :array, default: DEFAULT_FIELDS, required: false
|
@@ -130,7 +132,10 @@ class LogStash::Inputs::Cloudflare < LogStash::Inputs::Base
|
|
130
132
|
def cloudflare_api_call(endpoint, params, multi_line = false)
|
131
133
|
uri = _build_uri(endpoint, params)
|
132
134
|
@logger.info('Sending request to Cloudflare')
|
133
|
-
Net::HTTP.start(uri.hostname, uri.port,
|
135
|
+
Net::HTTP.start(uri.hostname, uri.port,
|
136
|
+
use_ssl: true,
|
137
|
+
open_timeout: @open_timeout,
|
138
|
+
read_timeout: @read_timeout) do |http|
|
134
139
|
request = Net::HTTP::Get.new(
|
135
140
|
uri.request_uri,
|
136
141
|
'Accept-Encoding' => 'gzip',
|
@@ -144,7 +149,11 @@ class LogStash::Inputs::Cloudflare < LogStash::Inputs::Base
|
|
144
149
|
|
145
150
|
def cloudflare_zone_id(domain)
|
146
151
|
params = { status: 'active', name: domain }
|
147
|
-
|
152
|
+
begin
|
153
|
+
response = cloudflare_api_call('/zones', params)
|
154
|
+
rescue Timeout::Error
|
155
|
+
raise 'Cloudflare API timed out. Consider adjusting the read_timeout.'
|
156
|
+
end
|
148
157
|
response['result'].each do |zone|
|
149
158
|
return zone['id'] if zone['name'] == domain
|
150
159
|
end
|
@@ -208,6 +217,9 @@ class LogStash::Inputs::Cloudflare < LogStash::Inputs::Base
|
|
208
217
|
"#{error['message']}")
|
209
218
|
end
|
210
219
|
entries = []
|
220
|
+
rescue Timeout::Error
|
221
|
+
@logger.error('Cloudflare API timed out. Consider adjusting the read_timeout.')
|
222
|
+
entries = []
|
211
223
|
end
|
212
224
|
return entries unless entries.empty?
|
213
225
|
@logger.info('No entries returned from Cloudflare')
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-input-cloudflare
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.9.
|
4
|
+
version: 0.9.11
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Igor Serko
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-
|
11
|
+
date: 2017-06-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|