logstash-input-azure_blob_storage 0.11.1 → 0.11.6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +46 -12
- data/README.md +90 -21
- data/lib/logstash/inputs/azure_blob_storage.rb +257 -101
- data/logstash-input-azure_blob_storage.gemspec +3 -3
- metadata +6 -26
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ececd96b04d2cab60eca54a0fe2a98c9ed093da2227e3568d4feea09264912fa
|
|
4
|
+
data.tar.gz: 7bcd39bc38d26a05da1275e5fb2317e41b5c2cddc6541535c7d166a69bb3cf62
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1bcbfab30de973e9eafee295221dc816411dca0e0f747a01c62bb48ec5c46eaf4db4162fdd5283611cd79da59910daab9e7c6e234df47f5ce7f320e65f7b8c69
|
|
7
|
+
data.tar.gz: 7bbdab8694d024b9c08cc89e13bc86aa8b90a536f5615565333593e0da7c3073d7c4cf3ad3f2b4005a90541de9693a93826158a18fbf9015234bee1812b3d46c
|
data/CHANGELOG.md
CHANGED
|
@@ -1,29 +1,63 @@
|
|
|
1
|
+
## 0.11.6
|
|
2
|
+
- fix in json head and tail learning the max_results
|
|
3
|
+
- broke out connection setup in order to call it again if connection exceptions come
|
|
4
|
+
- deal better with skipping of empty files.
|
|
5
|
+
|
|
6
|
+
## 0.11.5
|
|
7
|
+
- added optional addfilename to add filename in message
|
|
8
|
+
- NSGFLOWLOG version 2 uses 0 as value instead of NULL in src and dst values
|
|
9
|
+
- added connection exception handling when full_read files
|
|
10
|
+
- rewritten json header footer learning to ignore learning from registry
|
|
11
|
+
- plumbing for emulator
|
|
12
|
+
|
|
13
|
+
## 0.11.4
|
|
14
|
+
- fixed listing 3 times, rather than retrying to list max 3 times
|
|
15
|
+
- added option to migrate/save to using local registry
|
|
16
|
+
- rewrote interval timing
|
|
17
|
+
- reduced saving of registry to maximum once per interval, protect duplicate simultanious writes
|
|
18
|
+
- added debug_timer for better tracing how long operations take
|
|
19
|
+
- removing pipeline name from logfiles, logstash 7.6 and up have this in the log4j2 by default now
|
|
20
|
+
- moved initialization from register to run. should make logs more readable
|
|
21
|
+
|
|
22
|
+
## 0.11.3
|
|
23
|
+
- don't crash on failed codec, e.g. gzip_lines could sometimes have a corrupted file?
|
|
24
|
+
- fix nextmarker loop so that more than 5000 files (or 15000 if faraday doesn't crash)
|
|
25
|
+
|
|
26
|
+
## 0.11.2
|
|
27
|
+
- implemented path_filters to to use path filtering like this **/*.log
|
|
28
|
+
- implemented debug_until to debug only at the start of a pipeline until it processed enough messages
|
|
29
|
+
|
|
30
|
+
## 0.11.1
|
|
31
|
+
- copied changes from irnc fork (danke!)
|
|
32
|
+
- fixed trying to load the registry, three time is the charm
|
|
33
|
+
- logs are less chatty, changed info to debug
|
|
34
|
+
|
|
1
35
|
## 0.11.0
|
|
2
|
-
-
|
|
3
|
-
-
|
|
4
|
-
-
|
|
36
|
+
- implemented start_fresh to skip all previous logs and start monitoring new entries
|
|
37
|
+
- fixed the timer, now properly sleep the interval and check again
|
|
38
|
+
- work around for a Faraday Middleware v.s. Azure Storage Account bug in follow_redirect
|
|
5
39
|
|
|
6
40
|
## 0.10.6
|
|
7
|
-
-
|
|
41
|
+
- fixed the rootcause of the checking the codec. Now compare the classname.
|
|
8
42
|
|
|
9
43
|
## 0.10.5
|
|
10
|
-
-
|
|
44
|
+
- previous fix broke codec = "line"
|
|
11
45
|
|
|
12
46
|
## 0.10.4
|
|
13
|
-
-
|
|
47
|
+
- fixed JSON parsing error for partial files because somehow (logstash 7?) @codec.is_a? doesn't work anymore
|
|
14
48
|
|
|
15
49
|
## 0.10.3
|
|
16
|
-
-
|
|
50
|
+
- fixed issue-1 where iplookup confguration was removed, but still used
|
|
17
51
|
- iplookup is now done by a separate plugin named logstash-filter-weblookup
|
|
18
52
|
|
|
19
53
|
## 0.10.2
|
|
20
54
|
- moved iplookup to own plugin logstash-filter-lookup
|
|
21
55
|
|
|
22
56
|
## 0.10.1
|
|
23
|
-
-
|
|
24
|
-
-
|
|
25
|
-
-
|
|
57
|
+
- implemented iplookup
|
|
58
|
+
- fixed sas tokens (maybe)
|
|
59
|
+
- introduced dns_suffix
|
|
26
60
|
|
|
27
61
|
## 0.10.0
|
|
28
|
-
-
|
|
29
|
-
-
|
|
62
|
+
- plugin created with the logstash plugin generator
|
|
63
|
+
- reimplemented logstash-input-azureblob with incompatible config and data/registry
|
data/README.md
CHANGED
|
@@ -1,29 +1,81 @@
|
|
|
1
|
-
# Logstash
|
|
1
|
+
# Logstash
|
|
2
2
|
|
|
3
|
-
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
|
3
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash). It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way. All logstash plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/). Need generic logstash help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
## Documentation
|
|
8
|
-
|
|
9
|
-
All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
|
|
10
|
-
|
|
11
|
-
## Need Help?
|
|
12
|
-
|
|
13
|
-
Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum. For real problems or feature requests, raise a github issue [GITHUB/janmg/logstash-input-azure_blob_storage/](https://github.com/janmg/logstash-input-azure_blob_storage). Pull requests will ionly be merged after discussion through an issue.
|
|
5
|
+
For problems or feature requests with this specific plugin, raise a github issue [GITHUB/janmg/logstash-input-azure_blob_storage/](https://github.com/janmg/logstash-input-azure_blob_storage). Pull requests will also be welcomed after discussion through an issue.
|
|
14
6
|
|
|
15
7
|
## Purpose
|
|
16
|
-
This plugin can read from Azure Storage Blobs, for instance diagnostics logs for NSG flow logs or accesslogs from App Services.
|
|
8
|
+
This plugin can read from Azure Storage Blobs, for instance JSON diagnostics logs for NSG flow logs or LINE based accesslogs from App Services.
|
|
17
9
|
[Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
|
|
18
10
|
|
|
19
|
-
|
|
11
|
+
The plugin depends on the [Ruby library azure-storage-blon](https://rubygems.org/gems/azure-storage-blob/versions/1.1.0) from Microsoft, that depends on Faraday for the HTTPS connection to Azure.
|
|
12
|
+
|
|
13
|
+
The plugin executes the following steps
|
|
14
|
+
1. Lists all the files in the azure storage account. where the path of the files are matching pathprefix
|
|
15
|
+
2. Filters on path_filters to only include files that match the directory and file glob (e.g. **/*.json)
|
|
16
|
+
3. Save the listed files in a registry of known files and filesizes. (data/registry.dat on azure, or in a file on the logstash instance)
|
|
17
|
+
4. List all the files again and compare the registry with the new filelist and put the delta in a worklist
|
|
18
|
+
5. Process the worklist and put all events in the logstash queue.
|
|
19
|
+
6. if there is time left, sleep to complete the interval. If processing takes more than an inteval, save the registry and continue processing.
|
|
20
|
+
7. If logstash is stopped, a stop signal will try to finish the current file, save the registry and than quit
|
|
20
21
|
|
|
21
22
|
## Installation
|
|
22
23
|
This plugin can be installed through logstash-plugin
|
|
23
24
|
```
|
|
24
|
-
logstash-plugin install logstash-input-azure_blob_storage
|
|
25
|
+
/usr/share/logstash/bin/logstash-plugin install logstash-input-azure_blob_storage
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Minimal Configuration
|
|
29
|
+
The minimum configuration required as input is storageaccount, access_key and container.
|
|
30
|
+
|
|
31
|
+
/etc/logstash/conf.d/test.conf
|
|
32
|
+
```
|
|
33
|
+
input {
|
|
34
|
+
azure_blob_storage {
|
|
35
|
+
storageaccount => "yourstorageaccountname"
|
|
36
|
+
access_key => "Ba5e64c0d3=="
|
|
37
|
+
container => "insights-logs-networksecuritygroupflowevent"
|
|
38
|
+
}
|
|
39
|
+
}
|
|
25
40
|
```
|
|
26
41
|
|
|
42
|
+
## Additional Configuration
|
|
43
|
+
The registry keeps track of files in the storage account, their size and how many bytes have been processed. Files can grow and the added part will be processed as a partial file. The registry is saved todisk every interval.
|
|
44
|
+
|
|
45
|
+
The registry_create_policy determines at the start of the pipeline if processing should resume from the last known unprocessed file, or to start_fresh ignoring old files and start only processing new events that came after the start of the pipeline. Or start_over to process all the files ignoring the registry.
|
|
46
|
+
|
|
47
|
+
interval defines the minimum time the registry should be saved to the registry file (by default to 'data/registry.dat'), this is only needed in case the pipeline dies unexpectedly. During a normal shutdown the registry is also saved.
|
|
48
|
+
|
|
49
|
+
When registry_local_path is set to a directory, the registry is saved on the logstash server in that directory. The filename is the pipe.id
|
|
50
|
+
|
|
51
|
+
with registry_create_policy set to resume and the registry_local_path set to a directory where the registry isn't yet created, should load the registry from the storage account and save the registry on the local server. This allows for a migration to localstorage
|
|
52
|
+
|
|
53
|
+
For pipelines that use the JSON codec or the JSON_LINE codec, the plugin uses one file to learn how the JSON header and tail look like, they can also be configured manually. Using skip_learning the learning can be disabled.
|
|
54
|
+
|
|
55
|
+
## Running the pipeline
|
|
56
|
+
The pipeline can be started in several ways.
|
|
57
|
+
- On the commandline
|
|
58
|
+
```
|
|
59
|
+
/usr/share/logstash/bin/logtash -f /etc/logstash/conf.d/test.conf
|
|
60
|
+
```
|
|
61
|
+
- In the pipeline.yml
|
|
62
|
+
```
|
|
63
|
+
/etc/logstash/pipeline.yml
|
|
64
|
+
pipe.id = test
|
|
65
|
+
pipe.path = /etc/logstash/conf.d/test.conf
|
|
66
|
+
```
|
|
67
|
+
- As managed pipeline from Kibana
|
|
68
|
+
|
|
69
|
+
Logstash itself (so not specific to this plugin) has a feature where multiple instances can run on the same system. The default TCP port is 9600, but if it's already in use it will use 9601 (and up). To update a config file on a running instance on the commandline you can add the argument --config.reload.automatic and if you modify the files that are in the pipeline.yml you can send a SIGHUP channel to reload the pipelines where the config was changed.
|
|
70
|
+
[https://www.elastic.co/guide/en/logstash/current/reloading-config.html](https://www.elastic.co/guide/en/logstash/current/reloading-config.html)
|
|
71
|
+
|
|
72
|
+
## Internal Working
|
|
73
|
+
When the plugin is started, it will read all the filenames and sizes in the blob store excluding the directies of files that are excluded by the "path_filters". After every interval it will write a registry to the storageaccount to save the information of how many bytes per blob (file) are read and processed. After all files are processed and at least one interval has passed a new file list is generated and a worklist is constructed that will be processed. When a file has already been processed before, partial files are read from the offset to the filesize at the time of the file listing. If the codec is JSON partial files will be have the header and tail will be added. They can be configured. If logtype is nsgflowlog, the plugin will process the splitting into individual tuple events. The logtype wadiis may in the future be used to process the grok formats to split into log lines. Any other format is fed into the queue as one event per file or partial file. It's then up to the filter to split and mutate the file format.
|
|
74
|
+
|
|
75
|
+
By default the root of the json message is named "message" so you can modify the content in the filter block
|
|
76
|
+
|
|
77
|
+
The configurations and the rest of the code are in [https://github.com/janmg/logstash-input-azure_blob_storage/tree/master/lib/logstash/inputs](lib/logstash/inputs) [https://github.com/janmg/logstash-input-azure_blob_storage/blob/master/lib/logstash/inputs/azure_blob_storage.rb#L10](azure_blob_storage.rb)
|
|
78
|
+
|
|
27
79
|
## Enabling NSG Flowlogs
|
|
28
80
|
1. Enable Network Watcher in your regions
|
|
29
81
|
2. Create Storage account per region
|
|
@@ -39,7 +91,6 @@ logstash-plugin install logstash-input-azure_blob_storage
|
|
|
39
91
|
- Access key (key1 or key2)
|
|
40
92
|
|
|
41
93
|
## Troubleshooting
|
|
42
|
-
|
|
43
94
|
The default loglevel can be changed in global logstash.yml. On the info level, the plugin save offsets to the registry every interval and will log statistics of processed events (one ) plugin will print for each pipeline the first 6 characters of the ID, in DEBUG the yml log level debug shows details of number of events per (partial) files that are read.
|
|
44
95
|
```
|
|
45
96
|
log.level
|
|
@@ -50,10 +101,11 @@ The log level of the plugin can be put into DEBUG through
|
|
|
50
101
|
curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'{"logger.logstash.inputs.azureblobstorage" : "DEBUG"}'
|
|
51
102
|
```
|
|
52
103
|
|
|
104
|
+
Because logstash debug makes logstash very chatty, the option debug_until will for a number of processed events and stops debuging. One file can easily contain thousands of events. The debug_until is useful to monitor the start of the plugin and the processing of the first files.
|
|
53
105
|
|
|
54
|
-
|
|
55
|
-
The minimum configuration required as input is storageaccount, access_key and container.
|
|
106
|
+
debug_timer will show detailed information on how much time listing of files took and how long the plugin will sleep to fill the interval and the listing and processing starts again.
|
|
56
107
|
|
|
108
|
+
## Other Configuration Examples
|
|
57
109
|
For nsgflowlogs, a simple configuration looks like this
|
|
58
110
|
```
|
|
59
111
|
input {
|
|
@@ -77,6 +129,10 @@ filter {
|
|
|
77
129
|
}
|
|
78
130
|
}
|
|
79
131
|
|
|
132
|
+
output {
|
|
133
|
+
stdout { }
|
|
134
|
+
}
|
|
135
|
+
|
|
80
136
|
output {
|
|
81
137
|
elasticsearch {
|
|
82
138
|
hosts => "elasticsearch"
|
|
@@ -84,22 +140,35 @@ output {
|
|
|
84
140
|
}
|
|
85
141
|
}
|
|
86
142
|
```
|
|
87
|
-
|
|
88
|
-
It's possible to specify the optional parameters to overwrite the defaults. The iplookup, use_redis and iplist parameters are used for additional information about the source and destination ip address. Redis can be used for caching the results and iplist is to configure an array of ip addresses.
|
|
143
|
+
A more elaborate input configuration example
|
|
89
144
|
```
|
|
90
145
|
input {
|
|
91
146
|
azure_blob_storage {
|
|
147
|
+
codec => "json"
|
|
92
148
|
storageaccount => "yourstorageaccountname"
|
|
93
149
|
access_key => "Ba5e64c0d3=="
|
|
94
150
|
container => "insights-logs-networksecuritygroupflowevent"
|
|
95
|
-
codec => "json"
|
|
96
151
|
logtype => "nsgflowlog"
|
|
97
152
|
prefix => "resourceId=/"
|
|
153
|
+
path_filters => ['**/*.json']
|
|
154
|
+
addfilename => true
|
|
98
155
|
registry_create_policy => "resume"
|
|
156
|
+
registry_local_path => "/usr/share/logstash/plugin"
|
|
99
157
|
interval => 300
|
|
158
|
+
debug_timer => true
|
|
159
|
+
debug_until => 100
|
|
160
|
+
}
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
output {
|
|
164
|
+
elasticsearch {
|
|
165
|
+
hosts => "elasticsearch"
|
|
166
|
+
index => "nsg-flow-logs-%{+xxxx.ww}"
|
|
100
167
|
}
|
|
101
168
|
}
|
|
102
169
|
```
|
|
170
|
+
The configuration documentation is in the first 100 lines of the code
|
|
171
|
+
[GITHUB/janmg/logstash-input-azure_blob_storage/blob/master/lib/logstash/inputs/azure_blob_storage.rb](https://github.com/janmg/logstash-input-azure_blob_storage/blob/master/lib/logstash/inputs/azure_blob_storage.rb)
|
|
103
172
|
|
|
104
173
|
For WAD IIS and App Services the HTTP AccessLogs can be retrieved from a storage account as line based events and parsed through GROK. The date stamp can also be parsed with %{TIMESTAMP_ISO8601:log_timestamp}. For WAD IIS logfiles the container is wad-iis-logfiles. In the future grokking may happen already by the plugin.
|
|
105
174
|
```
|
|
@@ -138,7 +207,7 @@ filter {
|
|
|
138
207
|
remove_field => ["subresponse"]
|
|
139
208
|
remove_field => ["username"]
|
|
140
209
|
remove_field => ["clientPort"]
|
|
141
|
-
remove_field => ["port"]
|
|
210
|
+
remove_field => ["port"]:0
|
|
142
211
|
remove_field => ["timestamp"]
|
|
143
212
|
}
|
|
144
213
|
}
|
|
@@ -25,6 +25,9 @@ config :storageaccount, :validate => :string, :required => false
|
|
|
25
25
|
# DNS Suffix other then blob.core.windows.net
|
|
26
26
|
config :dns_suffix, :validate => :string, :required => false, :default => 'core.windows.net'
|
|
27
27
|
|
|
28
|
+
# For development this can be used to emulate an accountstorage when not available from azure
|
|
29
|
+
#config :use_development_storage, :validate => :boolean, :required => false
|
|
30
|
+
|
|
28
31
|
# The (primary or secondary) Access Key for the the storage account. The key can be found in the portal.azure.com or through the azure api StorageAccounts/ListKeys. For example the PowerShell command Get-AzStorageAccountKey.
|
|
29
32
|
config :access_key, :validate => :password, :required => false
|
|
30
33
|
|
|
@@ -39,6 +42,9 @@ config :container, :validate => :string, :default => 'insights-logs-networksecur
|
|
|
39
42
|
# The default, `data/registry`, it contains a Ruby Marshal Serialized Hash of the filename the offset read sofar and the filelength the list time a filelisting was done.
|
|
40
43
|
config :registry_path, :validate => :string, :required => false, :default => 'data/registry.dat'
|
|
41
44
|
|
|
45
|
+
# If registry_local_path is set to a directory on the local server, the registry is save there instead of the remote blob_storage
|
|
46
|
+
config :registry_local_path, :validate => :string, :required => false
|
|
47
|
+
|
|
42
48
|
# The default, `resume`, will load the registry offsets and will start processing files from the offsets.
|
|
43
49
|
# When set to `start_over`, all log files are processed from begining.
|
|
44
50
|
# when set to `start_fresh`, it will read log files that are created or appended since this start of the pipeline.
|
|
@@ -55,9 +61,21 @@ config :registry_create_policy, :validate => ['resume','start_over','start_fresh
|
|
|
55
61
|
# Z00000000000000000000000000000000 2 ]}
|
|
56
62
|
config :interval, :validate => :number, :default => 60
|
|
57
63
|
|
|
64
|
+
# add the filename into the events
|
|
65
|
+
config :addfilename, :validate => :boolean, :default => false, :required => false
|
|
66
|
+
|
|
67
|
+
# debug_until will for a maximum amount of processed messages shows 3 types of log printouts including processed filenames. This is a lightweight alternative to switching the loglevel from info to debug or even trace
|
|
68
|
+
config :debug_until, :validate => :number, :default => 0, :required => false
|
|
69
|
+
|
|
70
|
+
# debug_timer show time spent on activities
|
|
71
|
+
config :debug_timer, :validate => :boolean, :default => false, :required => false
|
|
72
|
+
|
|
58
73
|
# WAD IIS Grok Pattern
|
|
59
74
|
#config :grokpattern, :validate => :string, :required => false, :default => '%{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:instanceId} %{NOTSPACE:instanceId2} %{IPORHOST:ServerIP} %{WORD:httpMethod} %{URIPATH:requestUri} %{NOTSPACE:requestQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:httpVersion} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:host} %{NUMBER:httpStatus} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:sentBytes:int} %{NUMBER:receivedBytes:int} %{NUMBER:timeTaken:int}'
|
|
60
75
|
|
|
76
|
+
# skip learning if you use json and don't want to learn the head and tail, but use either the defaults or configure them.
|
|
77
|
+
config :skip_learning, :validate => :boolean, :default => false, :required => false
|
|
78
|
+
|
|
61
79
|
# The string that starts the JSON. Only needed when the codec is JSON. When partial file are read, the result will not be valid JSON unless the start and end are put back. the file_head and file_tail are learned at startup, by reading the first file in the blob_list and taking the first and last block, this would work for blobs that are appended like nsgflowlogs. The configuration can be set to override the learning. In case learning fails and the option is not set, the default is to use the 'records' as set by nsgflowlogs.
|
|
62
80
|
config :file_head, :validate => :string, :required => false, :default => '{"records":['
|
|
63
81
|
# The string that ends the JSON
|
|
@@ -76,64 +94,66 @@ config :file_tail, :validate => :string, :required => false, :default => ']}'
|
|
|
76
94
|
# For NSGFLOWLOGS a path starts with "resourceId=/", but this would only be needed to exclude other files that may be written in the same container.
|
|
77
95
|
config :prefix, :validate => :string, :required => false
|
|
78
96
|
|
|
97
|
+
config :path_filters, :validate => :array, :default => ['**/*'], :required => false
|
|
98
|
+
|
|
99
|
+
# TODO: Other feature requests
|
|
100
|
+
# show file path in logger
|
|
101
|
+
# add filepath as part of log message
|
|
102
|
+
# option to keep registry on local disk
|
|
79
103
|
|
|
80
104
|
|
|
81
105
|
public
|
|
82
106
|
def register
|
|
83
107
|
@pipe_id = Thread.current[:name].split("[").last.split("]").first
|
|
84
|
-
@logger.info("=== "+config_name
|
|
85
|
-
#@logger.info("ruby #{ RUBY_VERSION }p#{ RUBY_PATCHLEVEL } / #{Gem.loaded_specs[config_name].version.to_s}")
|
|
108
|
+
@logger.info("=== #{config_name} #{Gem.loaded_specs["logstash-input-"+config_name].version.to_s} / #{@pipe_id} / #{@id[0,6]} / ruby #{ RUBY_VERSION }p#{ RUBY_PATCHLEVEL } ===")
|
|
86
109
|
@logger.info("If this plugin doesn't work, please raise an issue in https://github.com/janmg/logstash-input-azure_blob_storage")
|
|
87
110
|
# TODO: consider multiple readers, so add pipeline @id or use logstash-to-logstash communication?
|
|
88
111
|
# TODO: Implement retry ... Error: Connection refused - Failed to open TCP connection to
|
|
112
|
+
end
|
|
113
|
+
|
|
89
114
|
|
|
115
|
+
|
|
116
|
+
def run(queue)
|
|
90
117
|
# counter for all processed events since the start of this pipeline
|
|
91
118
|
@processed = 0
|
|
92
119
|
@regsaved = @processed
|
|
93
120
|
|
|
94
|
-
|
|
95
|
-
# 1. storageaccount / sas_token
|
|
96
|
-
# 2. connection_string
|
|
97
|
-
# 3. storageaccount / access_key
|
|
98
|
-
|
|
99
|
-
unless connection_string.nil?
|
|
100
|
-
conn = connection_string.value
|
|
101
|
-
end
|
|
102
|
-
unless sas_token.nil?
|
|
103
|
-
unless sas_token.value.start_with?('?')
|
|
104
|
-
conn = "BlobEndpoint=https://#{storageaccount}.#{dns_suffix};SharedAccessSignature=#{sas_token.value}"
|
|
105
|
-
else
|
|
106
|
-
conn = sas_token.value
|
|
107
|
-
end
|
|
108
|
-
end
|
|
109
|
-
unless conn.nil?
|
|
110
|
-
@blob_client = Azure::Storage::Blob::BlobService.create_from_connection_string(conn)
|
|
111
|
-
else
|
|
112
|
-
@blob_client = Azure::Storage::Blob::BlobService.create(
|
|
113
|
-
storage_account_name: storageaccount,
|
|
114
|
-
storage_dns_suffix: dns_suffix,
|
|
115
|
-
storage_access_key: access_key.value,
|
|
116
|
-
)
|
|
117
|
-
end
|
|
121
|
+
connect
|
|
118
122
|
|
|
119
123
|
@registry = Hash.new
|
|
120
124
|
if registry_create_policy == "resume"
|
|
121
|
-
|
|
122
|
-
for counter in 0..3
|
|
125
|
+
for counter in 1..3
|
|
123
126
|
begin
|
|
124
|
-
|
|
125
|
-
|
|
127
|
+
if (!@registry_local_path.nil?)
|
|
128
|
+
unless File.file?(@registry_local_path+"/"+@pipe_id)
|
|
129
|
+
@registry = Marshal.load(@blob_client.get_blob(container, registry_path)[1])
|
|
130
|
+
#[0] headers [1] responsebody
|
|
131
|
+
@logger.info("migrating from remote registry #{registry_path}")
|
|
132
|
+
else
|
|
133
|
+
if !Dir.exist?(@registry_local_path)
|
|
134
|
+
FileUtils.mkdir_p(@registry_local_path)
|
|
135
|
+
end
|
|
136
|
+
@registry = Marshal.load(File.read(@registry_local_path+"/"+@pipe_id))
|
|
137
|
+
@logger.info("resuming from local registry #{registry_local_path+"/"+@pipe_id}")
|
|
138
|
+
end
|
|
139
|
+
else
|
|
140
|
+
@registry = Marshal.load(@blob_client.get_blob(container, registry_path)[1])
|
|
141
|
+
#[0] headers [1] responsebody
|
|
142
|
+
@logger.info("resuming from remote registry #{registry_path}")
|
|
143
|
+
end
|
|
144
|
+
break
|
|
126
145
|
rescue Exception => e
|
|
127
|
-
@logger.error(
|
|
146
|
+
@logger.error("caught: #{e.message}")
|
|
128
147
|
@registry.clear
|
|
129
|
-
@logger.error(
|
|
148
|
+
@logger.error("loading registry failed for attempt #{counter} of 3")
|
|
130
149
|
end
|
|
131
150
|
end
|
|
132
151
|
end
|
|
133
152
|
# read filelist and set offsets to file length to mark all the old files as done
|
|
134
153
|
if registry_create_policy == "start_fresh"
|
|
135
|
-
@logger.info(@pipe_id+" starting fresh")
|
|
136
154
|
@registry = list_blobs(true)
|
|
155
|
+
save_registry(@registry)
|
|
156
|
+
@logger.info("starting fresh, writing a clean registry to contain #{@registry.size} blobs/files")
|
|
137
157
|
end
|
|
138
158
|
|
|
139
159
|
@is_json = false
|
|
@@ -146,34 +166,41 @@ def register
|
|
|
146
166
|
@tail = ''
|
|
147
167
|
# if codec=json sniff one files blocks A and Z to learn file_head and file_tail
|
|
148
168
|
if @is_json
|
|
149
|
-
learn_encapsulation
|
|
150
169
|
if file_head
|
|
151
|
-
|
|
170
|
+
@head = file_head
|
|
152
171
|
end
|
|
153
172
|
if file_tail
|
|
154
|
-
|
|
173
|
+
@tail = file_tail
|
|
155
174
|
end
|
|
156
|
-
|
|
175
|
+
if file_head and file_tail and !skip_learning
|
|
176
|
+
learn_encapsulation
|
|
177
|
+
end
|
|
178
|
+
@logger.info("head will be: #{@head} and tail is set to #{@tail}")
|
|
157
179
|
end
|
|
158
|
-
end # def register
|
|
159
|
-
|
|
160
|
-
|
|
161
180
|
|
|
162
|
-
def run(queue)
|
|
163
181
|
newreg = Hash.new
|
|
164
182
|
filelist = Hash.new
|
|
165
183
|
worklist = Hash.new
|
|
166
|
-
|
|
184
|
+
@last = start = Time.now.to_i
|
|
185
|
+
|
|
186
|
+
# This is the main loop, it
|
|
187
|
+
# 1. Lists all the files in the remote storage account that match the path prefix
|
|
188
|
+
# 2. Filters on path_filters to only include files that match the directory and file glob (**/*.json)
|
|
189
|
+
# 3. Save the listed files in a registry of known files and filesizes.
|
|
190
|
+
# 4. List all the files again and compare the registry with the new filelist and put the delta in a worklist
|
|
191
|
+
# 5. Process the worklist and put all events in the logstash queue.
|
|
192
|
+
# 6. if there is time left, sleep to complete the interval. If processing takes more than an inteval, save the registry and continue.
|
|
193
|
+
# 7. If stop signal comes, finish the current file, save the registry and quit
|
|
167
194
|
while !stop?
|
|
168
|
-
chrono = Time.now.to_i
|
|
169
195
|
# load the registry, compare it's offsets to file list, set offset to 0 for new files, process the whole list and if finished within the interval wait for next loop,
|
|
170
196
|
# TODO: sort by timestamp ?
|
|
171
197
|
#filelist.sort_by(|k,v|resource(k)[:date])
|
|
172
198
|
worklist.clear
|
|
173
199
|
filelist.clear
|
|
174
200
|
newreg.clear
|
|
201
|
+
|
|
202
|
+
# Listing all the files
|
|
175
203
|
filelist = list_blobs(false)
|
|
176
|
-
# registry.merge(filelist) {|key, :offset, :length| :offset.merge :length }
|
|
177
204
|
filelist.each do |name, file|
|
|
178
205
|
off = 0
|
|
179
206
|
begin
|
|
@@ -182,63 +209,98 @@ def run(queue)
|
|
|
182
209
|
off = 0
|
|
183
210
|
end
|
|
184
211
|
newreg.store(name, { :offset => off, :length => file[:length] })
|
|
212
|
+
if (@debug_until > @processed) then @logger.info("2: adding offsets: #{name} #{off} #{file[:length]}") end
|
|
185
213
|
end
|
|
186
|
-
|
|
214
|
+
# size nilClass when the list doesn't grow?!
|
|
187
215
|
# Worklist is the subset of files where the already read offset is smaller than the file size
|
|
188
216
|
worklist.clear
|
|
217
|
+
chunk = nil
|
|
218
|
+
|
|
189
219
|
worklist = newreg.select {|name,file| file[:offset] < file[:length]}
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
220
|
+
if (worklist.size > 4) then @logger.info("worklist contains #{worklist.size} blobs") end
|
|
221
|
+
|
|
222
|
+
# Start of processing
|
|
223
|
+
# This would be ideal for threading since it's IO intensive, would be nice with a ruby native ThreadPool
|
|
224
|
+
if (worklist.size > 0) then
|
|
225
|
+
worklist.each do |name, file|
|
|
226
|
+
start = Time.now.to_i
|
|
227
|
+
if (@debug_until > @processed) then @logger.info("3: processing #{name} from #{file[:offset]} to #{file[:length]}") end
|
|
194
228
|
size = 0
|
|
195
229
|
if file[:offset] == 0
|
|
196
|
-
|
|
197
|
-
|
|
230
|
+
# This is where Sera4000 issue starts
|
|
231
|
+
# For an append blob, reading full and crashing, retry, last_modified? ... lenght? ... committed? ...
|
|
232
|
+
# length and skip reg value
|
|
233
|
+
if (file[:length] > 0)
|
|
234
|
+
begin
|
|
235
|
+
chunk = full_read(name)
|
|
236
|
+
size=chunk.size
|
|
237
|
+
rescue Exception => e
|
|
238
|
+
@logger.error("Failed to read #{name} because of: #{e.message} .. will continue and pretend this never happened")
|
|
239
|
+
end
|
|
240
|
+
else
|
|
241
|
+
@logger.info("found a zero size file #{name}")
|
|
242
|
+
chunk = nil
|
|
243
|
+
end
|
|
198
244
|
else
|
|
199
245
|
chunk = partial_read_json(name, file[:offset], file[:length])
|
|
200
|
-
@logger.debug(
|
|
246
|
+
@logger.debug("partial file #{name} from #{file[:offset]} to #{file[:length]}")
|
|
201
247
|
end
|
|
202
248
|
if logtype == "nsgflowlog" && @is_json
|
|
249
|
+
# skip empty chunks
|
|
250
|
+
unless chunk.nil?
|
|
203
251
|
res = resource(name)
|
|
204
252
|
begin
|
|
205
253
|
fingjson = JSON.parse(chunk)
|
|
206
|
-
@processed += nsgflowlog(queue, fingjson)
|
|
207
|
-
@logger.debug(
|
|
254
|
+
@processed += nsgflowlog(queue, fingjson, name)
|
|
255
|
+
@logger.debug("Processed #{res[:nsg]} [#{res[:date]}] #{@processed} events")
|
|
208
256
|
rescue JSON::ParserError
|
|
209
|
-
@logger.error(
|
|
257
|
+
@logger.error("parse error on #{res[:nsg]} [#{res[:date]}] offset: #{file[:offset]} length: #{file[:length]}")
|
|
210
258
|
end
|
|
259
|
+
end
|
|
211
260
|
# TODO: Convert this to line based grokking.
|
|
212
261
|
# TODO: ECS Compliance?
|
|
213
262
|
elsif logtype == "wadiis" && !@is_json
|
|
214
263
|
@processed += wadiislog(queue, name)
|
|
215
264
|
else
|
|
216
265
|
counter = 0
|
|
217
|
-
|
|
266
|
+
begin
|
|
267
|
+
@codec.decode(chunk) do |event|
|
|
218
268
|
counter += 1
|
|
269
|
+
if @addfilename
|
|
270
|
+
event.set('filename', name)
|
|
271
|
+
end
|
|
219
272
|
decorate(event)
|
|
220
273
|
queue << event
|
|
274
|
+
end
|
|
275
|
+
rescue Exception => e
|
|
276
|
+
@logger.error("codec exception: #{e.message} .. will continue and pretend this never happened")
|
|
277
|
+
@registry.store(name, { :offset => file[:length], :length => file[:length] })
|
|
278
|
+
@logger.debug("#{chunk}")
|
|
221
279
|
end
|
|
222
280
|
@processed += counter
|
|
223
281
|
end
|
|
224
282
|
@registry.store(name, { :offset => size, :length => file[:length] })
|
|
225
283
|
# TODO add input plugin option to prevent connection cache
|
|
226
284
|
@blob_client.client.reset_agents!
|
|
227
|
-
#@logger.info(
|
|
285
|
+
#@logger.info("name #{name} size #{size} len #{file[:length]}")
|
|
228
286
|
# if stop? good moment to stop what we're doing
|
|
229
287
|
if stop?
|
|
230
288
|
return
|
|
231
289
|
end
|
|
232
|
-
|
|
233
|
-
now = Time.now.to_i
|
|
234
|
-
if ((now - chrono) > interval)
|
|
290
|
+
if ((Time.now.to_i - @last) > @interval)
|
|
235
291
|
save_registry(@registry)
|
|
236
|
-
chrono += interval
|
|
237
292
|
end
|
|
293
|
+
end
|
|
294
|
+
end
|
|
295
|
+
# The files that got processed after the last registry save need to be saved too, in case the worklist is empty for some intervals.
|
|
296
|
+
now = Time.now.to_i
|
|
297
|
+
if ((now - @last) > @interval)
|
|
298
|
+
save_registry(@registry)
|
|
299
|
+
end
|
|
300
|
+
sleeptime = interval - ((now - start) % interval)
|
|
301
|
+
if @debug_timer
|
|
302
|
+
@logger.info("going to sleep for #{sleeptime} seconds")
|
|
238
303
|
end
|
|
239
|
-
# Save the registry and sleep until the remaining polling interval is over
|
|
240
|
-
save_registry(@registry)
|
|
241
|
-
sleeptime = interval - (Time.now.to_i - chrono)
|
|
242
304
|
Stud.stoppable_sleep(sleeptime) { stop? }
|
|
243
305
|
end
|
|
244
306
|
end
|
|
@@ -252,8 +314,54 @@ end
|
|
|
252
314
|
|
|
253
315
|
|
|
254
316
|
private
|
|
317
|
+
def connect
|
|
318
|
+
# Try in this order to access the storageaccount
|
|
319
|
+
# 1. storageaccount / sas_token
|
|
320
|
+
# 2. connection_string
|
|
321
|
+
# 3. storageaccount / access_key
|
|
322
|
+
|
|
323
|
+
unless connection_string.nil?
|
|
324
|
+
conn = connection_string.value
|
|
325
|
+
end
|
|
326
|
+
unless sas_token.nil?
|
|
327
|
+
unless sas_token.value.start_with?('?')
|
|
328
|
+
conn = "BlobEndpoint=https://#{storageaccount}.#{dns_suffix};SharedAccessSignature=#{sas_token.value}"
|
|
329
|
+
else
|
|
330
|
+
conn = sas_token.value
|
|
331
|
+
end
|
|
332
|
+
end
|
|
333
|
+
unless conn.nil?
|
|
334
|
+
@blob_client = Azure::Storage::Blob::BlobService.create_from_connection_string(conn)
|
|
335
|
+
else
|
|
336
|
+
# unless use_development_storage?
|
|
337
|
+
@blob_client = Azure::Storage::Blob::BlobService.create(
|
|
338
|
+
storage_account_name: storageaccount,
|
|
339
|
+
storage_dns_suffix: dns_suffix,
|
|
340
|
+
storage_access_key: access_key.value,
|
|
341
|
+
)
|
|
342
|
+
# else
|
|
343
|
+
# @logger.info("not yet implemented")
|
|
344
|
+
# end
|
|
345
|
+
end
|
|
346
|
+
end
|
|
347
|
+
|
|
255
348
|
def full_read(filename)
|
|
256
|
-
|
|
349
|
+
tries ||= 2
|
|
350
|
+
begin
|
|
351
|
+
return @blob_client.get_blob(container, filename)[1]
|
|
352
|
+
rescue Exception => e
|
|
353
|
+
@logger.error("caught: #{e.message} for full_read")
|
|
354
|
+
if (tries -= 1) > 0
|
|
355
|
+
if e.message = "Connection reset by peer"
|
|
356
|
+
connect
|
|
357
|
+
end
|
|
358
|
+
retry
|
|
359
|
+
end
|
|
360
|
+
end
|
|
361
|
+
begin
|
|
362
|
+
chuck = @blob_client.get_blob(container, filename)[1]
|
|
363
|
+
end
|
|
364
|
+
return chuck
|
|
257
365
|
end
|
|
258
366
|
|
|
259
367
|
def partial_read_json(filename, offset, length)
|
|
@@ -276,8 +384,7 @@ def strip_comma(str)
|
|
|
276
384
|
end
|
|
277
385
|
|
|
278
386
|
|
|
279
|
-
|
|
280
|
-
def nsgflowlog(queue, json)
|
|
387
|
+
def nsgflowlog(queue, json, name)
|
|
281
388
|
count=0
|
|
282
389
|
json["records"].each do |record|
|
|
283
390
|
res = resource(record["resourceId"])
|
|
@@ -290,9 +397,16 @@ def nsgflowlog(queue, json)
|
|
|
290
397
|
tups = tup.split(',')
|
|
291
398
|
ev = rule.merge({:unixtimestamp => tups[0], :src_ip => tups[1], :dst_ip => tups[2], :src_port => tups[3], :dst_port => tups[4], :protocol => tups[5], :direction => tups[6], :decision => tups[7]})
|
|
292
399
|
if (record["properties"]["Version"]==2)
|
|
400
|
+
tups[9] = 0 if tups[9].nil?
|
|
401
|
+
tups[10] = 0 if tups[10].nil?
|
|
402
|
+
tups[11] = 0 if tups[11].nil?
|
|
403
|
+
tups[12] = 0 if tups[12].nil?
|
|
293
404
|
ev.merge!( {:flowstate => tups[8], :src_pack => tups[9], :src_bytes => tups[10], :dst_pack => tups[11], :dst_bytes => tups[12]} )
|
|
294
405
|
end
|
|
295
406
|
@logger.trace(ev.to_s)
|
|
407
|
+
if @addfilename
|
|
408
|
+
ev.merge!( {:filename => name } )
|
|
409
|
+
end
|
|
296
410
|
event = LogStash::Event.new('message' => ev.to_json)
|
|
297
411
|
decorate(event)
|
|
298
412
|
queue << event
|
|
@@ -323,66 +437,108 @@ end
|
|
|
323
437
|
# list all blobs in the blobstore, set the offsets from the registry and return the filelist
|
|
324
438
|
# inspired by: https://github.com/Azure-Samples/storage-blobs-ruby-quickstart/blob/master/example.rb
|
|
325
439
|
def list_blobs(fill)
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
440
|
+
tries ||= 3
|
|
441
|
+
begin
|
|
442
|
+
return try_list_blobs(fill)
|
|
443
|
+
rescue Exception => e
|
|
444
|
+
@logger.error("caught: #{e.message} for list_blobs retries left #{tries}")
|
|
445
|
+
if (tries -= 1) > 0
|
|
446
|
+
retry
|
|
447
|
+
end
|
|
448
|
+
end
|
|
449
|
+
end
|
|
450
|
+
|
|
451
|
+
def try_list_blobs(fill)
|
|
452
|
+
# inspired by: http://blog.mirthlab.com/2012/05/25/cleanly-retrying-blocks-of-code-after-an-exception-in-ruby/
|
|
453
|
+
chrono = Time.now.to_i
|
|
454
|
+
files = Hash.new
|
|
455
|
+
nextMarker = nil
|
|
456
|
+
counter = 1
|
|
457
|
+
loop do
|
|
334
458
|
blobs = @blob_client.list_blobs(container, { marker: nextMarker, prefix: @prefix})
|
|
335
459
|
blobs.each do |blob|
|
|
336
|
-
|
|
337
|
-
|
|
460
|
+
# FNM_PATHNAME is required so that "**/test" can match "test" at the root folder
|
|
461
|
+
# FNM_EXTGLOB allows you to use "test{a,b,c}" to match either "testa", "testb" or "testc" (closer to shell behavior)
|
|
462
|
+
unless blob.name == registry_path
|
|
463
|
+
if @path_filters.any? {|path| File.fnmatch?(path, blob.name, File::FNM_PATHNAME | File::FNM_EXTGLOB)}
|
|
338
464
|
length = blob.properties[:content_length].to_i
|
|
339
|
-
|
|
465
|
+
offset = 0
|
|
340
466
|
if fill
|
|
341
467
|
offset = length
|
|
342
|
-
|
|
468
|
+
end
|
|
343
469
|
files.store(blob.name, { :offset => offset, :length => length })
|
|
470
|
+
if (@debug_until > @processed) then @logger.info("1: list_blobs #{blob.name} #{offset} #{length}") end
|
|
344
471
|
end
|
|
472
|
+
end
|
|
345
473
|
end
|
|
346
474
|
nextMarker = blobs.continuation_token
|
|
347
475
|
break unless nextMarker && !nextMarker.empty?
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
476
|
+
if (counter % 10 == 0) then @logger.info(" listing #{counter * 50000} files") end
|
|
477
|
+
counter+=1
|
|
478
|
+
end
|
|
479
|
+
if @debug_timer
|
|
480
|
+
@logger.info("list_blobs took #{Time.now.to_i - chrono} sec")
|
|
481
|
+
end
|
|
353
482
|
return files
|
|
354
483
|
end
|
|
355
484
|
|
|
356
485
|
# When events were processed after the last registry save, start a thread to update the registry file.
|
|
357
486
|
def save_registry(filelist)
|
|
358
|
-
#
|
|
487
|
+
# Because of threading, processed values and regsaved are not thread safe, they can change as instance variable @! Most of the time this is fine because the registry is the last resort, but be careful about corner cases!
|
|
359
488
|
unless @processed == @regsaved
|
|
360
489
|
@regsaved = @processed
|
|
361
|
-
|
|
362
|
-
|
|
490
|
+
unless (@busy_writing_registry)
|
|
491
|
+
Thread.new {
|
|
363
492
|
begin
|
|
364
|
-
@
|
|
493
|
+
@busy_writing_registry = true
|
|
494
|
+
unless (@registry_local_path)
|
|
495
|
+
@blob_client.create_block_blob(container, registry_path, Marshal.dump(filelist))
|
|
496
|
+
@logger.info("processed #{@processed} events, saving #{filelist.size} blobs and offsets to remote registry #{registry_path}")
|
|
497
|
+
else
|
|
498
|
+
File.open(@registry_local_path+"/"+@pipe_id, 'w') { |file| file.write(Marshal.dump(filelist)) }
|
|
499
|
+
@logger.info("processed #{@processed} events, saving #{filelist.size} blobs and offsets to local registry #{registry_local_path+"/"+@pipe_id}")
|
|
500
|
+
end
|
|
501
|
+
@busy_writing_registry = false
|
|
502
|
+
@last = Time.now.to_i
|
|
365
503
|
rescue
|
|
366
|
-
@logger.error(
|
|
504
|
+
@logger.error("Oh my, registry write failed, do you have write access?")
|
|
367
505
|
end
|
|
368
506
|
}
|
|
507
|
+
else
|
|
508
|
+
@logger.info("Skipped writing the registry because previous write still in progress, it just takes long or may be hanging!")
|
|
509
|
+
end
|
|
369
510
|
end
|
|
370
511
|
end
|
|
371
512
|
|
|
513
|
+
|
|
372
514
|
def learn_encapsulation
|
|
515
|
+
@logger.info("learn_encapsulation, this can be skipped by setting skip_learning => true. Or set both head_file and tail_file")
|
|
373
516
|
# From one file, read first block and last block to learn head and tail
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
517
|
+
begin
|
|
518
|
+
blobs = @blob_client.list_blobs(container, { max_results: 3, prefix: @prefix})
|
|
519
|
+
blobs.each do |blob|
|
|
520
|
+
unless blob.name == registry_path
|
|
521
|
+
begin
|
|
522
|
+
blocks = @blob_client.list_blob_blocks(container, blob.name)[:committed]
|
|
523
|
+
if blocks.first.name.start_with?('A00')
|
|
524
|
+
@logger.debug("using #{blob.name}/#{blocks.first.name} to learn the json header")
|
|
525
|
+
@head = @blob_client.get_blob(container, blob.name, start_range: 0, end_range: blocks.first.size-1)[1]
|
|
526
|
+
end
|
|
527
|
+
if blocks.last.name.start_with?('Z00')
|
|
528
|
+
@logger.debug("using #{blob.name}/#{blocks.last.name} to learn the json footer")
|
|
529
|
+
length = blob.properties[:content_length].to_i
|
|
530
|
+
offset = length - blocks.last.size
|
|
531
|
+
@tail = @blob_client.get_blob(container, blob.name, start_range: offset, end_range: length-1)[1]
|
|
532
|
+
@logger.debug("learned tail: #{@tail}")
|
|
533
|
+
end
|
|
534
|
+
rescue Exception => e
|
|
535
|
+
@logger.info("learn json one of the attempts failed #{e.message}")
|
|
536
|
+
end
|
|
537
|
+
end
|
|
538
|
+
end
|
|
539
|
+
rescue Exception => e
|
|
540
|
+
@logger.info("learn json header and footer failed because #{e.message}")
|
|
541
|
+
end
|
|
386
542
|
end
|
|
387
543
|
|
|
388
544
|
def resource(str)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Gem::Specification.new do |s|
|
|
2
2
|
s.name = 'logstash-input-azure_blob_storage'
|
|
3
|
-
s.version = '0.11.
|
|
3
|
+
s.version = '0.11.6'
|
|
4
4
|
s.licenses = ['Apache-2.0']
|
|
5
5
|
s.summary = 'This logstash plugin reads and parses data from Azure Storage Blobs.'
|
|
6
6
|
s.description = <<-EOF
|
|
@@ -22,6 +22,6 @@ EOF
|
|
|
22
22
|
# Gem dependencies
|
|
23
23
|
s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.1'
|
|
24
24
|
s.add_runtime_dependency 'stud', '~> 0.0.23'
|
|
25
|
-
s.add_runtime_dependency 'azure-storage-blob', '~> 1.
|
|
26
|
-
s.add_development_dependency 'logstash-devutils', '~>
|
|
25
|
+
s.add_runtime_dependency 'azure-storage-blob', '~> 1.1'
|
|
26
|
+
#s.add_development_dependency 'logstash-devutils', '~> 2'
|
|
27
27
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: logstash-input-azure_blob_storage
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.11.
|
|
4
|
+
version: 0.11.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Jan Geertsma
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2021-02-11 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -17,8 +17,8 @@ dependencies:
|
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
18
|
version: '2.1'
|
|
19
19
|
name: logstash-core-plugin-api
|
|
20
|
-
prerelease: false
|
|
21
20
|
type: :runtime
|
|
21
|
+
prerelease: false
|
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
|
23
23
|
requirements:
|
|
24
24
|
- - "~>"
|
|
@@ -31,8 +31,8 @@ dependencies:
|
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
32
|
version: 0.0.23
|
|
33
33
|
name: stud
|
|
34
|
-
prerelease: false
|
|
35
34
|
type: :runtime
|
|
35
|
+
prerelease: false
|
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
37
|
requirements:
|
|
38
38
|
- - "~>"
|
|
@@ -43,35 +43,15 @@ dependencies:
|
|
|
43
43
|
requirements:
|
|
44
44
|
- - "~>"
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
|
-
version: '1.
|
|
46
|
+
version: '1.1'
|
|
47
47
|
name: azure-storage-blob
|
|
48
|
-
prerelease: false
|
|
49
48
|
type: :runtime
|
|
50
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
51
|
-
requirements:
|
|
52
|
-
- - "~>"
|
|
53
|
-
- !ruby/object:Gem::Version
|
|
54
|
-
version: '1.0'
|
|
55
|
-
- !ruby/object:Gem::Dependency
|
|
56
|
-
requirement: !ruby/object:Gem::Requirement
|
|
57
|
-
requirements:
|
|
58
|
-
- - ">="
|
|
59
|
-
- !ruby/object:Gem::Version
|
|
60
|
-
version: 1.0.0
|
|
61
|
-
- - "~>"
|
|
62
|
-
- !ruby/object:Gem::Version
|
|
63
|
-
version: '1.0'
|
|
64
|
-
name: logstash-devutils
|
|
65
49
|
prerelease: false
|
|
66
|
-
type: :development
|
|
67
50
|
version_requirements: !ruby/object:Gem::Requirement
|
|
68
51
|
requirements:
|
|
69
|
-
- - ">="
|
|
70
|
-
- !ruby/object:Gem::Version
|
|
71
|
-
version: 1.0.0
|
|
72
52
|
- - "~>"
|
|
73
53
|
- !ruby/object:Gem::Version
|
|
74
|
-
version: '1.
|
|
54
|
+
version: '1.1'
|
|
75
55
|
description: " This gem is a Logstash plugin. It reads and parses data from Azure\
|
|
76
56
|
\ Storage Blobs. The azure_blob_storage is a reimplementation to replace azureblob\
|
|
77
57
|
\ from azure-diagnostics-tools/Logstash. It can deal with larger volumes and partial\
|