fluent-plugin-festival 0.0.7 → 0.0.8
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +3 -3
- data/examples/README.md +148 -0
- data/examples/fluent-festival.conf +71 -0
- data/examples/train_station-mapping.json +16 -0
- data/fluent-plugin-festival.gemspec +1 -1
- data/lib/fluent/plugin/festival_proxy.rb +3 -3
- metadata +5 -3
- data/docs/tutorial.md +0 -62
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2902595f635d8b7761cc3d3cd3c51b6a90d38964
|
|
4
|
+
data.tar.gz: 94a59fc85d2b1bb06ef56dae8fb306986c3dad57
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 96eb77aee601e9f82966bfda0f8fc81a3cecaf02f3f7834bbe6cd8c97362e1d7bc8df3ae3861060455544492f804b1d1bc76de553b8efdeeb90e635e82c1462f
|
|
7
|
+
data.tar.gz: 8bc9b24054897346c9fb2b7ae8b5bbb389e185231aa84679a80453321d980b7353ac1ade65329a430db8ca9e26b24ee805f5beacea56e99a8ab388fcb635428c
|
data/README.md
CHANGED
|
@@ -35,10 +35,10 @@ Input Plugin can receive events from FESTIVAL EaaS API Server. It can be used vi
|
|
|
35
35
|
password festival_portal_password
|
|
36
36
|
polling_interval 30
|
|
37
37
|
<resource>
|
|
38
|
-
path /aggregators/IOT-0/testbeds/jose/resources/
|
|
38
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_humidity-info-value/current_data
|
|
39
39
|
</resource>
|
|
40
40
|
<resource>
|
|
41
|
-
path /aggregators/IOT-0/testbeds/jose/resources/
|
|
41
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_barometer-info-value/current_data
|
|
42
42
|
</resource>
|
|
43
43
|
</source>
|
|
44
44
|
|
|
@@ -60,7 +60,7 @@ Input Plugin can receive events from FESTIVAL EaaS API Server. It can be used vi
|
|
|
60
60
|
- **polling_interval** (optional): Polling interval (seconds) for accessing EaaS API (default: 60 seconds)
|
|
61
61
|
- **resource** (at least one entry is required): The target resources to obtain sensor data. multiple resources can be specified by multiple <resource> tags. If a user wants to specify different polling interval for each resource, it must be specified different <source> tags.
|
|
62
62
|
- **path** (at least one entry is required): The target resource path name should be specified. The pathname should specify only under aggregator part and target data type. Currently, only "current_data" type is supported.
|
|
63
|
-
(e.g. /aggregators/IOT-0/testbeds/jose/resources/
|
|
63
|
+
(e.g. /aggregators/IOT-0/testbeds/jose/resources/train_station_hyogo001_barometer-info-value/current_data).
|
|
64
64
|
|
|
65
65
|
If the time field is empty, this plugin automatically set the finished time of data downloading. If multiple sensor data specified simultaneously, the time difference may become larger than single datum case. A sample data format is shown below.
|
|
66
66
|
|
data/examples/README.md
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
# fluent-plugin-festival tutorial
|
|
2
|
+
|
|
3
|
+
## Install fluentd
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
gem install fluent-plugin-festival
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
fluentd 0.14.x will be installed automatically.
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
## Create a sample configuration file
|
|
13
|
+
|
|
14
|
+
Create a sample configuration file as follows. You need to change festival_portal_login_name and festival_portal_password to your account information.
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
% vi fluent.conf
|
|
18
|
+
---
|
|
19
|
+
<source>
|
|
20
|
+
@type festival
|
|
21
|
+
tag test1
|
|
22
|
+
email festival_portal_registered_email_address
|
|
23
|
+
password festival_portal_password
|
|
24
|
+
polling_interval 30
|
|
25
|
+
<resource>
|
|
26
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_humidity-info-value/current_data
|
|
27
|
+
</resource>
|
|
28
|
+
<resource>
|
|
29
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_barometer-info-value/current_data
|
|
30
|
+
</resource>
|
|
31
|
+
</source>
|
|
32
|
+
|
|
33
|
+
<source>
|
|
34
|
+
@type festival
|
|
35
|
+
tag test2
|
|
36
|
+
email festival_portal_registered_email_address
|
|
37
|
+
password festival_portal_password
|
|
38
|
+
polling_interval 180
|
|
39
|
+
<resource>
|
|
40
|
+
path /aggregators/IOT-0/testbeds/smartsantander/resources/smartsantander_u7jcfa_f3176-chemicalAgentAtmosphericConcentration:airParticles-sensor/current_data
|
|
41
|
+
</resource>
|
|
42
|
+
</source>
|
|
43
|
+
|
|
44
|
+
<match *>
|
|
45
|
+
@type stdout
|
|
46
|
+
</match>
|
|
47
|
+
---
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
## Start fluentd with debug mode
|
|
52
|
+
|
|
53
|
+
Start fluentd with debug mode as follows.
|
|
54
|
+
|
|
55
|
+
```
|
|
56
|
+
fluentd -c fluent.conf -vvv
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Then, you can confirm the sensor values are output to your console.
|
|
60
|
+
|
|
61
|
+
You can try the other resources by browsing resources via FESTIVAL portal and change target resource URI to the ones listed in the resource information.
|
|
62
|
+
|
|
63
|
+
## Store data into Elasticsearch
|
|
64
|
+
|
|
65
|
+
About installation, please refer the following page.
|
|
66
|
+
|
|
67
|
+
https://www.elastic.co/downloads/elasticsearch
|
|
68
|
+
|
|
69
|
+
In the followings, how to store FESTIVAL platform data into Elasticsearch will be shown.
|
|
70
|
+
|
|
71
|
+
First of all, you need to create mapping in Elasticsearch. An Elasticsearch server is assumed to run in localhost.
|
|
72
|
+
|
|
73
|
+
```
|
|
74
|
+
vi train_station-mapping.json
|
|
75
|
+
---
|
|
76
|
+
{
|
|
77
|
+
"train_station": {
|
|
78
|
+
"properties": {
|
|
79
|
+
"timestamp": {
|
|
80
|
+
"type": "date",
|
|
81
|
+
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
|
|
82
|
+
},
|
|
83
|
+
"resourceName": {
|
|
84
|
+
"type": "string"
|
|
85
|
+
},
|
|
86
|
+
"dataValue": {
|
|
87
|
+
"type": "double"
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
curl -XPUT 'http://localhost:9200/festival'
|
|
95
|
+
curl -XPUT 'http://localhost:9200/festival/train_station/_mapping' -d @train_station-mapping.json
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
% vi fluent.conf
|
|
100
|
+
---
|
|
101
|
+
<source>
|
|
102
|
+
@type festival
|
|
103
|
+
tag test1
|
|
104
|
+
email festival_portal_registered_email_address
|
|
105
|
+
password festival_portal_password
|
|
106
|
+
polling_interval 30
|
|
107
|
+
<resource>
|
|
108
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_humidity-info-value/current_data
|
|
109
|
+
</resource>
|
|
110
|
+
<resource>
|
|
111
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_barometer-info-value/current_data
|
|
112
|
+
</resource>
|
|
113
|
+
@label @test0
|
|
114
|
+
</source>
|
|
115
|
+
|
|
116
|
+
<label @test0>
|
|
117
|
+
<filter test*>
|
|
118
|
+
@type record_transformer
|
|
119
|
+
enable_ruby true
|
|
120
|
+
auto_typecast true
|
|
121
|
+
<record>
|
|
122
|
+
timestamp ${time.strftime("%FT%T.%L%:z")}
|
|
123
|
+
</record>
|
|
124
|
+
</filter>
|
|
125
|
+
|
|
126
|
+
# If you want to output the messages not only into Elasticsearch
|
|
127
|
+
# but also into standard output, please enable comment out lines.
|
|
128
|
+
# Configurations for fluent-plugin-elasticsearch must be fluentd
|
|
129
|
+
# 0.12 compatible form because it doesn't support 0.14 yet.
|
|
130
|
+
<match test*>
|
|
131
|
+
@type copy
|
|
132
|
+
<store>
|
|
133
|
+
@type elasticsearch
|
|
134
|
+
host localhost
|
|
135
|
+
port 9200
|
|
136
|
+
index_name festival
|
|
137
|
+
type_name train_station
|
|
138
|
+
logstash_format false
|
|
139
|
+
include_tag_key true
|
|
140
|
+
time_key timestamp
|
|
141
|
+
flush_interval 30s
|
|
142
|
+
</store>
|
|
143
|
+
<store>
|
|
144
|
+
@type stdout
|
|
145
|
+
</store>
|
|
146
|
+
</match>
|
|
147
|
+
</label>
|
|
148
|
+
```
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
<source>
|
|
2
|
+
@type festival
|
|
3
|
+
tag test1
|
|
4
|
+
email festival_portal_registered_email_address
|
|
5
|
+
password festival_portal_password
|
|
6
|
+
polling_interval 30
|
|
7
|
+
<resource>
|
|
8
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_hyogo001_barometer-info-value/current_data
|
|
9
|
+
</resource>
|
|
10
|
+
<resource>
|
|
11
|
+
path /aggregators/IOT-0/testbeds/jose/resources/train_station_kyoto001_barometer-info-value/current_data
|
|
12
|
+
</resource>
|
|
13
|
+
@label @test0
|
|
14
|
+
</source>
|
|
15
|
+
|
|
16
|
+
<source>
|
|
17
|
+
@type festival
|
|
18
|
+
tag test2
|
|
19
|
+
email festival_portal_registered_email_address
|
|
20
|
+
password festival_portal_password
|
|
21
|
+
polling_interval 180
|
|
22
|
+
<resource>
|
|
23
|
+
path /aggregators/IOT-0/testbeds/smartsantander/resources/smartsantander_u7jcfa_f3176-chemicalAgentAtmosphericConcentration:airParticles-sensor/current_data
|
|
24
|
+
</resource>
|
|
25
|
+
@label @test0
|
|
26
|
+
</source>
|
|
27
|
+
|
|
28
|
+
#<match *>
|
|
29
|
+
# @type stdout
|
|
30
|
+
#</match>
|
|
31
|
+
|
|
32
|
+
# <label> is not required in a simple use case like this configuration
|
|
33
|
+
# You can just specify tag at filter and match directives
|
|
34
|
+
# time_key option seems not to work in the current fluent-plugin-elasticsearch
|
|
35
|
+
# with fluentd 0.14. Instead of that, record_transformer is used in this
|
|
36
|
+
# example.
|
|
37
|
+
<label @test0>
|
|
38
|
+
<filter test*>
|
|
39
|
+
@type record_transformer
|
|
40
|
+
enable_ruby true
|
|
41
|
+
auto_typecast true
|
|
42
|
+
<record>
|
|
43
|
+
# set timestamp granularity as second
|
|
44
|
+
#timestamp ${time.strftime("%FT%T%:z")}
|
|
45
|
+
# set timestamp granularity as millisecond
|
|
46
|
+
timestamp ${time.strftime("%FT%T.%L%:z")}
|
|
47
|
+
</record>
|
|
48
|
+
</filter>
|
|
49
|
+
|
|
50
|
+
# If you want to output the messages not only into Elasticsearch
|
|
51
|
+
# but also into standard output, please enable comment out lines.
|
|
52
|
+
# Configurations for fluent-plugin-elasticsearch must be fluentd
|
|
53
|
+
# 0.12 compatible form because it doesn't support 0.14 yet.
|
|
54
|
+
<match test*>
|
|
55
|
+
@type copy
|
|
56
|
+
<store>
|
|
57
|
+
@type elasticsearch
|
|
58
|
+
host localhost
|
|
59
|
+
port 9200
|
|
60
|
+
index_name festival
|
|
61
|
+
type_name train_station
|
|
62
|
+
logstash_format false
|
|
63
|
+
include_tag_key true
|
|
64
|
+
time_key timestamp
|
|
65
|
+
flush_interval 30s
|
|
66
|
+
</store>
|
|
67
|
+
<store>
|
|
68
|
+
@type stdout
|
|
69
|
+
</store>
|
|
70
|
+
</match>
|
|
71
|
+
</label>
|
|
@@ -121,7 +121,7 @@ module Fluent::Plugin
|
|
|
121
121
|
when "current_data" then
|
|
122
122
|
log.debug "get_data: request #{get_data_request(resource.path)}, #{get_data_header.inspect}"
|
|
123
123
|
get_data_res = @https.get(get_data_request(resource.path), get_data_header)
|
|
124
|
-
|
|
124
|
+
next if !error_handler(get_data_res,"get_data failed.")
|
|
125
125
|
log.debug "get_data: #{get_data_res.body}"
|
|
126
126
|
data << {
|
|
127
127
|
"resourceName": resource.path,
|
|
@@ -129,11 +129,11 @@ module Fluent::Plugin
|
|
|
129
129
|
}
|
|
130
130
|
when "historical_data" then
|
|
131
131
|
log.error "historical_data is not supported yet"
|
|
132
|
-
|
|
132
|
+
next
|
|
133
133
|
else
|
|
134
134
|
log.error "The other resource type is not supported yet"
|
|
135
135
|
log.error "resource_type: #{resource_type(resource.path)}"
|
|
136
|
-
|
|
136
|
+
next
|
|
137
137
|
end
|
|
138
138
|
end
|
|
139
139
|
if data.size > 1
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fluent-plugin-festival
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.0.
|
|
4
|
+
version: 0.0.8
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Toyokazu Akiyama
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2017-05-
|
|
11
|
+
date: 2017-05-15 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: fluentd
|
|
@@ -77,7 +77,9 @@ files:
|
|
|
77
77
|
- LICENSE
|
|
78
78
|
- README.md
|
|
79
79
|
- Rakefile
|
|
80
|
-
-
|
|
80
|
+
- examples/README.md
|
|
81
|
+
- examples/fluent-festival.conf
|
|
82
|
+
- examples/train_station-mapping.json
|
|
81
83
|
- fluent-plugin-festival.gemspec
|
|
82
84
|
- lib/fluent/plugin/festival_proxy.rb
|
|
83
85
|
- lib/fluent/plugin/in_festival.rb
|
data/docs/tutorial.md
DELETED
|
@@ -1,62 +0,0 @@
|
|
|
1
|
-
# fluent-plugin-festival tutorial
|
|
2
|
-
|
|
3
|
-
## Install fluentd
|
|
4
|
-
|
|
5
|
-
```
|
|
6
|
-
gem install fluent-plugin-festival
|
|
7
|
-
```
|
|
8
|
-
|
|
9
|
-
fluentd 0.14.x will be installed automatically.
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
## Create a sample configuration file
|
|
13
|
-
|
|
14
|
-
Create a sample configuration file as follows. You need to change festival_portal_login_name and festival_portal_password to your account information.
|
|
15
|
-
|
|
16
|
-
```
|
|
17
|
-
% vi fluent.conf
|
|
18
|
-
---
|
|
19
|
-
<source>
|
|
20
|
-
@type festival
|
|
21
|
-
tag test1
|
|
22
|
-
email festival_portal_registered_email_address
|
|
23
|
-
password festival_portal_password
|
|
24
|
-
polling_interval 30
|
|
25
|
-
<resource>
|
|
26
|
-
path /aggregators/IOT-0/testbeds/jose/resources/hyogo001_barometer-info-value/current_data
|
|
27
|
-
</resource>
|
|
28
|
-
<resource>
|
|
29
|
-
path /aggregators/IOT-0/testbeds/jose/resources/kyoto001_barometer-info-value/current_data
|
|
30
|
-
</resource>
|
|
31
|
-
</source>
|
|
32
|
-
|
|
33
|
-
<source>
|
|
34
|
-
@type festival
|
|
35
|
-
tag test2
|
|
36
|
-
email festival_portal_registered_email_address
|
|
37
|
-
password festival_portal_password
|
|
38
|
-
polling_interval 180
|
|
39
|
-
<resource>
|
|
40
|
-
path /aggregators/IOT-0/testbeds/smartsantander/resources/smartsantander_u7jcfa_f3176-chemicalAgentAtmosphericConcentration:airParticles-sensor/current_data
|
|
41
|
-
</resource>
|
|
42
|
-
</source>
|
|
43
|
-
|
|
44
|
-
<match *>
|
|
45
|
-
@type stdout
|
|
46
|
-
</match>
|
|
47
|
-
---
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
You sould not include "current_data" in the resource path.
|
|
51
|
-
|
|
52
|
-
## Start fluentd with debug mode
|
|
53
|
-
|
|
54
|
-
Start fluentd with debug mode as follows.
|
|
55
|
-
|
|
56
|
-
```
|
|
57
|
-
fluentd -c fluent.conf -vvv
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
Then, you can confirm the sensor values are output to your console.
|
|
61
|
-
|
|
62
|
-
You can try the other resources by browsing resources via FESTIVAL portal and change target resource URI to the ones listed in the resource information.
|