azure-loganalytics-datacollector-api 0.1.5 → 0.5.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 130505bbd3cda48ee4330fba0c5fe76814a6414d20676a7691734cd38c7a7714
4
- data.tar.gz: ff86325bc1df19f3c505efcb27c194684a9f2e2111149b2a1428ecfe044ea891
3
+ metadata.gz: 3d3abe715a08517a56495053c816fc36a814c4d373e82af5d96a108cc9ee5d31
4
+ data.tar.gz: db6499466afb04d793a80ca16ed91cc3654dbc7cb897c9e8e0a7dec3cce15d5f
5
5
  SHA512:
6
- metadata.gz: 7c6998cb89da3454ab39605c4c801d96880beee4e80a2630d66d84ae8a5c5d2675b741f49cc5d61cc7ef0730582b3a67dfdc2f123c93959f01e0d4dddf9e87b4
7
- data.tar.gz: 679d59d0c792db18c0be081f0baf450245d41ac70dc8c2778de52d9d58bbbdd22a32ef03e07f488fd034a3be85de384b6adde2df410eba23debd7d70bcde1d2f
6
+ metadata.gz: 76263c50638ec683371727e61240bb705242226757fb928c841896b6a0a22af48ce76cc9afefdf2a5271b2dee50e13ee2346d866ee35c3bfac6e985e7bc71cce
7
+ data.tar.gz: 87220e1418b867e5eb1a8e32a182c0a6e2ff813405d493e67473dc7f049bca382f14003c9548a2342a9f4b53563c8565ade2d79b393a7691f13c0f3580af61e7
@@ -1,10 +1,38 @@
1
+ # ChangeLog
2
+
3
+ ## 0.5.0
4
+
5
+ * Check body size not to exceed data limit of 30MB in Azure Monitor Data Collection API - [issue #12](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/12)
6
+
7
+ ## 0.4.0
8
+
9
+ * restclient retries request on the following status code - [issue #10](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/10)
10
+ * 429 - Too Many Requests
11
+ * 500 - Internal Server Error
12
+ * 503 - Service Unavailable
13
+
14
+ ## 0.3.0
15
+
16
+ * Enhance log type validation: check not only alpha but also numeric, underscore, and character length (may not exceed 100) - [issue #11](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/11)
17
+
18
+ ## 0.2.0
19
+
20
+ * Support setting the x-ms-AzureResourceId Header - [issue #8](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/8)
21
+
22
+ ## 0.1.6
23
+
24
+ * fix CVE-2020-8130 - [issue #7](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/7)
25
+
1
26
  ## 0.1.5
27
+
2
28
  * Add endpoint parameter with the default value for Azure public - [PR#5](https://github.com/yokawasa/azure-log-analytics-data-collector/pull/5)
3
29
 
4
30
  ## 0.1.4
31
+
5
32
  * Add `set_proxy` method to client - [issue#3](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/3)
6
33
 
7
34
  ## 0.1.2
35
+
8
36
  * fixedup bug - [issue #1](https://github.com/yokawasa/azure-log-analytics-data-collector/issues/1)
9
37
 
10
38
  ## 0.1.1
data/README.md CHANGED
@@ -1,5 +1,23 @@
1
1
  # azure-log-analytics-data-collector Ruby Client
2
- Azure Log Analytics Data Collector API Ruby Client
2
+
3
+ [Azure Log Analytics Data Collector API](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api) Client Libraries for Ruby. The repository was originally created for multiple programming languages, but it was refactored as a dedicated one for Ruby client. Python and PHP client libraries were moved to [azure-log-analytics-data-colloector-python](https://github.com/yokawasa/azure-log-analytics-data-collector-python) and [azure-log-analytics-data-colloector-php](https://github.com/yokawasa/azure-log-analytics-data-collector-php) respectively.
4
+
5
+
6
+ ## Retry policy
7
+
8
+ The client internal leverage [rest-client] to send HTTP request to the API. The client library retries request using the rest-client on the following status code (which is [recommended action](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api)).
9
+ * `429` - Too Many Requests
10
+ * `500` - Internal Server Error
11
+ * `503` - Service Unavailable
12
+
13
+ By default, the client library retres for a total of `3` times, sleeping `5 sec` between retries. The number of retries and sleeping time between retries can be changed with `set_retries` in the client class.
14
+
15
+ ```ruby
16
+ def set_retries(max_retries, retry_sleep_period)
17
+ @max_retries = max_retries
18
+ @retry_sleep_period = retry_sleep_period
19
+ end
20
+ ```
3
21
 
4
22
  ## Installation
5
23
  ```bash
@@ -78,7 +96,51 @@ else
78
96
  end
79
97
  ```
80
98
 
81
- ### Sample3 - use proxy to access the API
99
+ ### Sample3 - With time_generated_field and azure_resource_id option
100
+ Supported setting azure_resource_id option from version [0.2.0](https://github.com/yokawasa/azure-log-analytics-data-collector/releases/tag/v0.2.0)
101
+ ```ruby
102
+ require "azure/loganalytics/datacollectorapi/client"
103
+
104
+ customer_id = '<Customer ID aka WorkspaceID String>'
105
+ shared_key = '<The primary or the secondary Connected Sources client authentication key>'
106
+ log_type = "MyCustomLog"
107
+
108
+ posting_records = []
109
+ record1= {
110
+ :string => "MyText1",
111
+ :boolean => true,
112
+ :number => 100,
113
+ :timegen => "2017-11-23T11:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
114
+ }
115
+ record2= {
116
+ :string => "MyText2",
117
+ :boolean => false,
118
+ :number => 200,
119
+ :timegen => "2017-11-23T12:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
120
+ }
121
+ posting_records.push(record1)
122
+ posting_records.push(record2)
123
+
124
+ time_generated_field = "timegen"
125
+
126
+ # Azure Resource ID
127
+ # [Azure Resource ID Format]
128
+ # /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}
129
+ azure_resource_id ="/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/otherResourceGroup/providers/Microsoft.Storage/storageAccounts/examplestorage"
130
+
131
+ client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
132
+ res = client.post_data(log_type, posting_records, time_generated_field, azure_resource_id)
133
+ puts res
134
+ puts "res code=#{res.code}"
135
+
136
+ if Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
137
+ puts "operation was succeeded!"
138
+ else
139
+ puts "operation was failured!"
140
+ end
141
+ ```
142
+
143
+ ### Sample4 - use proxy to access the API
82
144
  ```ruby
83
145
  require "azure/loganalytics/datacollectorapi/client"
84
146
 
@@ -114,4 +176,3 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/yokawa
114
176
  ## License
115
177
 
116
178
  The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
117
-
@@ -0,0 +1,12 @@
1
+ # Sample codes for azure-log-analytics-data-collector
2
+ [Azure Log Analytics HTTP Data Collecotr API](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api) Client libraries
3
+
4
+ ## Client libraries and Sample code
5
+
6
+ | Language | Source Code | Project URL |
7
+ | ------------- | ------------- |------------- |
8
+ | Ruby | This repository | [Rubygem Package](https://rubygems.org/gems/azure-loganalytics-datacollector-api) |
9
+ | Python | [Source code](https://github.com/yokawasa/azure-log-analytics-data-collector-python) | [Python Package](https://pypi.python.org/pypi/azure-log-analytics-data-collector-api) |
10
+ | PHP | [Source code](https://github.com/yokawasa/azure-log-analytics-data-collector-php) | NONE |
11
+ | CSharp | [Sample Code](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api#sample-requests) | NONE |
12
+ | PowerShell | [Sample Code](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api#sample-requests) | NONE |
@@ -7,7 +7,7 @@ Gem::Specification.new do |spec|
7
7
  spec.name = "azure-loganalytics-datacollector-api"
8
8
  spec.version = Azure::Loganalytics::Datacollectorapi::VERSION
9
9
  spec.authors = ["Yoichi Kawasaki"]
10
- spec.email = ["yoichi.kawasaki@outlook.com"]
10
+ spec.email = ["yokawasa@gmail.com"]
11
11
  spec.summary = %q{Azure Log Analytics Data Collector API Ruby Client}
12
12
  spec.description = spec.summary
13
13
  spec.homepage = "https://github.com/yokawasa/azure-log-analytics-data-collector"
@@ -20,7 +20,6 @@ Gem::Specification.new do |spec|
20
20
  spec.require_paths = ["lib"]
21
21
 
22
22
  spec.add_dependency "rest-client"
23
- spec.add_development_dependency "bundler", "~> 1.13"
24
- spec.add_development_dependency "rake", "~> 10.0"
23
+ spec.add_development_dependency "rake", ">= 12.3.3"
25
24
  spec.add_development_dependency "rspec", "~> 3.0"
26
25
  end
@@ -8,7 +8,13 @@ module Azure
8
8
 
9
9
  class Client
10
10
 
11
- def initialize (customer_id, shared_key,endpoint ='ods.opinsights.azure.com')
11
+ DEFAUT_MAX_RETRIES = 3.freeze
12
+ DEFAULT_RETRY_SLEEP_PERIOD = 5.freeze
13
+ # API Data Limits
14
+ # https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api#data-limits
15
+ MAX_BODY_BYTE_SIZE = 31457280.freeze # 30 MB
16
+
17
+ def initialize (customer_id, shared_key, endpoint ='ods.opinsights.azure.com')
12
18
  require 'rest-client'
13
19
  require 'json'
14
20
  require 'openssl'
@@ -18,13 +24,21 @@ module Azure
18
24
  @customer_id = customer_id
19
25
  @shared_key = shared_key
20
26
  @endpoint = endpoint
27
+ @default_azure_resource_id = ''
28
+
29
+ @max_retries = DEFAUT_MAX_RETRIES
30
+ @retry_sleep_period = DEFAULT_RETRY_SLEEP_PERIOD
21
31
  end
22
32
 
23
- def post_data(log_type, json_records, record_timestamp ='')
33
+ def post_data(log_type, json_records, record_timestamp ='', azure_resource_id ='' )
24
34
  raise ConfigError, 'no log_type' if log_type.empty?
25
- raise ConfigError, 'log_type must be only alpha characters' if not is_alpha(log_type)
35
+ raise ConfigError, 'log_type must only contain alpha numeric and _, and not exceed 100 chars' if not is_valid_log_type(log_type)
26
36
  raise ConfigError, 'no json_records' if json_records.empty?
37
+
27
38
  body = json_records.to_json
39
+ body_size = body.bytesize
40
+ raise "too large payload (#{body_size})! max post data size is #{MAX_BODY_BYTE_SIZE} bytes!" if body_size >= MAX_BODY_BYTE_SIZE
41
+
28
42
  uri = sprintf("https://%s.%s/api/logs?api-version=%s",
29
43
  @customer_id, @endpoint, API_VERSION)
30
44
  date = rfc1123date()
@@ -35,16 +49,39 @@ module Azure
35
49
  'Authorization' => sig,
36
50
  'Log-Type' => log_type,
37
51
  'x-ms-date' => date,
52
+ 'x-ms-AzureResourceId' => azure_resource_id.empty? ? @default_azure_resource_id : azure_resource_id,
38
53
  'time-generated-field' => record_timestamp
39
54
  }
40
55
 
41
- res = RestClient.post( uri, body, headers)
42
- res
56
+ retries = 0
57
+ begin
58
+ res = RestClient.post( uri, body, headers)
59
+ res
60
+ rescue => e
61
+ c = e.response.code.to_i
62
+ if c == 429 || c == 500 || c==503
63
+ if retries < @max_retries
64
+ retries += 1
65
+ sleep(@retry_sleep_period)
66
+ retry
67
+ end
68
+ end
69
+ raise e
70
+ end
43
71
  end
44
72
 
45
73
  def set_proxy(proxy='')
46
74
  RestClient.proxy = proxy.empty? ? ENV['http_proxy'] : proxy
47
75
  end
76
+
77
+ def set_default_azure_resoruce_id(azure_resource_id)
78
+ @default_azure_resource_id = azure_resource_id
79
+ end
80
+
81
+ def set_retres(max_retries, retry_sleep_period)
82
+ @max_retries = max_retries
83
+ @retry_sleep_period = retry_sleep_period
84
+ end
48
85
 
49
86
  def self.is_success(res)
50
87
  return (res.code == 200) ? true : false
@@ -52,8 +89,8 @@ module Azure
52
89
 
53
90
  private
54
91
 
55
- def is_alpha(s)
56
- return (s.match(/^[[:alpha:]]+$/)) ? true : false
92
+ def is_valid_log_type(s)
93
+ return ( s.match(/^[a-zA-Z0-9_]+$/) && s.length <= 100 ) ? true : false
57
94
  end
58
95
 
59
96
  def rfc1123date()
@@ -1,7 +1,7 @@
1
1
  module Azure
2
2
  module Loganalytics
3
3
  module Datacollectorapi
4
- VERSION = "0.1.5"
4
+ VERSION = "0.5.0"
5
5
  end
6
6
  end
7
7
  end
@@ -5,15 +5,36 @@ describe Azure::Loganalytics::Datacollectorapi::Client do
5
5
  expect(Azure::Loganalytics::Datacollectorapi::VERSION).not_to be nil
6
6
  end
7
7
 
8
- #it "does something useful" do
9
- # expect(false).to eq(true)
10
- #end
8
+ customer_id = '<Customer ID aka WorkspaceID String>'
9
+ shared_key = '<Primary Key String>'
10
+ log_type = "MyCustomLog"
11
11
 
12
- it "posting data to datacollector api" do
13
- customer_id = '<Customer ID aka WorkspaceID String>'
14
- shared_key = '<Primary Key String>'
15
- log_type = "MyCustomLog"
12
+ it "log type validation" do
13
+ json_records = []
14
+ dummy= {
15
+ :string => "dummy title",
16
+ :number => 10
17
+ }
18
+ valid_log_type = "abcedefghijklmnopqrstuvwxyz1234567890_"
19
+ invalid_log_type1 = "*-|[]//"
20
+ invalid_log_type2 = "abcde__xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
21
+ json_records.push(dummy)
22
+ client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
23
+
24
+ res = client.post_data(valid_log_type, json_records)
25
+ expect(Azure::Loganalytics::Datacollectorapi::Client.is_success(res)).to eq(true)
16
26
 
27
+ expect{
28
+ client.post_data(invalid_log_type1, json_records)
29
+ }.to raise_error(Exception)
30
+
31
+ expect{
32
+ client.post_data(invalid_log_type2, json_records)
33
+ }.to raise_error(Exception)
34
+
35
+ end
36
+
37
+ it "posting data to datacollector api" do
17
38
  json_records = []
18
39
  record1= {
19
40
  :string => "MyText1",
@@ -33,4 +54,53 @@ describe Azure::Loganalytics::Datacollectorapi::Client do
33
54
  expect(Azure::Loganalytics::Datacollectorapi::Client.is_success(res)).to eq(true)
34
55
  end
35
56
 
57
+ it "posting data to datacollector api with time-generated-field" do
58
+ json_records = []
59
+ record1= {
60
+ :string => "MyText1",
61
+ :boolean => true,
62
+ :number => 100,
63
+ :timegen => "2020-05-05T11:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
64
+ }
65
+ record2= {
66
+ :string => "MyText2",
67
+ :boolean => false,
68
+ :number => 200,
69
+ :timegen => "2020-05-05T12:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
70
+ }
71
+ json_records.push(record1)
72
+ json_records.push(record2)
73
+
74
+ time_generated_field = "timegen"
75
+ client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
76
+ res = client.post_data(log_type, json_records, time_generated_field)
77
+ expect(Azure::Loganalytics::Datacollectorapi::Client.is_success(res)).to eq(true)
78
+ end
79
+
80
+ it "posting data to datacollector api with azure-resource-id" do
81
+ json_records = []
82
+ record1= {
83
+ :string => "MyText1",
84
+ :boolean => true,
85
+ :number => 100
86
+ }
87
+ record2= {
88
+ :string => "MyText2",
89
+ :boolean => false,
90
+ :number => 200
91
+ }
92
+ json_records.push(record1)
93
+ json_records.push(record2)
94
+
95
+ time_generated_field = ""
96
+ # Azure Resource ID
97
+ # https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-resource#resourceid
98
+ # /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}
99
+ azure_resource_id ="/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/otherResourceGroup/providers/Microsoft.Storage/storageAccounts/examplestorage"
100
+
101
+ client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
102
+ res = client.post_data(log_type, json_records, time_generated_field, azure_resource_id)
103
+ expect(Azure::Loganalytics::Datacollectorapi::Client.is_success(res)).to eq(true)
104
+ end
105
+
36
106
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: azure-loganalytics-datacollector-api
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.5
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Yoichi Kawasaki
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-06-06 00:00:00.000000000 Z
11
+ date: 2020-07-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: rest-client
@@ -24,34 +24,20 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: '0'
27
- - !ruby/object:Gem::Dependency
28
- name: bundler
29
- requirement: !ruby/object:Gem::Requirement
30
- requirements:
31
- - - "~>"
32
- - !ruby/object:Gem::Version
33
- version: '1.13'
34
- type: :development
35
- prerelease: false
36
- version_requirements: !ruby/object:Gem::Requirement
37
- requirements:
38
- - - "~>"
39
- - !ruby/object:Gem::Version
40
- version: '1.13'
41
27
  - !ruby/object:Gem::Dependency
42
28
  name: rake
43
29
  requirement: !ruby/object:Gem::Requirement
44
30
  requirements:
45
- - - "~>"
31
+ - - ">="
46
32
  - !ruby/object:Gem::Version
47
- version: '10.0'
33
+ version: 12.3.3
48
34
  type: :development
49
35
  prerelease: false
50
36
  version_requirements: !ruby/object:Gem::Requirement
51
37
  requirements:
52
- - - "~>"
38
+ - - ">="
53
39
  - !ruby/object:Gem::Version
54
- version: '10.0'
40
+ version: 12.3.3
55
41
  - !ruby/object:Gem::Dependency
56
42
  name: rspec
57
43
  requirement: !ruby/object:Gem::Requirement
@@ -68,7 +54,7 @@ dependencies:
68
54
  version: '3.0'
69
55
  description: Azure Log Analytics Data Collector API Ruby Client
70
56
  email:
71
- - yoichi.kawasaki@outlook.com
57
+ - yokawasa@gmail.com
72
58
  executables:
73
59
  - console
74
60
  - setup
@@ -82,6 +68,7 @@ files:
82
68
  - LICENSE.txt
83
69
  - README.md
84
70
  - Rakefile
71
+ - SampleCodes.md
85
72
  - azure-loganalytics-datacollector-api.gemspec
86
73
  - bin/console
87
74
  - bin/setup
@@ -108,7 +95,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
108
95
  - !ruby/object:Gem::Version
109
96
  version: '0'
110
97
  requirements: []
111
- rubygems_version: 3.0.3
98
+ rubygems_version: 3.1.4
112
99
  signing_key:
113
100
  specification_version: 4
114
101
  summary: Azure Log Analytics Data Collector API Ruby Client