logstash-input-googleanalytics 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '04491f6fbd0155db1436da6f0845bd63ce29636b7372b492de73763c56924ecc'
+  data.tar.gz: 7cb8556acc02f88b1eae332ddca42e0bc754723174069cba9afadec93629e865
+SHA512:
+  metadata.gz: b51b990b31c66d14479e154e4581c3bea8897777a4f98633774c7e07e3a128b4969b8c19d91a80cd0f737e0e3020c657f9d107061d997921128e2f278eb76222
+  data.tar.gz: cb6eb9728d995f98fbedc7796cb645e58e1634cb6763940a05bc5d8ea89bd6b81e3c5dc8770f237f8c9cdd125841b5f64c5d8ab18a0fbdfdd5bd4a9b2a262962

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 1.0.0
+ - Initial Release

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-input-googleanalytics
+Use this input to query data from Google Analytics into Logstash

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,94 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## SSL Errors
+
+Sometimes you will get an SSL Error. It can easily be solved by following the advice [here](https://github.com/jruby/jruby/issues/1055#issuecomment-38209934) and adding a new cacert.pem to your environment.
+```
+Error: certificate verify failed
+Exception: Faraday::SSLError
+```
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-input-googleanalytics", :path => "/your/local/logstash-input-googleanalytics"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/googleanalytics.rb

@@ -0,0 +1,239 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+require "stud/interval"
+
+# Generate a repeating message.
+#
+# This plugin is intented only as an example.
+
+class LogStash::Inputs::GoogleAnalytics < LogStash::Inputs::Base
+  config_name "googleanalytics"
+
+  # If undefined, Logstash will complain, even if codec is unused.
+  default :codec, "plain"
+
+  # Most of these inputs are described in the Google Analytics API docs.
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#q_details
+  # Any changes from the format described above have been noted.
+
+  # Type for logstash filtering
+  config :type, :validate => :string, :default => 'googleanalytics'
+  # A comma separated list of view (profile) ids, in the format 'ga:XXXX'
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#ids
+  config :ids, :validate => :string, :required => true
+  # In the format YYYY-MM-DD, or relative by using today, yesterday, or the NdaysAgo pattern
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#startDate
+  config :start_date, :validate => :string, :default => 'yesterday'
+  # In the format YYYY-MM-DD, or relative by using today, yesterday, or the NdaysAgo pattern
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#endDate
+  config :end_date, :validate => :string, :default => 'yesterday'
+  # The aggregated statistics for user activity to your site, such as clicks or pageviews.
+  # Maximum of 10 metrics for any query
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#metrics
+  # For a full list of metrics, see the documentation
+  # https://developers.google.com/analytics/devguides/reporting/core/dimsmets
+  config :metrics, :validate => :string, :required => true
+  # Breaks down metrics by common criteria; for example, by ga:browser or ga:city
+  # Maximum of 7 dimensions in any query
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#dimensions
+  # For a full list of dimensions, see the documentation
+  # https://developers.google.com/analytics/devguides/reporting/core/dimsmets
+  config :dimensions, :validate => :string, :default => nil
+  # Used to restrict the data returned from your request
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#filters
+  config :filters, :validate => :string, :default => nil
+  # A list of metrics and dimensions indicating the sorting order and sorting direction for the returned data
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#sort
+  config :sort, :validate => :string, :default => nil
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#segment
+  config :segment, :validate => :string, :default => nil
+  # Valid values are DEFAULT, FASTER, HIGHER_PRECISION
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#samplingLevel
+  config :sampling_level, :validate => :string, :default => nil
+  # This is the result to start with, beginning at 1
+  # You probably don't need to change this but it has been included here for completeness
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#startIndex
+  config :start_index, :validate => :number, :default => 1
+  # This is the number of results in a page. This plugin will start at
+  # @start_index and keep pulling pages of data until it has all results.
+  # You probably don't need to change this but it has been included here for completeness
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#maxResults
+  config :max_results, :validate => :number, :default => 10000
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#include-empty-rows
+  config :include_empty_rows, :validate => :boolean, :default => true
+
+  # These values need to be pulled from your Google Developers Console
+  # For more information, see the docs. Be sure to enable Google Analytics API
+  # access for your application.
+  # https://developers.google.com/identity/protocols/OAuth2ServiceAccount
+
+  # This should be the path to the public/private key as a standard P12 file
+  config :key_file_path, :validate => :string, :required => true
+  # The key secret doe the file above. If not prompted for a secret,
+  # it seems to default to notasecret
+  config :key_secret, :validate => :string, :default => 'notasecret'
+  # The service email account found in the Google Developers Console after
+  # generating the key file.
+  config :service_account_email, :validate => :string, :required => true
+
+  # The service name to connect to. Should not change unless Google changes something
+  config :service_name, :validate => :string, :default => 'analytics'
+  # The version of the API to use.
+  config :api_version, :validate => :string, :default => 'v3'
+
+  # This will store the query in the resulting logstash event
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#data_response
+  config :store_query, :validate => :boolean, :default => true
+  # This will store the profile information in the resulting logstash event
+  # https://developers.google.com/analytics/devguides/reporting/core/v3/reference#data_response
+  config :store_profile, :validate => :boolean, :default => true
+
+  # Interval to run the command. Value is in seconds. If no interval is given,
+  # this plugin only fetches data once.
+  config :interval, :validate => :number, :required => false
+
+
+  public
+  def register
+    require 'google/api_client'
+  end # def register
+
+  def run(queue)
+    # we can abort the loop if stop? becomes true
+    while !stop?
+      start = Time.now
+      client, analytics = get_service
+      results_index = @start_index
+      while !stop?
+        results = client.execute(
+          :api_method => analytics.data.ga.get,
+          :parameters => client_options(results_index))
+
+        if results.data.rows.first
+          query = results.data.query.to_hash
+          profile_info = results.data.profile_info.to_hash
+          column_headers = results.data.column_headers.map { |c|
+            c.name
+          }
+
+          results.data.rows.each do |r|
+            event = LogStash::Event.new()
+            decorate(event)
+            event['containsSampledData'] = results.data.containsSampledData
+            event['query'] = query if @store_query
+            event['profileInfo'] = profile_info if @store_profile
+            column_headers.zip(r).each do |head,data|
+              if is_num(data)
+                float_data = Float(data)
+                # Sometimes GA returns infinity. if so, the number it invalid
+                # so set it to zero.
+                if float_data == Float::INFINITY
+                  event[head.gsub(':','_')] = 0.0
+                else
+                  event[head.gsub(':','_')] = float_data
+                end
+              else
+                event[head.gsub(':','_')] = data
+              end
+            end
+            # Try to add a date unless it was already added
+            if @start_date == @end_date
+              if !event.include?('ga_date')
+                if @start_date == 'today'
+                  event['ga_date'] = Date.parse(Time.now().strftime("%F"))
+                elsif @start_date == 'yesterday'
+                  event['ga_date'] = Date.parse(Time.at(Time.now.to_i - 86400).strftime("%F"))
+                elsif @start_date.include?('daysAgo')
+                  days_ago = @start_date.sub('daysAgo','').to_i
+                  event['ga_date'] = Date.parse(Time.at(Time.now.to_i - (days_ago*86400)).strftime("%F"))
+                else
+                  event['ga_date'] = Date.parse(@start_date)
+                end
+              else
+                event['ga_date'] = Date.parse(event['ga_date'].to_i.to_s)
+              end
+            else
+              # Convert YYYYMMdd to YYYY-MM-dd
+              event['ga_date'] = Date.parse(event['ga_date'].to_i.to_s)
+            end
+            event['ga_date'] = event['ga_date'].to_s
+            queue << event
+          end
+        end
+        nextLink = results.data.nextLink rescue nil
+        if nextLink
+          start_index+=@max_results
+        else
+          break
+        end
+      end
+      if @interval.nil?
+        break
+      else
+        duration = Time.now - start
+        # Sleep for the remainder of the interval, or 0 if the duration ran
+        # longer than the interval.
+        sleeptime = [0, @interval - duration].max
+        if sleeptime == 0
+          @logger.warn("Execution ran longer than the interval. Skipping sleep.",
+                       :duration => duration,
+                       :interval => @interval)
+        else
+          Stud.stoppable_sleep(sleeptime) { stop? }
+        end
+      end
+    end # loop
+  end # def run
+
+  def stop
+  end
+
+  private
+  def client_options(results_index)
+    options = {
+      'ids' => @ids,
+      'start-date' => @start_date,
+      'end-date' => @end_date,
+      'metrics' => @metrics,
+      'max-results' => @max_results,
+      'output' => 'json',
+      'start-index' => results_index
+    }
+    options.merge!({ 'dimensions' => @dimensions }) if @dimensions
+    options.merge!({ 'filters' => @filters }) if @filters
+    options.merge!({ 'sort' => @sort }) if @sort
+    options.merge!({ 'segment' => @segment }) if @segment
+    options.merge!({ 'samplingLevel' => @sampling_level }) if @sampling_level
+    options.merge!({ 'include-empty-rows' => @include_empty_rows }) if !@include_empty_rows.nil?
+    return options
+  end
+
+  def get_service
+    client = Google::APIClient.new(
+      :application_name => 'Google Analytics Logstash Input',
+      :application_version => '1.0.0')
+
+    puts @key_file_path
+    puts @key_secret
+    puts @service_account_email
+    # Load our credentials for the service account
+    key = Google::APIClient::KeyUtils.load_from_pkcs12(@key_file_path, @key_secret)
+    client.authorization = Signet::OAuth2::Client.new(
+      :token_credential_uri => 'https://accounts.google.com/o/oauth2/token',
+      :audience => 'https://accounts.google.com/o/oauth2/token',
+      :scope => 'https://www.googleapis.com/auth/analytics.readonly',
+      :issuer => @service_account_email,
+      :signing_key => key)
+
+    # Request a token for our service account
+    client.authorization.fetch_access_token!
+    analytics = client.discovered_api(@service_name, @api_version)
+    return client, analytics
+  end
+
+  private
+  def is_num(a)
+    return (Float(a) and true) rescue false
+  end
+end # class LogStash::Inputs::GoogleAnalytics

data/logstash-input-googleanalytics.gemspec

@@ -0,0 +1,29 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-input-googleanalytics'
+  s.version         = '1.0.0'
+  s.licenses = ['Apache License (2.0)']
+  s.summary = "This input queries the Google Analytics API at a definable interval."
+  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors = ["Elastic","Russ Savage"]
+  s.email = 'russ@elastic.co'
+  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+  s.add_development_dependency 'logstash-devutils'
+  s.add_runtime_dependency 'logstash-codec-plain'
+  s.add_runtime_dependency 'stud', '>= 0.0.22'
+  s.add_runtime_dependency 'google-api-client', "~> 0.7.1"
+  s.add_development_dependency 'vcr'
+  s.add_development_dependency 'webmock'
+  s.add_development_dependency 'json'
+end

data/spec/inputs/googleanalytics_spec.rb

@@ -0,0 +1,47 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/googleanalytics"
+require "vcr"
+require "json"
+
+ENV['SSL_CERT_FILE'] = "/Users/rsavage/Downloads/cacert.pem"
+
+VCR.configure do |config|
+  config.cassette_library_dir = File.join(File.dirname(__FILE__), '..', 'fixtures', 'vcr_cassettes')
+  config.hook_into :webmock
+end
+
+RSpec.describe LogStash::Inputs::GoogleAnalytics do
+  describe "inputs/googleanalytics" do
+    context "get audience overview" do
+      let(:options) do
+        {
+          'ids' => 'ga:97869209',
+          'start_date' => '27daysAgo',
+          'end_date' => '27daysAgo',
+          'metrics' => 'ga:pageviews',
+          'dimensions' => 'ga:date',
+          'key_file_path' => '/Users/rsavage/workspace/logstash-1.4.2/logstash-input-googleanalytics-1c933e55eca6.p12',
+          'key_secret' => 'notasecret',
+          'service_account_email' => '759646568999-hto359j2lud906ae7ufts01djuu0n7j0@developer.gserviceaccount.com'
+        }
+      end
+      let(:input) { LogStash::Inputs::GoogleAnalytics.new(options) }
+      let(:expected_fields_result) { ["ga_pageviews"] }
+      let(:queue) { [] }
+      subject { input }
+      it "loads pageviews" do
+        #VCR.use_cassette("get_audience_overview") do
+          subject.register
+          subject.run(queue)
+          expect(queue.length).to eq(1)
+          e = queue.pop
+          expected_fields_result.each do |f|
+            expect(e.to_hash).to include(f)
+          end
+        #end
+      end
+    end
+  end
+
+end

metadata

@@ -0,0 +1,175 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-googleanalytics
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Elastic
+- Russ Savage
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2020-09-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: logstash-codec-plain
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: stud
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  name: google-api-client
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.7.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.7.1
+- !ruby/object:Gem::Dependency
+  name: vcr
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: webmock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not
+  a stand-alone program
+email: russ@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/logstash/inputs/googleanalytics.rb
+- logstash-input-googleanalytics.gemspec
+- spec/inputs/googleanalytics_spec.rb
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.1.2
+signing_key: 
+specification_version: 4
+summary: This input queries the Google Analytics API at a definable interval.
+test_files:
+- spec/inputs/googleanalytics_spec.rb