ruby-grafana-reporter 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '00728deba4250ed327170393bbeaac1b1d6d1002463a5a243285ac570f09231f'
-  data.tar.gz: 2985a11f6404e622ff279dbe3e328b38e4202bf665b2fef9e23746bbfd6c3459
+  metadata.gz: db7e3fbf0f8c1a3d7d9ce8d3e3e215b6f6bb5ec3c170252d575e0516b3312b20
+  data.tar.gz: b05cfa64a8d5bef650917df47ad9b84a140f302e87d2c6664c1fab3272e7212e
 SHA512:
-  metadata.gz: 92de552918a451f9bac1196dfbb05dc1ad64e0059f4c90fe06c079e5e181fee96a07858ae0c5815e3dbf37358c4aa271fcd8d129bb6721f3c8a53f285325fc25
-  data.tar.gz: aff95df6e4b5a8636905ec498433d1fcdae6d7b9aa2d26640815a68160c2f74c39078adfbe32a67c9115855ddd97683c6844ea0347e18cf010586e2bd036faed
+  metadata.gz: 3925cc108c3cedc51d6a37926b4ba937296cfd55137d8af40b9254bb92383cf253713321e95e6cfcfeb50de9adf7acd11777cf108227527d2c87658d74173a51
+  data.tar.gz: 79c345933fd16d7c2acfa24715ef09119c1973c605bb5a7b2caa959fcf08ae748e087338717d6e4343281e68a9a3ed9440d5c74e57846691f7033ef2f41e4079

data/README.md

@@ -9,15 +9,11 @@ Reporting Service for Grafana
 ## Table of Contents
 
 * [About the project](#about-the-project)
-* [Getting started](#getting-started)
-  * [Grafana integration](#grafana-integration)
-  * [Webservice overview](#webservice-overview)
-* [Documentation](#documentation)
 * [Features](#features)
+* [Quick Start](#quick-start)
+* [Grafana integration](#grafana-integration)
+* [Webservice overview](#webservice-overview)
 * [Roadmap](#roadmap)
-* [Contributing](#contributing)
-* [Licensing](#licensing)
-* [Acknowledgements](#acknowledgements)
 * [Donations](#donations)
 
 ## About the project
@@ -26,53 +22,88 @@ Did you ever want to create (professional) reports based on Grafana dashboards?
 I did so in order to being able to automatically get monthly reports of my
 home's energy usage. That's how it started.
 
-The reporter provides reporting capabilities for Grafana. It is based on
-(but not limited to) [asciidoctor](https://github.com/asciidoctor/asciidoctor)
-report templates, which can dynamically integrate Grafana panels, queries,
-images etc. to create dynamic PDF reports on the fly.
+## Features
+
+* Build PDF reports based on [grafana](https://github.com/grafana/grafana) dashboards
+(other formats supported)
+* Include dynamic content from grafana (see [function documentation](FUNCTION_CALLS.md)
+as a detailed reference):
+  * panels as images
+  * tables based on grafana panel queries or custom database queries (no images!)
+  * single values to be integrated in text, based on grafana panel queries or custom
+database queries
+* Multi purpose use of the reporter
+  * webservice to be called directly from grafana - it also runs without further
+dependencies in the standard asciidoctor docker container!
+  * standalone command line tool, e.g. to be automated with cron or bash scrips
+* Comes with a complete configuration wizard, including functionality to build a
+demo report on top of the configured grafana host
+* Supports all SQL based datasources, as well as graphite and prometheus
+* Solid as a rock, also in case of template errors and whatever else may happen
+* Full [API documentation](https://rubydoc.info/gems/ruby-grafana-reporter) available
 
-The report may also be returned in any other format that asciidoctor supports.
+## Quick Start
 
-The reporter can run standalone or as a webservice. It is built to
-integrate without further dependencies with the asciidoctor docker image.
+You don't have a grafana setup runnning already? No worries, just configure
+`https://play.grafana.org` in the configuration wizard and see the magic
+happen for that!
 
-Can't wait to see, what functions the reporter provides within the asciidoctor
-templates? Have a look at the [function documentation](FUNCTION_CALLS.md).
+If your grafana setup requires a login, you'll have to setup an api key for
+the reporter. Please follow the steps
+[described here](https://github.com/divinity666/ruby-grafana-reporter/issues/2#issuecomment-811836757)
+first.
 
-## Getting started
+**Windows:**
 
-There exist several ways of installing the reporter. If you need further
-installation help, or want to use a "baremetal" ruby setup or a docker
-integration, please have a look at the more extended
-[installation documentation](INSTALL.md).
+* [Download latest Windows executable](https://github.com/divinity666/ruby-grafana-reporter/releases/latest)
+* `ruby-grafana-reporter -w`
 
-Windows users may directly use the provided executable.
+Known issues:
+* images are currently not included in PDF conversions due to missing support in Prawn gem for windows;
+other target formats do work properly with images
 
-Following these steps sets up the reporter on a fresh Raspberry Pi installation:
+**Raspberry Pi:**
 
-    sudo apt-get install ruby
-    gem install ruby-grafana-reporter
+* `sudo apt-get install ruby`
+* `gem install ruby-grafana-reporter`
+* `ruby-grafana-reporter -w`
 
-That's it. Let's now configure a grafana setup with the configuration wizard:
+**Ruby environment:**
 
-    ruby-grafana-reporter -w
+* `gem install ruby-grafana-reporter`
+* `ruby-grafana-reporter -w`
 
-It is strongly recommended, to also create the demo PDF file, as stated at the end
-of the procedure, to get a detailed documentation of all the reporter capabilities.
-The whole [function documentation](FUNCTION_CALLS.md) is also available at the
-previous link.
+**Docker environment** (advanced users):
 
-To run the reporter as a service, you only need to call it like this:
+* [Download latest single-rb file](https://github.com/divinity666/ruby-grafana-reporter/releases/latest)
+to an empty folder
+* create a configuration file by calling `ruby ruby-grafana-reporter -w` (if in doubt,
+run the command within your docker container)
+* create file `/<<path-to-single-rb-file-folder>>/startup.sh` with the following
+content:
 
-    ruby-grafana-reporter
+```
+cd /documents
+ruby bin/ruby-grafana-reporter
+```
+* add asciidoctor your compose yaml:
 
-Neat, isn't it?
+```
+asciidoctor:
+  image: asciidoctor/docker-asciidoctor
+  container_name: asciidoctor
+  hostname: asciidoctor
+  volumes:
+    - /<<path-to-single-rb-file-folder>>:/documents
+  command:
+    sh /documents/startup.sh
+  restart: unless-stopped
+```
+* start/restart the asciidoctor docker container
 
-### Grafana integration
+## Grafana integration
 
-The key feature of the report is, that it can easily be integrated with grafana
-(I've not even been talking about the features it is providing for that, but
-you'll find them having a look in the example results above).
+The key feature of the report is, that it can easily be integrated with grafana.
 
 For accessing the reporter from grafana, you need to simply add a link to your
 grafana dashboard:
@@ -82,27 +113,27 @@ grafana dashboard:
 * Select `Add`
 * Fill out as following:
   * Type: `link`
-  * Url: `http://<<your-server-url>>:<<your-webservice-port>>/render?var-template=myfirsttemplate`
-  * Title: `MyFirstReport`
+  * Url: `http://<<your-server-url>>:<<your-webservice-port>>/render?var-template=demo_report`
+  * Title: `Demo Report`
   * Select `Time range`
   * Select `Variable values`
 * Select `Add`
 
-Now go back to your dashboard and click the newly generated 'MyFirstReport'
+Now go back to your dashboard and click the newly generated `Demo Report`
 link on it. Now the renderer should start it's task and show you the expected
 results.
 
-But now the fun just starts! Try out the functions stated in the
-'MyFirstReport' PDF file, to include the dynamic content in your asciidoctor
-template.
+Please note, that the reporter won't automatically refresh your screen to update
+the progress. Simply hit `F5` to refresh your browser. After the report has been
+successfully built, it will show the PDF after the next refresh automatically.
 
-Additionally you might want to make the selection of the template variable.
+You want to select a template in grafana, which shall then be rendered?
 Piece of cake: Just add a dashboard variable to your grafana dashboard named
 `template` and let the user select or enter a template name. To make use of it,
-you should change the link of the 'MyFirstReport' link to
-`http://<<your-server-url>>:<<your-webservice-port>>/render?`
-
-That's it. Let me know your feedback!
+you should change the link of the `Demo Report` link to
+`http://<<your-server-url>>:<<your-webservice-port>>/render?`. On
+hitting the new link in the dashboard, grafana will add the selected template as
+a variable and forward it to the reporter.
 
 ## Webservice overview
 
@@ -117,33 +148,13 @@ The main endpoint to call for report generation is configured in the previous ch
 
 However, if you would like to see, currently running report generations and previously generated reports, you may want to call the endpoint `/overview`.
 
-## Documentation
-
-The [function documentation](FUNCTION_CALLS.md) contains a complete overview of
-all possible grafana calls, to generate dynamic report templates.
-
-The [API documentation](https://rubydoc.info/gems/ruby-grafana-reporter) can be
-found here.
-
-## Features
-
-* Build report template including all imaginable grafana content:
-  * panels as images
-  * panel table query or custom query results as real document tables (not images!)
-  * single panel value or custom query single value result integrated in texts
-* Solid as a rock, also in case of template errors and whatever else may happen
-* Fully controllable as command line application or as a webservice
-* Seamlessly integrates with asciidoctor docker container
-* Developed to support other tools than asciidoctor as well
-
 ## Roadmap
 
 This is just a collection of things, I am heading for in future, without a schedule.
 
-* Add a simple plugin system to support specific asciidoctor modifications
+* Support all grafana datasources
 * Solve code TODOs
 * Become [rubocop](https://rubocop.org/) ready
-* Clean and properly setup test cases
 
 ## Contributing
 
@@ -167,4 +178,3 @@ If this project saves you as much time as I hope it does, and if you'd like to
 support my work, feel free donate, even a cup of coffee is appreciated :)
 
 [![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/donate?hosted_button_id=35LH6JNLPHPHQ)
-

data/bin/ruby-grafana-reporter

@@ -1,5 +1,5 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-require_relative '../lib/ruby-grafana-reporter'
-GrafanaReporter::Application::Application.new.configure_and_run(ARGV) unless defined?(Ocra)
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative '../lib/ruby_grafana_reporter'
+GrafanaReporter::Application::Application.new.configure_and_run(ARGV) unless defined?(Ocra)

data/lib/VERSION.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
 # Version information
-GRAFANA_REPORTER_VERSION = [0, 3, 0].freeze
-GRAFANA_REPORTER_RELEASE_DATE = '2021-03-02'
+GRAFANA_REPORTER_VERSION = [0, 4, 0].freeze
+# Release date
+GRAFANA_REPORTER_RELEASE_DATE = '2021-04-14'

data/lib/grafana/abstract_datasource.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+module Grafana
+  # This abstract class defines the base functionalities for the common datasource implementations.
+  # Additionally it provides a factory method to build a real datasource from a given specification.
+  class AbstractDatasource
+    attr_reader :model
+
+    # Factory method to build a datasource from a given datasource Hash description.
+    # @param ds_model [Hash] grafana specification of a single datasource
+    # @return [AbstractDatasource] instance of a fitting datasource implementation
+    def self.build_instance(ds_model)
+      raise InvalidDatasourceQueryProvidedError, ds_model unless ds_model.is_a?(Hash)
+      raise InvalidDatasourceQueryProvidedError, ds_model unless ds_model['meta']
+      raise InvalidDatasourceQueryProvidedError, ds_model unless ds_model['meta']['id']
+      raise InvalidDatasourceQueryProvidedError, ds_model unless ds_model['meta']['category']
+
+      return SqlDatasource.new(ds_model) if ds_model['meta']['category'] == 'sql'
+
+      case ds_model['meta']['id']
+      when 'graphite'
+        return GraphiteDatasource.new(ds_model)
+
+      when 'prometheus'
+        return PrometheusDatasource.new(ds_model)
+
+      end
+
+      raise DatasourceTypeNotSupportedError.new(ds_model['name'], ds_model['meta']['id'])
+    end
+
+    def initialize(model)
+      @model = model
+    end
+
+    # @return [String] name of the datasource
+    def name
+      @model['name']
+    end
+
+    # @return [Integer] ID of the datasource
+    def id
+      @model['id'].to_i
+    end
+
+    # @abstract
+    #
+    # Executes a request for the current database with the given options.
+    #
+    # Used format of the response will always be the following:
+    #
+    #   {
+    #     :header => [column_title_1, column_title_2],
+    #     :content => [
+    #                   [row_1_column_1, row_1_column_2],
+    #                   [row_2_column_1, row_2_column_2]
+    #                 ]
+    #   }
+    #
+    # @param query_description [Hash] query description, which will requested:
+    # @option [String] :from +from+ timestamp
+    # @option [String] :to +to+ timestamp
+    # @option [Integer] :timeout expected timeout for the request
+    # @option [WebRequest] :prepared_request prepared web request for relevant {Grafana} instance, if this is needed by datasource
+    # @option [String] :raw_query raw query, which shall be executed. May include variables, which will be replaced before execution
+    # @option [Hash<Variable>] :variables hash of variables, which can potentially be replaced in the given +:raw_query+
+    # @return [Hash] sql result formatted as stated above
+    def request(query_description)
+      raise NotImplementedError
+    end
+
+    # @abstract
+    #
+    # The different datasources supported by grafana use different ways to store the query in the
+    # panel's JSON model. This method extracts a query from that description, that can be used
+    # by the {AbstractDatasource} implementation of the datasource.
+    #
+    # @param panel_query_target [Hash] grafana panel target, which contains the query description
+    # @return [String] query string, which can be used as +raw_query+ in a {#request}
+    def raw_query_from_panel_model(panel_query_target)
+      raise NotImplementedError
+    end
+
+    private
+
+    # Replaces the grafana variables in the given string with their replacement value.
+    #
+    # @param string [String] string in which the variables shall be replaced
+    # @param variables [Hash<String,Variable>] Hash containing the variables, which shall be replaced in the
+    #  given string
+    # @return [String] string in which all variables are properly replaced
+    def replace_variables(string, variables = {})
+      res = string
+      repeat = true
+      repeat_count = 0
+
+      # TODO: find a proper way to replace variables recursively instead of over and over again
+      # TODO: add tests for recursive replacement of variable
+      while repeat && (repeat_count < 3)
+        repeat = false
+        repeat_count += 1
+        variables.each do |var_name, obj|
+          # only set ticks if value is string
+          variable = var_name.gsub(/^var-/, '')
+          res = res.gsub(/(?:\$\{#{variable}(?::(?<format>\w+))?\}|\$#{variable})/) do
+            # TODO: respect datasource requirements for formatting here
+            obj.value_formatted($LAST_MATCH_INFO ? $LAST_MATCH_INFO[:format] : nil)
+          end
+        end
+        repeat = true if res.include?('$')
+      end
+
+      res
+    end
+  end
+end

data/lib/grafana/dashboard.rb

@@ -53,9 +53,7 @@ module Grafana
       list = @model['templating']['list']
       return unless list.is_a? Array
 
-      list.each do |item|
-        @variables << Variable.new(item)
-      end
+      list.each { |item| @variables << Variable.new(item) }
     end
 
     # read panels

data/lib/grafana/errors.rb

@@ -63,4 +63,19 @@ module Grafana
       super('No SQL statement has been specified.')
     end
   end
+
+  # Raised if a datasource shall be queried, which is not (yet) supported by the reporter
+  class DatasourceTypeNotSupportedError < GrafanaError
+    def initialize(name, type)
+      super("The configured datasource with name '#{name}' is of type '#{type}', which is currently "\
+            'not supported by ruby-grafana-reporter. It will only be usable in panel image queries.')
+    end
+  end
+
+  # Raised if a datasource shall be queried, which is not (yet) supported by the reporter
+  class InvalidDatasourceQueryProvidedError < GrafanaError
+    def initialize(query)
+      super("The datasource query provided, does not look like a grafana datasource target (received: #{query}).")
+    end
+  end
 end

data/lib/grafana/grafana.rb

@@ -13,12 +13,11 @@ module Grafana
     #   trailing slash, e.g. +https://localhost:3000+.
     # @param key [String] API key for the grafana instance, if required
     # @param opts [Hash] additional options.
-    #   Currently supporting +:logger+ and +:ssl_cert+.
+    #   Currently supporting +:logger+.
     def initialize(base_uri, key = nil, opts = {})
       @base_uri = base_uri
       @key = key
       @dashboards = {}
-      @ssl_cert = opts[:ssl_cert]
       @logger = opts[:logger] || ::Logger.new(nil)
 
       initialize_datasources unless @base_uri.empty?
@@ -31,33 +30,52 @@ module Grafana
     #
     # @return [String] +Admin+, +NON-Admin+ or +Failed+ is returned, depending on the test results
     def test_connection
-      if execute_http_request('/api/datasources').is_a?(Net::HTTPOK)
+      if prepare_request({ relative_url: '/api/datasources' }).execute.is_a?(Net::HTTPOK)
         # we have admin rights
         @logger.warn('Reporter is running with Admin privileges on grafana. This is a potential security risk.')
         return 'Admin'
       end
       # check if we have lower rights
-      return 'Failed' unless execute_http_request('/api/dashboards/home').is_a?(Net::HTTPOK)
+      return 'Failed' unless prepare_request({ relative_url: '/api/dashboards/home' }).execute.is_a?(Net::HTTPOK)
 
       @logger.info('Reporter is running with NON-Admin privileges on grafana.')
       'NON-Admin'
     end
 
-    # Returns the ID of a datasource, which has been queried by the datasource name.
+    # Returns the datasource, which has been queried by the datasource name.
     #
-    # @return [Integer] ID for the specified datasource name
-    def datasource_id(datasource_name)
-      datasource_name ||= 'default'
-      return @datasources[datasource_name] if @datasources[datasource_name]
+    # @param datasource_name [String] name of the searched datasource
+    # @return [Datasource] Datasource for the specified datasource name
+    def datasource_by_name(datasource_name)
+      datasource_name = 'default' if datasource_name.to_s.empty?
+      raise DatasourceDoesNotExistError.new('name', datasource_name) unless @datasources[datasource_name]
 
-      raise DatasourceDoesNotExistError.new('name', datasource_name)
+      @datasources[datasource_name]
     end
 
-    # Returns if the given datasource ID exists for the grafana instance.
+    # Returns the datasource, which has been queried by the datasource id.
     #
-    # @return [Boolean] true if exists, false otherwise
-    def datasource_id_exists?(datasource_id)
-      @datasources.value?(datasource_id)
+    # @param datasource_id [Integer] id of the searched datasource
+    # @return [Datasource] Datasource for the specified datasource id
+    def datasource_by_id(datasource_id)
+      datasource = @datasources.select { |_name, ds| ds.id == datasource_id.to_i }.values.first
+      raise DatasourceDoesNotExistError.new('id', datasource_id) unless datasource
+
+      datasource
+    end
+
+    # @return [Array] Array of dashboard uids within the current grafana object
+    def dashboard_ids
+      response = prepare_request({ relative_url: '/api/search' }).execute
+      return [] unless response.is_a?(Net::HTTPOK)
+
+      dashboards = JSON.parse(response.body)
+
+      dashboards.each do |dashboard|
+        @dashboards[dashboard['uid']] = nil unless @dashboards[dashboard['uid']]
+      end
+
+      @dashboards.keys
     end
 
     # @param dashboard_uid [String] UID of the searched {Dashboard}
@@ -65,54 +83,27 @@ module Grafana
     def dashboard(dashboard_uid)
       return @dashboards[dashboard_uid] unless @dashboards[dashboard_uid].nil?
 
-      response = execute_http_request("/api/dashboards/uid/#{dashboard_uid}")
-      model = JSON.parse(response.body)['dashboard']
-
-      raise DashboardDoesNotExistError, dashboard_uid if model.nil?
+      response = prepare_request({ relative_url: "/api/dashboards/uid/#{dashboard_uid}" }).execute
+      raise DashboardDoesNotExistError, dashboard_uid unless response.is_a?(Net::HTTPOK)
 
       # cache dashboard for reuse
+      model = JSON.parse(response.body)['dashboard']
       @dashboards[dashboard_uid] = Dashboard.new(model, self)
 
       @dashboards[dashboard_uid]
     end
 
-    # Runs a specific HTTP request against the current grafana instance.
+    # Prepares a {WebRequest} object for the current {Grafana} instance, which may be enriched
+    # with further properties and can then run {WebRequest#execute}.
     #
-    # Default (can be overridden, by specifying the options Hash):
-    #   accept: 'application/json'
-    #   request: Net::HTTP::Get
-    #   content_type: 'application/json'
-    #
-    # @param relative_uri [String] relative URL with a leading slash, which shall be queried
-    # @param options [Hash] options, which shall be merged to the request.
-    # @param timeout [Integer] number of seconds to wait, before the http request is cancelled, defaults to 60 seconds
-    def execute_http_request(relative_uri, options = {}, timeout = nil)
-      uri = URI.parse(@base_uri + relative_uri)
-      default_options = { accept: 'application/json', request: Net::HTTP::Get, content_type: 'application/json' }
-      options = default_options.merge(options)
-
-      http = Net::HTTP.new(uri.host, uri.port)
-      if @base_uri =~ /^https/
-        http.use_ssl = true
-        http.verify_mode = OpenSSL::SSL::VERIFY_PEER
-        if @ssl_cert && !File.exist?(@ssl_cert)
-          @logger.warn('SSL certificate file does not exist.')
-        elsif @ssl_cert
-          http.cert_store = OpenSSL::X509::Store.new
-          http.cert_store.set_default_paths
-          http.cert_store.add_file(@ssl_cert)
-        end
-      end
-      http.read_timeout = timeout.to_i if timeout
-
-      request = options[:request].new(uri.request_uri)
-      request['Accept'] = options[:accept]
-      request['Content-Type'] = options[:content_type]
-      request['Authorization'] = "Bearer #{@key}" unless @key.nil?
-      request.body = options[:body]
-
-      @logger.debug("Requesting #{relative_uri} with '#{options[:body]}' and timeout '#{http.read_timeout}'")
-      http.request(request)
+    # @option options [Hash] :relative_url relative URL with a leading slash, which shall be queried
+    # @option options [Hash] :accept
+    # @option options [Hash] :body
+    # @option options [Hash] :content_type
+    # @return [WebRequest] webrequest prepared for execution
+    def prepare_request(options = {})
+      auth = @key ? { authorization: "Bearer #{@key}" } : {}
+      WebRequest.new(@base_uri, auth.merge({ logger: @logger }).merge(options))
     end
 
     private
@@ -120,12 +111,18 @@ module Grafana
     def initialize_datasources
       @datasources = {}
 
-      settings = execute_http_request('/api/frontend/settings')
+      settings = prepare_request({ relative_url: '/api/frontend/settings' }).execute
       return unless settings.is_a?(Net::HTTPOK)
 
       json = JSON.parse(settings.body)
       json['datasources'].select { |_k, v| v['id'].to_i.positive? }.each do |ds_name, ds_value|
-        @datasources[ds_name] = ds_value['id'].to_i
+        begin
+          @datasources[ds_name] = AbstractDatasource.build_instance(ds_value)
+        rescue DatasourceTypeNotSupportedError => e
+          # an unsupported datasource type has been configured in the dashboard
+          # - no worries here
+          @logger.warn(e.message)
+        end
       end
       @datasources['default'] = @datasources[json['defaultDatasource']]
     end