superset 0.1.6

data/doc/setting_up_personal_api_credentials.md

@@ -0,0 +1,127 @@
+# API Credentials
+
+Superset API Credentials are essentially the username, password and host of the Superset host users account.
+
+If you know these already, plug them and your host value into the `.env` and it should all just work.  A sample env.sample is provided as a template.
+
+Create your own .env file with
+
+```
+cp env.sample .env
+```
+
+Adjust .env as required.
+```
+SUPERSET_HOST="https://your-superset-host.com"
+SUPERSET_API_USERNAME="your-username"
+SUPERSET_API_PASSWORD="your-password"
+```
+
+If you have multiple superset hosts across various environments you also have the option
+to create individual env files per environment.  More details below.
+
+Starting a ruby console with `bin/console` will auto load the env vars.
+
+## What is my user name?
+
+If your Superset instance is setup to authenicate via SSO then your authenticating agent will most likely have provided a username for you in the form of a UUID value.
+
+This is easily retrieved on you User Profile page in Superset.
+
+Optionally use jinja template macro in sql lab.
+
+```
+select '{{ current_username() }}' as current_username;
+```
+
+## Creating / Updating your password via Swagger interface
+
+A common setup is to use SSO to enable user access in Superset.  This would mean your authenticating agent is your SSO provider service ( ie Azure ) and most likely you do not have / need a password configured for your Superset user for GUI access.
+
+If this is the case, you will need to add your own password via hitting the superset API using the swagger interface.
+
+Firstly you will need your superset user id, which is the superset users table PK that is assigned to you.
+
+It appears this user id value is not exposed on the Users profile page in Superset. Depending on your level of access within your Superset instance you could:
+- access the Users List page, find your user, and mouse over the edit button to reveal the Url and user id param value.  
+- got to sql lab and use a jinja template predefined macro to retrieve your users id.
+```
+select '{{ current_user_id() }}' as current_user_id;
+```
+- ask your superset admin to tell you what your personal superset user id is.
+
+Once you have your user id value, open the Swagger API page on you superset host.  
+`https://{your-superset-host}/swagger/v1`
+
+Scroll down to the 'Security Users' area and find the PUT request that will update your users record.
+
+PUT `/api/v1/security/users/{pk}`
+
+Click 'Try it Out' and add your users ID in the PK input box.  
+
+Edit the params to only consist of only the password field and the value of your new password.  
+
+```
+{
+  "password": "{some-long-complex-random-password-value}"
+}
+```
+
+And click Execute.
+
+Within your `.env` now add your username and password.
+
+# Accessing API across Multiple Environments
+
+Given some local development requirements where you have to access multiple superset hosts across various environments with different credentials you can setup the env creds as follows.
+
+Just set the overall superset environment in `.env`
+
+```
+# .env file holding one setting for the overall superset environment
+SUPERSET_ENVIRONMENT='staging'
+```
+
+Then create a new file called `.env-staging` that holds your superset staging host and credentials.
+
+```
+# specific settings for the superset staging host
+SUPERSET_HOST="https://your-staging-superset-host.com"
+SUPERSET_API_USERNAME="staging-user-here"
+SUPERSET_API_PASSWORD="set-password-here"
+```
+
+Do the same for production env.  
+Create a new file called `.env-production` that holds your superset production host and credentials.
+
+```
+# specific settings for the superset production host
+SUPERSET_HOST="https://your-production-superset-host.com"
+SUPERSET_API_USERNAME="production-user-here"
+SUPERSET_API_PASSWORD="set-password-here"
+```
+
+The command `bin/console` will then load your env file depending on the value in ENV['SUPERSET_ENVIRONMENT'] from the primary `.env`.
+
+When you need to switch envs, exit the console, edit the .env to your desired value, eg `production`, then open a console again.
+
+Bonus is the Pry prompt will now also include the `SUPERSET_ENVIRONMENT` value.
+
+```
+bin/console
+ENV configuration loaded from from .env-staging
+[1] (ENV:STAGING)> Superset::Dashboard::List.new(title_contains: 'video').list
+
+Happi: GET https://your-staging-superset-host.com/api/v1/dashboard/?q=(filters:!((col:dashboard_title,opr:ct,value:'video')),page:0,page_size:100), {}
++----+------------------+-----------+------------------------------------------------------------------+
+|                                      Superset::Dashboard::List                                       |
++----+------------------+-----------+------------------------------------------------------------------+
+| Id | Dashboard title  | Status    | Url                                                              |
++----+------------------+-----------+------------------------------------------------------------------+
+| 6  | Video Game Sales | published | https://your-staging-superset-host.com/superset/dashboard/6/ |
++----+------------------+-----------+------------------------------------------------------------------+
+```
+
+
+
+

data/docker-compose.override.yml

@@ -0,0 +1,10 @@
+version: '3'
+
+volumes:
+  bundle-data:
+    driver: local
+
+services:
+  app:
+    volumes:
+      - bundle-data:/usr/local/bundle

data/docker-compose.yml

@@ -0,0 +1,8 @@
+version: '3'
+
+services:
+  app:
+    build: .
+    command: setup
+    volumes:
+      - .:/app

data/env.sample

@@ -0,0 +1,9 @@
+SUPERSET_HOST="https://your-superset-host.com"
+
+# general api calls
+SUPERSET_API_USERNAME="USERNAME"
+SUPERSET_API_PASSWORD="PASSWORD"
+
+# embedded user guest token api calls
+# SUPERSET_EMBEDDED_USERNAME="EMBEDDED-USERNAME"
+# SUPERSET_EMBEDDED_PASSWORD="EMBEDDED-PASSWORD"

data/lib/loggers/duplicate_dashboard_logger.rb

@@ -0,0 +1,15 @@
+module Superset
+  module DuplicateDashboardLogger
+   
+    
+    def logger
+      @logger ||= begin
+        if defined?(Rails)
+          Rails.try(:logger) || Logger.new(STDOUT)
+        else
+          Logger.new(STDOUT)
+        end
+      end
+    end
+  end
+end

data/lib/superset/authenticator.rb

@@ -0,0 +1,55 @@
+module Superset
+  class Authenticator
+    class CredentialMissingError < StandardError; end
+
+    attr_reader :credentials
+
+    def initialize(credentials)
+      @credentials = credentials
+    end
+
+    def self.call(credentials)
+      self.new(credentials).access_token
+    end
+
+    def access_token
+      response_body['access_token']
+    end
+
+    def refresh_token
+      response_body['refresh_token']
+    end
+
+    def validate_credential_existance
+      raise CredentialMissingError, 'password not set' unless credentials[:password].present?
+      raise CredentialMissingError, 'username not set' unless credentials[:username].present?
+    end
+
+    def superset_host
+      raise CredentialMissingError, "SUPERSET_HOST not found" unless ENV['SUPERSET_HOST'].present?
+
+      ENV['SUPERSET_HOST']
+    end
+
+    private
+
+    def response
+      validate_credential_existance
+      @response ||= connection.post(route,
+                                    credentials.to_json,
+                                    'Content-Type' => 'application/json')
+    end
+
+    def route
+      'api/v1/security/login'
+    end
+
+    def response_body
+      JSON.parse(response.env['body'] || response.env['response_body'])
+    end
+
+    def connection
+      Faraday.new(superset_host)
+    end
+  end
+end

data/lib/superset/chart/bulk_delete.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+# TODO: the gui delete has a confirmation step, this API call does not.
+# Potentially we could add a confirm_delete parameter to the constructor that would ensure that all charts either
+#   1 belong to only an expected dashboard before deleting
+#   2 or do not belong to any dashboards
+#  ( not sure if this needed at this point )
+
+module Superset
+  module Chart
+    class BulkDelete < Superset::Request
+      attr_reader :chart_ids
+
+      def initialize(chart_ids: [])
+        @chart_ids = chart_ids
+      end
+
+      def perform
+        raise InvalidParameterError, "chart_ids array of integers expected" unless chart_ids.is_a?(Array)
+        raise InvalidParameterError, "chart_ids array must contain Integer only values" unless chart_ids.all? { |item| item.is_a?(Integer) }
+
+        logger.info("Attempting to delete charts with id: #{chart_ids.join(', ')}")
+        response
+      end
+
+      def response
+        @response ||= client.delete(route, params)
+      end
+
+      private
+
+      def params
+        { q: "!(#{chart_ids.join(',')})" }
+      end
+
+      def route
+        "chart/"
+      end
+    end
+  end
+end

data/lib/superset/chart/delete.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Superset
+  module Chart
+    class Delete < Superset::Request
+      attr_reader :chart_id
+
+      def initialize(chart_id: )
+        @chart_id = chart_id
+      end
+
+      def perform
+        raise InvalidParameterError, "chart_id integer is required" unless chart_id.present? && chart_id.is_a?(Integer)
+
+        logger.info("Attempting to delete chart with id: #{chart_id}")
+        response
+      end
+
+      def response
+        @response ||= client.delete(route)
+      end
+
+      private
+
+      def route
+        "chart/#{chart_id}"
+      end
+    end
+  end
+end

data/lib/superset/chart/get.rb

@@ -0,0 +1,56 @@
+module Superset
+  module Chart
+    class Get < Superset::Request
+
+      attr_reader :id
+
+      def initialize(id)
+        @id = id
+      end
+
+      def self.call(id)
+        self.new(id).list
+      end
+
+      def perform
+        response
+        self
+      end
+
+      def result
+        [ super ]
+      end
+
+      def datasource_id
+        if result.first['query_context'].present? && JSON.parse(result.first['query_context'])['datasource'].present?
+          JSON.parse(result.first['query_context'])['datasource']['id']
+        elsif result.first['params'].present? && JSON.parse(result.first['params'])['datasource'].present?
+          JSON.parse(result.first['params'])['datasource'].match(/^\d+/)[0].to_i
+        end
+      end
+
+      def owner_ids
+        result.first['owners'].map{|o| o['id']}
+      end
+
+      def params
+        JSON.parse(result.first['params']) if result.first['params'].present?
+      end
+
+      def query_context
+        JSON.parse(result.first['query_context']) if result.first['query_context'].present?
+      end
+
+      private
+
+      def route
+        "chart/#{id}"
+      end
+
+      def list_attributes
+        %w(id slice_name owners)
+      end
+
+    end
+  end
+end

data/lib/superset/chart/list.rb

@@ -0,0 +1,59 @@
+module Superset
+  module Chart
+    class List < Superset::Request
+
+      attr_reader :name_contains, :dashboard_id_eq, :dataset_id_eq
+
+      def initialize(page_num: 0, name_contains: '', dashboard_id_eq: '', dataset_id_eq: '')
+        @name_contains = name_contains
+        @dashboard_id_eq = dashboard_id_eq
+        @dataset_id_eq = dataset_id_eq
+        super(page_num: page_num)
+      end
+
+      def self.call
+        self.new.list
+      end
+
+      def list_with_dashboards
+        puts Terminal::Table.new(
+          title: title,
+          headings: list_dashboard_attributes.map(&:to_s).map(&:humanize),
+          rows: rows_with_dashboards
+        )
+      end
+
+      def rows_with_dashboards
+        result.map do |d|
+          list_dashboard_attributes.map { |la| d[la].to_s }
+        end
+      end
+
+      private
+
+      def route
+        "chart/?q=(#{query_params})"
+      end
+
+      def filters
+        # TODO filtering across all classes needs a refactor to support multiple options in a more flexible way
+        filter_set = []
+        filter_set << "(col:slice_name,opr:ct,value:'#{name_contains}')" if name_contains.present?
+        filter_set << "(col:dashboards,opr:rel_m_m,value:#{dashboard_id_eq})" if dashboard_id_eq.present? # rel many to many
+        filter_set << "(col:datasource_id,opr:eq,value:#{dataset_id_eq})" if dataset_id_eq.present?
+
+        unless filter_set.empty?
+          "filters:!(" + filter_set.join(',') + "),"
+        end
+      end
+
+      def list_attributes
+        ['id', 'slice_name', 'datasource_id', 'datasource_name_text', 'created_by_name']
+      end
+
+      def list_dashboard_attributes
+        ['id', 'slice_name', 'dashboards']
+      end
+    end
+  end
+end

data/lib/superset/chart/update_dataset.rb

@@ -0,0 +1,90 @@
+# In the context of a chart, dataset and datasource are the same thing
+
+module Superset
+  module Chart
+    class UpdateDataset < Superset::Request
+
+      attr_reader :chart_id, :target_dataset_id, :target_dashboard_id
+
+      def initialize(chart_id: , target_dataset_id: , target_dashboard_id: nil)
+        @chart_id = chart_id
+        @target_dataset_id = target_dataset_id
+        @target_dashboard_id = target_dashboard_id
+      end
+
+      def perform
+        validate_proposed_changes
+
+        response
+
+        if result['datasource_id'] == target_dataset_id
+          "Successfully updated chart #{chart_id} to the target dataset #{target_dataset_id}"
+        else
+          "Error: Failed to update chart #{chart_id} to #{target_dataset_id}"
+        end
+
+      end
+
+      def response
+        @response ||= client.put(route, params_updated)
+      end
+
+      def params_updated
+        @params_updated ||= begin
+          new_params = {}
+          # chart updates to point to the new target dataset
+          new_params.merge!("datasource_id": target_dataset_id)          # point to the new dataset id
+          new_params.merge!("datasource_type": 'table')                  # type of dataset ?? not sure of other options
+          new_params.merge!("owners": chart.owner_ids )                  # expects an array of user ids
+
+          new_params.merge!("params": updated_chart_params.to_json)      # updated to point to the new params
+          query_context = updated_chart_query_context
+          if query_context
+            new_params.merge!("query_context": query_context.to_json) # update to point to the new query context
+            new_params.merge!("query_context_generation": true)            # new param set to true to regenerate the query context
+          end
+         
+          new_params
+        end
+      end
+
+      private
+
+      def chart
+        @chart ||= Superset::Chart::Get.new(chart_id).perform
+      end
+
+      def validate_proposed_changes
+        raise "Error: chart_id integer is required" unless chart_id.present? && chart_id.is_a?(Integer)
+        raise "Error: target_dataset_id integer is required" unless target_dataset_id.present? && target_dataset_id.is_a?(Integer)
+        # validate schema ???
+      end
+
+      def updated_chart_params
+        chart_params = chart.params # init with current chart params
+        chart_params['datasource'] = chart_params['datasource'].sub(source_dataset_id.to_s, target_dataset_id.to_s) # update to point to the new dataset
+        chart_params['dashboards'] = [target_dashboard_id] if target_dashboard_id.present?                          # update to point to the new dashboard
+        chart_params
+      end
+
+      def updated_chart_query_context
+        if chart.query_context.present?                                           # not all charts have a query context
+          chart_query_context = chart.query_context                               # init with source chart query context
+          chart_query_context['datasource']['id'] = target_dataset_id             # update to point to the new dataset
+          chart_query_context['form_data']['datasource'] = chart_query_context['form_data']['datasource']
+            .sub(source_dataset_id.to_s, target_dataset_id.to_s)                  # update to point to the new dataset
+            chart_query_context['form_data']['dashboards'] = [target_dashboard_id] if target_dashboard_id.present? # update to point to the new dashboard
+          chart_query_context
+        end
+      end
+
+      def source_dataset_id
+        chart.datasource_id
+      end
+
+      def route
+        "chart/#{chart_id}"
+      end
+    end
+  end
+end

data/lib/superset/client.rb

@@ -0,0 +1,53 @@
+module Superset
+  class Client < Happi::Client
+    include Credential::ApiUser
+
+    attr_reader :authenticator
+
+    def initialize
+      @authenticator = Superset::Authenticator.new(credentials)
+      super(log_level: :debug, host: superset_host)
+    end
+
+    def access_token
+      @access_token ||= authenticator.access_token
+    end
+
+    def superset_host
+      @superset_host ||= authenticator.superset_host
+    end
+
+    # TODO: Happi has not got a put method yet
+    def put(resource, params = {})
+      call(:put, url(resource), param_check(params))
+        .body.with_indifferent_access
+    end
+
+    # TODO: Happi is not surfacing the errors correctly overriding raise_error for now
+    def raise_error(response)
+      message =
+      if response.body['errors']
+        response.body['errors']
+      else
+        response.body
+      end
+
+      puts "API Error: #{message}"  # display the error message for console debugging
+      # binding.pry                 # helpfull to debug the response
+
+      raise errors[response.status].new(message, response)  # message is not being surfaced from Happi correctly, :(
+    end
+
+    private
+
+    def connection
+      @connection ||= Faraday.new(superset_host) do |f|
+        f.authorization :Bearer, access_token
+        f.use FaradayMiddleware::ParseJson, content_type: 'application/json'
+        f.request :json
+
+        f.adapter :net_http
+      end
+    end
+  end
+end

data/lib/superset/credential/api_user.rb

@@ -0,0 +1,25 @@
+module Superset
+  module Credential
+    module ApiUser
+
+      def credentials
+        {
+          "username": api_username,
+          "password": api_password,
+          "provider": "db",
+          "refresh":  false
+        }
+      end
+
+      private
+
+      def api_username
+        ENV['SUPERSET_API_USERNAME']
+      end
+
+      def api_password
+        ENV['SUPERSET_API_PASSWORD']
+      end
+    end
+  end
+end

data/lib/superset/credential/embedded_user.rb

@@ -0,0 +1,25 @@
+module Superset
+  module Credential
+    module EmbeddedUser
+
+      def credentials
+        {
+          "username": embedded_username,
+          "password": embedded_password,
+          "provider": "db",
+          "refresh":  false
+        }
+      end
+
+      private
+
+      def embedded_username
+        ENV['SUPERSET_EMBEDDED_USERNAME']
+      end
+
+      def embedded_password
+        ENV['SUPERSET_EMBEDDED_PASSWORD']
+      end
+    end
+  end
+end

data/lib/superset/dashboard/bulk_delete.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+# TODO: the gui delete has a confirmation step, this API call does not.
+# Potentially we could add a confirm_delete parameter to the constructor that would ensure that all dashboards either
+#   1 have an expected set of charts or filters before deleting
+#   2 or do not have any charts or filters linked to them
+#  ( not sure if this needed at this point )
+
+# NOTE: deletes the Dashboard Only. Use Dashboard::BulkDeleteCascade to delete all related objects
+module Superset
+  module Dashboard
+    class BulkDelete < Superset::Request
+      attr_reader :dashboard_ids
+
+      def initialize(dashboard_ids: [])
+        @dashboard_ids = dashboard_ids
+      end
+
+      def perform
+        raise InvalidParameterError, "dashboard_ids array of integers expected" unless dashboard_ids.is_a?(Array)
+        raise InvalidParameterError, "dashboard_ids array must contain Integer only values" unless dashboard_ids.all? { |item| item.is_a?(Integer) }
+
+        logger.info("Attempting to delete dashboards with id: #{dashboard_ids.join(', ')}")
+        response
+      end
+
+      def response
+        @response ||= client.delete(route, params)
+      end
+
+      private
+
+      def params
+        { q: "!(#{dashboard_ids.join(',')})" }
+      end
+
+      def route
+        "dashboard/"
+      end
+    end
+  end
+end