superset 0.1.6

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 16bb352b533737ed4faed7d5b4360c850619d60b7997b12638019681a8b36bfd
+  data.tar.gz: e7dd50138ea34bd23969a61bf6f69f7eb98cc6174f67abdb05fdb9ba4eb60197
+SHA512:
+  metadata.gz: ed7199d606cb33b4f30e79d6816ccd2d00c63061c88d630b8e9b538e540cbf4ae3ea50bfccf3acb8ba7e6d92ba09e51871a58b0ea3a84cbfc37d16641923d737
+  data.tar.gz: 50e1ed97a6fb3b698136c313b0384755f42eecf6b171aa7a02021919e59fc9d2bed6991fb03129d8f37a47df6ca4d825b542fe8d36b1410d1bbd284570014e01

data/.buildkite/pipeline.yml

@@ -0,0 +1,16 @@
+steps:
+  - label: ":rspec:"
+    command: bin/setup && bundle exec rspec
+    plugins:
+      docker-compose#v3.0.3:
+        run: app
+        volumes:
+          - ./coverage:/app/coverage
+          - ./log:/app/log
+    timeout: 10
+    agents:
+      queue: docker-heavy
+    artifact_paths:
+      - "coverage/.resultset*.json"
+      - "log/*.log"
+      - "log/*.xml"

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,13 @@
+AllCops:
+  TargetRubyVersion: 2.6
+
+Style/StringLiterals:
+  Enabled: true
+  EnforcedStyle: double_quotes
+
+Style/StringLiteralsInInterpolation:
+  Enabled: true
+  EnforcedStyle: double_quotes
+
+Layout/LineLength:
+  Max: 120

data/CHANGELOG.md

@@ -0,0 +1,48 @@
+## Change Log
+
+## 0.1.6 - 2024-07-10
+
+* added a class **WarmUpCache** to hit the 'api/v1/dataset/warm_up_cache' endpoint to warm up the cache of all the datasets for a particular dashaboard being passed to the class - https://github.com/rdytech/superset-client/pull/28
+
+## 0.1.5 - 2024-05-10
+
+* add multi config for multi env creds https://github.com/rdytech/superset-client/pull/22
+* add endpoint for sqllab/execute https://github.com/rdytech/superset-client/pull/22
+* add endpoint for database/list https://github.com/rdytech/superset-client/pull/22
+* add delete cascade endpoint by @jbat in https://github.com/rdytech/superset-client/pull/21
+
+## 0.1.4 - 2024-05-01
+
+* Filter dashboards by array of tags by @jbat in https://github.com/rdytech/superset-client/pull/20
+* adds endpoints for Delete of dashboards, charts, datasets @jbat in https://github.com/rdytech/superset-client/pull/20
+* adds endpoints for BulkDelete of dashboards, charts, datasets @jbat in https://github.com/rdytech/superset-client/pull/20
+
+## 0.1.3 - 2024-04-23
+
+* duplicate dashboard should also create embedded setting by @jbat in https://github.com/rdytech/superset-client/pull/14
+* Validate and duplicate filters to new dashboard by @vidishaweddy-readytech in https://github.com/rdytech/superset-client/pull/17
+* duplicate cross filters by @jbat in https://github.com/rdytech/superset-client/pull/18
+
+## 0.1.2 - 2024-03-22
+
+* adds export endpoint
+
+## 0.1.1 - 2024-03-14
+
+* superset pipeline part 1 with supported endpoints by @jbat in https://github.com/rdytech/superset-client/pull/4
+* API update chart to new dataset by @hanpeic in https://github.com/rdytech/superset-client/pull/5
+* Adds DuplicateDashboard class and fixes by @jbat in https://github.com/rdytech/superset-client/pull/6
+* Update Docs by @jbat in https://github.com/rdytech/superset-client/pull/7
+* update cred docs by @jbat in https://github.com/rdytech/superset-client/pull/8
+* more updates to DuplicateDashboard, extra endpoints by @jbat in https://github.com/rdytech/superset-client/pull/9
+
+## 0.1.0 - 2023-12-12
+
+- add base classes for credentials, authentication, client, request
+- add dashboard endpoints
+- add security/user endpoints
+- add security/role endpoints
+- add security/role/permission endpoints
+
+
+

data/Dockerfile

@@ -0,0 +1,17 @@
+# 1: Use ruby 2.7 as base:
+FROM ruby:2.7
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    build-essential
+
+ENV BUNDLER_VERSION='2.4.22'
+RUN gem install bundler -v ${BUNDLER_VERSION}
+
+# 2: We'll set the application path as the working directory
+WORKDIR /app
+
+# 3: We'll add the app's binaries path to $PATH:
+ENV PATH=$PATH:/app/bin
+
+ENV BUNDLE_GITHUB__HTTPS=true BUNDLE_MAJOR_DEPRECATIONS=true

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 Jonathon Batson, ReadyTech
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,205 @@
+# Superset Client
+
+[![Build status](https://badge.buildkite.com/fc7ee4a03e119a5d859472865fc0bdc9a6e46d51b7f5b8cd62.svg)](https://buildkite.com/jobready/superset-client)
+
+The Repo is `superset-client` with the ruby gem named `superset`
+
+All ruby classes are namespaced under `Superset::`
+
+# Installation
+
+## Docker Setup
+
+Build, bundle and open a ruby console
+
+```
+docker-compose build
+docker-compose run --rm app bundle install
+docker-compose run --rm app bin/console
+```
+
+Run specs
+
+```
+docker-compose run --rm app rspec
+# or 
+docker-compose run --rm app bash      # then run 'bundle exec rspec' from the container.
+```
+
+## Local setup or including in a Ruby/Rails app
+
+Add to your Gemfile `gem 'superset'`  
+And then execute: `bundle install`  
+Or install it yourself as `gem install superset`
+
+## Setup API Credentials
+
+Follow this doc setup your users API creds [setting_up_personal_api_credentials](https://github.com/rdytech/superset-client/tree/develop/doc/setting_up_personal_api_credentials.md)
+
+Short version is .. copy the `env.sample` to `.env` and add edit values where applicable.  Opening a console with `bin/console` will then auto load the `.env` file.
+
+## Usage
+
+Experiment with the API calls directly by open a pry console using  `bin/console`
+
+
+
+
+### API calls
+
+Quickstart examples
+
+```ruby
+Superset::Database::List.call
+Superset::Database::GetSchemas.new(1).list # get schemas for database 1
+
+Superset::Dashboard::List.call
+Superset::Dashboard::List.new(title_contains: 'Sales').list
+
+Superset::Dashboard::BulkDelete.new(dashboard_ids: [1,2,3]).perform        # Dashboards only ( leaves all charts, datasets in place)
+Superset::Dashboard::BulkDeleteCascade.new(dashboard_ids: [1,2,3]).perform # Dashboards and related charts and datasets.
+
+Superset::Sqllab::Execute.new(database_id: 1, schema: 'public', query: 'select count(*) from birth_names').perform
+
+Superset::Dashboard::Export.new(dashboard_id: 1, destination_path: '/tmp').perform
+
+Superset::RouteInfo.new(route: 'dashboard/_info').perform # Get info on an API endpoint .. handy for getting available filters
+Superset::RouteInfo.new(route: 'chart/_info').filters     # OR just get the filters for an endpoint
+
+superset_class_list # helper method to list all classes under Superset::
+
+```
+
+### Duplicating Dashboards
+
+Primary motivation behind this library was to use the Superset API to duplicate dashboards, charts, datasets across multiple database connections.  
+See examples in [Duplicate Dashboards](https://github.com/rdytech/superset-client/tree/develop/doc/duplicate_dashboards.md)
+
+### API Examples with results
+
+Generally classes follow the convention/path of the Superset API strucuture as per the swagger docs.
+
+ref https://superset.apache.org/docs/api/
+
+Limited support for filters is available on some list pages, primarily through param `title_contains`.  
+Pagination is supported via `page_num` param.
+
+Primary methods across majority of api calls are
+- response : the full API response
+- result : just the result attribute array
+- list : displays a formatted output to console, handy for quick investigation of objects
+- call : is a class method to list on Get and List requests
+
+```ruby
+# List all Databases
+Superset::Database::List.call
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/database/?q=(page:0,page_size:100), {}
++----+------------------------------------+------------+------------------+
+|                        Superset::Database::List                         |
++----+------------------------------------+------------+------------------+
+| Id | Database name                      | Backend    | Expose in sqllab |
++----+------------------------------------+------------+------------------+
+| 1  | examples                           | postgresql | true             |
++----+------------------------------------+------------+------------------+
+
+# List database schemas for Database 1
+Superset::Database::GetSchemas.new(1).list
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/database/1/schemas/, {}
+=> ["information_schema", "public"]
+
+# List dashboards
+Superset::Dashboard::List.call
+# PAGE_SIZE is set to 100, so get the second set of 100 dashboards with
+Superset::Dashboard::List.new(page_num: 1).list
+# OR filter by title
+Superset::Dashboard::List.new(title_contains: 'Sales').list
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/dashboard/?q=(filters:!((col:dashboard_title,opr:ct,value:'Sales')),page:0,page_size:100), {}
+
++-----+------------------------------+-----------+--------------------------------------------------------------------+
+|                                              Superset::Dashboard::List                                              |
++-----+------------------------------+-----------+--------------------------------------------------------------------+
+| Id  | Dashboard title              | Status    | Url                                                                |
++-----+------------------------------+-----------+--------------------------------------------------------------------+
+| 6   | Video Game Sales             | published | https://your-superset-host/superset/dashboard/6/                   |
+| 8   | Sales Dashboard              | published | https://your-superset-host/superset/dashboard/8/                   |
++-----+------------------------------+-----------+--------------------------------------------------------------------+
+
+
+Superset::Dashboard::Get.call(1)  # same as Superset::Dashboard::Get.new(1).list
++----------------------------+
+|     World Banks Data      |
++----------------------------+
+| Charts                     |
++----------------------------+
+| % Rural                    |
+| Region Filter              |
+| Life Expectancy VS Rural % |
+| Box plot                   |
+| Most Populated Countries   |
+| Worlds Population          |
+| Worlds Pop Growth          |
+| Rural Breakdown            |
+| Treemap                    |
+| Growth Rate                |
++----------------------------+
+
+
+```
+## Optional Credential setup for Embedded User
+
+Primary usage is for api calls and/or for guest token retrieval when setting up applications to use the superset embedded dashboard workflow.
+
+The Superset API users fall into 2 categories  
+- a user for general api calls to endpoints for Dashboards, Datasets, Charts, Users, Roles etc.  
+  ref `Superset::Credential::ApiUser`  
+  which pulls credentials from  `ENV['SUPERSET_API_USERNAME']` and `ENV['SUPERSET_API_PASSWORD']`
+
+- a user for guest token api call to use when embedding dashboards in a host application.  
+  ref `Superset::Credential::EmbeddedUser`
+  which pulls credentials from  `ENV['SUPERSET_EMBEDDED_USERNAME']` and `ENV['SUPERSET_EMBEDDED_PASSWORD']`
+
+
+### Fetch a Guest Token for Embedded user
+
+Assuming you have setup your Dashboard in Superset to be embedded and that your creds are setup in  
+`ENV['SUPERSET_EMBEDDED_USERNAME']` and `ENV['SUPERSET_EMBEDDED_PASSWORD']`
+
+```
+Superset::GuestToken.new(embedded_dashboard_id: '15').guest_token
+=> "eyJ0eXAiOi............VV4mrMfsvg"
+```
+
+## Releasing a new version
+
+On develop branch make sure to update `Superset::VERSION` and `CHANGELOG.md` with the new version number and changes.  
+Build the new version and upload to gemfury.
+
+`gem build superset.gemspec`
+
+### Publishing to RubyGems
+
+WIP .. direction is for this gem to be made public
+
+### ReadyTech private Gemfury repo
+
+ReadyTech hosts its own private gemfury remote repo.
+
+Get the latest develop into master
+
+    git checkout master
+    git pull
+    git fetch
+    git merge origin/develop
+
+Tag the version and push to github
+
+    git tag -a -m "Version 0.1.0" v0.1.0
+    git push origin master --tags
+
+Push to gemfury or upload the build manually in the gemfury site.
+
+    git push fury master
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/doc/duplicate_dashboards.md

@@ -0,0 +1,214 @@
+# Duplicating Dashboards to a new database or schema
+
+The problem!
+
+Given a multi tenant configuration where clients databases are logically separate dbs and or schemas all configured in Superset,  
+copying or duplicating an existing Superset dashboard to a new database is a fairly laborious operation with many manual steps.
+
+Usually run in the Superset GUI something like :
+- Create Dashboard (template)
+- Duplicate the template in GUI with Edit->Save As (duplicate charts checked)
+- Duplicate all Datasets from template, Edit each and point to the new target database
+- Edit all the new charts created and link them to the new datasets.
+- Setup the embedded settings for new dashboard
+- Add new tags for the new dashboard
+
+Give the need for multiple clients, 10s or 100s, or 1000s, this quickly becomes a laborious and time consuming feat.
+
+Superset API to the rescue.
+
+Note .. requires setup of your [superset environment credentials](https://github.com/rdytech/superset-client/blob/develop/doc/setting_up_personal_api_credentials.md)
+
+## The Solution
+
+Essentially we perform the same points above but all via the Superset API.
+
+The examples video game dashboard was adjusted to have only 1 chart.  
+Output is new new_dashboard_id and url.  Logs provided in `log/superset-client.log`
+
+Given you have a dashboard created, ie `source_dashboard_id`  
+and you know your `target_schema`  
+as well as your `target_database_id`  
+then you could go ahead and run something like this.  
+
+
+
+```ruby
+Superset::Services::DuplicateDashboard.new(
+    source_dashboard_id: 90,
+    target_schema: 'public',
+    target_database_id: 2
+  ).perform
+
+=> {:new_dashboard_id=>401, :new_dashboard_url=>"https://your-superset-host/superset/dashboard/401/", :published=>false}
+
+# logfile shows the steps taken
+
+# cat log/superset-client.log
+# INFO -- : >>>>>>>>>>>>>>>>> Starting DuplicateDashboard Service <<<<<<<<<<<<<<<<<<<<<<
+# INFO -- : Source Dashboard URL: https://your-superset-host/superset/dashboard/90/
+# INFO -- : Duplicating dashboard 90 into Target Schema: public in database 2
+# INFO -- :   Copy Dashboard/Charts Completed - New Dashboard ID: 401
+# INFO -- : Duplicating Source Dataset examples.video_game_sales with id 11
+# INFO -- :     Finished. Duplicate Dataset Name video_game_sales-example_two with id 542
+# INFO -- :     Validating Dataset ID: 542 schema update to public on Database: 2
+# INFO -- :     Successfully updated dataset schema to public on Database: 2
+# INFO -- : Updating Charts to point to New Datasets and updating Dashboard json_metadata ...
+# INFO -- :   Update Chart 55752 to new dataset_id 542
+# INFO -- :   Updated new Dashboard json_metadata charts with new dataset ids
+# INFO -- : Duplication Successful. New Dashboard URL: https://your-superset-host/superset/dashboard/401/
+# INFO -- : >>>>>>>>>>>>>>>>> Finished DuplicateDashboard Service <<<<<<<<<<<<<<<<<<<<<<
+
+```
+
+## Other options for embedded workflow and tags
+
+If your using the embedded dashboards you can also provied attributes for 
+- allowed domains for embeded dashboard settings
+- database tags for ease of searching
+- option to publish
+
+```ruby
+Superset::Services::DuplicateDashboard.new(
+    source_dashboard_id: 37,
+    target_schema: 'public',
+    target_database_id: 2,
+    allowed_domains: ["https://wylee-coyote-domain/"],
+    tags: ["product:acme_fu", "client:wylee_coyote", "embedded"],
+    publish: true
+  ).perform
+```
+
+### What is my Database ID ?
+
+``` ruby
+# list your available databases with
+Superset::Database::List.call
+
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/database/?q=(page:0,page_size:100), {}
++----+------------------------------------+------------+------------------+
+|                        Superset::Database::List                         |
++----+------------------------------------+------------+------------------+
+| Id | Database name                      | Backend    | Expose in sqllab |
++----+------------------------------------+------------+------------------+
+| 1  | examples                           | postgresql | true             |
+| 2  | examples_two                       | postgresql | true             |
++----+------------------------------------+------------+------------------+
+
+# optionally provide a title filter
+Superset::Database::List.new(title_contains: 'examples_two').list
+
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/database/?q=(filters:!((col:database_name,opr:ct,value:'examples')),page:0,page_size:100), {}
++----+------------------------------------+------------+------------------+
+|                        Superset::Database::List                         |
++----+------------------------------------+------------+------------------+
+| Id | Database name                      | Backend    | Expose in sqllab |
++----+------------------------------------+------------+------------------+
+| 2  | examples_two                       | postgresql | true             |
++----+------------------------------------+------------+------------------+
+
+```
+
+### What Dashboards do I have access to ?
+
+```ruby
+# list dashboard with
+Superset::Dashboard::List.call
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/dashboard/?q=(page:0,page_size:100), {}
++-----+------------------------------------------------------+-----------+--------------------------------------------------------------------+
+|                                                          Superset::Dashboard::List                                                          |
++-----+------------------------------------------------------+-----------+--------------------------------------------------------------------+
+| Id  | Dashboard title                                      | Status    | Url                                                                |
++-----+------------------------------------------------------+-----------+--------------------------------------------------------------------+
+| 20  | Baby Names                                           | draft     | https://your-superset-host/superset/dashboard/20/                  |
+| 6   | Video Game Sales                                     | published | https://your-superset-host/superset/dashboard/6/                   |
+| 5   | COVID Vaccine Dashboard                              | published | https://your-superset-host/superset/dashboard/5/                   |
+| 9   | Superset Project Slack Dashboard                     | published | https://your-superset-host/superset/dashboard/9/                   |
++-----+------------------------------------------------------+-----------+--------------------------------------------------------------------+
+
+# or filter by title
+Superset::Dashboard::List.new(title_contains: 'video').list
+# DEBUG -- : Happi: GET https://your-superset-host/api/v1/dashboard/?q=(filters:!((col:dashboard_title,opr:ct,value:'video')),page:0,page_size:100), {}
++----+------------------+-----------+------------------------------------------------------------------+
+|                                      Superset::Dashboard::List                                       |
++----+------------------+-----------+------------------------------------------------------------------+
+| Id | Dashboard title  | Status    | Url                                                              |
++----+------------------+-----------+------------------------------------------------------------------+
+| 6  | Video Game Sales | published | https://your-superset-host/superset/dashboard/6/ |
++----+------------------+-----------+------------------------------------------------------------------+
+
+```
+
+### Replicate a Dashboard across all schemas
+
+With a bit of ruby ...
+
+Duplicate dashboard across all schemas in acme pools 1,2,3.
+
+```ruby
+acme_dbs = Superset::Database::List.new(title_contains: 'acme').rows
+=> [["7", "acme-pool1", "postgresql", "true"],
+    ["8", "acme-pool2", "postgresql", "true"], 
+    ["9", "acme-pool3", "postgresql", "true"]]
+
+ignore_system_tables = ['information_schema', 'shared_extensions']   # postgres system schemas
+
+db_with_schemas = acme_dbs.map do |db_conn|
+  Superset::Database::GetSchemas.new(db_conn[0]).list.map do |schema|
+    { database_id: db_conn[0], schema: schema, database_name: db_conn[1] } unless ignore_system_tables.include?(schema)
+  end.compact
+end.flatten
+
+=>[{:database_id=>"7", :schema=>"client1", :database_name=>"acme-pool1"},
+   {:database_id=>"7", :schema=>"client2", :database_name=>"acme-pool1"},
+   {:database_id=>"7", :schema=>"client3", :database_name=>"acme-pool1"},
+   {:database_id=>"8", :schema=>"client4", :database_name=>"acme-pool1"},
+   {:database_id=>"8", :schema=>"client5", :database_name=>"acme-pool2"},
+   {:database_id=>"8", :schema=>"client6", :database_name=>"acme-pool2"},
+   {:database_id=>"9", :schema=>"client7", :database_name=>"acme-pool3"},
+   {:database_id=>"9", :schema=>"client8", :database_name=>"acme-pool3"},
+   {:database_id=>"9", :schema=>"client9", :database_name=>"acme-pool3"},
+   {:database_id=>"9", :schema=>"client10", :database_name=>"acme-pool3"}]
+
+db_with_schemas.each do |conn|
+  Superset::Services::DuplicateDashboard.new(
+    source_dashboard_id: 90,
+    target_schema:       conn[:schema],
+    target_database_id:  conn[:database_id]
+  ).perform
+end
+
+```
+
+## TODO / ISSUES
+
+### Handling Change
+
+Dashboards are an ever evolving animal and they are expected to change.
+
+This raises the question, given I have a template dashboard and I have X number of replicas of that dashboard  
+how do I make a change to the template and get the change updated to each of the replicas?
+
+Current direction is to separate these "changes" in to 2 categories.
+
+- Firstly: minor chages to a Dataset query that will not result in breaking a chart.  
+  ie .. adjusting the logic of the query but not the output attributes.
+- Secondly: ... everything else.
+  ie .. editing/adding charts, formating the dashboard, updating the datasets with new attributes for new charts
+
+For the first case, the Superset API can easily locate each Dashboards Dataset and update the query with the changes.  
+This is a fairly simple procedure.
+
+For the second case, currently we can see no easy/clear direction forward.  
+Very happy to have others with more experience in Superset pose suggestions.
+
+Putting it simply, the current thinking is to delete all the replica dashboards and recreate them.
+
+### Bringing the Duplicate Dashboard process into Superset core
+
+(WIP) The goal would be to have the DuplicateDashboard process as a part of the core superset codebase.
+
+To that end this Superset Improvement Proposal (SIP) .. is a starting point.
+
+{add SIP request here}
+