drone-hunter 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 656d8a52817c26e8a3dc5d956cd458b4c3c6ae8354523968bfdfba0acf3d6358
+  data.tar.gz: df60380cea37bcd2817000df50fd60050c1a3e0e8ba6e6675050a74f325007f9
+SHA512:
+  metadata.gz: fd3f1e3932c15708c5afcff7405bfd534f7779c7e36d66f305566cfab96ce0820299a4edacb2069df8b8fd95ba6f0c63574c21ad63c8dbd8faa9d707c99ac2b8
+  data.tar.gz: 46525d26161abbbbd93401936ef6a3a38254d693761f2c26884b97baafcef57c5cc778377af8e184d1015ace32eeff8ab99d709a436e60baeac331048cf2809b

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+
+The MIT License (MIT)
+Copyright © 2022 Chris Olstrom <chris@olstrom.com>
+Copyright © 2022 SUSE LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the “Software”), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,81 @@
+# drone-hunter - finds your dronefiles when you have too many repositories.
+
+## Overview
+
+`drone-hunter` is a small utility that trawls whatever GitHub accounts you give it, looking for Drone CI configs.
+
+## Why does this exist?
+
+Because I needed to search across hundreds of repositories across multiple GitHub organizations, searching for drone configs so I could review and compare them. I considered Terraform for this, but opted for Ruby instead, because caching makes iterative development much faster, and I didn't know how to make Terraform do that. Also the GitHub Provider for Terraform didn't handle full repository names for several data sources.
+
+## How do I obtain this majestic tool?
+
+```shell
+gem install drone-hunter
+```
+
+## How do I use it?
+
+```shell
+drone-hunter <owner ...>
+```
+
+Where `owner` is either a user or an organization on GitHub.
+
+`drone-hunter` will use the GitHub API to find all the repositories that are visible to you in the accounts you indicated. This means that if you're using private repositories, you'll need a `GITHUB_TOKEN` in the environment that has access to whatever repositories you want it to read from.
+
+With this list of repositories, `drone-hunter` will find the default branches for everything, fetch an index of the tree for the commit at the tip of that branch, dig around for stuff that looks like it might be a drone config, and download all of them.
+
+All of this will be cached.
+
+The current version will output something like this (this may change in future releases):
+
+```json
+[
+  {
+    "repository": "rancherlabs/drone-plugin-golangci-lint",
+    "path": ".drone.yml",
+    "sha": "d95f9416c3521bdcb4acc0146d9af9fbe42ff165",
+    "content": "---\nkind: pipeline\nname: golangci-lint\n\nsteps:\n- name: golangci-lint-run\n  image: rancher/drone-golangci-lint:latest\n  failure: ignore\n\n---\nkind: pipeline\nname: docker\n\nsteps:\n- name: publish\n  image: plugins/docker\n  settings:\n    username:\n      from_secret: docker_username\n    password:\n      from_secret: docker_password\n    repo: rancher/drone-golangci-lint\n    tags: latest\n  when:\n    instance:\n      - drone-publish.rancher.io\n    ref:\n      include:\n        - \"refs/heads/*\"\n        - \"refs/tags/*\"\n        - \"refs/pull/*\"\n    event:\n      - push\n      - tag\n\n"
+  }
+]
+```
+
+By default, it produces JSON output. You can change this with the `--output-format` option. Maybe you want YAML, since that's what drone configs are...
+
+```yaml
+- repository: rancherlabs/drone-plugin-golangci-lint
+  path: ".drone.yml"
+  sha: d95f9416c3521bdcb4acc0146d9af9fbe42ff165
+  content: |+
+    ---kind: pipelinename: golangci-lintsteps:- name: golangci-lint-run  image: rancher/drone-golangci-lint:latest  failure: ignore---kind: pipelinename: dockersteps:- name: publish  image: plugins/docker  settings:    username:      from_secret: docker_username    password:      from_secret: docker_password    repo: rancher/drone-golangci-lint    tags: latest  when:    instance:      - drone-publish.rancher.io    ref:      include:        - "refs/heads/*"        - "refs/tags/*"        - "refs/pull/*"    event:      - push      - tag
+```
+
+Or maybe you want to dump a copy of every dronefile onto the filesystem. Set `--output-format=files` and you'll get something like this:
+
+```shell
+find drone-hunter.output -type f
+```
+
+```text
+drone-hunter.output/rancherlabs/drone-plugin-golangci-lint/.drone.yml
+drone-hunter.output/rancherlabs/ssh-pub-keys/.drone.yml
+drone-hunter.output/rancherlabs/drone-runner-laboratory/.drone_lab.yml
+drone-hunter.output/rancherlabs/rio-website/.drone.yml
+drone-hunter.output/rancherlabs/rancher-catalog-stats/.drone.yml
+drone-hunter.output/rancherlabs/drone-plugin-fossa/.drone.yml
+drone-hunter.output/rancherlabs/swiss-army-knife/.drone.ci.yml
+drone-hunter.output/rancherlabs/swiss-army-knife/.drone.yml
+drone-hunter.output/rancherlabs/huawei-ui/.drone.yml
+drone-hunter.output/rancherlabs/drone-runner-docker/.drone.yml
+drone-hunter.output/rancherlabs/k3s-website/.drone.yml
+drone-hunter.output/rancherlabs/k3os-website/.drone.yml
+drone-hunter.output/rancherlabs/website-theme/.drone.yml
+drone-hunter.output/rancherlabs/support-tools/.drone.yml
+```
+
+The only limits are your imagination (and the GitHub API Rate Limit).
+
+## License
+
+`drone-hunter` is available under the [MIT License](https://tldrlegal.com/license/mit-license). See `LICENSE.txt` for the full text.

data/bin/drone-hunter

@@ -0,0 +1,151 @@
+#! /usr/bin/env ruby
+# SPDX-License-Identifier: MIT
+
+#########################
+# Ruby Standard Library #
+#########################
+
+require "logger"
+require "optparse"
+
+#########################
+# Internal Dependencies #
+#########################
+
+require_relative "../lib/drone_hunter"
+
+def github_access_token_from_environment
+    # prefer a token designated for this program
+    ENV.fetch("DRONE_HUNTER_GITHUB_ACCESS_TOKEN") do
+        # but accept one that isn't designated for any particular use
+        ENV.fetch("GITHUB_ACCESS_TOKEN") do
+            # or accept one designated for the underlying library
+            ENV.fetch("OCTOKIT_ACCESS_TOKEN") do
+                # or play nice with Terraform's GitHub Provider
+                ENV.fetch("GITHUB_TOKEN") do
+                    # well... a token isn't strictly REQUIRED...
+                    nil
+                end
+            end
+        end
+    end
+end
+
+def output_format_from(input)
+    case input
+    when /j(son)?/i  then :JSON
+    when /y(a?ml)?/i then :YAML
+    when /f(ile)?(s(ystem)?)?/i then :FileSystem
+    else raise NotImplementedError
+    end
+end
+
+def log_level_from(input)
+    case input
+    when /warn/i  then Logger::WARN
+    when /info/i  then Logger::INFO
+    when /debug/i then Logger::DEBUG
+    else raise NotImplementedError
+    end
+end
+
+#########################
+# Default Configuration #
+#########################
+
+config = {
+    hacking: (ENV["HACKING"] ? true : false),
+    cache: {
+        dir: File.expand_path(ENV.fetch("DRONE_HUNTER_CACHE_DIR", './drone-hunter.cache'))
+    },
+    github: {
+        auto_paginate: true,
+        access_token: github_access_token_from_environment
+    },
+    log: {
+        level: log_level_from(ENV.fetch("DRONE_HUNTER_LOG_LEVEL", "info"))
+    },
+    output: {
+        format: output_format_from(ENV.fetch("DRONE_HUNTER_OUTPUT_FORMAT", "json")),
+        path: File.expand_path(ENV.fetch("DRONE_HUNTER_OUTPUT_PATH", "./drone-hunter.output")),
+    }
+}
+
+################################
+# Options from the Commandline #
+################################
+
+OptionParser.new do |options|
+    options.on("-C", "--cache-dir=DIR",             "env: DRONE_HUNTER_CACHE_DIR")           { |argument| config[:cache][:dir]           = File.expand_path(argument) }
+    options.on(      "--github-access-token=TOKEN", "env: DRONE_HUNTER_GITHUB_ACCESS_TOKEN") { |argument| config[:github][:access_token] = argument }
+    options.on("-L", "--log-level=LEVEL",           "env: DRONE_HUNTER_LOG_LEVEL")           { |argument| config[:log][:level]           = log_level_from(argument) }
+    options.on("-o", "--output-format=FORMAT",      "env: DRONE_HUNTER_OUTPUT_FORMAT")       { |argument| config[:output][:format]       = output_format_from(argument) }
+    options.on("-p", "--output-path=PATH",          "env: DRONE_HUNTER_OUTPUT_PATH")         { |argument| config[:output][:path]         = File.expand_path(argument) }
+end.parse!
+
+#################
+# Sanity Checks #
+#################
+
+if ARGV.empty?
+    STDERR.puts "#{File.basename($PROGRAM_NAME)} <owner ...>"
+    exit Errno::EINVAL::Errno 
+end
+
+#################
+# Logging Setup #
+#################
+
+log = Logger::new(STDERR)
+log.level = config[:log][:level]
+
+################
+# GitHub Setup #
+################
+
+# the GitHub API has a pretty low rate limit for unauthenticated calls,
+# and we can exhaust that very quickly when doing what we need to do.
+# Warn the user about this missing configuration as a courtesy.
+unless config.dig(:github, :access_token)
+    log.warn("No GitHub Access Token provided. Expect rate limiting.")
+end
+
+github = Octokit::Client.new(**config[:github])
+
+#################
+# Caching Setup #
+#################
+
+cache = Moneta.new(:File, dir: config[:cache][:dir])
+
+################
+# Main Program #
+################
+
+hunt = DroneHunter.new(owners: ARGV, log: log, github: github, cache: cache)
+
+if config[:hacking]
+    require "pry"
+    binding.pry
+else
+    hunt.dronefiles.then do |dronefiles|
+        case config[:output][:format]
+        when :JSON
+            require "json"
+            STDOUT.puts JSON.pretty_generate dronefiles
+        when :YAML
+            require "yaml"
+            STDOUT.puts YAML.dump dronefiles
+        when :FileSystem
+            require "fileutils"
+            Dir.chdir(File.expand_path(*FileUtils.mkdir_p(config[:output][:path]))) do |output|
+                dronefiles.each do |file|
+                    Dir.chdir(File.expand_path(*FileUtils.mkdir_p(File.join(output, file[:repository])))) do |repo|
+                        File.write(File.join(".", file[:path]), file[:content])
+                    end
+                end
+            end
+        else raise NotImplementedError
+        end
+    end
+end

data/drone-hunter.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |gem|
+    tag = `git describe --tags --abbrev=0`.chomp
+  
+    gem.name          = 'drone-hunter'
+    gem.homepage      = 'https://github.com/colstrom/drone-hunter'
+    gem.summary       = 'Hunts for Drone CI files across many repositories'
+  
+    gem.version       = "#{tag}"
+    gem.licenses      = ['MIT']
+    gem.authors       = ['Chris Olstrom']
+    gem.email         = 'chris@olstrom.com'
+  
+    # gem.cert_chain    = ['trust/certificates/colstrom.pem']
+    # gem.signing_key   = File.expand_path ENV.fetch 'GEM_SIGNING_KEY'
+  
+    gem.files         = `git ls-files -z`.split("\x0")
+    gem.test_files    = `git ls-files -z -- {test,spec,features}/*`.split("\x0")
+    gem.executables   = `git ls-files -z -- bin/*`.split("\x0").map { |f| File.basename(f) }
+  
+    gem.require_paths = ['lib']
+  
+    gem.add_runtime_dependency 'octokit', '~> 4.25', '>= 4.25.1'
+    gem.add_runtime_dependency 'moneta',  '~> 1.5',  '>= 1.5.1'
+end

data/lib/drone_hunter.rb

@@ -0,0 +1,107 @@
+# SPDX-License-Identifier: MIT
+
+#########################
+# Ruby Standard Library #
+#########################
+
+require "base64"
+require "logger"
+require "set"
+
+#########################
+# External Dependencies #
+#########################
+
+require "moneta"
+require "octokit"
+
+class DroneHunter
+    def initialize(**options)
+        @log    ||= options.fetch(:log) { Logger.new(STDERR) }
+        @github ||= options.fetch(:client) { Octokit::Client.new(auto_paginate: true) }
+        @cache  ||= options.fetch(:cache) { Moneta.new(:File, dir: 'drone-hunter.cache') }
+        @owners ||= Set.new(options.fetch(:owners, []))
+    end
+
+    attr_reader :log
+    attr_reader :github
+    attr_reader :cache
+    attr_reader :owners
+
+    def cached(key, *rest, &block)
+        if cache.key?(key)
+            log.debug("(cache) #{key}")
+            cache.fetch(key)
+        else
+            log.info("(fetch) #{key}")
+            cache.store(key, block.call(*rest))
+        end
+    end
+
+    def repositories(owner = nil)
+        case owner
+        when String then cached("repositories/#{owner}") { github.repositories(owner) }
+        when nil    then owners.flat_map { |owner| repositories(owner) }
+        else raise TypeError
+        end
+    end
+
+    def branches(repo = nil)
+        case repo
+        when String            then cached("branches/#{repo}") { github.branches(repo) }
+        when Sawyer::Resource  then branches(repo.full_name)
+        when nil               then repositories.flat_map { |repo| branches(repo) }
+        else raise TypeError
+        end
+    end
+
+    def trees
+        repositories.map do |repo|
+            tree_sha = branches(repo.full_name).find do |branch|
+                branch.name == repo.default_branch
+            end&.commit&.sha
+            
+            next unless tree_sha
+            repo = repo.full_name
+
+            {
+                repo => cached("tree/#{repo}/#{tree_sha}") { github.tree(repo, tree_sha) }.tree
+            }
+        end.compact.reduce(&:merge)
+    end
+
+    def blobs
+        trees.map do |repo, tree|
+            blobs = tree
+                .select { |entry| entry.path.match?(/ya?ml$/) }
+                .select { |entry| entry.path.match?(/drone/) }
+                .map do |entry| 
+                    {
+                        entry.path => cached("blob/#{repo}/#{entry.sha}") { github.blob(repo, entry.sha) }
+                    }
+                end
+            next if blobs.empty?
+
+            {
+                repo => blobs.reduce(&:merge)
+            }
+        end.compact.reduce(&:merge)
+    end
+
+    def dronefiles
+        blobs.flat_map do |repo, blobs|
+            blobs.map do |path, blob|
+                case blob[:encoding]
+                when "base64"
+                    {
+                        "repository" => repo,
+                        "path"       => path,
+                        "sha"        => blob[:sha],
+                        "content"    => Base64::decode64(blob[:content])
+                    }
+                else raise EncodingError
+                end
+            end
+        end
+    end
+end

metadata

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification
+name: drone-hunter
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Chris Olstrom
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2022-08-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: octokit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.25'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.25.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.25'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 4.25.1
+- !ruby/object:Gem::Dependency
+  name: moneta
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.5.1
+description: 
+email: chris@olstrom.com
+executables:
+- drone-hunter
+extensions: []
+extra_rdoc_files: []
+files:
+- Gemfile
+- LICENSE.txt
+- README.md
+- bin/drone-hunter
+- drone-hunter.gemspec
+- lib/drone_hunter.rb
+homepage: https://github.com/colstrom/drone-hunter
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3.1
+signing_key: 
+specification_version: 4
+summary: Hunts for Drone CI files across many repositories
+test_files: []