github-ripper 0.1.0

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+
+This changelog was automatically generated using [Caretaker](https://github.com/DevelopersToolbox/caretaker) by [Wolf Software](https://github.com/WolfSoftware)
+
+### [v0.1.0](https://github.com/DevelopersToolbox/github-ripper/releases/v0.1.0)
+
+> Released on March, 12th 2021
+
+- The initial commit [`[head]`](https://github.com/DevelopersToolbox/github-ripper/commit/)
+

data/Gemfile

@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+
+git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
+
+# Specify your gem's dependencies in slackit.gemspec
+gemspec

data/LICENSE.md

@@ -0,0 +1,25 @@
+The MIT License (MIT)
+=====================
+
+Copyright © `2009-2021` `Wolf Software Limited`
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the “Software”), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,139 @@
+<h1 align="center">
+	<a href="https://github.com/WolfSoftware">
+		<img src="https://raw.githubusercontent.com/WolfSoftware/branding/master/images/general/banners/64/black-and-white.png" alt="Wolf Software Logo" />
+	</a>
+	<br>
+	GitHub Ripper - Bulk Cloning Tool
+</h1>
+
+<p align="center">
+	<a href="https://travis-ci.com/DevelopersToolbox/github-ripper">
+		<img src="https://img.shields.io/travis/com/DevelopersToolbox/github-ripper/master?style=for-the-badge&logo=travis" alt="Build Status">
+	</a>
+	<a href="https://github.com/DevelopersToolbox/github-ripper/releases/latest">
+		<img src="https://img.shields.io/github/v/release/DevelopersToolbox/github-ripper?color=blue&style=for-the-badge&logo=github&logoColor=white&label=Latest%20Release" alt="Release">
+	</a>
+	<a href="https://github.com/DevelopersToolbox/github-ripper/releases/latest">
+		<img src="https://img.shields.io/github/commits-since/DevelopersToolbox/github-ripper/latest.svg?color=blue&style=for-the-badge&logo=github&logoColor=white" alt="Commits since release">
+	</a>
+	<a href="LICENSE.md">
+		<img src="https://img.shields.io/badge/license-MIT-blue?style=for-the-badge&logo=read-the-docs&logoColor=white" alt="Software License">
+	</a>
+	<br>
+	<a href=".github/CODE_OF_CONDUCT.md">
+		<img src="https://img.shields.io/badge/Code%20of%20Conduct-blue?style=for-the-badge&logo=read-the-docs&logoColor=white" />
+	</a>
+	<a href=".github/CONTRIBUTING.md">
+		<img src="https://img.shields.io/badge/Contributing-blue?style=for-the-badge&logo=read-the-docs&logoColor=white" />
+	</a>
+	<a href=".github/SECURITY.md">
+		<img src="https://img.shields.io/badge/Report%20Security%20Concern-blue?style=for-the-badge&logo=read-the-docs&logoColor=white" />
+	</a>
+	<a href=".github/SUPPORT.md">
+		<img src="https://img.shields.io/badge/Get%20Support-blue?style=for-the-badge&logo=read-the-docs&logoColor=white" />
+	</a>
+</p>
+
+## Overview
+
+Ghrip is a command line tool for ripping (cloning) repositories from GitHub in bulk. It allows you to clone all repositories owned by a named user or users, or from an organisation or organisations.
+
+To make the cloning as fast as possible we make extensive use of the [Parallel](https://rubygems.org/gems/parallel) gem to run as many clones as possible in parallel.
+
+### Tokens
+
+There is no requirement to supply a token when using ghrip, however there are numerous benefits. including but not limited to:
+
+1. Less impact from connection throttling (Unauthenticated connections are throttled more heavily by GitHub)
+2. Ability to clone private repositories
+3. Simple to clone all of your own repositories (no need to specify a username)
+
+You could for example clone ALL the repositories that you own with a single command.
+
+```
+# ghrip -t <Your token goes here> -A
+```
+
+> If you just wanted your personal repositories you could use -U instead of -A, or -M if you just wanted the repositories for organisations that you are a member of.
+
+If you do not supply token you will be limited to just cloning public repositories.
+
+### No Passwords?
+
+We intentionally do **NOT** handle username + password combinations when talking to GitHub, this removes some security risks and also removes the issue of having to handle MFA logins. It allows you to create a `read-only` token for cloning only to further improve your security position.
+
+### Example
+
+The following is an example of how to rip all of the **public** repositories owned by Wolf Software.
+
+```shell
+# ghrip -o WolfSoftware
+```
+
+## Usage
+
+```shell
+Usage: ghrip
+    -h, --help                       Display this screen
+
+Parameters:
+    -t, --token <token>              GitHub personal access token (PAT)
+    -b, --base-dir <path>            The base directory to download to
+    -g, --use-git                    Use git instead of https to clone the repositories
+
+Cloning Parameters:
+    -u, --user <names>               Github username(s) to rip
+    -o, --org <names>                Github organisation(s) to rip
+    -U, --user-repos                 Rip all of the repositories for the named user(s)
+    -M, --org-member-repos           Rip all of the repositories for all organisation the user(s) is a member of
+    -A, --all-repos                  Same as running -U -M
+    -O, --org-repos                  Rip all of the repositories for the named organisation(s)
+
+Flags:
+    -d, --dry-run                    Show a list of repositories that WOULD be ripped
+    -f, --full                       Show status of all repositories in post run report
+    -q, --quiet                      Suppress the showing of the post run report
+    -s, --silent                     Suppress all output
+```
+
+> If you just supply a username and nothing else it will default to just the users own repositories (same as -U)
+
+### Default Values
+
+| Option Name      | Default Value  | Purpose |
+| ---------------- | -------------- | ------- |
+| -t, --token            | No default     | The token used to authenticate yourself (not required but helps remove/mitigate throttling issues) | 
+| -b, --base-dir         | ~/Downloads/   | The directory to place the cloned repos in |
+| -g, --use-git          | False          | Use git instead of https to clone the repos, this requires your public key to be configured on GitHub |
+| -u, --user             | No default     | The user(s) you wish to rip the repo of (if not supplied defaults to yourself **IF** a token is used) |
+| -o, --org              | No default     | The organisation(s) you want to rip the repos of |
+| -U, --user-repos       | False          | Download all repositories belonging to the named user(s) |
+| -M, --org-member-repos | False          | Download all repositories belonging to any organisation the named user(s) is a member of |
+| -A, --all-repos        | False          | Works the same as --user-repo and --org-member-repos combined |
+| -O, --org-repos        | False          | Download all repositories belonging to the named organisation(s) |
+| -d, --dry-run          | False          | Display the list of repositories that **would** be downloaded |
+| -f, --full             | False          | Show all repositories in the post run report instead of just errors |
+| -q, --quiet            | False          | Suppress the showing of the post run report |
+| -s, --silent           | False          | Suppress all output |
+
+> users and orgs can be either a single entry or comma-separated list of entries.
+
+### Error Handling
+
+All errors raised from github-core-lister are caught and re-thrown as StandardErrors in order to simplify the catching.
+
+## Contributors
+
+<p>
+	<a href="https://github.com/TGWolf">
+		<img src="https://img.shields.io/badge/Wolf-black?style=for-the-badge" />
+	</a>
+</p>
+
+## Show Support
+
+<p>
+	<a href="https://ko-fi.com/wolfsoftware">
+		<img src="https://img.shields.io/badge/Ko%20Fi-blue?style=for-the-badge&logo=ko-fi&logoColor=white" />
+	</a>
+</p>

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/VERSION.txt

@@ -0,0 +1 @@
+0.1.0

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'github-ripper'
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/exe/ghrip

@@ -0,0 +1,106 @@
+#!/usr/bin/env ruby
+
+require 'github-ripper'
+require 'json'
+require 'optparse'
+require 'pp'
+
+MANDATORY_PARAMETERS = [].freeze
+DEFAULT_VALUES = {}.freeze
+
+def init_default_parser
+    parser = OptionParser.new
+
+    parser.banner = "Usage: #{$PROGRAM_NAME}"
+
+    parser.on('-h', '--help', 'Display this screen') do
+        puts parser
+        exit(1)
+    end
+    parser
+end
+
+# This method reeks of :reek:UtilityFunction, :reek:TooManyStatements
+def add_parameters(parser, options)
+    parser.separator ''
+    parser.separator 'Parameters:'
+    parser.on('-t', '--token <token>', 'GitHub personal access token (PAT)') { |token| options[:token] = token }
+    parser.on('-b', '--base-dir <path>', "The base directory to download to [default: #{File.expand_path('~')}/Downloads/]") { |base_dir| options[:base_dir] = base_dir }
+    parser.on('-g', '--use-git', TrueClass, 'Use git instead of https to clone the repositories') { |_errors| options[:use_git] = true }
+    [parser, options]
+end
+
+# This method reeks of :reek:UtilityFunction, :reek:TooManyStatements
+def add_repo_flag_parameters(parser, options)
+    parser.separator ''
+    parser.separator 'Cloning Parameters:'
+    parser.on('-u', '--user <names>', 'Github username(s) to rip') { |user| options[:user] = user }
+    parser.on('-o', '--org <names>', 'Github organisation(s) to rip') { |org| options[:org] = org }
+    parser.on('-U', '--user-repos', TrueClass, 'Rip all of the repositories for the named user(s)') { |_dry_run| options[:user_repos] = true }
+    parser.on('-M', '--org-member-repos', TrueClass, 'Rip all of the repositories for all organisation the user(s) is a member of') { |_quiet| options[:org_members_repos] = true }
+    parser.on('-A', '--all-repos', TrueClass, 'Same as running -U -M') { |_silent| options[:all_repos] = true }
+    parser.on('-O', '--org-repos', TrueClass, 'Rip all of the repositories for the named organisation(s)') { |_errors| options[:org_repos] = true }
+    [parser, options]
+end
+
+# This method reeks of :reek:UtilityFunction, :reek:TooManyStatements
+def add_flag_parameters(parser, options)
+    parser.separator ''
+    parser.separator 'Flags:'
+    parser.on('-d', '--dry-run', 'Show a list of repositories that WOULD be ripped') { |_dry_run| options[:dry_run] = true }
+    parser.on('-f', '--full', 'Show status of all repositories in post run report') { |_errors| options[:full_report] = true }
+    parser.on('-q', '--quiet', 'Suppress the showing of the post run report') { |_quiet| options[:quiet] = true }
+    parser.on('-s', '--silent', 'Suppress all output') { |_silent| options[:silent] = true }
+    [parser, options]
+end
+
+def create_parser
+    options = DEFAULT_VALUES.dup
+
+    parser = init_default_parser
+    parser, options = add_parameters(parser, options)
+    parser, options = add_repo_flag_parameters(parser, options)
+    parser, options = add_flag_parameters(parser, options)
+
+    [parser, options]
+end
+
+def process_parser(parser, options)
+    begin
+        parser.parse!
+        missing = MANDATORY_PARAMETERS.reject { |param| options.include?(param) }
+        raise OptionParser::MissingArgument.new(missing.join(', ')) unless missing.empty?
+    rescue OptionParser::InvalidOption, OptionParser::InvalidArgument, OptionParser::MissingArgument => exception
+        exit_with_error(parser, exception)
+    end
+    #
+    # The parser will use :"some-symbol" so we convert it to :some_symbol
+    #
+    options.transform_keys(&:to_s).transform_keys { |key| key.gsub('-', '_') }.transform_keys(&:to_sym)
+end
+
+def exit_with_error(parser, exception = nil)
+    puts exception if exception
+    puts parser.help
+    exit
+end
+
+def process_arguments
+    parser, options = create_parser
+    options = process_parser(parser, options)
+
+    exit_with_error(parser) if options.empty?
+    options
+end
+
+def main
+    options = process_arguments
+
+    begin
+        GithubRipper.rip(options)
+    rescue StandardError => exception
+        puts exception
+    end
+end
+
+main

data/github-ripper.gemspec

@@ -0,0 +1,49 @@
+lib = File.expand_path('lib', __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+require 'github-ripper/version'
+
+Gem::Specification.new do |spec|
+    spec.name          = 'github-ripper'
+    spec.version       = GithubRipper::VERSION
+    spec.authors       = ['Tim Gurney aka Wolf']
+    spec.email         = ['wolf@tgwolf.com']
+
+    spec.summary       = 'Ghrip is a command line tool for ripping (cloning) repositories from GitHub in bulk.'
+    spec.description   = 'Ghrip is a command line tool for ripping (cloning) repositories from GitHub in bulk. It allows you to clone all repositories owned by a named user or users, or from an organisation or organisations.'
+    spec.homepage      = 'https://github.com/DevelopersToolbox/github-ripper'
+    spec.license       = 'MIT'
+
+    spec.required_ruby_version = Gem::Requirement.new('>= 2.5.0')
+
+    spec.metadata['homepage_uri'] = spec.homepage
+    spec.metadata['source_code_uri'] = 'https://github.com/DevelopersToolbox/github-ripper'
+    spec.metadata['changelog_uri'] = 'https://github.com/DevelopersToolbox/github-ripper/blob/master/CHANGELOG.md'
+
+    # Specify which files should be added to the gem when it is released.
+    # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+    # rubocop:disable Layout/ExtraSpacing, Layout/SpaceAroundOperators
+    spec.files         = Dir.chdir(File.expand_path(__dir__)) do
+        `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+    end
+    # rubocop:enable Layout/ExtraSpacing, Layout/SpaceAroundOperators
+
+    spec.bindir        = 'exe'
+    spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+    spec.require_paths = ['lib']
+
+    spec.add_development_dependency 'bundler', '~> 2'
+    spec.add_development_dependency 'github-lister-core', '~> 0.1'
+    spec.add_development_dependency 'parallel', '~> 1.2'
+    spec.add_development_dependency 'rainbow', '~> 3'
+    spec.add_development_dependency 'rake', '~> 12.3.3'
+    spec.add_development_dependency 'rspec', '~> 3.0'
+    spec.add_development_dependency 'ruby-progressbar', '~> 1'
+    spec.add_development_dependency 'terminal-table', '~> 3'
+
+    spec.add_runtime_dependency 'github-lister-core', '~> 0.1'
+    spec.add_runtime_dependency 'parallel', '~> 1.2'
+    spec.add_runtime_dependency 'rainbow', '~> 3'
+    spec.add_runtime_dependency 'ruby-progressbar', '~> 1'
+    spec.add_runtime_dependency 'terminal-table', '~> 3'
+end

data/lib/github-ripper.rb

@@ -0,0 +1,36 @@
+require 'fileutils'
+require 'github-lister-core'
+require 'json'
+require 'open3'
+require 'rainbow'
+require 'terminal-table'
+
+require_relative 'github-ripper/function-maps'
+require_relative 'github-ripper/git'
+require_relative 'github-ripper/report'
+require_relative 'github-ripper/repos'
+require_relative 'github-ripper/table'
+require_relative 'github-ripper/utils'
+require_relative 'github-ripper/version'
+require_relative 'github-ripper/wrapper'
+
+#
+# Class level docs
+#
+class GithubRipper
+    class << self
+        #
+        # Do something ...
+        #
+        def rip(options = {})
+            # The following is to force github-lister-core to only retun repo slugs
+            options[:use_slugs] = true
+            options[:base_dir] = "#{File.expand_path('~')}/Downloads" unless get_option(options, :base_dir)
+
+            repos = get_repo_list(options)
+            results = rip_repos(options, repos)
+            results, repo_count, error_count = process_results(results, options)
+            draw_report(results, repo_count, error_count, options)
+        end
+    end
+end

data/lib/github-ripper/function-maps.rb

@@ -0,0 +1,31 @@
+#
+# Class docs
+#
+class GithubRipper
+    #
+    # Lookup the github-lister-core function from the options given
+    #
+    FUNCTION_MAP = {
+        :user_repos        => 'user_repos',
+        :org_repos         => 'org_repos',
+        :org_members_repos => 'org_members_repos',
+        :all_repos         => 'all_repos'
+    }.freeze
+
+    class << self
+        #
+        # Everything below here is private
+        #
+
+        private
+
+        def function_map_lookup(options)
+            return FUNCTION_MAP[:user_repos] if flag_set?(options, :user_repos) || get_option(options, :user)
+            return FUNCTION_MAP[:org_members_repos] if flag_set?(options, :org_members_repos)
+            return FUNCTION_MAP[:all_repos] if flag_set?(options, :all_repos)
+            return FUNCTION_MAP[:org_repos] if flag_set?(options, :org_repos) || get_option(options, :org)
+
+            nil
+        end
+    end
+end