gitrob 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 2644a28d02e265e64b4217080b1b3dece92f499d
+  data.tar.gz: e6540cad01070d9bc0c7329c90e70ce0f498b53b
+SHA512:
+  metadata.gz: df03cc1140011bfacab7d828b31af4c2ed62acd3d86ffbdaf1ab33c40a5ba580b1aaac24b2cef43ba400926c9d2f108a6439474a311fe6554db49fc0cce6437e
+  data.tar.gz: 772f06c99dd5b3c3012e676629fad0cdfc0478f3dfa0e066647828adb910a32b13b0be1314143389a7520295a29223624a1698da5fe6c07450462026df29f682

data/.gitignore

@@ -0,0 +1,18 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+*.swp
+mkmf.log
+.ruby-version
+.ruby-gemset
+agreement

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in gitrob.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Michael Henriksen
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,47 @@
+# Gitrob
+
+Developers generally like to share their code, and many of them do so by open sourcing it on GitHub, a social code hosting and collaboration service. Many companies also use GitHub as a convenient place to host both private and public code repositories by creating GitHub organizations where employees can be joined.
+
+Sometimes employees might publish things that should not be publicly available, things that contain sensitive information or things that could even lead to direct compromise of a system. This can happen by accident or because the employee does not know the sensitivity of the information.
+
+Gitrob is a command line tool that can help organizations and security professionals find such sensitive information. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files, that typically contain sensitive or dangerous information.
+
+Read the [blog post](http://michenriksen.com/blog/gitrob-putting-the-open-source-in-osint/) for more information and screenshots.
+
+## How it works
+
+Looking for sensitive information in GitHub repositories is not a new thing, it has been [known for a while](http://blog.conviso.com.br/2013/06/github-hacking-for-fun-and-sensitive.html) that things such as private keys and credentials can be found with GitHub's search functionality, however Gitrob makes it easier to focus the effort on a specific organization.
+
+The first thing the tool does is to collect all public repositories of the organization itself. It then goes on to collect all the organization members and their public repositories, in order to compile a list of repositories that might be related or have relevance to the organization.
+
+When the list of repositories has been compiled, it proceeds to gather all the filenames in each repository and runs them through a series of observers that will flag the files, if they match any patterns of known sensitive files. This step might take a while if the organization is big or if the members have a lot of public repositories.
+
+All of the members, repositories and files will be saved to a PostgreSQL database. When everything has been sifted through, it will start a Sinatra web server locally on the machine, which will serve a simple web application to present the collected data for analysis.
+
+## Installation
+
+Gitrob is written in Ruby and requires at least version 1.9.3 or above. If you are on an older version, it is very easy to install newer versions with [RVM](http://rvm.io/). If you are installing Gitrob on [Kali](http://www.kali.org/), you are almost good to go, you just need to update Bundler with `gem install bundler` and install a PostgreSQL dependency with `apt-get install libpq-dev` in a terminal.
+
+Gitrob is a Ruby gem, so installation is a simple `gem install gitrob` in a terminal. This will automatically install all the code dependencies as well.
+
+A [PostgreSQL](http://www.postgresql.org/) database is also needed for Gitrob to store its data. Installing PostgreSQL is pretty straight forward; here is an installation guide for [Mac OS X](http://www.gotealeaf.com/blog/how-to-install-postgresql-on-a-mac) and one for [Ubuntu/Debian](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-14-04) based Linux. If you're installing Gitrob on Kali, you already have PostgreSQL installed, however you need to start the server with `service postgresql start` in a terminal.
+
+The last thing we need is a GitHub access token in order to be able to talk to their API. The easiest way is to create a [personal access token](https://github.com/settings/applications). If you plan on using Gitrob extensively or on a very big organization, it might be necessary to have multiple access tokens to prevent running into rate limiting, but they need to be from different user accounts.
+
+When everything is ready, simply run `gitrob --configure` and you will be presented with a configuration wizard that asks you for database connection details and GitHub access tokens. All of this configuration can be changed by running the same command again. The configuration will be saved in `~/.gitrobrc` - and yes, Gitrob is looking for this file too so watch out.
+
+When everything is set up, you can start analyzing organizations by running `gitrob -o <orgname>` in a terminal. To see options, use `gitrob --help`.
+
+## Contributing
+
+Gitrob should be considered Beta and there is probably a good amount of bugs. Bug reports and suggestions for improvements are welcome!
+
+Another way to help out is to contribute new patterns for sensitive files. If you know of any sensitive files that are not already identified, please submit them in a pull request. I am especially interested in sensitive web framework files and configuration files. Have a look at the [patterns.json](https://github.com/michenriksen/gitrob/blob/master/patterns.json) file to see what is already looked for.
+
+### How to make a pull request:
+
+1. Fork it ( https://github.com/michenriksen/gitrob/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/bin/gitrob

@@ -0,0 +1,258 @@
+#!/usr/bin/env ruby
+# coding: utf-8
+
+require 'gitrob'
+
+class App
+  include Methadone::Main
+
+  leak_exceptions true
+
+  main do
+    Thread.abort_on_exception = true
+    Paint.mode = 0 if options.include?('no-color')
+
+    puts Paint[Gitrob::banner, :bright, :blue] unless options.include?('no-banner')
+    Gitrob::status("Starting Gitrob version #{Gitrob::VERSION} at #{Time.now.strftime("%Y-%m-%d %I:%S %Z")}")
+
+    if !Gitrob::agreement_accepted?
+      puts Gitrob::agreement
+      print Paint["\n\nDo you agree to the terms of use? [y/n]: ", :bright, :green]
+      choice = $stdin.gets.chomp
+      if %w{y yes}.include?(choice)
+        Gitrob::agreement_accepted
+      else
+        Gitrob::fatal("Exiting Gitrob.")
+      end
+    end
+
+    if !Gitrob::configured? || options.include?('configure')
+      Gitrob::status("Starting Gitrob configuration wizard\n\n")
+
+      hostname = ask("#{Paint[" Enter PostgreSQL hostname:", :bright, :white]} ") { |q| q.default = "localhost" }
+      port     = ask("#{Paint[" Enter PostgreSQL port:", :bright, :white]} ", Integer) { |q| q.default = 5432; q.in = 1..65535 }
+      username = ask("#{Paint[" Enter PostgreSQL username:", :bright, :white]} ")
+      password = ask("#{Paint[" Enter PostgreSQL password for #{username} (masked):", :bright, :white]} ") { |q| q.echo = 'x' }
+      database = ask("#{Paint[" Enter PostgreSQL database name:", :bright, :white]} ") { |q| q.default = 'gitrob' }
+
+      access_tokens = Array.new
+      while access_tokens.uniq.empty?
+        access_tokens = ask(Paint[" Enter GitHub access tokens (blank line to stop):", :bright, :white],
+                          lambda { |ans| ans =~ /[a-f0-9]{40}/ ? ans : nil } ) do |q|
+          q.gather = ""
+        end
+      end
+
+      config = {
+        'sql_connection_uri'   => "postgres://#{username}:#{password}@#{hostname}:#{port}/#{database}",
+        'github_access_tokens' => access_tokens.uniq
+      }
+
+      Gitrob::task("Saving configuration to file...") do
+        Gitrob::save_configuration!(config)
+      end
+
+      exit unless options['organization']
+    end
+
+    Gitrob::task("Loading configuration...") do
+      Gitrob::load_configuration!
+    end
+
+    Gitrob::task("Preparing SQL database...") do
+      Gitrob::prepare_database!
+    end
+
+    if options['reset-db']
+      Gitrob::task("Resetting database tables...") do
+        DataMapper::auto_migrate!
+      end
+      exit
+    end
+
+    if options['delete']
+      Gitrob::delete_organization(options['delete'])
+      exit
+    end
+
+    if options['start-server']
+      Gitrob::status("Starting web application on http://#{options['bind-address']}:#{options['port']}/...")
+      puts "\n\n"
+      Gitrob::WebApp.run!(:port => options['port'].to_i, :bind => options['bind-address'])
+      exit
+    end
+
+    org_name    = options['organization']
+    repo_count  = 0
+    members     = Array.new
+    http_client = Gitrob::Github::HttpClient.new({:access_tokens => Gitrob::configuration['github_access_tokens']})
+    observers   = Gitrob::Observers.constants.collect { |c| Gitrob::Observers::const_get(c) }
+
+    Gitrob::task("Loading file patterns...") do
+      Gitrob::Observers::SensitiveFiles::load_patterns!
+    end
+
+    begin
+      org = Gitrob::Github::Organization.new(org_name, http_client)
+
+      Gitrob::delete_organization(org.login)
+
+      Gitrob::task("Collecting organization repositories...") do
+        repo_count = org.repositories.count
+      end
+    rescue Gitrob::Github::HttpClient::ClientError => e
+      if e.status == 404
+        Gitrob::fatal("Cannot find GitHub organization with that name; exiting.")
+      else
+        raise e
+      end
+    end
+
+    Gitrob::task("Collecting organization members...") do
+      members = org.members
+    end
+
+    progress = Gitrob::ProgressBar.new("Collecting member repositories...",
+      :total => members.count
+    )
+    thread_pool = Thread.pool(options['threads'].to_i)
+
+    members.each do |member|
+      thread_pool.process do
+        if member.repositories.count > 0
+          repo_count += member.repositories.count
+          progress.log("Collected #{Gitrob::Util::pluralize(member.repositories.count, 'repository', 'repositories')} from #{Paint[member.username, :bright, :white]}")
+        end
+        progress.increment
+      end
+    end
+
+    thread_pool.shutdown
+
+    if repo_count.zero?
+      Gitrob::fatal("Organization has no repositories to check; exiting.")
+    end
+
+    progress = Gitrob::ProgressBar.new("Processing repositories...",
+      :total => repo_count
+    )
+
+    db_org = org.save_to_database!
+    thread_pool = Thread.pool(options['threads'].to_i)
+
+    org.repositories.each do |repo|
+      thread_pool.process do
+        begin
+          if repo.contents.count > 0
+            db_repo  = repo.save_to_database!(db_org)
+            findings = 0
+
+            repo.contents.each do |blob|
+              db_blob = blob.to_model(db_org, db_repo)
+
+              observers.each do |observer|
+                observer.observe(db_blob)
+              end
+
+              db_blob.findings.each do |f|
+                db_blob.findings_count += 1
+                findings += 1
+                f.organization = db_org
+                f.repo         = db_repo
+              end
+
+              db_blob.save
+            end
+            progress.log("Processed #{Gitrob::Util::pluralize(repo.contents.count, 'file', 'files')} from #{Paint[repo.full_name, :bright, :white]} with #{findings.zero? ? 'no findings' : Paint[Gitrob::Util.pluralize(findings, 'finding', 'findings'), :yellow]}")
+          end
+          progress.increment
+        rescue Exception => e
+          progress.log_error("Encountered error when processing #{Paint[repo.full_name, :bright, :white]} (#{e.class.name})")
+          progress.increment
+        end
+      end
+    end
+
+    org.members.each do |member|
+      thread_pool.process do
+        begin
+          db_user = member.save_to_database!(db_org)
+
+          member.repositories.each do |repo|
+            if repo.contents.count > 0
+              db_repo  = repo.save_to_database!(db_org, db_user)
+              findings = 0
+
+              repo.contents.each do |blob|
+                db_blob = blob.to_model(db_org, db_repo)
+
+                observers.each do |observer|
+                  observer.observe(db_blob)
+                end
+
+                db_blob.findings.each do |f|
+                  db_blob.findings_count += 1
+                  findings += 1
+                  f.organization = db_org
+                  f.repo         = db_repo
+                  f.user         = db_user
+                end
+
+                db_blob.save
+              end
+              progress.log("Processed #{Gitrob::Util::pluralize(repo.contents.count, 'file', 'files')} from #{Paint[repo.full_name, :bright, :white]} with #{findings.zero? ? 'no findings' : Paint[Gitrob::Util.pluralize(findings, 'finding', 'findings'), :yellow]}")
+            end
+            progress.increment
+          end
+        rescue Exception => e
+          progress.log_error("Encountered error when processing #{Paint[repo.full_name, :bright, :white]} (#{e.class.name})")
+          progress.increment
+        end
+      end
+    end
+
+    thread_pool.shutdown
+
+    if !options.include?('no-server')
+      Gitrob::status("Starting web application on port #{options['port']}...")
+      Gitrob::status("Browse to http://#{options['bind-address']}:#{options['port']}/ to see results!")
+      puts "\n\n"
+      Gitrob::WebApp.run!(:port => options[:port].to_i, :bind => options['bind-address'])
+      exit
+    end
+  end
+
+  version Gitrob::VERSION
+  description "Reconnaissance tool for GitHub organizations."
+
+  options['bind-address'] = '127.0.0.1'
+  options['port']         = 9393
+  options['threads']      = 3
+
+  on('-o', '--organization NAME', "Name of GitHub organization")
+  on('-s', '--start-server', "Start web server, don't run any checks")
+  on('-p', '--port PORT', "Port to bind web server to")
+  on('-b', '--bind-address ADDRESS', "Address to bind web server to")
+  on('-t', '--threads THREADS', "Number of threads to use")
+  on('--delete NAME', "Delete an organization in the database")
+  on('--reset-db', "Resets the database")
+  on('--configure', "Start configuration wizard")
+  on('--no-server', "Don't start the server when finished")
+  on('--no-color', "Don't colorize output")
+  on('--no-banner', "Don't print Gitrob banner")
+
+  begin
+    go!
+  rescue Gitrob::Github::HttpClient::MissingAccessTokensError
+    Gitrob::fatal("Configuration file does not contain any GitHub access tokens. Run Gitrob with --configure flag to set it up.")
+  rescue Gitrob::Github::HttpClient::AccessTokensDepletedError
+    Gitrob::fatal("All GitHub access tokens under rate limiting. Go have a cup of coffee and try again.")
+  rescue Gitrob::Github::HttpClient::RequestError => e
+    Gitrob::fatal("A request to the GitHub API failed: #{e.message}")
+  rescue Interrupt
+    print "\b\b\n"; # Remove ^C from screen
+    Gitrob::fatal("Caught interrupt; exiting.")
+  rescue => e
+    Gitrob::fatal("An error occurred: #{e.class.name}: #{e.message}")
+  end
+end

data/gitrob.gemspec

@@ -0,0 +1,36 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'gitrob/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "gitrob"
+  spec.version       = Gitrob::VERSION
+  spec.authors       = ["Michael Henriksen"]
+  spec.email         = ["michenriksen@neomailbox.ch"]
+  spec.summary       = %q{Reconnaissance tool for GitHub organizations.}
+  spec.description   = %q{Reconnaissance tool for GitHub organizations.}
+  spec.homepage      = "https://github.com/michenriksen/gitrob"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "httparty", "~> 0.13"
+  spec.add_dependency "methadone", "~> 1.7"
+  spec.add_dependency "highline", "~> 1.6"
+  spec.add_dependency "paint", "~> 0.8"
+  spec.add_dependency "ruby-progressbar", "~> 1.6"
+  spec.add_dependency "thread", "~> 0.1"
+  spec.add_dependency "sinatra", "~> 1.4"
+  spec.add_dependency "thin", "~> 1.6"
+  spec.add_dependency "datamapper", "~> 1.2"
+  spec.add_dependency "dm-postgres-adapter"
+
+  spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec", "~> 3.1"
+  spec.add_development_dependency "webmock", "~> 1.20"
+end

data/lib/gitrob.rb

@@ -0,0 +1,116 @@
+require 'json'
+require 'cgi'
+
+require 'methadone'
+require 'highline/import'
+require 'thread/pool'
+require 'httparty'
+require 'ruby-progressbar'
+require 'paint'
+require 'sinatra/base'
+require 'data_mapper'
+
+require 'gitrob/version'
+require 'gitrob/util'
+require 'gitrob/progressbar'
+require 'gitrob/github/http_client'
+require 'gitrob/github/repository'
+require 'gitrob/github/blob'
+require 'gitrob/github/organization'
+require 'gitrob/github/user'
+require 'gitrob/observers/sensitive_files'
+require 'gitrob/webapp'
+
+require "#{File.dirname(__FILE__)}/../models/organization"
+require "#{File.dirname(__FILE__)}/../models/repo"
+require "#{File.dirname(__FILE__)}/../models/user"
+require "#{File.dirname(__FILE__)}/../models/blob"
+require "#{File.dirname(__FILE__)}/../models/finding"
+
+module Gitrob
+  def self.task(message)
+    print " #{Paint['[*]', :bright, :blue]} #{Paint[message, :bright, :white]}"
+    yield
+    puts Paint[" done", :bright, :green]
+  rescue => e
+    puts Paint[" failed", :bright, :red]
+    puts "#{Paint[' [!]', :bright, :red]} #{Paint[e.class, :bright, :white]}: #{e.message}"
+    exit!
+  end
+
+  def self.status(message)
+    puts " #{Paint['[*]', :bright, :blue]} #{Paint[message, :bright, :white]}"
+  end
+
+  def self.fatal(message)
+    puts " #{Paint['[!]', :bright, :red]} #{Paint[message, :bright, :white]}"
+    exit!
+  end
+
+  def self.prepare_database!
+    DataMapper::Model.raise_on_save_failure = true
+    DataMapper::Property.auto_validation(false)
+    DataMapper.setup(:default, configuration['sql_connection_uri'])
+    DataMapper.finalize
+    DataMapper.auto_upgrade!
+  end
+
+  def self.delete_organization(org)
+    orgs = Gitrob::Organization.all(:login => org)
+    if orgs.count > 0
+      task("Deleting existing #{org} organization...") do
+        orgs.destroy
+      end
+    end
+  end
+
+  def self.agreement_accepted?
+    File.exists?("#{File.dirname(__FILE__)}/../agreement")
+  end
+
+  def self.agreement
+    "\n#{self.license}\n\n" +
+
+    Paint["Gitrob is designed for security professionals. If you use any information\n" +
+          "found through this tool for malicious purposes that are not authorized by\n" +
+          "the organization, you are violating the terms of use and license of this\n" +
+          "tool. By typing y/yes, you agree to the terms of use and that you will use\n" +
+          "this tool for lawful purposes only.",
+          :bright, :red]
+  end
+
+  def self.agreement_accepted
+    File.open("#{File.dirname(__FILE__)}/../agreement", 'w') { |file| file.write("user accepted") }
+  end
+
+  def self.license
+    File.read("#{File.dirname(__FILE__)}/../LICENSE.txt")
+  end
+
+  def self.configured?
+    File.exists?("#{Dir.home}/.gitrobrc")
+  end
+
+  def self.configuration
+    @config ||= load_configuration!
+  end
+
+  def self.load_configuration!
+    YAML.load_file("#{Dir.home}/.gitrobrc")
+  end
+
+  def self.save_configuration!(config)
+    @config = config
+    File.open("#{Dir.home}/.gitrobrc", 'w') { |f| f.write YAML.dump(config) }
+  end
+
+  def self.banner
+<<-BANNER
+      _ _           _
+  ___|_| |_ ___ ___| |_
+ | . | |  _|  _| . | . |
+ |_  |_|_| |_| |___|___|
+ |___| #{Paint["By @michenriksen", :bright, :white]}
+BANNER
+  end
+end