githubtakeout 0.1.0__tar.gz → 0.1.1__tar.gz

{githubtakeout-0.1.0 → githubtakeout-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: githubtakeout
-Version: 0.1.0
+Version: 0.1.1
 Summary: Archive public Git Repos and Gists from GitHub
 Author: Corey Goldberg
 Maintainer: Corey Goldberg
@@ -27,7 +27,7 @@ Dynamic: license-file
 
 # githubtakeout
 
-## Archive public Git Repos and Gists from GitHub
+## Archive Git Repos and Gists from GitHub
 
 ---
 
@@ -41,11 +41,11 @@ Dynamic: license-file
 ## About:
 
 `githubtakeout` is a data export tool for archiving Git repositories hosted on GitHub.
-It clones a user's public repos and creates an archive of each.
+It clones a user's repos and creates an archive of each.
 
-By default, it doesn't save commit history or branches (`.git` directory), or Gist
-repositories (both can be enabled with command line options). It also doesn't access
-private repositories or secret gists.
+It supports public/private repos and public/secret gists. By default, it doesn't save
+commit history or branches (`.git` directory), or Gist repositories (both can be enabled
+with command line options).
 
 When you run the program, archives of your repos will be saved in a directory named
 `backups` inside your current working directory, unless a different location is specified
@@ -62,38 +62,32 @@ as tarballs (`.tar.gz`) using the `--format=tar` option.
     - GitPython
     - PyGithub
 
-## Installing and Running:
+## Installation:
 
-### Install from PyPI:
+Install from [PyPI](https://pypi.org/project/githubtakeout):
 
 ```
 pip install githubtakeout
 ```
 
-### Create/Activate Virtual Environment, Install from PyPI, Run:
+## Authentication:
 
-```
-python3 -m venv venv
-source venv/bin/activate
-pip install githubtakeout
-githubtakeout <github username>
-```
+By default, `githubtakeout` will only retrieve an account's public repos. To access
+private repos and secret gists, you need to authenticate.
 
-### Clone Repo, Create/Activate Virtual Environment, Install from Source, Run:
+First, you must create a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
+  on Github (either a fine-grained or classic personal access token). Once you have a token, you can set the `GITHUB_TOKEN` environment variable:
 
 ```
-git clone git@github.com:cgoldberg/githubtakeout.git
-cd ./githubtakeout
-python3 -m venv venv
-source venv/bin/activate
-pip install .
-githubtakeout <github username>
+$ export GITHUB_TOKEN=<auth token created on GitHub>
 ```
 
-### Usage:
+If you prefer to be prompted for your token each time you run the program, use the `--token` argument.
+
+### CLI Options:
 
 ```
-usage: githubtakeout [-h] [--dir DIR] [--format FORMAT] [--gists] [--history] [--list] username
+usage: githubtakeout [-h] [--dir DIR] [--format FORMAT] [--gists] [--history] [--list] [--token] username
 
 positional arguments:
   username         GitHub username
@@ -105,4 +99,28 @@ options:
   --gists          include gists
   --history        include commit history and branches (.git directory)
   --list           list repos only
+  --token          prompt for auth toke
+```
+
+
+## Usage Examples:
+
+### Create/Activate Virtual Environment, Install from PyPI, Run:
+
+```
+python3 -m venv venv
+source venv/bin/activate
+pip install githubtakeout
+githubtakeout <github username>
+```
+
+### Clone Repo, Create/Activate Virtual Environment, Install from Source, Run:
+
+```
+git clone git@github.com:cgoldberg/githubtakeout.git
+cd ./githubtakeout
+python3 -m venv venv
+source venv/bin/activate
+pip install .
+githubtakeout <github username>
 ```

{githubtakeout-0.1.0 → githubtakeout-0.1.1}/README.md

@@ -1,6 +1,6 @@
 # githubtakeout
 
-## Archive public Git Repos and Gists from GitHub
+## Archive Git Repos and Gists from GitHub
 
 ---
 
@@ -14,11 +14,11 @@
 ## About:
 
 `githubtakeout` is a data export tool for archiving Git repositories hosted on GitHub.
-It clones a user's public repos and creates an archive of each.
+It clones a user's repos and creates an archive of each.
 
-By default, it doesn't save commit history or branches (`.git` directory), or Gist
-repositories (both can be enabled with command line options). It also doesn't access
-private repositories or secret gists.
+It supports public/private repos and public/secret gists. By default, it doesn't save
+commit history or branches (`.git` directory), or Gist repositories (both can be enabled
+with command line options).
 
 When you run the program, archives of your repos will be saved in a directory named
 `backups` inside your current working directory, unless a different location is specified
@@ -35,38 +35,32 @@ as tarballs (`.tar.gz`) using the `--format=tar` option.
     - GitPython
     - PyGithub
 
-## Installing and Running:
+## Installation:
 
-### Install from PyPI:
+Install from [PyPI](https://pypi.org/project/githubtakeout):
 
 ```
 pip install githubtakeout
 ```
 
-### Create/Activate Virtual Environment, Install from PyPI, Run:
+## Authentication:
 
-```
-python3 -m venv venv
-source venv/bin/activate
-pip install githubtakeout
-githubtakeout <github username>
-```
+By default, `githubtakeout` will only retrieve an account's public repos. To access
+private repos and secret gists, you need to authenticate.
 
-### Clone Repo, Create/Activate Virtual Environment, Install from Source, Run:
+First, you must create a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
+  on Github (either a fine-grained or classic personal access token). Once you have a token, you can set the `GITHUB_TOKEN` environment variable:
 
 ```
-git clone git@github.com:cgoldberg/githubtakeout.git
-cd ./githubtakeout
-python3 -m venv venv
-source venv/bin/activate
-pip install .
-githubtakeout <github username>
+$ export GITHUB_TOKEN=<auth token created on GitHub>
 ```
 
-### Usage:
+If you prefer to be prompted for your token each time you run the program, use the `--token` argument.
+
+### CLI Options:
 
 ```
-usage: githubtakeout [-h] [--dir DIR] [--format FORMAT] [--gists] [--history] [--list] username
+usage: githubtakeout [-h] [--dir DIR] [--format FORMAT] [--gists] [--history] [--list] [--token] username
 
 positional arguments:
   username         GitHub username
@@ -78,4 +72,28 @@ options:
   --gists          include gists
   --history        include commit history and branches (.git directory)
   --list           list repos only
+  --token          prompt for auth toke
+```
+
+
+## Usage Examples:
+
+### Create/Activate Virtual Environment, Install from PyPI, Run:
+
+```
+python3 -m venv venv
+source venv/bin/activate
+pip install githubtakeout
+githubtakeout <github username>
+```
+
+### Clone Repo, Create/Activate Virtual Environment, Install from Source, Run:
+
+```
+git clone git@github.com:cgoldberg/githubtakeout.git
+cd ./githubtakeout
+python3 -m venv venv
+source venv/bin/activate
+pip install .
+githubtakeout <github username>
 ```

{githubtakeout-0.1.0 → githubtakeout-0.1.1}/githubtakeout.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: githubtakeout
-Version: 0.1.0
+Version: 0.1.1
 Summary: Archive public Git Repos and Gists from GitHub
 Author: Corey Goldberg
 Maintainer: Corey Goldberg
@@ -27,7 +27,7 @@ Dynamic: license-file
 
 # githubtakeout
 
-## Archive public Git Repos and Gists from GitHub
+## Archive Git Repos and Gists from GitHub
 
 ---
 
@@ -41,11 +41,11 @@ Dynamic: license-file
 ## About:
 
 `githubtakeout` is a data export tool for archiving Git repositories hosted on GitHub.
-It clones a user's public repos and creates an archive of each.
+It clones a user's repos and creates an archive of each.
 
-By default, it doesn't save commit history or branches (`.git` directory), or Gist
-repositories (both can be enabled with command line options). It also doesn't access
-private repositories or secret gists.
+It supports public/private repos and public/secret gists. By default, it doesn't save
+commit history or branches (`.git` directory), or Gist repositories (both can be enabled
+with command line options).
 
 When you run the program, archives of your repos will be saved in a directory named
 `backups` inside your current working directory, unless a different location is specified
@@ -62,38 +62,32 @@ as tarballs (`.tar.gz`) using the `--format=tar` option.
     - GitPython
     - PyGithub
 
-## Installing and Running:
+## Installation:
 
-### Install from PyPI:
+Install from [PyPI](https://pypi.org/project/githubtakeout):
 
 ```
 pip install githubtakeout
 ```
 
-### Create/Activate Virtual Environment, Install from PyPI, Run:
+## Authentication:
 
-```
-python3 -m venv venv
-source venv/bin/activate
-pip install githubtakeout
-githubtakeout <github username>
-```
+By default, `githubtakeout` will only retrieve an account's public repos. To access
+private repos and secret gists, you need to authenticate.
 
-### Clone Repo, Create/Activate Virtual Environment, Install from Source, Run:
+First, you must create a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
+  on Github (either a fine-grained or classic personal access token). Once you have a token, you can set the `GITHUB_TOKEN` environment variable:
 
 ```
-git clone git@github.com:cgoldberg/githubtakeout.git
-cd ./githubtakeout
-python3 -m venv venv
-source venv/bin/activate
-pip install .
-githubtakeout <github username>
+$ export GITHUB_TOKEN=<auth token created on GitHub>
 ```
 
-### Usage:
+If you prefer to be prompted for your token each time you run the program, use the `--token` argument.
+
+### CLI Options:
 
 ```
-usage: githubtakeout [-h] [--dir DIR] [--format FORMAT] [--gists] [--history] [--list] username
+usage: githubtakeout [-h] [--dir DIR] [--format FORMAT] [--gists] [--history] [--list] [--token] username
 
 positional arguments:
   username         GitHub username
@@ -105,4 +99,28 @@ options:
   --gists          include gists
   --history        include commit history and branches (.git directory)
   --list           list repos only
+  --token          prompt for auth toke
+```
+
+
+## Usage Examples:
+
+### Create/Activate Virtual Environment, Install from PyPI, Run:
+
+```
+python3 -m venv venv
+source venv/bin/activate
+pip install githubtakeout
+githubtakeout <github username>
+```
+
+### Clone Repo, Create/Activate Virtual Environment, Install from Source, Run:
+
+```
+git clone git@github.com:cgoldberg/githubtakeout.git
+cd ./githubtakeout
+python3 -m venv venv
+source venv/bin/activate
+pip install .
+githubtakeout <github username>
 ```

{githubtakeout-0.1.0 → githubtakeout-0.1.1}/githubtakeout.py

@@ -1,10 +1,12 @@
 #!/usr/bin/env python3
 
 import argparse
+import getpass
 import logging
 import os
 import shutil
 import tarfile
+import urllib
 import zipfile
 
 from pathlib import Path
@@ -18,6 +20,17 @@ logging.basicConfig(level=logging.INFO, format='%(message)s')
 logger = logging.getLogger(__name__)
 
 
+def add_creds(url, username, token):
+    if token is None:
+        new_url = url
+    else:
+        username = urllib.parse.quote(username)
+        url_parts = urllib.parse.urlparse(url)
+        netloc = f'{username}:{token}@{url_parts.netloc}'
+        new_url = urllib.parse.urlunparse(url_parts._replace(netloc=netloc))
+    return new_url
+
+
 def archive(local_repo_dir, archive_format='zip', is_gist=False):
     if archive_format not in ('tar', 'zip'):
         raise ValueError(f'{archive_format} is not a valid archive format')
@@ -46,7 +59,8 @@ def clone_and_archive_repo(repo_url, local_repo_dir, archive_format, include_his
         shutil.rmtree(local_repo_dir)
     except FileNotFoundError:
         pass
-    logger.info(f'cloning repo: {repo_url} to {local_repo_dir}')
+    repo_path = urllib.parse.urlparse(repo_url).path
+    logger.info(f'cloning repo: {repo_path} to {local_repo_dir}')
     start = default_timer()
     try:
         if include_history:
@@ -73,48 +87,88 @@ def clone_and_archive_repo(repo_url, local_repo_dir, archive_format, include_his
         pass
     logger.info('')
 
-def run(username, base_dir, archive_format, include_gists, include_history, list):
+
+def get_user(username, prompt_token):
+    if prompt_token:
+        token = getpass.getpass('Token:')
+        if not token:
+            print(f'Error: auth token cannot be empty')
+            exit(1)
+        auth = github.Auth.Token(token)
+        gh = github.Github(auth=auth)
+    else:
+        token = os.getenv('GITHUB_TOKEN')
+        if token is not None:
+            auth = github.Auth.Token(token)
+            gh = github.Github(auth=auth)
+        else:
+            gh = github.Github()
+    if token is None:
+        try:
+            user = gh.get_user(username)
+        except github.GithubException as e:
+            if e.data['status'] == '404':
+                print(f'Error: user "{username}" not found on GitHub')
+            else:
+                raise e
+            exit(1)
+        repos = user.get_repos()
+    else:
+        # you need to be authenticated and then call the API
+        # with no username to get private repos
+        user = gh.get_user()
+        repos = user.get_repos(affiliation='owner')
+        try:
+            # this just makes an API request so we can exit if unauthorized
+            repos.totalCount
+        except github.GithubException as e:
+            if e.data['status'] == '401':
+                print(f'Error: invalid auth token for user "{username}" on GitHub')
+            else:
+                raise e
+            exit(1)
+    gists = user.get_gists()
+    return user, repos, gists, token
+
+
+def run(username, base_dir, archive_format, include_gists, include_history, list, prompt_token):
     working_dir = os.path.join(base_dir, 'backups')
-    gh = github.Github()
-    try:
-        user = gh.get_user(username)
-    except github.GithubException:
-        print(f'Error: user "{username}" not found on GitHub')
-        exit(1)
-    repos = user.get_repos()
-    num_repos = len([1 for _ in repos])
+    user, repos, gists, token = get_user(username, prompt_token)
+    num_repos = repos.totalCount
     if not list:
         logger.info(f'creating archives in: {working_dir}\n')
     logger.info(f'found {num_repos} repos for user "{username}"\n')
     for repo in repos:
         local_repo_dir = os.path.join(working_dir, repo.name)
+        url = add_creds(repo.clone_url, username, token)
         if list:
-            logger.info(repo.clone_url)
+            logger.info(url)
         else:
             clone_and_archive_repo(
-                repo.clone_url,
+                url,
                 local_repo_dir,
                 archive_format,
                 include_history
             )
     if include_gists:
-        gists = user.get_gists()
-        num_gists = len([1 for _ in gists])
+        num_gists = gists.totalCount
         logger.info('')
         logger.info(f'found {num_gists} gists for user "{username}"\n')
         for gist in gists:
             local_repo_dir = os.path.join(working_dir, gist.id)
+            url = add_creds(gist.git_pull_url, username, token)
             if list:
-                logger.info(gist.git_pull_url)
+                logger.info(url)
             else:
                 clone_and_archive_repo(
-                    gist.git_pull_url,
+                    url,
                     local_repo_dir,
                     archive_format,
                     include_history,
                     is_gist=True
                 )
 
+
 def main():
     parser = argparse.ArgumentParser()
     parser.add_argument(
@@ -149,6 +203,12 @@ def main():
         action='store_true',
         help='list repos only'
     )
+    parser.add_argument(
+        '--token',
+        default=False,
+        action='store_true',
+        help='prompt for auth token'
+    )
     args = parser.parse_args()
     run(
         username=args.username,
@@ -156,7 +216,8 @@ def main():
         archive_format=args.format,
         include_gists=args.gists,
         include_history=args.history,
-        list=args.list
+        list=args.list,
+        prompt_token=args.token
     )
 
 

{githubtakeout-0.1.0 → githubtakeout-0.1.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "githubtakeout"
-version = "0.1.0"
+version = "0.1.1"
 description = "Archive public Git Repos and Gists from GitHub"
 license = "MIT"
 license-files = ["LICENSE"]