github-to-sqlite 2.8.3__tar.gz → 2.9.1__tar.gz

{github-to-sqlite-2.8.3/github_to_sqlite.egg-info → github_to_sqlite-2.9.1}/PKG-INFO

@@ -1,14 +1,19 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: github-to-sqlite
-Version: 2.8.3
+Version: 2.9.1
 Summary: Save data from GitHub to a SQLite database
-Home-page: https://github.com/dogsheep/github-to-sqlite
 Author: Simon Willison
-License: Apache License, Version 2.0
-Platform: UNKNOWN
-Description-Content-Type: text/markdown
-Provides-Extra: test
+License-Expression: Apache-2.0
 License-File: LICENSE
+Requires-Dist: sqlite-utils>4
+Requires-Dist: requests
+Requires-Dist: pyyaml
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/dogsheep/github-to-sqlite
+Project-URL: Changelog, https://github.com/dogsheep/github-to-sqlite/releases
+Project-URL: Issues, https://github.com/dogsheep/github-to-sqlite/issues
+Project-URL: CI, https://github.com/dogsheep/github-to-sqlite/actions
+Description-Content-Type: text/markdown
 
 # github-to-sqlite
 
@@ -94,13 +99,25 @@ You can use the `--pull-request` option one or more times to load specific pull
 
 Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.
 
+You can load only pull requests in a certain state with the `--state` option:
+
+    $ github-to-sqlite pull-requests --state=open github.db simonw/datasette
+
+Pull requests across an entire organization (or more than one) can be loaded with `--org`:
+
+    $ github-to-sqlite pull-requests --state=open --org=psf --org=python github.db
+
+You can use a search query to find pull requests.  Note that no more than 1000 will be loaded (this is a GitHub API limitation), and some data will be missing (base and head SHAs).  When using searches, other filters are ignored; put all criteria into the search itself:
+
+    $ github-to-sqlite pull-requests --search='org:python defaultdict state:closed created:<2023-09-01' github.db
+
 Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)
 
 ## Fetching issue comments for a repository
 
 The `issue-comments` command retrieves all of the comments on all of the issues in a repository.
 
-It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.
+It is recommended you run `issues` first, so that each imported comment can have a foreign key pointing to its issue.
 
     $ github-to-sqlite issues github.db simonw/datasette
     $ github-to-sqlite issue-comments github.db simonw/datasette
@@ -113,7 +130,7 @@ Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/iss
 
 ## Fetching commits for a repository
 
-The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
+The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the SHA, commit message and author and committer details; it does not retrieve the full commit body.
 
     $ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
 
@@ -168,7 +185,7 @@ You can pass more than one username to fetch for multiple users or organizations
 
     $ github-to-sqlite repos github.db simonw dogsheep
 
-Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.
+Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a column called `readme_html`.
 
 Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)
 
@@ -228,7 +245,7 @@ You can fetch a list of every emoji supported by GitHub using the `emojis` comma
 
     $ github-to-sqlite emojis github.db
 
-This will create a table callad `emojis` with a primary key `name` and a `url` column.
+This will create a table called `emojis` with a primary key `name` and a `url` column.
 
 If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:
 
@@ -247,7 +264,7 @@ The `github-to-sqlite get` command provides a convenient shortcut for making aut
 
 This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
 
-You can ommit the `https://api.github.com/` prefix, for example:
+You can omit the `https://api.github.com/` prefix, for example:
 
     $ github-to-sqlite get /gists
 
@@ -258,5 +275,3 @@ Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.
 You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.
 
     $ github-to-sqlite get /users/simonw/repos --nl
-
-

github-to-sqlite-2.8.3/PKG-INFO → github_to_sqlite-2.9.1/README.md

@@ -1,15 +1,3 @@
-Metadata-Version: 2.1
-Name: github-to-sqlite
-Version: 2.8.3
-Summary: Save data from GitHub to a SQLite database
-Home-page: https://github.com/dogsheep/github-to-sqlite
-Author: Simon Willison
-License: Apache License, Version 2.0
-Platform: UNKNOWN
-Description-Content-Type: text/markdown
-Provides-Extra: test
-License-File: LICENSE
-
 # github-to-sqlite
 
 [![PyPI](https://img.shields.io/pypi/v/github-to-sqlite.svg)](https://pypi.org/project/github-to-sqlite/)
@@ -94,13 +82,25 @@ You can use the `--pull-request` option one or more times to load specific pull
 
 Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.
 
+You can load only pull requests in a certain state with the `--state` option:
+
+    $ github-to-sqlite pull-requests --state=open github.db simonw/datasette
+
+Pull requests across an entire organization (or more than one) can be loaded with `--org`:
+
+    $ github-to-sqlite pull-requests --state=open --org=psf --org=python github.db
+
+You can use a search query to find pull requests.  Note that no more than 1000 will be loaded (this is a GitHub API limitation), and some data will be missing (base and head SHAs).  When using searches, other filters are ignored; put all criteria into the search itself:
+
+    $ github-to-sqlite pull-requests --search='org:python defaultdict state:closed created:<2023-09-01' github.db
+
 Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)
 
 ## Fetching issue comments for a repository
 
 The `issue-comments` command retrieves all of the comments on all of the issues in a repository.
 
-It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.
+It is recommended you run `issues` first, so that each imported comment can have a foreign key pointing to its issue.
 
     $ github-to-sqlite issues github.db simonw/datasette
     $ github-to-sqlite issue-comments github.db simonw/datasette
@@ -113,7 +113,7 @@ Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/iss
 
 ## Fetching commits for a repository
 
-The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
+The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the SHA, commit message and author and committer details; it does not retrieve the full commit body.
 
     $ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
 
@@ -168,7 +168,7 @@ You can pass more than one username to fetch for multiple users or organizations
 
     $ github-to-sqlite repos github.db simonw dogsheep
 
-Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.
+Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a column called `readme_html`.
 
 Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)
 
@@ -228,7 +228,7 @@ You can fetch a list of every emoji supported by GitHub using the `emojis` comma
 
     $ github-to-sqlite emojis github.db
 
-This will create a table callad `emojis` with a primary key `name` and a `url` column.
+This will create a table called `emojis` with a primary key `name` and a `url` column.
 
 If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:
 
@@ -247,7 +247,7 @@ The `github-to-sqlite get` command provides a convenient shortcut for making aut
 
 This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
 
-You can ommit the `https://api.github.com/` prefix, for example:
+You can omit the `https://api.github.com/` prefix, for example:
 
     $ github-to-sqlite get /gists
 
@@ -258,5 +258,3 @@ Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.
 You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.
 
     $ github-to-sqlite get /users/simonw/repos --nl
-
-

{github-to-sqlite-2.8.3 → github_to_sqlite-2.9.1}/github_to_sqlite/cli.py

@@ -1,5 +1,6 @@
 import click
 import datetime
+import itertools
 import pathlib
 import textwrap
 import os
@@ -104,19 +105,54 @@ def issues(db_path, repo, issue_ids, auth, load):
     type=click.Path(file_okay=True, dir_okay=False, allow_dash=True, exists=True),
     help="Load pull-requests JSON from this file instead of the API",
 )
-def pull_requests(db_path, repo, pull_request_ids, auth, load):
+@click.option(
+    "--org",
+    "orgs",
+    help="Fetch all pull requests from this GitHub organization",
+    multiple=True,
+)
+@click.option(
+    "--state",
+    help="Only fetch pull requests in this state",
+)
+@click.option(
+    "--search",
+    help="Find pull requests with a search query",
+)
+def pull_requests(db_path, repo, pull_request_ids, auth, load, orgs, state, search):
     "Save pull_requests for a specified repository, e.g. simonw/datasette"
     db = sqlite_utils.Database(db_path)
     token = load_token(auth)
-    repo_full = utils.fetch_repo(repo, token)
-    utils.save_repo(db, repo_full)
     if load:
+        repo_full = utils.fetch_repo(repo, token)
+        utils.save_repo(db, repo_full)
         pull_requests = json.load(open(load))
+        utils.save_pull_requests(db, pull_requests, repo_full)
+    elif search:
+        repos_seen = set()
+        search += " is:pr"
+        pull_requests = utils.fetch_searched_pulls_or_issues(search, token)
+        for pull_request in pull_requests:
+            pr_repo_url = pull_request["repository_url"]
+            if pr_repo_url not in repos_seen:
+                pr_repo = utils.fetch_repo(url=pr_repo_url)
+                utils.save_repo(db, pr_repo)
+                repos_seen.add(pr_repo_url)
+            utils.save_pull_requests(db, [pull_request], pr_repo)
     else:
-        pull_requests = utils.fetch_pull_requests(repo, token, pull_request_ids)
-
-    pull_requests = list(pull_requests)
-    utils.save_pull_requests(db, pull_requests, repo_full)
+        if orgs:
+            repos = itertools.chain.from_iterable(
+                utils.fetch_all_repos(token=token, org=org) for org in orgs
+            )
+        else:
+            repos = [utils.fetch_repo(repo, token)]
+        for repo_full in repos:
+            utils.save_repo(db, repo_full)
+            repo = repo_full["full_name"]
+            pull_requests = utils.fetch_pull_requests(
+                repo, state, token, pull_request_ids
+            )
+            utils.save_pull_requests(db, pull_requests, repo_full)
     utils.ensure_db_shape(db)
 
 
@@ -464,7 +500,9 @@ def scrape_dependents(db_path, repos, auth, verbose):
                     {
                         "repo": repo_full["id"],
                         "dependent": dependent_id,
-                        "first_seen_utc": datetime.datetime.utcnow().isoformat(),
+                        "first_seen_utc": datetime.datetime.now(datetime.timezone.utc)
+                        .replace(tzinfo=None)
+                        .isoformat(),
                     },
                     pk=("repo", "dependent"),
                     foreign_keys=(

{github-to-sqlite-2.8.3 → github_to_sqlite-2.9.1}/github_to_sqlite/utils.py

@@ -2,6 +2,7 @@ import base64
 import requests
 import re
 import time
+import urllib.parse
 import yaml
 
 FTS_CONFIG = {
@@ -74,16 +75,17 @@ FOREIGN_KEYS = [
 
 
 class GitHubError(Exception):
-    def __init__(self, message, status_code):
+    def __init__(self, message, status_code, headers=None):
         self.message = message
         self.status_code = status_code
+        self.headers = headers
 
     @classmethod
     def from_response(cls, response):
         message = response.json()["message"]
         if "git repository is empty" in message.lower():
             cls = GitHubRepositoryEmpty
-        return cls(message, response.status_code)
+        return cls(message, response.status_code, response.headers)
 
 
 class GitHubRepositoryEmpty(GitHubError):
@@ -169,8 +171,11 @@ def save_pull_requests(db, pull_requests, repo):
         # Add repo key
         pull_request["repo"] = repo["id"]
         # Pull request _links can be flattened to just their URL
-        pull_request["url"] = pull_request["_links"]["html"]["href"]
-        pull_request.pop("_links")
+        if "_links" in pull_request:
+            pull_request["url"] = pull_request["_links"]["html"]["href"]
+            pull_request.pop("_links")
+        else:
+            pull_request["url"] = pull_request["pull_request"]["html_url"]
         # Extract user
         pull_request["user"] = save_user(db, pull_request["user"])
         labels = pull_request.pop("labels")
@@ -178,8 +183,9 @@ def save_pull_requests(db, pull_requests, repo):
         if pull_request.get("merged_by"):
             pull_request["merged_by"] = save_user(db, pull_request["merged_by"])
         # Head sha
-        pull_request["head"] = pull_request["head"]["sha"]
-        pull_request["base"] = pull_request["base"]["sha"]
+        if "head" in pull_request:
+            pull_request["head"] = pull_request["head"]["sha"]
+            pull_request["base"] = pull_request["base"]["sha"]
         # Extract milestone
         if pull_request["milestone"]:
             pull_request["milestone"] = save_milestone(
@@ -223,6 +229,11 @@ def save_pull_requests(db, pull_requests, repo):
 
 
 def save_user(db, user):
+    # Under some conditions, GitHub caches removed repositories with
+    # stars and ends up leaving dangling `None` user references.
+    if user is None:
+        return None
+
     # Remove all url fields except avatar_url and html_url
     to_save = {
         key: value
@@ -286,12 +297,13 @@ def save_issue_comment(db, comment):
     return last_pk
 
 
-def fetch_repo(full_name, token=None):
+def fetch_repo(full_name=None, token=None, url=None):
     headers = make_headers(token)
     # Get topics:
     headers["Accept"] = "application/vnd.github.mercy-preview+json"
-    owner, slug = full_name.split("/")
-    url = "https://api.github.com/repos/{}/{}".format(owner, slug)
+    if url is None:
+        owner, slug = full_name.split("/")
+        url = "https://api.github.com/repos/{}/{}".format(owner, slug)
     response = requests.get(url, headers=headers)
     response.raise_for_status()
     return response.json()
@@ -352,7 +364,7 @@ def fetch_issues(repo, token=None, issue_ids=None):
             yield from issues
 
 
-def fetch_pull_requests(repo, token=None, pull_request_ids=None):
+def fetch_pull_requests(repo, state=None, token=None, pull_request_ids=None):
     headers = make_headers(token)
     headers["accept"] = "application/vnd.github.v3+json"
     if pull_request_ids:
@@ -364,11 +376,20 @@ def fetch_pull_requests(repo, token=None, pull_request_ids=None):
             response.raise_for_status()
             yield response.json()
     else:
-        url = "https://api.github.com/repos/{}/pulls?state=all&filter=all".format(repo)
+        state = state or "all"
+        url = f"https://api.github.com/repos/{repo}/pulls?state={state}"
         for pull_requests in paginate(url, headers):
             yield from pull_requests
 
 
+def fetch_searched_pulls_or_issues(query, token=None):
+    headers = make_headers(token)
+    url = "https://api.github.com/search/issues?"
+    url += urllib.parse.urlencode({"q": query})
+    for pulls_or_issues in paginate(url, headers):
+        yield from pulls_or_issues["items"]
+
+
 def fetch_issue_comments(repo, token=None, issue=None):
     assert "/" in repo
     headers = make_headers(token)
@@ -439,13 +460,17 @@ def fetch_stargazers(repo, token=None):
         yield from stargazers
 
 
-def fetch_all_repos(username=None, token=None):
-    assert username or token, "Must provide username= or token= or both"
+def fetch_all_repos(username=None, token=None, org=None):
+    assert (
+        username or token or org
+    ), "Must provide username= or token= or org= or a combination"
     headers = make_headers(token)
     # Get topics for each repo:
     headers["Accept"] = "application/vnd.github.mercy-preview+json"
     if username:
         url = "https://api.github.com/users/{}/repos".format(username)
+    elif org:
+        url = "https://api.github.com/orgs/{}/repos".format(org)
     else:
         url = "https://api.github.com/user/repos"
     for repos in paginate(url, headers):
@@ -463,6 +488,7 @@ def fetch_user(username=None, token=None):
 
 
 def paginate(url, headers=None):
+    url += ("&" if "?" in url else "?") + "per_page=100"
     while url:
         response = requests.get(url, headers=headers)
         # For HTTP 204 no-content this yields an empty list
@@ -671,7 +697,10 @@ def ensure_foreign_keys(db):
     for expected_foreign_key in FOREIGN_KEYS:
         table, column, table2, column2 = expected_foreign_key
         if (
-            expected_foreign_key not in db[table].foreign_keys
+            expected_foreign_key not in {
+                (fk.table, fk.column, fk.other_table, fk.other_column)
+                for fk in db[table].foreign_keys
+            }
             and
             # Ensure all tables and columns exist
             db[table].exists()
@@ -726,7 +755,7 @@ def scrape_dependents(repo, verbose=False):
         yield from repos
         # next page?
         try:
-            next_link = soup.select(".paginate-container")[0].find("a", text="Next")
+            next_link = soup.select(".paginate-container")[0].find("a", string="Next")
         except IndexError:
             break
         if next_link is not None:

github_to_sqlite-2.9.1/pyproject.toml

@@ -0,0 +1,39 @@
+[project]
+name = "github-to-sqlite"
+version = "2.9.1"
+description = "Save data from GitHub to a SQLite database"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "Apache-2.0"
+license-files = ["LICENSE"]
+dependencies = [
+    "sqlite-utils>4",
+    "requests",
+    "PyYAML",
+]
+
+[[project.authors]]
+name = "Simon Willison"
+
+[project.urls]
+Homepage = "https://github.com/dogsheep/github-to-sqlite"
+Changelog = "https://github.com/dogsheep/github-to-sqlite/releases"
+Issues = "https://github.com/dogsheep/github-to-sqlite/issues"
+CI = "https://github.com/dogsheep/github-to-sqlite/actions"
+
+[project.scripts]
+github-to-sqlite = "github_to_sqlite.cli:cli"
+
+[dependency-groups]
+dev = [
+    "pytest",
+    "requests-mock",
+    "bs4",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.18,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = ""

github_to_sqlite-2.9.1/pyproject.toml.orig

@@ -0,0 +1,37 @@
+[project]
+name = "github-to-sqlite"
+version = "2.9.1"
+description = "Save data from GitHub to a SQLite database"
+readme = "README.md"
+requires-python = ">=3.10"
+authors = [{name = "Simon Willison"}]
+license = "Apache-2.0"
+license-files = ["LICENSE"]
+dependencies = [
+    "sqlite-utils>4",
+    "requests",
+    "PyYAML",
+]
+
+[project.urls]
+Homepage = "https://github.com/dogsheep/github-to-sqlite"
+Changelog = "https://github.com/dogsheep/github-to-sqlite/releases"
+Issues = "https://github.com/dogsheep/github-to-sqlite/issues"
+CI = "https://github.com/dogsheep/github-to-sqlite/actions"
+
+[project.scripts]
+github-to-sqlite = "github_to_sqlite.cli:cli"
+
+[dependency-groups]
+dev = [
+    "pytest",
+    "requests-mock",
+    "bs4",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.18,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = ""

github-to-sqlite-2.8.3/README.md

@@ -1,248 +0,0 @@
-# github-to-sqlite
-
-[![PyPI](https://img.shields.io/pypi/v/github-to-sqlite.svg)](https://pypi.org/project/github-to-sqlite/)
-[![Changelog](https://img.shields.io/github/v/release/dogsheep/github-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/github-to-sqlite/releases)
-[![Tests](https://github.com/dogsheep/github-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest)
-[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE)
-
-Save data from GitHub to a SQLite database.
-
-<!-- toc -->
-
-- [Demo](#demo)
-- [How to install](#how-to-install)
-- [Authentication](#authentication)
-- [Fetching issues for a repository](#fetching-issues-for-a-repository)
-- [Fetching pull requests for a repository](#fetching-pull-requests-for-a-repository)
-- [Fetching issue comments for a repository](#fetching-issue-comments-for-a-repository)
-- [Fetching commits for a repository](#fetching-commits-for-a-repository)
-- [Fetching releases for a repository](#fetching-releases-for-a-repository)
-- [Fetching tags for a repository](#fetching-tags-for-a-repository)
-- [Fetching contributors to a repository](#fetching-contributors-to-a-repository)
-- [Fetching repos belonging to a user or organization](#fetching-repos-belonging-to-a-user-or-organization)
-- [Fetching specific repositories](#fetching-specific-repositories)
-- [Fetching repos that have been starred by a user](#fetching-repos-that-have-been-starred-by-a-user)
-- [Fetching users that have starred specific repos](#fetching-users-that-have-starred-specific-repos)
-- [Fetching GitHub Actions workflows](#fetching-github-actions-workflows)
-- [Scraping dependents for a repository](#scraping-dependents-for-a-repository)
-- [Fetching emojis](#fetching-emojis)
-- [Making authenticated API calls](#making-authenticated-api-calls)
-
-<!-- tocstop -->
-
-## Demo
-
-https://github-to-sqlite.dogsheep.net/ hosts a [Datasette](https://datasette.io/) demo of a database created by [running this tool](https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60) against all of the repositories in the [Dogsheep GitHub organization](https://github.com/dogsheep), plus the [datasette](https://github.com/simonw/datasette) and [sqlite-utils](https://github.com/simonw/sqlite-utils) repositories.
-
-## How to install
-
-    $ pip install github-to-sqlite
-
-## Authentication
-
-Create a GitHub personal access token: https://github.com/settings/tokens
-
-Run this command and paste in your new token:
-
-    $ github-to-sqlite auth
-
-This will create a file called `auth.json` in your current directory containing the required value. To save the file at a different path or filename, use the `--auth=myauth.json` option.
-
-As an alternative to using an `auth.json` file you can add your access token to an environment variable called `GITHUB_TOKEN`.
-
-## Fetching issues for a repository
-
-The `issues` command retrieves all of the issues belonging to a specified repository.
-
-    $ github-to-sqlite issues github.db simonw/datasette
-
-If an `auth.json` file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.
-
-You can point to a different location of `auth.json` using `-a`:
-
-    $ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
-
-You can use the `--issue` option one or more times to load specific issues:
-
-    $ github-to-sqlite issues github.db simonw/datasette --issue=1
-
-Example: [issues table](https://github-to-sqlite.dogsheep.net/github/issues)
-
-## Fetching pull requests for a repository
-
-While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.
-
-Following the API of issues, the `pull-requests` command retrieves all of the pull requests belonging to a specified repository.
-
-    $ github-to-sqlite pull-requests github.db simonw/datasette
-
-You can use the `--pull-request` option one or more times to load specific pull request:
-
-    $ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81
-
-Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.
-
-Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)
-
-## Fetching issue comments for a repository
-
-The `issue-comments` command retrieves all of the comments on all of the issues in a repository.
-
-It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.
-
-    $ github-to-sqlite issues github.db simonw/datasette
-    $ github-to-sqlite issue-comments github.db simonw/datasette
-
-You can use the `--issue` option to only load comments for a specific issue within that repository, for example:
-
-    $ github-to-sqlite issue-comments github.db simonw/datasette --issue=1
-
-Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/issue_comments)
-
-## Fetching commits for a repository
-
-The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
-
-    $ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
-
-The command accepts one or more repositories.
-
-By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using `--all`.
-
-Example: [commits table](https://github-to-sqlite.dogsheep.net/github/commits)
-
-## Fetching releases for a repository
-
-The `releases` command retrieves the releases for one or more repositories.
-
-    $ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils
-
-The command accepts one or more repositories.
-
-Example: [releases table](https://github-to-sqlite.dogsheep.net/github/releases)
-
-## Fetching tags for a repository
-
-The `tags` command retrieves all of the tags for one or more repositories.
-
-    $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils
-
-Example: [tags table](https://github-to-sqlite.dogsheep.net/github/tags)
-
-## Fetching contributors to a repository
-
-The `contributors` command retrieves details of all of the contributors for one or more repositories.
-
-    $ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils
-
-The command accepts one or more repositories. It populates a `contributors` table, with foreign keys to `repos` and `users` and a `contributions` table listing the number of commits to that repository for each contributor.
-
-Example: [contributors table](https://github-to-sqlite.dogsheep.net/github/contributors)
-
-## Fetching repos belonging to a user or organization
-
-The `repos` command fetches repos belonging to a user or organization.
-
-Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:
-
-    $ github-to-sqlite repos github.db
-
-To fetch repos belonging to a specific user or organization, provide their username as an argument:
-
-    $ github-to-sqlite repos github.db dogsheep # organization
-    $ github-to-sqlite repos github.db simonw # user
-
-You can pass more than one username to fetch for multiple users or organizations at once:
-
-    $ github-to-sqlite repos github.db simonw dogsheep
-
-Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.
-
-Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)
-
-## Fetching specific repositories
-
-You can use `-r` with the `repos` command one or more times to fetch just specific repositories.
-
-    $ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite
-
-## Fetching repos that have been starred by a user
-
-The `starred` command fetches the repos that have been starred by a user.
-
-    $ github-to-sqlite starred github.db simonw
-
-If you are using an `auth.json` file you can omit the username to retrieve the starred repos for the authenticated user.
-
-Example: [stars table](https://github-to-sqlite.dogsheep.net/github/stars)
-
-## Fetching users that have starred specific repos
-
-The `stargazers` command fetches the users that have starred the specified repos.
-
-    $ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite
-
-You can specify one or more repository using `owner/repo` syntax.
-
-Users fetched using this command will be inserted into the `users` table. Many-to-many records showing which repository they starred will be added to the `stars` table.
-
-## Fetching GitHub Actions workflows
-
-The `workflows` command fetches the YAML workflow configurations from each repository's `.github/workflows` directory and parses them to populate `workflows`, `jobs` and `steps` tables.
-
-    $ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite
-
-You can specify one or more repository using `owner/repo` syntax.
-
-Example: [workflows table](https://github-to-sqlite.dogsheep.net/github/workflows), [jobs table](https://github-to-sqlite.dogsheep.net/github/jobs), [steps table](https://github-to-sqlite.dogsheep.net/github/steps)
-
-## Scraping dependents for a repository
-
-The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example [simonw/datasette/network/dependents](https://github.com/simonw/datasette/network/dependents).
-
-This data is not yet available through the GitHub API. The `scrape-dependents` command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.
-
-    $ github-to-sqlite scrape-dependents github.db simonw/datasette
-
-The command accepts one or more repositories.
-
-Add `-v` for verbose output.
-
-Example: [dependents table](https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc)
-
-## Fetching emojis
-
-You can fetch a list of every emoji supported by GitHub using the `emojis` command:
-
-    $ github-to-sqlite emojis github.db
-
-This will create a table callad `emojis` with a primary key `name` and a `url` column.
-
-If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:
-
-    $ github-to-sqlite emojis emojis.db -f
-    [########----------------------------]  397/1799   22%  00:03:43
-
-You can then use the [datasette-render-images](https://github.com/simonw/datasette-render-images) plugin to browse them visually.
-
-Example: [emojis table](https://github-to-sqlite.dogsheep.net/github/emojis)
-
-## Making authenticated API calls
-
-The `github-to-sqlite get` command provides a convenient shortcut for making authenticated calls to the API. Once you have created your `auth.json` file (or set a `GITHUB_TOKEN` environment variable) you can use it like this:
-
-    $ github-to-sqlite get https://api.github.com/gists
-
-This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
-
-You can ommit the `https://api.github.com/` prefix, for example:
-
-    $ github-to-sqlite get /gists
-
-Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.com/en/rest/guides/traversing-with-pagination). You can follow this pagination and output a list of all of the resulting items using `--paginate`:
-
-    $ github-to-sqlite get /users/simonw/repos --paginate
-
-You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.
-
-    $ github-to-sqlite get /users/simonw/repos --nl

github-to-sqlite-2.8.3/github_to_sqlite.egg-info/SOURCES.txt

@@ -1,12 +0,0 @@
-LICENSE
-README.md
-setup.py
-github_to_sqlite/__init__.py
-github_to_sqlite/cli.py
-github_to_sqlite/utils.py
-github_to_sqlite.egg-info/PKG-INFO
-github_to_sqlite.egg-info/SOURCES.txt
-github_to_sqlite.egg-info/dependency_links.txt
-github_to_sqlite.egg-info/entry_points.txt
-github_to_sqlite.egg-info/requires.txt
-github_to_sqlite.egg-info/top_level.txt

github-to-sqlite-2.8.3/github_to_sqlite.egg-info/dependency_links.txt

@@ -1 +0,0 @@
-

github-to-sqlite-2.8.3/github_to_sqlite.egg-info/entry_points.txt

@@ -1,4 +0,0 @@
-
-        [console_scripts]
-        github-to-sqlite=github_to_sqlite.cli:cli
-    

github-to-sqlite-2.8.3/github_to_sqlite.egg-info/requires.txt

@@ -1,8 +0,0 @@
-sqlite-utils>=2.7.2
-requests
-PyYAML
-
-[test]
-pytest
-requests-mock
-bs4

github-to-sqlite-2.8.3/github_to_sqlite.egg-info/top_level.txt

@@ -1 +0,0 @@
-github_to_sqlite

github-to-sqlite-2.8.3/setup.cfg

@@ -1,4 +0,0 @@
-[egg_info]
-tag_build = 
-tag_date = 0
-

github-to-sqlite-2.8.3/setup.py

@@ -1,32 +0,0 @@
-from setuptools import setup
-import os
-
-VERSION = "2.8.3"
-
-
-def get_long_description():
-    with open(
-        os.path.join(os.path.dirname(os.path.abspath(__file__)), "README.md"),
-        encoding="utf8",
-    ) as fp:
-        return fp.read()
-
-
-setup(
-    name="github-to-sqlite",
-    description="Save data from GitHub to a SQLite database",
-    long_description=get_long_description(),
-    long_description_content_type="text/markdown",
-    author="Simon Willison",
-    url="https://github.com/dogsheep/github-to-sqlite",
-    license="Apache License, Version 2.0",
-    version=VERSION,
-    packages=["github_to_sqlite"],
-    entry_points="""
-        [console_scripts]
-        github-to-sqlite=github_to_sqlite.cli:cli
-    """,
-    install_requires=["sqlite-utils>=2.7.2", "requests", "PyYAML"],
-    extras_require={"test": ["pytest", "requests-mock", "bs4"]},
-    tests_require=["github-to-sqlite[test]"],
-)