Project: github-ripper - The Ruby Toolbox

Overview

Ghrip is a command line tool for ripping (cloning) repositories from GitHub in bulk. It allows you to clone all repositories owned by a named user or users, or from an organisation or organisations.

To make the cloning as fast as possible we make extensive use of the Parallel gem to run as many clones as possible in parallel.

Tokens

There is no requirement to supply a token when using ghrip, however there are numerous benefits. including but not limited to:

Less impact from connection throttling (Unauthenticated connections are throttled more heavily by GitHub)
Ability to clone private repositories
Simple to clone all of your own repositories (no need to specify a username)

You could for example clone ALL the repositories that you own with a single command.

# ghrip -t <Your token goes here> -A

If you just wanted your personal repositories you could use -U instead of -A, or -M if you just wanted the repositories for organisations that you are a member of.

If you do not supply token you will be limited to just cloning public repositories.

No Passwords?

We intentionally do NOT handle username + password combinations when talking to GitHub, this removes some security risks and also removes the issue of having to handle MFA logins. It allows you to create a read-only token for cloning only to further improve your security position.

Example

The following is an example of how to rip all of the public repositories owned by Wolf Software.

# ghrip -o WolfSoftware

Usage

Usage: ghrip
    -h, --help                       Display this screen

Parameters:
    -t, --token <token>              GitHub personal access token (PAT)
    -b, --base-dir <path>            The base directory to download to
    -g, --use-git                    Use git instead of https to clone the repositories

Cloning Parameters:
    -u, --user <names>               Github username(s) to rip
    -o, --org <names>                Github organisation(s) to rip
    -U, --user-repos                 Rip all of the repositories for the named user(s)
    -M, --org-member-repos           Rip all of the repositories for all organisation the user(s) is a member of
    -A, --all-repos                  Same as running -U -M
    -O, --org-repos                  Rip all of the repositories for the named organisation(s)

Flags:
    -d, --dry-run                    Show a list of repositories that WOULD be ripped
    -f, --full                       Show status of all repositories in post run report
    -q, --quiet                      Suppress the showing of the post run report
    -s, --silent                     Suppress all output

If you just supply a username and nothing else it will default to just the users own repositories (same as -U)

Default Values

Option Name	Default Value	Purpose
-t, --token	No default	The token used to authenticate yourself (not required but helps remove/mitigate throttling issues)
-b, --base-dir	~/Downloads/repos	The directory to place the cloned repos in
-g, --use-git	False	Use git instead of https to clone the repos, this requires your public key to be configured on GitHub
-u, --user	No default	The user(s) you wish to rip the repo of (if not supplied defaults to yourself IF a token is used)
-o, --org	No default	The organisation(s) you want to rip the repos of
-U, --user-repos	False	Download all repositories belonging to the named user(s)
-M, --org-member-repos	False	Download all repositories belonging to any organisation the named user(s) is a member of
-A, --all-repos	False	Works the same as --user-repo and --org-member-repos combined
-O, --org-repos	False	Download all repositories belonging to the named organisation(s)
-d, --dry-run	False	Display the list of repositories that would be downloaded
-f, --full	False	Show all repositories in the post run report instead of just errors
-q, --quiet	False	Suppress the showing of the post run report
-s, --silent	False	Suppress all output

users and orgs can be either a single entry or comma-separated list of entries.

Error Handling

All errors raised from github-core-lister are caught and re-thrown as StandardErrors in order to simplify the catching.

github-ripper

Development