Snapcrawl - crawl a website and take screenshots

Snapcrawl is a command line utility for crawling a website and saving screenshots.

Snapcrawl relies on two deprecated libraries:

• Webshot (last updated in August 2020)
• PhantomJS (last updated around 2020)

As such, the project is no longer actively maintained and is unlikely to receive updates or bug fixes.

If you are interested in contributing and have ideas for replacing these libraries with modern alternatives, you are welcome to propose changes via pull requests or issues.

• Crawls a website to any given depth and saves screenshots
• Can capture the full length of the page
• Can use a specific resolution for screenshots
• Skips capturing if the screenshot was already saved recently
• Uses local caching to avoid expensive crawl operations if not needed
• Reports broken links

Using Docker

You can run Snapcrawl by using this docker image (which contains all the necessary prerequisites):

$ alias snapcrawl='docker run --rm -it --network host --volume "$PWD:/app" dannyben/snapcrawl'

For more information on the Docker image, refer to the docker-snapcrawl repository.

Using Ruby

$ gem install snapcrawl

Note that Snapcrawl requires PhantomJS and ImageMagick.

Snapcrawl can be configured either through a configuration file (YAML), or by specifying options in the command line.

$ snapcrawl
Usage:
  snapcrawl URL [--config FILE] [SETTINGS...]
  snapcrawl -h | --help
  snapcrawl -v | --version

The default configuration filename is snapcrawl.yml.

Using the --config flag will create a template configuration file if it is not present:

$ snapcrawl example.com --config snapcrawl

All configuration options can be specified in the command line as key=value pairs:

$ snapcrawl example.com log_level=0 depth=2 width=1024

# All values below are the default values

# log level (0-4) 0=DEBUG 1=INFO 2=WARN 3=ERROR 4=FATAL
log_level: 1

# log_color (yes, no, auto)
# yes  = always show log color
# no   = never use colors
# auto = only use colors when running in an interactive terminal
log_color: auto

# number of levels to crawl, 0 means capture only the root URL
depth: 1

# screenshot width in pixels
width: 1280

# screenshot height in pixels, 0 means the entire height
height: 0

# number of seconds to consider the page cache and its screenshot fresh
cache_life: 86400

# where to store the HTML page cache
cache_dir: cache

# where to store screenshots
snaps_dir: snaps

# screenshot filename template, where '%{url}' will be replaced with a 
# slug version of the URL (no need to include the .png extension)
name_template: '%{url}'

# urls not matching this regular expression will be ignored
url_whitelist: 

# urls matching this regular expression will be ignored
url_blacklist: 

# take a screenshot of this CSS selector only
css_selector: 

# when true, ignore SSL related errors
skip_ssl_verification: false

# set to any number of seconds to wait for the page to load before taking
# a screenshot, leave empty to not wait at all (only needed for pages with
# animations or other post-load events).
screenshot_delay: 

If you experience any issue, have a question or a suggestion, or if you wish to contribute, feel free to open an issue.