html2rss is a Ruby gem that generates RSS 2.0 feeds from websites automatically, and as a fallback via feed config.

With the feed config, you provide a URL to scrape and CSS selectors for extracting information (like title, URL, etc.). The gem builds the RSS feed accordingly. Extractors and chainable post processors make information extraction, processing, and sanitizing a breeze. The gem also supports scraping JSON responses and setting HTTP request headers.

Looking for a ready-to-use app to serve generated feeds via HTTP? Check out html2rss-web!

Support the development by sponsoring this project on GitHub. Thank you! 💓

Install Ruby (latest version is recommended) on your machine and run gem install html2rss in your terminal.

After the installation has finished, html2rss help will print usage information.

html2rss offers an automatic RSS generation feature. Try it with:

html2rss auto https://unmatchedstyle.com/

If the results are not to your satisfaction, you can create a feed config file.

Create a file called my_config_file.yml with this sample content:

channel:
  url: https://unmatchedstyle.com
selectors:
  items:
    selector: "article[id^='post-']"
  title:
    selector: h2
  link:
    selector: a
    extractor: href
  description:
    selector: ".post-content"
    post_process:
      - name: sanitize_html

Build the feed from this config with: html2rss feed ./my_config_file.yml.

You can also install it as a dependency in your Ruby project:

Here's a minimal working example using Ruby:

require 'html2rss'

rss =
  Html2rss.feed(
    channel: { url: 'https://stackoverflow.com/questions' },
    selectors: {
      items: { selector: '#hot-network-questions > ul > li' },
      title: { selector: 'a' },
      link: { selector: 'a', extractor: 'href' }
    }
  )

puts rss

A feed config consists of a channel and a selectors hash. The contents of both hashes are explained below.

Good to know:

• You'll find extensive example feed configs at spec/*.test.yml.
• See html2rss-configs for ready-made feed configs!
• If you've created feed configs, you're invited to send a PR to html2rss-configs to make your config available to the public.

Alright, let's move on.

Sometimes there are structurally similar pages with different URLs. In such cases, you can add dynamic parameters to the channel's attributes.

Example of a dynamic id parameter in the channel URLs:

channel:
  url: "http://domainname.tld/whatever/%<id>s.html"

Command line usage example:

html2rss feed the_feed_config.yml id=42

config = Html2rss::Config.new({ channel: { url: 'http://domainname.tld/whatever/%<id>s.html' } }, {}, { id: 42 })
Html2rss.feed(config)

See the more complex formatting options of the sprintf method.

First, you must give an items selector hash, which contains a CSS selector. The selector selects a collection of HTML tags from which the RSS feed items are built. Except for the items selector, all other keys are scoped to each item of the collection.

To build a valid RSS 2.0 item, you need at least a title or a description. You can have both.

Having an items and a title selector is enough to build a simple feed.

Your selectors hash can contain arbitrary named selectors, but only a few will make it into the RSS feed (due to the RSS 2.0 specification):

Every named selector (i.e. title, description, see table above) in your selectors hash can have these attributes:

Extractors help with extracting the information from the selected HTML tag.

• The default extractor is text, which returns the tag's inner text.
• The html extractor returns the tag's outer HTML.
• The href extractor returns a URL from the tag's href attribute and corrects relative ones to absolute ones.
• The attribute extractor returns the value of that tag's attribute.
• The static extractor returns the configured static value (it doesn't extract anything).
• See file list of extractors.

Extractors might need extra attributes on the selector hash. 👉 Read their docs for usage examples.

Html2rss.feed(
  channel: {}, selectors: { link: { selector: 'a', extractor: 'href' } }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  link:
    selector: "a"
    extractor: "href"

Extracted information can be further manipulated with post processors.

⚠️ Always make use of the sanitize_html post processor for HTML content. Never trust the internet! ⚠️

Pass an array to post_process to chain the post processors.

channel:
  # ... omitted
selectors:
  # ... omitted
  price:
    selector: '.price'
  description:
    selector: '.section'
    post_process:
      - name: template
        string: |
          # %{self}

          Price: %{price}
      - name: markdown_to_html

The post processor gsub makes use of Ruby's gsub method.

Html2rss.feed(
  channel: {},
  selectors: {
    title: { selector: 'a', post_process: [{ name: 'gsub', pattern: 'foo', replacement: 'bar' }] }
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  title:
    selector: "a"
    post_process:
      - name: "gsub"
        pattern: "foo"
        replacement: "bar"

The categories selector takes an array of selector names. Each value of those selectors will become a <category> on the RSS item.

Html2rss.feed(
  channel: {},
  selectors: {
    genre: {
      # ... omitted
      selector: '.genre'
    },
    branch: { selector: '.branch' },
    categories: %i[genre branch]
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  genre:
    selector: ".genre"
  branch:
    selector: ".branch"
  categories:
    - genre
    - branch

By default, html2rss generates a GUID from the title or description.

If this does not work well, you can choose other attributes from which the GUID is build. The principle is the same as for the categories: pass an array of selectors names.

In all cases, the GUID is a SHA1-encoded string.

Html2rss.feed(
  channel: {},
  selectors: {
    title: {
      # ... omitted
      selector: 'h1'
    },
    link: { selector: 'a', extractor: 'href' },
    guid: %i[link]
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  title:
    selector: "h1"
  link:
    selector: "a"
    extractor: "href"
  guid:
    - link

An enclosure can be any file, e.g. a image, audio or video - think Podcast.

The enclosure selector needs to return a URL of the content to enclose. If the extracted URL is relative, it will be converted to an absolute one using the channel's URL as base.

Since html2rss does no further inspection of the enclosure, its support comes with trade-offs:

1. The content-type is guessed from the file extension of the URL, unless one is specified in content_type.
2. If the content-type guessing fails, it will default to application/octet-stream.
3. The content-length will always be undetermined and therefore stated as 0 bytes.

Read the RSS 2.0 spec for further information on enclosing content.

Html2rss.feed(
  channel: {},
  selectors: {
    enclosure: {
      selector: 'audio',
      extractor: 'attribute',
      attribute: 'src',
      content_type: 'audio/mp3'
    }
  }
)

channel:
  # ... omitted
selectors:
  # ... omitted
  enclosure:
    selector: "audio"
    extractor: "attribute"
    attribute: "src"
    content_type: "audio/mp3"

By default, html2rss assumes the URL responds with HTML. However, it can also handle JSON responses. The JSON response must be an Array or Hash.

The JSON is converted to XML which you can query using CSS selectors.

Html2rss.feed(
  channel: { url: 'http://domainname.tld/whatever.json', json: true },
  selectors: { title: { selector: 'foo' } }
)

channel:
  url: "http://domainname.tld/whatever.json"
  json: true
selectors:
  title:
    selector: "foo"

By default, html2rss issues a naiive HTTP request and extracts information from the response. That is performant and works for many websites.

However, modern websites often do not render much HTML on the server, but evaluate JavaScript on the client to create the HTML. In such cases, the default strategy will not find the "juicy content".

You can use Browserless.io to run a Chrome browser and return the website's source code after the website generated it. For this, you can either run your own Browserless.io instance (Docker image available -- read their license!) or pay them for a hosted instance.

To run a local Browserless.io instance, you can use the following Docker command:

docker run \
  --rm \
  -p 3000:3000 \
  -e "CONCURRENT=10" \
  -e "TOKEN=6R0W53R135510" \
  ghcr.io/browserless/chromium

To make html2rss use your instance,

1. specify the environment variables accordingly, and
2. use the browserless strategy for those websites.

When running locally with commands from above, you can skip setting the environment variables, as they are aligned with the default values.

BROWSERLESS_IO_WEBSOCKET_URL="ws://127.0.0.1:3000" BROWSERLESS_IO_API_TOKEN="6R0W53R135510" \
  html2rss auto --strategy=browserless https://example.com

When using traditional feed configs, inside your channel config set strategy: browserless.

channel:
  url: https://www.imdb.com/user/ur67728460/ratings
  time_zone: UTC
  ttl: 1440
  strategy: browserless
  headers:
    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
selectors:
  items:
    selector: "li.ipc-metadata-list-summary-item"
  title:
    selector: ".ipc-title__text"
    post_process:
      - name: gsub
        pattern: "/^(\\d+.)\\s/"
        replacement: ""
      - name: template
        string: "%{self} rated with: %{user_rating}"
  link:
    selector: "a.ipc-title-link-wrapper"
    extractor: "href"
  user_rating:
    selector: "[data-testid='ratingGroup--other-user-rating'] > .ipc-rating-star--rating"

To set HTTP request headers, you can add them to the channel's headers hash. This is useful for APIs that require an Authorization header.

channel:
  url: "https://example.com/api/resource"
  headers:
    Authorization: "Bearer YOUR_TOKEN"
selectors:
  # ... omitted

Or for setting a User-Agent:

channel:
  url: "https://example.com"
  headers:
    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
selectors:
  # ... omitted

This step is not required to work with this gem. If you're using html2rss-web and want to create your private feed configs, keep on reading!

First, create a YAML file, e.g. feeds.yml. This file will contain your global config and multiple feed configs under the key feeds.

Example:

headers:
  "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1"
feeds:
  myfeed:
    channel:
    selectors:
  myotherfeed:
    channel:
    selectors:

Your feed configs go below feeds. Everything else is part of the global config.

Find a full example of a feeds.yml at spec/fixtures/feeds.test.yml.

Now you can build your feeds like this:

require 'html2rss'

myfeed = Html2rss.feed_from_yaml_config('feeds.yml', 'myfeed')
myotherfeed = Html2rss.feed_from_yaml_config('feeds.yml', 'myotherfeed')

html2rss feed feeds.yml myfeed
html2rss feed feeds.yml myotherfeed

To display RSS feeds nicely in a web browser, you can:

• add a plain old CSS stylesheet, or
• use XSLT (eXtensible Stylesheet Language Transformations).

A web browser will apply these stylesheets and show the contents as described.

In a CSS stylesheet, you'd use element selectors to apply styles.

If you want to do more, then you need to create a XSLT. XSLT allows you to use a HTML template and to freely design the information of the RSS, including using JavaScript and external resources.

You can add as many stylesheets and types as you like. Just add them to your global configuration.

config = Html2rss::Config.new(
  { channel: {}, selectors: {} }, # omitted
  {
    stylesheets: [
      {
        href: '/relative/base/path/to/style.xls',
        media: :all,
        type: 'text/xsl'
      },
      {
        href: 'http://example.com/rss.css',
        media: :all,
        type: 'text/css'
      }
    ]
  }
)

Html2rss.feed(config)

stylesheets:
  - href: "/relative/base/path/to/style.xls"
    media: "all"
    type: "text/xsl"
  - href: "http://example.com/rss.css"
    media: "all"
    type: "text/css"
feeds:
  # ... omitted

Recommended further readings:

• How to format RSS with CSS on lifewire.com
• XSLT: Extensible Stylesheet Language Transformations on MDN
• The XSLT used by html2rss-web

• Check that the channel URL does not redirect to a mobile page with a different markup structure.
• Do not rely on your web browser's developer console. html2rss does not execute JavaScript.
• Fiddling with curl and pup to find the selectors seems efficient (curl URL | pup).
• CSS selectors are versatile. Here's an overview.

Find ideas what to contribute in:

1. https://github.com/orgs/html2rss/discussions
2. the issues tracker: https://github.com/html2rss/html2rss/issues

To submit changes:

1. Fork this repo ( https://github.com/html2rss/html2rss/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Implement a commit your changes (git commit -am 'feat: add XYZ')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request using the Github web UI

1. bin/setup: installs dependencies and sets up the development environment.
2. for a modern Ruby development experience: install ruby-lsp and integrate it to your IDE.

For example: Ruby in Visual Studio Code.