Ruby C Extensions, Explained

• Background
• How To Use This Repository
• Strategies
  • Strategy 0, "isolated"
  • Strategy 1, "system"
  • Strategy 2a, "packaged_source"
  • Strategy 2b, "packaged_tarball"
  • Strategy 3, "precompiled"
• Strategy Combinations
  • "system", fallback to "packaged"
  • "precompiled", fall back to "packaged", leave option for "system"
• FAQ
  • How do you test cross-compiling gems on Github Actions?
  • What's the significance of the flowers you're using as a background image for your RubyKaigi slides?

Hello there! Welcome to Ruby C Extensions, Explained.

This repository, and the example ruby gems in it, were originally written as companion materials for some conference talks being given in 2021 by Mike Dalessio (@flavorjones).

(Video)

This talk focuses mostly on the mechanics of the gems in this repository.

(Video)

This talk covers additional material, including how to test extensions, how to think about Trust in the context of precompilation, and why Nokogiri chose to start precompiling.

Each of the gems in this repository demonstrate a different strategy for integrating with (or packaging) third-party libraries. A brief explanation of the strategies is below, and the code for each gem is in a distinct subdirectory, along with a README that contains a more complete explanation.

Each gem builds on the previous one if you tackle them in this order:

• /isolated
• /system
• /packaged_source
• /packaged_tarball
• /precompiled

These are actual, working gems! And they've all been pushed to rubygems.org:

• gem install rcee_isolated
• gem install rcee_system
• gem install rcee_packaged_source
• gem install rcee_packaged_tarball
• gem install rcee_precompiled

Or they can all be installed as dependencies of the meta-gem rcee:

• gem install rcee

Also note that these gems are tested. In particular, the precompiled native gems have a rather complex setup to build, install, and test across platforms using rake-compiler-dock and Github Actions. See the Actions running at https://github.com/flavorjones/ruby-c-extensions-explained/actions

A few commonly-followed approaches have emerged in the Ruby ecosystem. Broadly categorized, the strategies are:

A self-contained C extension (no third-party libraries).

See subdirectory /isolated.

Real-world examples:

• https://github.com/bcrypt-ruby/bcrypt-ruby

Find and use an third-party library already installed on the target system.

See subdirectory /system.

Real-world examples:

• https://github.com/rmagick/rmagick
• https://github.com/sparklemotion/sqlite3-ruby

Package the source code for the third-party library, and compile it into the C extension during installation.

See subdirectory /packaged_source.

Real-world examples:

• https://github.com/ruby/psych
• https://github.com/rubys/nokogumbo
• https://github.com/sass/sassc-ruby

Package the tarball distribution for the third-party library, use its configuration toolchain and build a library that the C extension can link against.

See subdirectory /packaged_tarball.

Real-world examples:

• https://github.com/sparklemotion/nokogiri
• https://github.com/kwilczynski/ruby-magic

Precompile the library and C extension ahead-of-time for a subset of platforms; fall back to packaged_tarball strategy for other platforms.

See subdirectory /precompiled

Real-world examples:

• https://github.com/sparklemotion/nokogiri
• https://github.com/kwilczynski/ruby-magic
• https://github.com/grpc/grpc/tree/master/src/ruby

These strategies can also be combined!

Real-world examples:

• https://github.com/ruby/psych

Real-world examples:

• https://github.com/sparklemotion/nokogiri
• https://github.com/kwilczynski/ruby-magic

I'll try to answer questions asked during conference Q&A here.

Nokogiri uses the rake-compiler-dock OCI images directly to build the gem; and then uses the upload-artifact/download-artifact actions to install that gem on the target system.

See https://github.com/sparklemotion/nokogiri/blob/main/.github/workflows/gem-install.yml for the concrete implementation!

I tweeted a brief teaser video in which I explained the joke!

When I edited my RubyKaigi talk about C extensions, some bits had to be cut for time. Here's a fun teaser that I put together from some of those bits. [video link] pic.twitter.com/wZjSepHNpq

— mike dalessio (@flavorjones) September 9, 2021

In short, my friend Mark once sent me flowers for my birthday along with a card that read:

Fetching nokogiri-1.8.0.gem (100%)
Building native extensions. This could take a while...

Mark is funny. This was the inspiration for the talk's title.