Standardizes different weekly snippet formats into a common format, munges snippet text according to user-supplied rules, performs redaction of internal information, and publishes snippets in plaintext or Markdown format.

Downloads and API docs are available on the weekly_snippets RubyGems page. API documentation is written using YARD markup.

Contributed by the 18F team, part of the United States General Services Administration: https://18f.gsa.gov/

This gem was extracted from the 18F Hub Joiner plugin. That plugin manipulates Jekyll-imported data by removing or promoting private data, building indices, and performing joins between different data files so that the results appear as unified collections in Jekyll's site.data object. It serves as the first stage in a pipeline that also builds cross-references and canonicalizes data before generating static HTML pages and other artifacts.

Add this line to your application's Gemfile:

gem 'weekly_snippets'

And then execute:

$ bundle

Or install it yourself as:

$ gem install weekly_snippets

The 18F Hub processes snippet data as CSV files harvested from a web-based spreadsheet, stored using timestamped filenames in the Jekyll _data folder. Since we have experimented with different CSV column formats, we keep the data files corresponding to each version in separate directories:

$ ls -1d _data/private/snippets/*
_data/private/snippets/v1/
_data/private/snippets/v2/
_data/private/snippets/v3/

The content of the lattermost v3 directory as of writing:

$ ls -1 _data/private/snippets/v3
20141208.csv
20141215.csv
20141222.csv

With this data in-place, the Hub performs the following steps:

The 18F Hub joiner.rb plugin defines this map from version names to Version objects:

# Used to standardize snippet data of different versions before joining
# and publishing.
SNIPPET_VERSIONS = {
  'v1' => WeeklySnippets::Version.new(
    version_name:'v1',
    field_map:{
      'Username' => 'username',
      'Timestamp' => 'timestamp',
      'Name' => 'full_name',
      'Snippets' => 'last-week',
      'No This Week' => 'this-week',
    }
  ),
  'v2' => WeeklySnippets::Version.new(
    version_name:'v2',
    field_map:{
      'Timestamp' => 'timestamp',
      'Public vs. Private' => 'public',
      'Last Week' => 'last-week',
      'This Week' => 'this-week',
      'Username' => 'username',
    },
    markdown_supported: true
  ),
  'v3' => WeeklySnippets::Version.new(
    version_name:'v3',
    field_map:{
      'Timestamp' => 'timestamp',
      'Public' => 'public',
      'Username' => 'username',
      'Last week' => 'last-week',
      'This week' => 'this-week',
    },
    public_field: 'public',
    public_value: 'Public',
    markdown_supported: true
  ),
}

This map is then used to standardize batches of weekly snippets, converting each different version to a common format, before joining the data with team member information:

# Snippet data is expected to be stored in files matching the pattern:
# _data/@source/snippets/[version]/[YYYYMMDD].csv
#
# resulting in the initial structure:
# site.data[@source][snippets][version][YYYYMMDD] = Array<Hash>
#
# After this function returns, the `standardized` will be of the form:
# site.data[snippets][YYYYMMDD] = Array<Hash>
standardized = ::WeeklySnippets::Version.standardize_versions(
  @data[@source]['snippets'], snippet_versions)

To accommodate the preferred formats employed by some team members, the 18F Hub snippets.rb plugin defines a Ruby block to munge the snippet data before converting it to a uniform Markdown representation:

MARKDOWN_SNIPPET_MUNGER = Proc.new do |text|
  text.gsub!(/^::: (.*) :::$/, "#{HEADLINE} \\1") # For jtag. ;-)
  text.gsub!(/^\*\*\*/, HEADLINE) # For elaine. ;-)
end

This block is then passed as an argument to WeeklySnippets::Publisher.new(), discussed in the Publish section below.

Text that should be available when published internally, but redacted from publicly-published snippets, can be surrounded by {{ and }} tokens:

> require 'weekly_snippets/publisher'

# Instantiate a publisher for internally-visible snippets
> publisher = WeeklySnippets::Publisher.new(
    headline: "\n####", public_mode: false)

> snippets = [
    '- Did stuff{{ including private details}}',
    '{{- Did secret stuff}}',
    '- Did more stuff',
    '{{- Did more secret stuff',
    '- Yet more secret stuff}}',
  ]

# For internally-visible snippets, the text inside the `{{` and `}}`
# tokens will be preserved.
> puts publisher.redact! snippets.join("\n")

- Did stuff including private details
- Did secret stuff
- Did more stuff
- Did more secret stuff
- Yet more secret stuff

# Instantiate a publisher for publicly-visible snippets
> publisher = WeeklySnippets::Publisher.new(
    headline: "\n####", public_mode: true)

# For publicly-visible snippets, the text inside the `{{` and `}}`
# tokens will be removed.
> puts publisher.redact! snippets.join("\n")

- Did stuff
- Did more stuff

WeeklySnippets::Publisher::publish() automatically calls WeeklySnippets::Publisher::redact!, so it shouldn't be necessary to call it directly.

This is how the 18F Hub snippets.rb plugin creates a WeeklySnippets::Publisher object and uses its publish() method to process snippets and store the result in the Jekyll site.data object:

publisher = ::WeeklySnippets::Publisher.new(
  headline: HEADLINE, public_mode: site.config['public'],
  markdown_snippet_munger: MARKDOWN_SNIPPET_MUNGER)
site.data['snippets'] = publisher.publish site.data['snippets']

1. Fork the repo ( https://github.com/18F/weekly_snippets/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Feel free to ping @mbland with any questions you may have, especially if the current documentation should've addressed your needs, but didn't.

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.