weekly_snippets 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 56b395cc3e7141db6ffdd1c1fab46b846c70fab6
+  data.tar.gz: 6c86ae22375a0bc52522595ffe84cad841251b55
+SHA512:
+  metadata.gz: 0e118fcbf069ab6d4d3a097c7c0f3af0977db2fb674102e6b160c97075ba10c6f79fe1369eca65f0795429c488be46ec8edb4bbd8f8669e82dface6413b03a41
+  data.tar.gz: df30c05e750d18d0857d82bb99b15fd2356edad0d7e5ce29c0edf797ee327524f636f2faa86f6702e4cf216acf0ee8910967e32efbb4f50859734c6a07a86990

data/README.md

@@ -0,0 +1,244 @@
+## weekly_snippets Gem
+
+[![Gem Version](https://badge.fury.io/rb/weekly_snippets.svg)](https://badge.fury.io/rb/weekly_snippets)
+[![Build Status](https://travis-ci.org/18F/weekly_snippets.svg?branch=master)](https://travis-ci.org/18F/weekly_snippets)
+[![Code Climate](https://codeclimate.com/github/18F/weekly_snippets/badges/gpa.svg)](https://codeclimate.com/github/18F/weekly_snippets)
+[![Test Coverage](https://codeclimate.com/github/18F/weekly_snippets/badges/coverage.svg)](https://codeclimate.com/github/18F/weekly_snippets)
+
+Standardizes different weekly snippet formats into a common format,
+[munges](http://en.wikipedia.org/wiki/Mung_(computer_term)) snippet text
+according to user-supplied rules, performs redaction of internal information,
+and publishes snippets in plaintext or Markdown format.
+
+Downloads and API docs are available on the [weekly_snippets RubyGems
+page](https://rubygems.org/gems/weekly_snippets). API documentation is written
+using [YARD markup](http://yardoc.org/).
+
+Contributed by the 18F team, part of the United States General Services
+Administration: https://18f.gsa.gov/
+
+### Motivation
+
+This gem was extracted from [the 18F Hub Joiner
+plugin](https://github.com/18F/hub/blob/master/_plugins/joiner.rb). That
+plugin manipulates [Jekyll-imported data](http://jekyllrb.com/docs/datafiles/)
+by removing or promoting private data, building indices, and performing joins
+between different data files so that the results appear as unified collections
+in Jekyll's `site.data` object. It serves as the first stage in a pipeline
+that also builds cross-references and canonicalizes data before generating
+static HTML pages and other artifacts.
+
+### Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'weekly_snippets'
+```
+
+And then execute:
+```
+$ bundle
+```
+
+Or install it yourself as:
+```
+$ gem install weekly_snippets
+```
+
+### Usage
+
+The [18F Hub](https://github.com/18F/hub) processes snippet data as [CSV
+files](https://en.wikipedia.org/wiki/Comma-separated_values) harvested from a
+web-based spreadsheet, stored using [timestamped
+filenames](http://en.wikipedia.org/wiki/ISO_8601#Calendar_dates) in the
+[Jekyll _data folder](http://jekyllrb.com/docs/datafiles/). Since we have
+experimented with different CSV column formats, we keep the data files
+corresponding to each version in separate directories:
+
+```
+$ ls -1d _data/private/snippets/*
+_data/private/snippets/v1/
+_data/private/snippets/v2/
+_data/private/snippets/v3/
+```
+
+The content of the lattermost `v3` directory as of writing:
+
+```
+$ ls -1 _data/private/snippets/v3
+20141208.csv
+20141215.csv
+20141222.csv
+```
+
+With this data in-place, the Hub performs the following steps:
+
+#### Standardize versions
+
+The [18F Hub joiner.rb
+plugin](https://github.com/18F/hub/blob/master/_plugins/joiner.rb) defines
+this map from version names to `Version` objects:
+
+```ruby
+# Used to standardize snippet data of different versions before joining
+# and publishing.
+SNIPPET_VERSIONS = {
+  'v1' => WeeklySnippets::Version.new(
+    version_name:'v1',
+    field_map:{
+      'Username' => 'username',
+      'Timestamp' => 'timestamp',
+      'Name' => 'full_name',
+      'Snippets' => 'last-week',
+      'No This Week' => 'this-week',
+    }
+  ),
+  'v2' => WeeklySnippets::Version.new(
+    version_name:'v2',
+    field_map:{
+      'Timestamp' => 'timestamp',
+      'Public vs. Private' => 'public',
+      'Last Week' => 'last-week',
+      'This Week' => 'this-week',
+      'Username' => 'username',
+    },
+    markdown_supported: true
+  ),
+  'v3' => WeeklySnippets::Version.new(
+    version_name:'v3',
+    field_map:{
+      'Timestamp' => 'timestamp',
+      'Public' => 'public',
+      'Username' => 'username',
+      'Last week' => 'last-week',
+      'This week' => 'this-week',
+    },
+    public_field: 'public',
+    public_value: 'Public',
+    markdown_supported: true
+  ),
+}
+```
+
+This map is then used to standardize batches of weekly snippets, converting
+each different version to a common format, before joining the data with team
+member information:
+
+```ruby
+# Snippet data is expected to be stored in files matching the pattern:
+# _data/@source/snippets/[version]/[YYYYMMDD].csv
+#
+# resulting in the initial structure:
+# site.data[@source][snippets][version][YYYYMMDD] = Array<Hash>
+#
+# After this function returns, the `standardized` will be of the form:
+# site.data[snippets][YYYYMMDD] = Array<Hash>
+standardized = ::WeeklySnippets::Version.standardize_versions(
+  @data[@source]['snippets'], snippet_versions)
+```
+
+#### Munge
+
+To accommodate the preferred formats employed by some team members, the
+[18F Hub snippets.rb plugin](https://github.com/18F/hub/blob/master/_plugins/snippets.rb)
+defines a Ruby block to
+[munge](http://en.wikipedia.org/wiki/Mung_(computer_term)) the snippet data
+before converting it to a uniform
+[Markdown](http://daringfireball.net/projects/markdown/syntax) representation:
+
+```ruby
+MARKDOWN_SNIPPET_MUNGER = Proc.new do |text|
+  text.gsub!(/^::: (.*) :::$/, "#{HEADLINE} \\1") # For jtag. ;-)
+  text.gsub!(/^\*\*\*/, HEADLINE) # For elaine. ;-)
+end
+```
+
+This block is then passed as an argument to `WeeklySnippets::Publisher.new()`,
+discussed in the **Publish** section below.
+
+#### Redact internal info
+
+Text that should be available when published internally, but redacted from
+publicly-published snippets, can be surrounded by `{{` and `}}` tokens:
+
+```ruby
+> require 'weekly_snippets/publisher'
+
+# Instantiate a publisher for internally-visible snippets
+> publisher = WeeklySnippets::Publisher.new(
+    headline: "\n####", public_mode: false)
+
+> snippets = [
+    '- Did stuff{{ including private details}}',
+    '{{- Did secret stuff}}',
+    '- Did more stuff',
+    '{{- Did more secret stuff',
+    '- Yet more secret stuff}}',
+  ]
+
+# For internally-visible snippets, the text inside the `{{` and `}}`
+# tokens will be preserved.
+> puts publisher.redact! snippets.join("\n")
+
+- Did stuff including private details
+- Did secret stuff
+- Did more stuff
+- Did more secret stuff
+- Yet more secret stuff
+
+# Instantiate a publisher for publicly-visible snippets
+> publisher = WeeklySnippets::Publisher.new(
+    headline: "\n####", public_mode: true)
+
+# For publicly-visible snippets, the text inside the `{{` and `}}`
+# tokens will be removed.
+> puts publisher.redact! snippets.join("\n")
+
+- Did stuff
+- Did more stuff
+```
+
+`WeeklySnippets::Publisher::publish()` automatically calls
+`WeeklySnippets::Publisher::redact!`, so it shouldn't be necessary to call it
+directly.
+
+#### Publish
+
+This is how the 
+[18F Hub snippets.rb plugin](https://github.com/18F/hub/blob/master/_plugins/snippets.rb)
+creates a `WeeklySnippets::Publisher` object and uses its `publish()` method
+to process snippets and store the result in the [Jekyll site.data
+object](http://jekyllrb.com/docs/datafiles/):
+
+```ruby
+publisher = ::WeeklySnippets::Publisher.new(
+  headline: HEADLINE, public_mode: site.config['public'],
+  markdown_snippet_munger: MARKDOWN_SNIPPET_MUNGER)
+site.data['snippets'] = publisher.publish site.data['snippets']
+```
+
+### Contributing
+
+1. Fork the repo ( https://github.com/18F/weekly_snippets/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request
+
+Feel free to ping [@mbland](https://github.com/mbland) with any questions you
+may have, especially if the current documentation should've addressed your
+needs, but didn't.
+
+### Public domain
+
+This project is in the worldwide [public domain](LICENSE.md). As stated in
+[CONTRIBUTING](CONTRIBUTING.md):
+
+> This project is in the public domain within the United States, and copyright
+> and related rights in the work worldwide are waived through the
+> [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+>
+> All contributions to this project will be released under the CC0 dedication.
+> By submitting a pull request, you are agreeing to comply with this waiver of
+> copyright interest.

data/lib/weekly_snippets.rb

@@ -0,0 +1,18 @@
+# 18F Snippets - Standardize, munge, redact, and publish weekly snippets
+#
+# Written in 2014 by Mike Bland (michael.bland@gsa.gov)
+# on behalf of the 18F team, part of the US General Services Administration:
+# https://18f.gsa.gov/
+#
+# To the extent possible under law, the author(s) have dedicated all copyright
+# and related and neighboring rights to this software to the public domain
+# worldwide. This software is distributed without any warranty.
+#
+# You should have received a copy of the CC0 Public Domain Dedication along
+# with this software. If not, see
+# <https://creativecommons.org/publicdomain/zero/1.0/>.
+#
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require "weekly_snippets/version"
+require "weekly_snippets/publisher"

data/lib/weekly_snippets/publisher.rb

@@ -0,0 +1,110 @@
+# 18F Snippets - Standardize, munge, redact, and publish weekly snippets
+#
+# Written in 2014 by Mike Bland (michael.bland@gsa.gov)
+# on behalf of the 18F team, part of the US General Services Administration:
+# https://18f.gsa.gov/
+#
+# To the extent possible under law, the author(s) have dedicated all copyright
+# and related and neighboring rights to this software to the public domain
+# worldwide. This software is distributed without any warranty.
+#
+# You should have received a copy of the CC0 Public Domain Dedication along
+# with this software. If not, see
+# <https://creativecommons.org/publicdomain/zero/1.0/>.
+#
+# @author Mike Bland (michael.bland@gsa.gov)
+
+module WeeklySnippets
+  class Publisher
+    # @param headline [String] Markdown string used to markup each section
+    #   title within a block of snippet text
+    # @param public_mode [true, false, or nil] indicates whether or not the
+    #   snippets are to be published publicly; private snippets and redacted
+    #   spans of text will not be published when +public_mode+ is true
+    # @param markdown_snippet_munger [Proc] the code block that will be called
+    #   with a String of snippet text after redaction and before Markdown
+    #   preparation, to modify the snippet text String in-place; will not be
+    #   called if the snippet's version doesn't support Markdown syntax
+    def initialize(headline:, public_mode:, markdown_snippet_munger:nil)
+      @headline = headline
+      @public_mode = public_mode
+      @markdown_snippet_munger = markdown_snippet_munger
+    end
+
+    # Processes +snippets+ entries for publication. Any snippets that should
+    # not appear when in +public_mode+ are removed from +snippets+
+    def publish(snippets)
+      result = {}
+      snippets.each do |timestamp, snippet_batch|
+        published = []
+        snippet_batch.each do |snippet|
+          unless @public_mode and !snippet['public']
+            publish_snippet(snippet, published)
+          end
+        end
+        result[timestamp] = published unless published.empty?
+      end
+      result
+    end
+
+    # Parses and publishes a snippet. Filters out snippets rendered empty
+    # after redaction.
+    # @param snippet [Hash<String,String>] snippet hash with two fields:
+    #   +last-week+ and +this-week+
+    # @param published [Array<Hash<String,String>>] array of published snippets
+    def publish_snippet(snippet, published)
+      ['last-week', 'this-week'].each do |field|
+        text = snippet[field] || ''
+        redact! text
+        if snippet['markdown']
+          @markdown_snippet_munger.yield text if @markdown_snippet_munger
+          text = prepare_markdown text
+        end
+        snippet[field] = text.empty? ? nil : text
+      end
+
+      is_empty = (snippet['last-week'] || '').empty? && (
+        snippet['this-week'] || '').empty?
+      published << snippet unless is_empty
+    end
+
+    # Parses "{{" and "}}" redaction markers. For public snippets, will redact
+    # everything between each set of markers. For internal snippets, will only
+    # remove the markers.
+    def redact!(text)
+      if @public_mode
+        text.gsub!(/\n?\{\{.*?\}\}/m,'')
+      else
+        text.gsub!(/(\{\{|\}\})/,'')
+      end
+    end
+
+    # Processes snippet text in Markdown format to smooth out any anomalies
+    # before rendering. Also translates arbitrary plaintext to Markdown.
+    #
+    # @param text [String] snippet text
+    # @return [String]
+    def prepare_markdown(text)
+      parsed = []
+      uses_item_markers = (text =~ /^[-*]/)
+
+      text.each_line do |line|
+        line.rstrip!
+        # Convert headline markers.
+        line.sub!(/^(#+)/, @headline)
+
+        # Add item markers for those who used plaintext and didn't add them;
+        # add headline markers for those who defined different sections and
+        # didn't add them.
+        if line =~ /^([A-Za-z0-9])/
+          line = uses_item_markers ? "#{@headline} #{line}" : "- #{line}"
+        end
+
+        # Fixup item markers missing a space.
+        line.sub!(/^[-*]([^ ])/, '- \1')
+        parsed << line unless line.empty?
+      end
+      parsed.join("\n")
+    end
+  end
+end

data/lib/weekly_snippets/version.rb

@@ -0,0 +1,172 @@
+# 18F Snippets - Standardize, munge, redact, and publish weekly snippets
+#
+# Written in 2014 by Mike Bland (michael.bland@gsa.gov)
+# on behalf of the 18F team, part of the US General Services Administration:
+# https://18f.gsa.gov/
+#
+# To the extent possible under law, the author(s) have dedicated all copyright
+# and related and neighboring rights to this software to the public domain
+# worldwide. This software is distributed without any warranty.
+#
+# You should have received a copy of the CC0 Public Domain Dedication along
+# with this software. If not, see
+# <https://creativecommons.org/publicdomain/zero/1.0/>.
+#
+# @author Mike Bland (michael.bland@gsa.gov)
+
+module WeeklySnippets
+  VERSION = "0.0.1"
+
+  # Encapsulates the mapping from actual snippet data fields to a standardized
+  # set of data fields for each version of snippets.
+  #
+  # Since 18F experimented with a handful of different snippet formats, with
+  # slightly different field names and semantics, we needed a way to transform
+  # each batch into a common format before generating Hub pages, to streamline
+  # the logic and possibly allow for even more formats in the future without
+  # requiring version-specific hacks.
+  #
+  # The common format fields are:
+  # - username: identifies the snippet author
+  # - last-week: summary of last week's activity
+  # - this-week: summary of this week's anticipated activity
+  # - timestamp: identifies when the snippet was reported; might not
+  #   necessarily match the timestamp of the batch in which it appears
+  # - public: true if the snippet may be published publicly
+  # - markdown: true if the snippet version supports Markdown syntax
+  #
+  # When using Jekyll, snippet data should be stored in the _data directory
+  # with versioned subdirectories containing timestamped Comma Separated Value
+  # (CSV) files, e.g.:
+  # - _data/snippets/v1/20141110.csv
+  # - _data/snippets/v2/20141201.csv
+  # - _data/snippets/v3/20141208.csv
+  #
+  # Jekyll imports this data into a Hash structure (site.data['snippets'])
+  # resembling:
+  # - version => { timestamp => [ snippets ] }
+  #
+  # Version.standardize_versions will convert this structure into a Hash
+  # resembling:
+  # - timestamp => [ snippets ]
+  class Version
+    # Set of field name values that the +field_map+ argument of +initialize+
+    # must map to.
+    FIELD_NAMES = ['username', 'last-week', 'this-week', 'timestamp']
+
+    # Raised by +initialize+ when the initialization parameters are flawed.
+    class InitError < ::Exception
+    end
+
+    attr_reader(:version_name, :field_map, :public_field, :public_value,
+      :markdown_supported)
+
+    # @param version_name [String] identifies the version, e.g. "v3"
+    # @param field_map [Hash<String, String>] contains the mapping from the
+    #   field name in the original data file to the standardized internal
+    #   field name
+    # @param public_field [String] if present, the field that indicates whether
+    #   or not a snippet can be published in public mode; if not present, no
+    #   snippets matching this version should be published publicly
+    # @param public_value [String] if present, the value for +public_field+
+    #   that indicates whether or not a snippet should be published in public
+    #   mode
+    # @param markdown_supported [true,false] indicates whether or not the
+    #   snippet version supports Markdown syntax
+    # @raise [InitError] if +field_map+ does not contain mappings for every
+    #   element of FIELD_NAMES
+    # @raise [InitError] if one of +public_field+ or +public_value+ is set, but
+    #   not the other
+    def initialize(version_name:, field_map:, public_field:nil,
+      public_value:nil, markdown_supported: false)
+
+      expected = FIELD_NAMES.sort
+      actual = field_map.values.sort
+      intersection = expected & actual
+
+      unless intersection == expected
+        raise InitError.new("Snippet version \"#{version_name}\" " +
+          "missing mappings for fields: #{expected - intersection}")
+      end
+
+      unless (public_field == nil and public_value == nil) or (
+        public_field != nil and public_value != nil)
+        raise InitError.new("Snippet version \"#{version_name}\" has " +
+          "public_field and public_value mismatched: " +
+          "public_field == #{public_field ? "\"#{public_field}\"" : 'nil'}; " +
+          "public_value == #{public_value ? "\"#{public_value}\"" : 'nil'}")
+      end
+
+      @version_name = version_name
+      @field_map = field_map
+      @public_field = public_field
+      @public_value = public_value
+      @markdown_supported = markdown_supported
+    end
+
+    # Raised by +standardize+ when a snippet contains fields not contained in
+    # +field_map+.
+    class UnknownFieldError < ::Exception
+    end
+
+    # Converts the field names within +snippet+ to standardized names using
+    # +field_map+, and sets snippet[public] and snippet[markdown].
+    #
+    # @param snippet [Hash<String, String>] snippet data to evaluate
+    # @return [Hash<String,String>] +snippet+
+    # @raise [UnknownFieldError] if +snippet+ contains fields not contained in
+    #   +field_map+
+    def standardize(snippet)
+      snippet.keys.each do |k|
+        unless @field_map.member? k
+          raise UnknownFieldError.new("Snippet field not recognized by " +
+            "version \"#{@version_name}\": #{k}")
+        end
+        snippet[@field_map[k]] = snippet.delete k
+      end
+      snippet['public'] = (@public_field and
+        snippet[@public_field] == @public_value) ? true : false
+      snippet['markdown'] = @markdown_supported
+      snippet
+    end
+
+    # Raised by +standardize_versions+ if a snippet version is unknown.
+    class UnknownVersionError < ::Exception
+    end
+
+    # Transforms snippets of different versions into a standard format.
+    #
+    # The keys of +snippets_by_version+ should indicate the version of the
+    # corresponding batch of snippets, and also match the keys of
+    # +snippet_versions+. Each batch of snippets for each version should be a
+    # Hash from a timestamp string (typically YYYYMMDD) to an Array of Hashes
+    # representing individual snippet entries.
+    #
+    # The resulting Hash will map from timestamp string to an Array of
+    # standardized snippet Hashes, eliminating the now unnecessary version
+    # information.
+    #
+    # @param snippets_by_version [Hash<String, Hash<String, Array<Hash>>>]
+    #   contains: version => { timestamp => [ snippets ] }
+    # @param snippet_versions [Hash<String,Snippets::Version>] mapping from
+    #   snippet version name to the corresponding Snippets::Version object
+    # @return [Hash<String, Array<Hash>>] a mapping from a (weekly) timestamp
+    #   to a corresponding set of standardized snippets
+    #
+    # @raise [UnknownVersionError] if any snippets correspond to versions not
+    #   in +snippet_versions+
+    def self.standardize_versions(snippets_by_version, snippet_versions)
+      result = {}
+      snippets_by_version.each do |version, batch|
+        v = snippet_versions[version]
+        unless v
+          raise UnknownVersionError.new("Unknown snippet version: #{version}")
+        end
+        batch.each do |timestamp, snippets|
+          result[timestamp] = snippets.each {|s| v.standardize s}
+        end
+      end
+      result
+    end
+  end
+end

metadata

@@ -0,0 +1,106 @@
+--- !ruby/object:Gem::Specification
+name: weekly_snippets
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Mike Bland
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-01-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: codeclimate-test-reporter
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Standardizes different weekly snippet formats into a common format, munges
+  snippet text according to user-supplied rules, performs redaction of internal information,
+  and publishes snippets in plaintext or Markdown format.
+email:
+- michael.bland@gsa.gov
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/weekly_snippets.rb
+- lib/weekly_snippets/publisher.rb
+- lib/weekly_snippets/version.rb
+homepage: https://github.com/18F/weekly_snippets
+licenses:
+- CC0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Standardize, munge, redact, and publish weekly snippets
+test_files: []