team_api 0.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 6740624d81a90a97d8d0f86e4495d74fd26d1b54
+  data.tar.gz: 3fc55eee6ee7d09912249b15962525fe9add3da7
+SHA512:
+  metadata.gz: 474780225962c7350a00ea29b8e1fa80dc6b062918b4746a8e6984f0bcd7577319febf787cbf29dd70c8c3aabf34e9d993c3dd6ec3c7851c3c6880c4fe97e905
+  data.tar.gz: 7c53f0fcb51b54a3e30d62f93d48068e5c3e5b497c715d8c5f9d66f4b9eed4b02608dce56463eb3949973c23b7fbadf6e6aa6475f2b435d71f161e722b5748af

data/CONTRIBUTING.md

@@ -0,0 +1,15 @@
+## Welcome!
+
+We're so glad you're thinking about contributing to an 18F open source project! If you're unsure or afraid of anything, just ask or submit the issue or pull request anyways. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contribution, and don't want a wall of rules to get in the way of that.
+
+Before contributing, we encourage you to read our CONTRIBUTING policy (you are here), our LICENSE, and our README, all of which should be in this repository. If you have any questions, or want to read more about our underlying policies, you can consult the 18F Open Source Policy GitHub repository at https://github.com/18f/open-source-policy, or just shoot us an email/official government letterhead note to [18f@gsa.gov](mailto:18f@gsa.gov).
+
+## Public domain
+
+This project is in the public domain within the United States, and
+copyright and related rights in the work worldwide are waived through
+the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+
+All contributions to this project will be released under the CC0
+dedication. By submitting a pull request, you are agreeing to comply
+with this waiver of copyright interest.

data/LICENSE.md

@@ -0,0 +1,31 @@
+As a work of the United States Government, this project is in the
+public domain within the United States.
+
+Additionally, we waive copyright and related rights in the work
+worldwide through the CC0 1.0 Universal public domain dedication.
+
+## CC0 1.0 Universal Summary
+
+This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode).
+
+### No Copyright
+
+The person who associated a work with this deed has dedicated the work to
+the public domain by waiving all of his or her rights to the work worldwide
+under copyright law, including all related and neighboring rights, to the
+extent allowed by law.
+
+You can copy, modify, distribute and perform the work, even for commercial
+purposes, all without asking permission.
+
+### Other Information
+
+In no way are the patent or trademark rights of any person affected by CC0,
+nor are the rights that other persons may have in the work or in how the
+work is used, such as publicity or privacy rights.
+
+Unless expressly stated otherwise, the person who associated a work with
+this deed makes no warranties about the work, and disclaims liability for
+all uses of the work, to the fullest extent permitted by applicable law.
+When using or citing the work, you should not imply endorsement by the
+author or the affirmer.

data/README.md

@@ -0,0 +1,53 @@
+# 18F Team API
+
+Compiles information about team members, projects, etc. and exposes it via a
+JSON API.
+
+Targeted consumers of this API include:
+
+- [18F Hub](https://github.com/18F/hub)
+- [18F Dashboard](https://github.com/18F/dashboard)
+- [18F.gsa.gov](https://github.com/18F/18f.gsa.gov)
+
+## Installation
+
+This gem currently serves as a [Jekyll](https://jekyllrb.com/) plugin, though
+it may become decoupled in the future. Presuming you're using
+[Bundler](http://bundler.io) in your Jekyll project, add the following to your
+`Gemfile`:
+
+```ruby
+group :jekyll_plugins do
+  gem 'team_api'
+end
+```
+
+Then, make sure to add an entry for `api_index_layout:` to your `_config.yml`
+file. The index page will have an `endpoints` collection with one entry per
+data collection, where each element has:
+
+* `endpoint`: the URL of the collection's JSON endpoint
+* `title`: title of the collection
+* `description`: description of the collection
+
+Here's a sample bare-bones template you can drop into your prefered layout:
+
+```html
+<h1>API Endpoint Index</h1>
+<br/>{% for i in page.endpoints %}
+<div class="api_endpoint_desc">
+<h2><a href="{{ i.endpoint }}"<code>{{ i.endpoint }}</code></a> - {{ i.title }}</h2>
+<p>{{ i.description }}</p>
+</div>
+{% endfor %}
+```
+
+## Public domain
+
+This project is in the worldwide [public domain](LICENSE.md). As stated in [CONTRIBUTING](CONTRIBUTING.md):
+
+> This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+>
+> All contributions to this project will be released under the CC0
+>dedication. By submitting a pull request, you are agreeing to comply
+>with this waiver of copyright interest.

data/lib/team_api/README.md

@@ -0,0 +1,33 @@
+## 18F Team API Plugins
+
+Plugins are used to create data joins and cross-references needed to produce
+the API. The basic flow is:
+
+* Join private data with public data
+* Process snippet data
+* Build cross-references between data elements
+* Perform canonicalization of names and their ordering
+* Generate API endpoints based on the joined, cross-referenced data
+
+[generator.rb](generator.rb) is the entry point for this entire process. It
+contains `TeamApi::Generator`, which performs all of the above steps in order.
+
+### Data Joining
+
+[joiner.rb](joiner.rb) contains the plugins that join public, private, and
+local data into the `site.data` object.
+
+### Cross-Referencing
+
+[cross_referencer.rb](cross_referencer.rb) builds links between `site.data`
+data collections which are used to generate cross-referenced pages.
+
+### Canonicalization
+
+[canonicalizer.rb](canonicalizer.rb) contains functions used to canonicalize
+names and the sort order of collections in `site.data`.
+
+### API Endpoint Generation
+
+[api.rb](api.rb) generates all API endpoints and provides an index under
+`/api`.

data/lib/team_api/api.rb

@@ -0,0 +1,217 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require_relative 'canonicalizer'
+require_relative 'config'
+require 'jekyll'
+require 'json'
+require 'safe_yaml'
+
+module TeamApi
+  class IndexPage < ::Jekyll::Page
+    private_class_method :new
+
+    def initialize(site)
+      @site = site
+      @base = site.source
+      @dir = File.join site.config['baseurl'], Api::BASEURL
+      @name = 'index.html'
+      @data = {}
+    end
+
+    def self.create(site, index_endpoints)
+      index_page = new site
+      index_page.process index_page.name
+      layout = site.config['api_index_layout']
+      fail '`api_index_layout:` not defined in _config.yml' unless layout
+      index_page.read_yaml File.join(site.source, '_layouts'), layout
+      index_page.data['endpoints'] = index_endpoints
+      site.pages << index_page
+    end
+  end
+
+  class Endpoint < ::Jekyll::Page
+    private_class_method :new
+
+    def initialize(site, endpoint_path)
+      @site = site
+      @base = site.source
+      @dir = endpoint_path
+      @name = 'api.json'
+      @data = {}
+    end
+
+    def self.create(site, endpoint_path, data)
+      endpoint = new site, endpoint_path
+      endpoint.process endpoint.name
+      endpoint.content = data.to_json
+      site.pages << endpoint
+    end
+  end
+
+  # Functions for generating JSON objects as part of an API
+  class Api
+    BASEURL = 'api'
+
+    # Generates all of the API endpoints.
+    # +site+:: Jekyll site object
+    def self.generate_api(site)
+      impl = ApiImpl.new site, BASEURL
+      generate_collection_endpoints impl
+      generate_tag_category_endpoints impl
+      impl.generate_snippets_endpoints
+      IndexPage.create site, impl.index_endpoints
+    end
+
+    # Calculates the full URL prefix for every API endpoint, used to generate
+    # `self:` links. It is generated by concatenating the `url:` and
+    # `baseurl:` values from _config.yml, and the Api::BASEURL value, e.g.
+    # localhost:4001/api, https://team-api.18f.gov/public/api.
+    def self.baseurl(site)
+      File.join site.config['url'], site.config['baseurl'], BASEURL
+    end
+
+    def self.add_self_links(site)
+      baseurl = self.baseurl site
+      Config.endpoint_config.each do |endpoint_info|
+        collection_name = endpoint_info['collection']
+        (site.data[collection_name] || {}).values.each do |item|
+          slug = Canonicalizer.canonicalize item[endpoint_info['item_id']]
+          item['self'] = File.join baseurl, collection_name, slug
+        end
+      end
+    end
+
+    def self.generate_collection_endpoints(impl)
+      Config.endpoint_config.each do |endpoint_info|
+        impl.generate_index_endpoint_for_collection endpoint_info
+        impl.generate_item_endpoints endpoint_info['collection']
+      end
+    end
+    private_class_method :generate_collection_endpoints
+
+    def self.generate_tag_category_endpoints(impl)
+      %w(Skills Interests).each do |tag_category|
+        impl.generate_tag_category_endpoint tag_category
+        impl.generate_item_endpoints Canonicalizer.canonicalize(tag_category)
+      end
+    end
+    private_class_method :generate_tag_category_endpoints
+  end
+
+  module ApiImplSnippetHelpers
+    private
+
+    def snippet_dates
+      @snippet_dates ||= (data['snippets'] || {}).keys.sort.reverse
+    end
+
+    def snippets
+      @snippets ||= snippet_dates.map { |t| [t, data['snippets'][t]] }.to_h
+    end
+
+    def snippets_by_user
+      @snippets_by_user ||= snippets
+        .flat_map { |date, batch| batch.map { |snippet| [date, snippet] } }
+        .group_by { |_date, snippet| snippet['name'] }
+        .map { |name, mapping| [name, mapping.to_h] }
+        .to_h
+    end
+
+    def snippets_summary
+      @snippet_summary ||= {
+        'latest' => snippet_dates.first,
+        'all' => snippet_dates,
+        'users' => Canonicalizer.team_xrefs(
+          data['team'], snippets_by_user.keys),
+      }
+    end
+
+    def generate_latest_snippet_endpoint
+      return if snippets.empty?
+      latest = snippets.first
+      endpoint = 'snippets/latest'
+      Endpoint.create(site, "#{baseurl}/#{endpoint}",
+        { 'datestamp' => latest[0] }.merge(envelop(endpoint, latest[1])))
+    end
+
+    def generate_snippets_by_date_endpoints
+      snippets.each do |timestamp, batch|
+        endpoint = "snippets/#{timestamp}"
+        Endpoint.create site, "#{baseurl}/#{endpoint}", envelop(endpoint, batch)
+      end
+    end
+
+    def generate_snippets_by_user_endpoints
+      snippets_by_user.each do |name, batch|
+        Endpoint.create site, "#{baseurl}/snippets/#{name}", batch
+        Endpoint.create(
+          site, "#{baseurl}/snippets/#{name}/latest", [batch.first].to_h)
+      end
+    end
+
+    def generate_snippets_index_summary_endpoint
+      generate_index_endpoint(
+        'snippets', 'Snippets', 'Summary of all available snippets',
+        snippets_summary) unless snippets.empty?
+    end
+  end
+
+  class ApiImpl
+    attr_accessor :site, :data, :index_endpoints, :baseurl
+    include ApiImplSnippetHelpers
+
+    def initialize(site, baseurl)
+      @site = site
+      @data = site.data
+      @index_endpoints = []
+      @baseurl = baseurl
+    end
+
+    def self_link(endpoint)
+      File.join site.config['url'], baseurl, endpoint
+    end
+
+    def envelop(endpoint, items)
+      return if items.nil? || items.empty?
+      { 'self' => self_link(endpoint), 'results' => items }
+    end
+
+    def generate_index_endpoint(endpoint, title, description, items)
+      return if items.nil? || items.empty?
+      Endpoint.create site, "#{baseurl}/#{endpoint}", items
+      index_endpoints << {
+        'endpoint' => endpoint, 'title' => title, 'description' => description
+      }
+    end
+
+    def generate_tag_category_endpoint(category)
+      canonicalized = Canonicalizer.canonicalize(category)
+      generate_index_endpoint(canonicalized, category,
+        "Index of team members by #{category.downcase}",
+        envelop(canonicalized, (data[canonicalized] || {}).values))
+    end
+
+    def generate_index_endpoint_for_collection(endpoint_info)
+      collection = endpoint_info['collection']
+      generate_index_endpoint(
+        endpoint_info['collection'], endpoint_info['title'],
+        endpoint_info['description'],
+        envelop(collection, (data[collection] || {}).values))
+    end
+
+    def generate_item_endpoints(collection_name)
+      (data[collection_name] || {}).each do |identifier, value|
+        identifier = Canonicalizer.canonicalize(identifier)
+        url = "#{baseurl}/#{collection_name}/#{identifier}"
+        Endpoint.create site, url, value
+      end
+    end
+
+    def generate_snippets_endpoints
+      generate_latest_snippet_endpoint
+      generate_snippets_by_date_endpoints
+      generate_snippets_by_user_endpoints
+      generate_snippets_index_summary_endpoint
+    end
+  end
+end

data/lib/team_api/canonicalizer.rb

@@ -0,0 +1,125 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require_relative 'config'
+require_relative 'cross_referencer'
+
+module TeamApi
+  # Contains utility functions for canonicalizing names and the order of data.
+  class Canonicalizer
+    # Canonicalizes the order and names of certain fields within site_data.
+    def self.canonicalize_data(site_data)
+      sort_collections site_data
+      canonicalize_tag_category site_data['skills']
+      canonicalize_tag_category site_data['interests']
+    end
+
+    def self.sort_collections(site_data)
+      Config.endpoint_config.each do |endpoint_info|
+        collection = endpoint_info['collection']
+        next unless site_data.member? collection
+        sorted = sort_collection_values(endpoint_info,
+          site_data[collection].values)
+        sort_item_xrefs endpoint_info, sorted
+        item_id_field = endpoint_info['item_id']
+        site_data[collection] = sorted.map { |i| [i[item_id_field], i] }.to_h
+      end
+    end
+
+    def self.sort_collection_values(endpoint_info, values)
+      sort_by_field = endpoint_info['sort_by']
+      if sort_by_field == 'last_name'
+        sort_by_last_name values
+      else
+        values.sort_by { |i| (i[sort_by_field] || '').downcase }
+      end
+    end
+    private_class_method :sort_collection_values
+
+    def self.sort_item_xrefs(endpoint_info, collection)
+      collection.each do |item|
+        sortable_item_fields(item, endpoint_info).each do |field, field_info|
+          item[field] = sort_collection_values field_info, item[field]
+        end
+      end
+    end
+    private_class_method :sort_item_xrefs
+
+    def self.sortable_item_fields(item, collection_endpoint_info)
+      collection_endpoint_info['item_collections'].map do |item_spec|
+        field, endpoint_info = parse_collection_spec item_spec
+        [field, endpoint_info] if item[field]
+      end.compact
+    end
+    private_class_method :sortable_item_fields
+
+    def self.parse_collection_spec(collection_spec)
+      if collection_spec.instance_of? Hash
+        [collection_spec['field'],
+         Config.endpoint_info_by_collection[collection_spec['collection']]]
+      else
+        [collection_spec, Config.endpoint_info_by_collection[collection_spec]]
+      end
+    end
+
+    # Returns a canonicalized, URL-friendly substitute for an arbitrary string.
+    # +s+:: string to canonicalize
+    def self.canonicalize(s)
+      s.downcase.gsub(/\s+/, '-')
+    end
+
+    def self.comparable_name(person)
+      if person['last_name']
+        [person['last_name'].downcase, person['first_name'].downcase]
+      else
+        # Trim off title suffix, if any.
+        full_name = person['full_name'].downcase.split(',')[0]
+        last_name = full_name.split.last
+        [last_name, full_name]
+      end
+    end
+    private_class_method :comparable_name
+
+    # Sorts an array of team member data hashes based on the team members'
+    # last names.
+    # +team+:: An array of team member data hashes
+    def self.sort_by_last_name(team)
+      team.sort_by { |member| comparable_name member }
+    end
+
+    def self.team_xrefs(team, usernames)
+      fields = CrossReferencer::TEAM_FIELDS
+      usernames
+        .map { |username| team[username] }
+        .compact
+        .map { |member| member.select { |field, _| fields.include? field } }
+        .sort_by { |member| comparable_name member }
+    end
+
+    # Breaks a YYYYMMDD timestamp into a hyphenated version: YYYY-MM-DD
+    # +timestamp+:: timestamp in the form YYYYMMDD
+    def self.hyphenate_yyyymmdd(timestamp)
+      "#{timestamp[0..3]}-#{timestamp[4..5]}-#{timestamp[6..7]}"
+    end
+
+    # Consolidate tags entries that are not exactly the same. Selects the
+    # lexicographically smaller version of the tag as a standard.
+    #
+    # In the future, we may just consider raising an error if there are two
+    # different strings for the same thing.
+    def self.canonicalize_tag_category(tags_xrefs)
+      return if tags_xrefs.nil? || tags_xrefs.empty?
+      tags_xrefs.replace(CrossReferencer.map_reduce(tags_xrefs.values,
+        ->(xref) { [[xref['slug'], xref]] }, method(:consolidate_xrefs)).to_h)
+    end
+
+    def self.consolidate_xrefs(slug, xrefs)
+      xrefs.sort_by! { |xref| xref['name'] }
+      result = xrefs.each_with_object(xrefs.shift) do |xref, consolidated|
+        consolidated['members'].concat xref['members']
+      end
+      result['members'].sort_by! { |member| comparable_name member }
+      [slug, result]
+    end
+    private_class_method :consolidate_xrefs
+  end
+end

data/lib/team_api/config.rb

@@ -0,0 +1,18 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+module TeamApi
+  class Config
+    def self.endpoint_config
+      @endpoint_config ||= begin
+        endpoint_config_path = File.join File.dirname(__FILE__), 'endpoints.yml'
+        SafeYAML.load_file endpoint_config_path, safe: true
+      end
+    end
+
+    def self.endpoint_info_by_collection
+      @endpoint_info_by_collection ||= Config.endpoint_config.map do |item|
+        [item['collection'], item]
+      end.to_h
+    end
+  end
+end

data/lib/team_api/cross_referencer.rb

@@ -0,0 +1,202 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require_relative 'api'
+require_relative 'canonicalizer'
+
+module TeamApi
+  # Signals that a cross-reference ID value in one object is not present in
+  # the target collection. Only raised in "private" mode, since "public" mode
+  # may legitimately filter out data.
+  class UnknownCrossReferenceTargetId < StandardError
+  end
+
+  # Provides a collection with the ability to replace identifiers with more
+  # detailed cross-reference values from another collection, and with the
+  # ability to construct its own cross-reference values to assign to values
+  # from other collections.
+  #
+  # The intent is to provide enough cross-reference information to surface in
+  # an API without requiring the client to join the data necessary to produce
+  # cross-links. For example, instead of surfacing `['mbland']` in a list of
+  # team members, this class will produce `[{'name' => 'mbland', 'full_name'
+  # => 'Mike Bland', 'first_name' => 'Mike', 'last_name' => 'Bland'}]`, which
+  # the client can use to more easily sort multiple values and transform into:
+  # `<a href="https://hub.18f.gov/team/mbland/">Mike Bland</a>`.
+  class CrossReferenceData
+    attr_accessor :collection_name, :data, :item_xref_fields, :public_mode
+
+    # @param site [Jekyll::Site] site object
+    # @param collection_name [String] name of collection within site.data
+    # @param field_to_xref [String] name of the field to cross-reference
+    # @param item_xref_fields [Array<String>] list of fields from which to
+    #   produce cross-references for this collection
+    def initialize(site, collection_name, item_xref_fields)
+      @collection_name = collection_name
+      @data = site.data[collection_name] || {}
+      @item_xref_fields = item_xref_fields
+      @public_mode = site.config['public']
+    end
+
+    # Selects fields from `item` to produce a smaller hash as a
+    # cross-reference.
+    def item_to_xref(item)
+      item.select { |field, _| item_xref_fields.include? field }
+    end
+
+    # Translates identifiers into cross-reference values in both this object's
+    # collection and the `target` collection.
+    #
+    # This object's collection is considered the "source", and references to
+    # its values will be injected into "target". For each "source" object,
+    # `source[target.collection_name]` should be an existing field containing
+    # identifiers that are keys into `target.data`. The `target` collection
+    # values should not contain a `target[source.collection_name]` field; that
+    # field will be created by this method.
+    #
+    # @param target [CrossReferenceData] contains data to cross-reference with
+    #   items from this object's collection
+    # @param source_to_target_field [String] if specified, the field from this
+    #   collection's objects that contain identifiers of objects stored within
+    #   target; if not specified, target.collection_name will be used instead
+    def create_xrefs(target, source_to_target_field: nil)
+      target_collection_field = source_to_target_field || target.collection_name
+      data.values.each do |source|
+        create_xrefs_for_source source, target_collection_field, target
+      end
+      target.data.values.each { |item| (item[collection_name] || []).uniq! }
+    end
+
+    private
+
+    def create_xrefs_for_source(source, target_collection_field, target)
+      source_xref = item_to_xref source
+      target_ids = filter_target_ids target, source, target_collection_field
+      link_source_to_targets source_xref, target_ids, target
+      source[target_collection_field] = target_xrefs target, target_ids
+    end
+
+    def filter_target_ids(target_xref, source_item, target_collection_field)
+      (source_item[target_collection_field] || []).map do |target_id|
+        if target_xref.data.member? target_id
+          target_id
+        elsif !public_mode
+          fail UnknownCrossReferenceTargetId, unknown_cross_reference_msg(
+            collection_name, source_item, target_collection_field,
+            target_xref, target_id)
+        end
+      end.compact
+    end
+
+    def unknown_cross_reference_msg(collection_name,
+      source_item, target_collection_field, target_xref, target_id)
+      "source collection: \"#{collection_name}\" " \
+        "source xref: #{item_to_xref source_item} " \
+        "target collection field: \"#{target_collection_field}\" " \
+        "target collection: \"#{target_xref.collection_name}\" " \
+        "target ID: \"#{target_id}\""
+    end
+
+    def link_source_to_targets(source_xref, target_ids, target_xref)
+      target_ids.each do |target_id|
+        (target_xref.data[target_id][collection_name] ||= []) << source_xref
+      end
+    end
+
+    def target_xrefs(target_xref, target_ids)
+      target_ids.map { |id| target_xref.item_to_xref target_xref.data[id] }
+    end
+  end
+
+  # Builds cross-references between data sets.
+  class CrossReferencer
+    TEAM_FIELDS = %w(name last_name first_name full_name self)
+    PROJECT_FIELDS = %w(name project self)
+    WORKING_GROUP_FIELDS = %w(name full_name self)
+    GUILD_FIELDS = %w(name full_name self)
+    TAG_CATEGORIES = %w(skills interests)
+
+    # Build cross-references between data sets.
+    # +site_data+:: Jekyll +site.data+ object
+    def self.build_xrefs(site)
+      team, projects, working_groups, guilds = create_xref_data site
+
+      projects.create_xrefs team
+      [working_groups, guilds].each do |grouplet|
+        grouplet.create_xrefs team, source_to_target_field: 'leads'
+        grouplet.create_xrefs team, source_to_target_field: 'members'
+      end
+
+      xref_tags_and_team_members site, TAG_CATEGORIES, team
+      xref_locations site.data, team, [projects, working_groups, guilds]
+    end
+
+    def self.create_xref_data(site)
+      [CrossReferenceData.new(site, 'team', TEAM_FIELDS),
+       CrossReferenceData.new(site, 'projects', PROJECT_FIELDS),
+       CrossReferenceData.new(site, 'working-groups', WORKING_GROUP_FIELDS),
+       CrossReferenceData.new(site, 'guilds', GUILD_FIELDS),
+      ]
+    end
+    private_class_method :create_xref_data
+
+    def self.xref_tags_and_team_members(site, tag_categories, team_xref)
+      tag_categories.each do |category|
+        xrefs = create_tag_xrefs(site, (site.data['team'] || {}).values,
+          category, team_xref)
+        site.data[category] = xrefs unless xrefs.empty?
+      end
+    end
+
+    def self.create_tag_xrefs(site, items, category, xref_data)
+      map_items_to_tags = lambda do |item|
+        item_xref = xref_data.item_to_xref item
+        item[category].map { |tag| [tag, item_xref] } unless item[category].nil?
+      end
+      create_tag_xrefs = lambda do |tag, item_xrefs|
+        [tag, tag_xref(site, category, tag, item_xrefs)]
+      end
+      map_reduce(items, map_items_to_tags, create_tag_xrefs).to_h
+    end
+
+    # Returns an Array of objects after mapping and reducing items.
+    # mapper takes a single item and returns an Array of [key, value] pairs.
+    # reducer takes a [key, Array of values] pair and returns a single item.
+    def self.map_reduce(items, mapper, reducer)
+      items.flat_map { |item| mapper.call(item) }.compact
+        .each_with_object({}) { |kv, shuffle| (shuffle[kv[0]] ||= []) << kv[1] }
+        .map { |key, values| reducer.call(key, values) }.compact
+    end
+
+    def self.tag_xref(site, category, tag, members)
+      category_slug = Canonicalizer.canonicalize category
+      tag_slug = Canonicalizer.canonicalize tag
+      { 'name' => tag,
+        'slug' => tag_slug,
+        'self' => File.join(Api.baseurl(site), category_slug, tag_slug),
+        'members' => Canonicalizer.sort_by_last_name(members || []),
+      }
+    end
+
+    def self.group_names_to_team_xrefs(team, collection_xrefs)
+      collection_xrefs.map do |xref|
+        xrefs = team.flat_map { |i| i[xref.collection_name] }.compact.uniq
+        [xref.collection_name, xrefs] unless xrefs.empty?
+      end.compact.to_h
+    end
+
+    # Produces an array of locations containing cross references to team
+    # members and all projects, working groups, guilds, etc. associated with
+    # each team member. All team member cross-references must already exist.
+    def self.xref_locations(site_data, team_xref, collection_xrefs)
+      location_xrefs = site_data['team'].values.group_by { |i| i['location'] }
+        .map do |location_code, team|
+          [location_code,
+           {
+             'team' => team.map { |member| team_xref.item_to_xref member },
+           }.merge(group_names_to_team_xrefs(team, collection_xrefs)),
+          ] unless location_code.nil?
+        end
+      HashJoiner.deep_merge site_data['locations'], location_xrefs.compact.to_h
+    end
+  end
+end

data/lib/team_api/endpoints.yml

@@ -0,0 +1,58 @@
+- collection: team
+  title: Team
+  description: Team member info, indexed by username
+  item_id: name
+  sort_by: last_name
+  item_collections:
+  - projects
+  - working-groups
+  - guilds
+
+- collection: locations
+  title: Locations
+  description: Index of team members by location code
+  item_id: code
+  sort_by: label
+  item_collections:
+  - team
+  - projects
+  - working-groups
+  - guilds
+
+- collection: projects
+  title: Projects
+  description: Project info, indexed by short project name
+  item_id: name
+  sort_by: full_name
+  item_collections:
+  - team
+
+- collection: departments
+  title: Departments
+  description: Department info, indexed by department name
+  item_id: name
+  sort_by: name
+  item_collections:
+  - team
+
+- collection: working-groups
+  title: Working Groups
+  description: Working Groups info, indexed by name
+  item_id: name
+  sort_by: full_name
+  item_collections:
+  - field: leads
+    collection: team
+  - field: members
+    collection: team
+
+- collection: guilds
+  title: Guilds
+  description: Guilds info, indexed by name
+  item_id: name
+  sort_by: full_name
+  item_collections:
+  - field: leads
+    collection: team
+  - field: members
+    collection: team

data/lib/team_api/front_matter.rb

@@ -0,0 +1,33 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require 'safe_yaml'
+
+module TeamApi
+  class FrontMatter
+    class Error < StandardError
+    end
+
+    MARKER = '---'
+    START_MARKER = "#{MARKER}\n"
+    END_MARKER = "\n#{MARKER}\n"
+
+    def self.update_front_matter(filename)
+      end_front_matter = front_matter_end_index filename, content
+      front_matter = content[0..end_front_matter]
+      content = content[end_front_matter..-1]
+      front_matter = SafeYAML.load front_matter, safe: true
+      yield front_matter
+      File.write filename, "#{front_matter.to_yaml}#{content}"
+    end
+
+    def self.front_matter_end_index(filename, content)
+      unless content.start_with? START_MARKER
+        fail Error, "#{filename}: contains no front matter"
+      end
+      end_front_matter = content.index END_MARKER, START_MARKER.size
+      return end_front_matter unless end_front_matter.nil?
+      fail Error, "#{filename}: front matter does not end with '#{MARKER}'"
+    end
+    private_class_method :front_matter_end_index
+  end
+end

data/lib/team_api/generator.rb

@@ -0,0 +1,28 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require_relative 'api'
+require_relative 'canonicalizer'
+require_relative 'cross_referencer'
+require_relative 'joiner'
+require_relative 'snippets'
+require 'hash-joiner'
+require 'jekyll'
+
+module TeamApi
+  # Processes site data, generates authorization artifacts, publishes an API,
+  # and generates cross-linked Hub pages.
+  class Generator < ::Jekyll::Generator
+    safe true
+
+    # Executes all of the data processing and artifact/page generation phases
+    # for the Hub.
+    def generate(site)
+      Joiner.join_data(site)
+      Snippets.publish(site)
+      CrossReferencer.build_xrefs(site)
+      Canonicalizer.canonicalize_data(site.data)
+      ::HashJoiner.prune_empty_properties(site.data)
+      Api.generate_api(site)
+    end
+  end
+end

data/lib/team_api/joiner.rb

@@ -0,0 +1,151 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require_relative 'api'
+require 'hash-joiner'
+
+module TeamApi
+  class UnknownTeamMemberReferenceError < StandardError
+  end
+
+  class UnknownSnippetUsernameError < StandardError
+  end
+
+  # Joins the data from collections into +site.data+. Also filters out private
+  # data when +site.config[+'public'] is +true+ (aka "public mode").
+  class Joiner
+    # Executes all of the steps to join the different data sources into
+    # +site.data+ and filters out private data when in public mode.
+    #
+    # +site+:: Jekyll site data object
+    def self.join_data(site)
+      impl = JoinerImpl.new site
+      site.data.merge! impl.collection_data
+      impl.promote_or_remove_data
+      impl.join_project_data
+      Api.add_self_links site
+      impl.join_snippet_data
+    end
+  end
+
+  # Implements Joiner operations.
+  class JoinerImpl
+    attr_reader :site, :data, :public_mode
+
+    # +site+:: Jekyll site data object
+    def initialize(site)
+      @site = site
+      @data = site.data
+      @public_mode = site.config['public']
+    end
+
+    def collection_data
+      @collection_data ||= site.collections.map do |data_class, collection|
+        groups = groups collection
+        result = (groups[:public] || {})
+        result.merge!('private' => groups[:private]) if groups[:private]
+        [data_class, result] unless result.empty?
+      end.compact.to_h
+    end
+
+    def groups(collection)
+      collection.docs
+        .select { |doc| doc.data['published'] != false }
+        .group_by { |doc| doc_visibility doc }
+        .map { |group, docs| [group, docs_data(docs)] }
+        .to_h
+    end
+
+    def doc_visibility(doc)
+      parent = File.basename File.dirname(doc.cleaned_relative_path)
+      (parent == 'private') ? :private : :public
+    end
+
+    def docs_data(docs)
+      docs.map { |doc| [doc.basename_without_ext, doc.data] }.to_h
+    end
+
+    def promote_or_remove_data
+      private_data_method = public_mode ? :remove_data : :promote_data
+      HashJoiner.send private_data_method, data, 'private'
+    end
+
+    def join_project_data
+      # A little bit of project data munging. Can go away after the .about.yml
+      # convention takes hold, hopefully.
+      projects = (data['projects'] ||= {})
+      projects.delete_if { |_, p| p['status'] == 'Hold' } if @public_mode
+      projects.values.each { |p| join_team_list p['team'] }
+    end
+
+    def team
+      data['team'] ||= {}
+    end
+
+    # Returns an index of team member usernames keyed by email address.
+    def team_by_email
+      @team_by_email ||= team_index_by_field 'email'
+    end
+
+    # Returns an index of team member usernames keyed by email address.
+    def team_by_github
+      @team_by_github ||= team_index_by_field 'github'
+    end
+
+    # Returns an index of team member usernames keyed by a particular field.
+    def team_index_by_field(field)
+      team.values.map do |member|
+        value = member[field]
+        [value, member['name']] unless value.nil?
+      end.compact.to_h
+    end
+
+    # Replaces each member of team_list with a key into the team hash.
+    # Values can be:
+    # - Strings that are already team hash keys
+    # - Strings that are email addresses
+    # - Strings that are GitHub usernames
+    # - Hashes that contain an 'email' property
+    # - Hashes that contain a 'github' property
+    def join_team_list(team_list)
+      (team_list || []).map! do |reference|
+        member = team_member_from_reference reference
+        if member.nil?
+          fail UnknownTeamMemberReferenceError, member unless public_mode
+        else
+          member['name']
+        end
+      end.compact
+    end
+
+    def team_member_from_reference(reference)
+      key = (reference.instance_of? String) ? reference : (
+        reference['email'] || reference['github'])
+      team[key] || team[team_by_email[key] || team_by_github[key]]
+    end
+
+    SNIPPET_JOIN_FIELDS = %w(name full_name first_name last_name self)
+
+    # Joins snippet data into +site.data[+'snippets'] and filters out snippets
+    # from team members not appearing in +site.data[+'team'] or
+    # +team_by_email+.
+    def join_snippet_data
+      data['snippets'] = data['snippets'].map do |timestamp, snippets|
+        joined = snippets.map { |snippet| join_snippet snippet }
+          .compact.each { |i| i.delete 'username' }
+        [timestamp, joined] unless joined.empty?
+      end.compact.to_h
+    end
+
+    def join_snippet(snippet)
+      username = snippet['username']
+      member = team[username] || team[team_by_email[username]]
+
+      if member.nil?
+        fail UnknownSnippetUsernameError, username unless public_mode
+      else
+        member = member.select { |k, _| SNIPPET_JOIN_FIELDS.include? k }
+        snippet.merge member
+      end
+    end
+  end
+end

data/lib/team_api/snippets.rb

@@ -0,0 +1,24 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require 'weekly_snippets/publisher'
+
+module TeamApi
+  class Snippets
+    # Used to convert snippet headline markers to h4, since the layout uses
+    # h3.
+    HEADLINE = "\n####"
+
+    MARKDOWN_SNIPPET_MUNGER = proc do |text|
+      text.gsub!(/^::: (.*) :::$/, "#{HEADLINE} \\1") # For jtag. ;-)
+      text.gsub!(/^\*\*\*/, HEADLINE) # For elaine. ;-)
+    end
+
+    # TODO(mbland): Push this to the snippet import script.
+    def self.publish(site)
+      publisher = ::WeeklySnippets::Publisher.new(
+        headline: HEADLINE, public_mode: site.config['public'],
+        markdown_snippet_munger: MARKDOWN_SNIPPET_MUNGER)
+      site.data['snippets'] = publisher.publish site.data['snippets']
+    end
+  end
+end

data/lib/team_api/tag_categories.yml

@@ -0,0 +1,9 @@
+- category: skills
+  title: Skills
+  description: Index of team members by technical skills
+  source: team
+
+- category: interests
+  title: Interests
+  description: Index of team members by general interests
+  source: team

data/lib/team_api/version.rb

@@ -0,0 +1,5 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+module TeamApi
+  VERSION = '0.0.0'
+end

data/lib/team_api.rb

@@ -0,0 +1,11 @@
+# @author Mike Bland (michael.bland@gsa.gov)
+
+require_relative 'team_api/api'
+require_relative 'team_api/canonicalizer'
+require_relative 'team_api/config'
+require_relative 'team_api/cross_referencer'
+require_relative 'team_api/front_matter'
+require_relative 'team_api/generator'
+require_relative 'team_api/joiner'
+require_relative 'team_api/snippets'
+require_relative 'team_api/version'

metadata

@@ -0,0 +1,229 @@
+--- !ruby/object:Gem::Specification
+name: team_api
+version: !ruby/object:Gem::Version
+  version: 0.0.0
+platform: ruby
+authors:
+- Mike Bland
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-08-30 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: safe_yaml
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: jekyll
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: weekly_snippets
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: hash-joiner
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: go_script
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.4'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.4'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: codeclimate-test-reporter
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: coveralls
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: about_yml
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Compiles information about team members, projects, etc. and exposes it
+  via a JSON API.
+email:
+- michael.bland@gsa.gov
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CONTRIBUTING.md
+- LICENSE.md
+- README.md
+- lib/team_api.rb
+- lib/team_api/README.md
+- lib/team_api/api.rb
+- lib/team_api/canonicalizer.rb
+- lib/team_api/config.rb
+- lib/team_api/cross_referencer.rb
+- lib/team_api/endpoints.yml
+- lib/team_api/front_matter.rb
+- lib/team_api/generator.rb
+- lib/team_api/joiner.rb
+- lib/team_api/snippets.rb
+- lib/team_api/tag_categories.yml
+- lib/team_api/version.rb
+homepage: https://github.com/18F/team_api
+licenses:
+- CC0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5.1
+signing_key: 
+specification_version: 4
+summary: Compiles team information and publishes it as a JSON API
+test_files: []