nanoc-conref-fs 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 35dc702fdbe8ffa00711be829afcf827fe5565fe
-  data.tar.gz: 83767a5ade9f4945c99f25d3a03122e1ff3467e7
+  metadata.gz: fdb225742e914ae1ac3f39c539c290a2c91aabf0
+  data.tar.gz: 694937b4c238491409411370271cc9d5d061c0c3
 SHA512:
-  metadata.gz: bf4dd287a13d192f38777a6dc31211976ba3e6398ce7b9f53a06769740086dc706217788e05b1a954c35425863b75a41bea4937eaaa801eef7a2c87487b56289
-  data.tar.gz: 37c643e5e0b2ac33cb40aa87025146d3549aad0d48f7215ce4cbb5f41aa79a65f2580104e4ae78fd7a54343fe5c466b1a6eb70a657e10d83b0c6b98256d69d62
+  metadata.gz: 9edf0b62eb28ea4fb128303e7cb442f79b32916b6b562c08f784e09b3d4396e16043a9c96d43837f4b341d4be3a62668168c4a0335c75d42b572318040e34e27
+  data.tar.gz: f6cbdd94a675dadc4b3dc2b91704909b415f91c718baf57a85e10e8581d665b32aa93e7f174d5bef783c41b98f4a38649cce52ed3b4a463515f9f69d629c81a2

data/README.md

@@ -1,12 +1,12 @@
 # nanoc-conref-fs
 
-This gem adds a new Filesystem type called `ConrefFS` to nanoc.
+This gem adds a new Filesystem type to Nanoc called `ConrefFS`. This filesystem permits you to use reusable content and Liquid variables in your content to generate multiple outputs from a single source. It makes heavy use of item representations (or `rep`s for short).
 
-The idea is that you have a set of YAML files in a data folder which act as your reusables. You can apply thse reusables throughout your content and layout.
+The idea is that you have a set of YAML files in a data folder which act as your reusables. You can apply these reusables throughout your content.
 
 [![Build Status](https://travis-ci.org/gjtorikian/nanoc-conref-fs.svg)](https://travis-ci.org/gjtorikian/nanoc-conref-fs)
 
-Set the data source in your *nanoc.yaml* file:
+To get started, set the data source in your *nanoc.yaml* file:
 
 ``` yml
 data_sources:
@@ -14,7 +14,18 @@ data_sources:
     type: conref-fs
 ```
 
-**NOTE:** If you use this library with nanoc's ERB filter, and want to use `render`, you'll need to monkey-patch an alias to avoid conflicts:
+You'll probably also want to provide a list of `rep`s which define all the item reps available to your site. For example:
+
+``` yml
+data_sources:
+  -
+    type: conref-fs
+    reps:
+      - :default
+      - :X
+```
+
+**NOTE:** If you use this library with Nanoc's ERB filter, and want to use `render`, you'll need to monkey-patch an alias to avoid conflicts with Liquid:
 
 ``` ruby
 require 'nanoc-conref-fs'
@@ -27,64 +38,100 @@ Nanoc::Helpers::Rendering.module_eval do
 end
 ```
 
-Then, use `renderp` instead of `render`.
+Then, you can use `renderp` just as you would with `render`.
 
 ## Usage
 
-Nearly all the usage of this gem relies on a *data* folder, sibling to your *content* and *layouts* folders. See [the test fixture](test/fixtures/data) for an example.
-
-You'll also need some relevant keys added to your *nanoc.yaml* file:
-
-* The `data_variables` key applies additional/dynamic values to your data files, based on their path. For example, the following `data_variables` configuration adds a `version` attribute to every data file, whose value is `dotcom`:
-
-   ``` yaml
-   data_variables:
-     -
-       scope:
-         path: ""
-       values:
-         version: "dotcom"
-   ```
-
-   You could add to this key to indicate that any data file called *changed* instead has a version of `something_different`:
-
-   ``` yaml
-   data_variables:
-     -
-       scope:
-         path: ""
-       values:
-         version: "dotcom"
-     -
-       scope:
-         path: "changed"
-       values:
-         version: "2.0"
-  ```
-
-* Similarly, the `page_variables` also use `scope`s and `value`s to determine variables:
-
-  ``` yaml
-  page_variables:
-    -
-      scope:
-        path: ""
-      values:
-        version: "dotcom"
-    -
-      scope:
-        path: "different"
-      values:
-        version: "2.0"
-  ```
-
-  In this case, every file path will get a `version` of `dotcom`, but any file matching `different` will get a version of 2.0.
+Nearly all the usage of this gem relies on a *data* folder, sibling to your *content* and *layouts* folders. See [the test fixture](test/fixtures/data) for an example. You can change this with the `data_dir` config option:
+
+``` yml
+data_sources:
+  -
+    type: conref-fs
+    data_dir: 'somewhere_else'
+    reps:
+      - :default
+      - :X
+```
+
+Then, you can [construct a data folder filled with YAML files](https://github.com/gjtorikian/nanoc-conref-fs/tree/master/test/fixtures/data). These act as the source of your reusable content.
+
+Finally, you'll need some relevant keys added to your *nanoc.yaml* file.
+
+### Data folder variables
+
+The `data_variables` key applies additional/dynamic values to your data files, based on their path. For example, the following `data_variables` configuration adds a `version` attribute to every data file, whose value is `dotcom`:
+
+ ``` yaml
+ data_variables:
+   -
+     scope:
+       path: ""
+     values:
+       version: "dotcom"
+ ```
+
+ You could add to this key to indicate that any data file called *changed* instead has a version of `something_different`:
+
+ ``` yaml
+ data_variables:
+   -
+     scope:
+       path: ""
+     values:
+       version: "dotcom"
+   -
+     scope:
+       path: "changed"
+     values:
+       version: "2.0"
+```
+
+In addition to `path`, you can provide an array of `reps` to define which `reps` should receive which version:
+
+``` yaml
+data_variables:
+  -
+    scope:
+      path: "feature"
+    values:
+      version: "dotcom"
+  -
+    scope:
+      path: "feature"
+      reps:
+        - :X
+    values:
+      version: "something_else"
+```
+
+In this example, any data folder with a path containing "feature" will have `page.version` equal to `dotcom`. However, if you're constructing for the `:X` item representation, `page.version` becomes `something_else`.
+
+### Page variables
+
+Similarly, the `page_variables` also use `scope`s, `rep`s, and `value`s to determine variables:
+
+``` yaml
+page_variables:
+  -
+    scope:
+      path: ""
+    values:
+      version: "dotcom"
+  -
+    scope:
+      path: "different"
+    values:
+      version: "2.0"
+```
+
+In this case, every file path will get a `version` of `dotcom`, but any file matching `different` will get a version of 2.0.
 
 See the tests for further usage of these conditionals. In both cases, `path` is converted into a Ruby regular expression before being matched against a filename.
 
 ### Associating files with data
 
-If you have a special `data_association` value, additional metadata to items will be applied:
+If you have a special `data_association` value in your `scope`, additional metadata to items will be applied:
 
 * An attribute called `:parents`, which adds the parent "map topic" to an item.
 * An attribute called `:children`, which adds any children of a "map topic."
@@ -96,3 +143,25 @@ You can retrieve the stored data at any time (for example, in a layout) by calli
 ### Retrieving data files
 
 You can fetch anything in the *data* folder by passing in a string, demarcated with `.`s, to `VariableMixin.fetch_data_file`. For example, `VariableMixin.fetch_data_file('reusables.intro')` will fetch the file in *data/reusables/intro.yml*.
+
+### Ignoring content
+
+You can use the `create_ignore_rules` method to construct ignore rules for your content. For example, say you have a category file that looks like this:
+
+``` yaml
+Something:
+  - Blah1
+  {% if page.version == 'dotcom' %}
+  - Blah2
+  {% endif %}
+```
+
+By default, Nanoc will try to compile and layout a page for `Blah2` for every item rep, even though it probably shouldn't.
+
+You can use `create_ignore_rules`, passing in an item rep and a category file to generate ignore rules. For example:
+
+``` ruby
+ConrefFS.create_ignore_rules(:X, 'categories.category').each do |f|
+  ignore f, rep: :X
+end
+```

data/lib/nanoc-conref-fs/ancestry.rb

@@ -0,0 +1,98 @@
+module NanocConrefFS
+  module Ancestry
+    def create_parents(toc, meta)
+      if toc.is_a?(Array)
+        find_array_parents(toc, meta[:title])
+      elsif toc.is_a?(Hash)
+        find_hash_parents(toc, meta[:title])
+      end
+    end
+    module_function :create_parents
+
+    def create_children(toc, meta)
+      if toc.is_a?(Array)
+        find_array_children(toc, meta[:title])
+      elsif toc.is_a?(Hash)
+        find_hash_children(toc, meta[:title])
+      end
+    end
+    module_function :create_children
+
+    # Given a category file that's an array, this method finds
+    # the parent of an item
+    def find_array_parents(toc, title)
+      parents = ''
+      toc.each do |item|
+        if item.is_a?(Hash)
+          parents = find_hash_parents(item, title)
+          break unless parents.empty?
+        end
+      end
+      parents
+    end
+    module_function :find_array_parents
+
+    # Given a category file that's a hash, this method finds
+    # the parent of an item
+    def find_hash_parents(toc, title)
+      parents = ''
+      toc.each_key do |key|
+        toc[key].each do |item|
+          if item.is_a?(Hash)
+            if item.keys.include?(title)
+              parents = key
+              break
+            else
+              if item[item.keys.first].include?(title)
+                parents = key
+                break
+              end
+            end
+          elsif title == item
+            parents = key
+            break
+          end
+        end
+        break unless parents.empty?
+      end
+      parents
+    end
+    module_function :find_hash_parents
+
+    # Given a category file that's an array, this method finds
+    # the children of an item, probably a map topic
+    def find_array_children(toc, title)
+      children = ''
+      toc.each do |item|
+        next unless item.is_a?(Hash)
+        item.each_pair do |key, values|
+          if key == title
+            children = values.flatten
+            break
+          end
+        end
+        break unless children.empty?
+      end
+      children
+    end
+    module_function :find_array_children
+
+    # Given a category file that's a hash, this method finds
+    # the children of an item, probably a map topic
+    def find_hash_children(toc, title)
+      children = ''
+      toc.each_key do |key|
+        toc[key].each do |item|
+          next unless item.is_a?(Hash)
+          unless item[title].nil?
+            children = item.values.flatten
+            break
+          end
+        end
+        break unless children.empty?
+      end
+      children
+    end
+    module_function :find_hash_children
+  end
+end

data/lib/nanoc-conref-fs/conref-filter.rb

@@ -0,0 +1,27 @@
+module Nanoc
+  class ItemView
+    # allows apply_attributes to create new item attributes
+    def []=(key, value)
+      unwrap.attributes[key] = value
+    end
+  end
+end
+
+module Nanoc::HashExtensions
+  alias_method :original_freeze, :__nanoc_freeze_recursively
+
+  # prevents the item's frozen attributes from remaining frozen
+  def __nanoc_freeze_recursively
+    return if caller.first =~ %r{base/entities/document.rb}
+    original_freeze
+  end
+end
+
+class ConrefFSFilter < Nanoc::Filter
+  identifier :'conref-fs-filter'
+
+  def run(content, _)
+    ConrefFS.apply_attributes(@config, item, @rep.name)
+    NanocConrefFS::Conrefifier.liquify(@config, path: @item[:filename], content: content, rep: @rep.name)
+  end
+end

data/lib/nanoc-conref-fs/conref-fs.rb

@@ -1,202 +1,145 @@
 require_relative 'conrefifier'
-
-# Unsure why attr_accessor does not work here
-module VariableMixin
-  def self.variables
-    @variables
-  end
-
-  def self.variables=(variables)
-    @variables = variables
-  end
-
-  def self.fetch_data_file(association)
-    reference = association.split('.')
-    variables = VariableMixin.variables
-    data = variables['site']['data']
-    while key = reference.shift
-      data = data[key]
-    end
-    data
-  end
-end
+require 'active_support/core_ext/string'
 
 class ConrefFS < Nanoc::DataSource
   include Nanoc::DataSources::Filesystem
-  include VariableMixin
+  include NanocConrefFS::Variables
+  include NanocConrefFS::Ancestry
 
   identifier :'conref-fs'
 
   # Before iterating over the file objects, this method loads the data folder
   # and applies it to an ivar for later usage.
-  def load_objects(dir_name, kind, klass)
-    if klass == Nanoc::Int::Item && @variables.nil?
-      data = Datafiles.process(@site_config)
-      config = @site_config.to_h
-      @variables = { 'site' => { 'config' => config, 'data' => data } }
-      VariableMixin.variables = @variables
-    end
-    super
+  def up
+    data_files = NanocConrefFS::Datafiles.collect_data(data_dir_name)
+    NanocConrefFS::Variables.data_files = data_files
+    NanocConrefFS::Variables.variables = {}
+    reps = @config[:reps] || [:default]
+    reps.each { |rep| ConrefFS.load_data_folder(@site_config, rep) }
   end
 
-  # This function calls the parent super, then adds additional metadata to the item.
-  def parse(content_filename, meta_filename, _kind)
-    meta, content = super
-    page_vars = Conrefifier.file_variables(@site_config[:page_variables], content_filename)
+  def data_dir_name
+    config.fetch(:data_dir) { |_| 'data' }
+  end
+
+  def self.load_data_folder(config, rep)
+    return unless NanocConrefFS::Variables.variables[rep].nil?
+    data_files = NanocConrefFS::Variables.data_files
+    data = NanocConrefFS::Datafiles.process(data_files, config, rep)
+    NanocConrefFS::Variables.variables[rep] = { 'site' => { 'config' => config, 'data' => data } }
+  end
+
+  def self.apply_attributes(config, item, rep)
+    page_vars = NanocConrefFS::Conrefifier.file_variables(config[:page_variables], item[:filename], rep)
+
+    frontmatter_vars = { :page => page_vars }.merge(NanocConrefFS::Variables.variables[rep])
+
     unless page_vars[:data_association].nil?
       association = page_vars[:data_association]
-      toc = VariableMixin.fetch_data_file(association)
-      meta[:parents] = if toc.is_a?(Array)
-                         find_array_parents(toc, meta['title'])
-                       elsif toc.is_a?(Hash)
-                         find_hash_parents(toc, meta['title'])
-                       end
-
-      meta[:children] = if toc.is_a?(Array)
-                          find_array_children(toc, meta['title'])
-                        elsif toc.is_a?(Hash)
-                          find_hash_children(toc, meta['title'])
-                        end
-    end
-    page_vars.each_pair do |name, value|
-      meta[name.to_s] = value
+      toc = NanocConrefFS::Variables.fetch_data_file(association, rep)
+      item[:parents] = NanocConrefFS::Ancestry::create_parents(toc, item.attributes)
+      item[:children] = NanocConrefFS::Ancestry::create_children(toc, item.attributes)
     end
-    [meta, content]
-  end
 
-  # Given a category file that's an array, this method finds
-  # the parent of an item
-  def find_array_parents(toc, title)
-    parents = ''
-    toc.each do |item|
-      if item.is_a?(Hash)
-        parents = find_hash_parents(item, title)
-        break unless parents.empty?
-      end
+    page_vars.each_pair do |key, value|
+      item[key] = value
     end
-    parents
-  end
 
-  # Given a category file that's a hash, this method finds
-  # the parent of an item
-  def find_hash_parents(toc, title)
-    parents = ''
-    toc.keys.each do |key|
-      toc[key].each do |item|
-        if item.is_a?(Hash)
-          if item.keys.include?(title)
-            parents = key
-            break
-          else
-            if item[item.keys.first].include?(title)
-              parents = key
-              break
-            end
-          end
-        elsif title == item
-          parents = key
-          break
+    item.attributes.each_pair do |key, value|
+      if value.is_a?(Array)
+        item[key] = value.map do |arr_v|
+          return arr_v unless arr_v =~ NanocConrefFS::Conrefifier::SINGLE_SUB
+          NanocConrefFS::Conrefifier.apply_liquid(arr_v, frontmatter_vars)
+        end
+      else
+        if value =~ NanocConrefFS::Conrefifier::SINGLE_SUB
+          value = NanocConrefFS::Conrefifier.apply_liquid(value, frontmatter_vars)
+          item[key] = value
         end
       end
-      break unless parents.empty?
     end
-    parents
   end
 
-    # Given a category file that's an array, this method finds
-    # the children of an item, probably a map topic
-    def find_array_children(toc, title)
-      children = ''
-      toc.each do |item|
-        next unless item.is_a?(Hash)
-        item.each_pair do |key, values|
-          if key == title
-            children = values.flatten
-            break
-          end
-        end
-        break unless children.empty?
-      end
-      children
+  # There are a lot of problems when trying to parse liquid
+  # out of the frontmatter—all of them dealing with collision between
+  # the { character in Liquid and its signfigance in YAML. We'll overload
+  # the parse method here to resolve those issues ahead of time.
+  def parse(content_filename, meta_filename, _kind)
+    # Read data
+    data = read(content_filename)
+
+    # Check presence of metadata section
+    return [{}, data] if data !~ /\A-{3,5}\s*$/
+
+    # Split data
+    pieces = data.split(/^(-{5}|-{3})[ \t]*\r?\n?/, 3)
+    if pieces.size < 4
+      raise RuntimeError.new(
+        "The file '#{content_filename}' appears to start with a metadata section (three or five dashes at the top) but it does not seem to be in the correct format.",
+      )
     end
 
-    # Given a category file that's a hash, this method finds
-    # the children of an item, probably a map topic
-    def find_hash_children(toc, title)
-      children = ''
-      toc.keys.each do |key|
-        toc[key].each do |item|
-          next unless item.is_a?(Hash)
-          unless item[title].nil?
-            children = item.values.flatten
-            break
-          end
-        end
-        break unless children.empty?
-      end
-      children
-    end
+    # N.B. the only change to the original function
+    pieces[2].gsub!(/^([^:]+): (\{\{.+)/, '\1: \'\2\'')
 
-  # This file reads each piece of content as it comes in. It also applies the conref variables
-  # (demarcated by Liquid's {{ }} tags) using both the data/ folder and any variables defined
-  # within the nanoc.yaml config file
-  def read(filename)
-    content = ''
+    # Parse
     begin
-      page_vars = Conrefifier.file_variables(@site_config[:page_variables], filename)
-      page_vars = { :page => page_vars }.merge(@variables)
-
-      content = File.read(filename)
-      return content unless filename.start_with?('content', 'layouts')
-
-      # we must obfuscate essential ExtendedMarkdownFilter content
-      content = content.gsub(/\{\{\s*#(\S+)\s*\}\}/, '[[#\1]]')
-      content = content.gsub(/\{\{\s*\/(\S+)\s*\}\}/, '[[/\1]]')
-      content = content.gsub(/\{\{\s*(octicon-\S+\s*[^\}]+)\s*\}\}/, '[[\1]]')
-    rescue => e
-      raise "Could not read #{filename}: #{e.inspect}"
+      meta = YAML.load(pieces[2]) || {}
+    rescue Exception => e
+      raise "Could not parse YAML for #{content_filename}: #{e.message}"
     end
+    verify_meta(meta, content_filename)
+    content = pieces[4]
 
-    begin
-      result = content
+    # Done
+    [meta, content]
+  end
 
-      # This pass replaces any matched conditionals
-      if result =~ Conrefifier::BLOCK_SUB || result =~ Conrefifier::SINGLE_SUB
-        result = Conrefifier.apply_liquid(result, page_vars)
-      end
+  def self.create_ignore_rules(rep, file)
+    current_articles = NanocConrefFS::Variables.fetch_data_file(file, rep)
+    current_articles = flatten_list(current_articles).flatten
+    current_articles = fix_nested_content(current_articles)
 
-      # This second pass renders any previously inserted
-      # data conditionals within the body. If a Liquid parse
-      # returns a blank string, we'll return the original
-      if result =~ Conrefifier::SINGLE_SUB
-        result = result.gsub(Conrefifier::SINGLE_SUB) do |match|
-          liquified = Conrefifier.apply_liquid(match, page_vars)
-          liquified.empty? ? match : liquified
-        end
-      end
+    basic_yaml = NanocConrefFS::Variables.data_files["data/#{file.tr!('.', '/')}.yml"]
+    basic_yaml.gsub!(/\{%.+/, '')
+    full_file = YAML.load(basic_yaml)
+
+    full_user_articles = flatten_list(full_file).flatten
+    full_user_articles = fix_nested_content(full_user_articles)
+
+    blacklisted_articles = full_user_articles - current_articles
+    blacklisted_articles.map do |article|
+      "#{article.parameterize}.md"
+    end
+  end
 
-      # This converts ": *" frontmatter strings into HTML equivalents;
-      # otherwise, ": *" messes the YAML parsing
-      result = result.gsub(/\A---\s*\n(.*?\n?)^---\s*$\n?/m) do |frontmatter|
-        frontmatter.gsub(/:\s*(\*.+)/) do |_|
-          asterisk_match = Regexp.last_match[1]
-          if asterisk_match[1] != '*'
-            asterisk_match = asterisk_match.sub(/\*(.+?)\*/, ': <em>\1</em>')
-          else
-            asterisk_match = asterisk_match.sub(/\*{2}(.+?)\*{2}/, ': <strong>\1</strong>')
-          end
-          asterisk_match
+  def self.flatten_list(arr)
+    result = []
+    return result unless arr
+    arr.each do |item|
+      if item.is_a?(Hash)
+        item.each_pair do |key, value|
+          result << key
+          result.concat(value)
         end
+      else
+        result << item
       end
+    end
 
-      result
-    rescue Liquid::SyntaxError => e
-      # unrecognized Liquid, so just return the content
-      STDERR.puts "Could not convert #{filename}: #{e.message}"
-      result
-    rescue => e
-      raise "#{e.message}: #{e.inspect}"
+    result
+  end
+
+  def self.fix_nested_content(articles)
+    articles.delete_if do |i|
+      if i.is_a? Hash
+        articles.concat(i.keys.concat(i.values).flatten)
+        true
+      else
+        false
+      end
     end
+    articles
   end
 
   # This method is extracted from the Nanoc default FS