yarrow 0.8.7 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 89667b38c1ebf2da983a381e20be9d8e0904bdb69137bb86a1bce6ee9507fbf1
-  data.tar.gz: 364c196cefad90398b10c387edb925b055490163be854b814f3522a6333234ef
+  metadata.gz: 96b575b4a93031397d38c09ab442ce577a98378114941c6be3dcaf4c2a71a3ac
+  data.tar.gz: 15d01eb7b8f710aee890c8879dc202a12ecab3f3cca0030cdc7df8e61619ed75
 SHA512:
-  metadata.gz: 113ce8dc745ec3e83121d5229beeed755671a7902e1330373ef7b129d7138048d090bbdb1abedd2458f0faca3f86f4433c30a855cc6a53119ba59576699f7002
-  data.tar.gz: 6b24b087284b3be6f2d55a8aa3afc25d0e3cc38dbbb2bd80bd9005487b9b95b3bf65d218a47c65b6ac9f4fd54e830a01984292bc1f0c4c8d1455b7af3a5b5049
+  metadata.gz: be90c39d718849c20db481c2f90d105400af237c7232d4cdd091e7260fffe3d09e4828c4d50f473424daa1d223d2cf5259ed55f4120534ad884ea7ca4d2124dc
+  data.tar.gz: b4e5150cd8a3da147489fe0f8077520df328ae2fc5018e4ea102ca171625ada864da3a08aeb59860a98d04b7515e8f0868a023f85fb10c1718592a066d818b1b

data/CONTENT.md

@@ -0,0 +1,228 @@
+# Content Management
+
+## Source Map
+
+The source map specifies how content is expanded into addressable resources that define the information structure of a project.
+
+Currently only website projects are supported, so the organisation is slightly skewed towards URL-centric representations (this can be tidied up in future in order to represent e-books and print publications).
+
+The source map is made up of a set of expansion policies, listed in the configuration as key-value pairs.
+
+### Shorthand Format
+
+The configuration format supports a shorthand for declaring policies based on defaults without needing to provide a specification with every single attribute filled in.
+
+Although it’s not quite implemented this way in Ruby (yet?), you can think of the policy spec in the config as being a union of `String | Symbol | SpecStruct` where all `SpecStruct` attributes are also optional.
+
+The simplest possible config shorthand assumes that the policy label is a plural reference to the collection type that gets expanded.
+
+If the spec value is a symbol, this is interpreted as a singular reference to the entity type that gets attached to the parent collection for each matched content object. In this case, the policy label is overloaded to match the source directory root.
+
+```yaml
+content:
+  source_map:
+    pages: :page
+    blog: :post
+    gallery: :photo
+```
+
+This source map will be interpreted as:
+
+- Expand a `Pages` collection of `Page` objects from the source directory `./pages`
+- Expand a `Blog` collection of `Post` objects from the source directory `./blog`
+- Expand a `Gallery` collection of `Photo` objects from the source directory `./gallery`
+
+If the spec value is a string, this is interpreted as matching the source directory root for the traversal with the policy label referring to the collection type only and the entity type being a singular conversion of the plural policy label.
+
+```yaml
+content:
+  source_map:
+    pages: "about"
+    notes: "archive"
+    photos: "gallery"
+```
+
+This source map will be interpreted as:
+
+- Expand a `Pages` collection of `Page` objects from the source directory `./about`
+- Expand a `Notes` collection of `Note` objects from the source directory `./archive`
+- Expand a `Photos` collection of `Photo` objects from the source directory `./gallery`
+
+### Attribute Precedence
+
+The shorthand format offers limited possibilities for customisation, so in many situations it’s best to provide a more detailed policy spec. All attributes specified directly take precedence over shorthand conventions and defaults.
+
+```yaml
+content:
+  source_map:
+    site:
+      collection: :pages
+      entity: :page
+      source_path: "about"
+    blog:
+      entity: :post
+      source_path: "archive"
+    gallery:
+      collection: :photos
+```
+
+This will be interpreted as:
+
+- Expand a `Pages` collection of `Page` objects from the source directory `./about`
+- Expand a `Blog` collection of `Post` objects from the source directory `./archive`
+- Expand a `Photos` collection of `Photo` objects from the source directory `./gallery`
+
+### Default Spec
+
+```yaml
+content:
+  source_map:
+    site:
+      container: :pages
+      collection: :pages
+      entity: :page
+      source_path: "."
+
+```
+
+## Expansion Strategies
+
+Each policy specified in the source map triggers a traversal of the source content graph using an event-driven pattern to report each relevant file and directory to an aggregator component which expands these into collections and entity resources.
+
+Changing the aggregator enables a variety of different structures of nested directories and files to be expanded into a well-organised content model, providing a more flexible and creative foundation for publishing and editorial design than the rigid conventions of most other static-site generators.
+
+### Filename Map
+
+```yml
+aggregator: :filename_map
+```
+
+Expands nested directories and files with a simple 1:1 mapping to collections and files. Each matching filename becomes a single item of content.
+
+#### Example
+
+Source:
+
+```
+🖿 content
+└──🖿 pages
+   ├──🗎page1.md
+   ├──🗎page2.md
+   ├──🗎page3.md
+   └──🖿 children
+      ├──🗎page4.md
+      └──🗎page5.md
+```
+
+Policy:
+
+```
+content:
+  source_map:
+    pages:
+      aggregator: :filename_map
+```
+
+Expansion:
+
+```mermaid
+graph TD;
+  pages((Pages: pages))-->page1(Page: page1);
+  pages-->page2(Page: page2);
+  pages-->page3(Page: page3);
+  pages-->children((Pages: children));
+  children-->page4(Page: page4);
+  children-->page5(Page: page5);
+```
+
+### Directory Merge
+
+```yml
+aggregator: :directory_merge
+```
+
+Expands from a list of directories, with each directory representing a single item of content.
+
+This is a good choice for rich-content websites with more complex content layouts for articles or interactive essays where a variety of assets are developed alongside the main manuscript file.
+
+#### Example
+
+Source:
+
+```
+🖿 content
+└──🖿 essays
+   └──🖿 concept1
+      ├──🗎concept-1.md
+      ├──🖻image1.png
+      ├──🖻image2.svg
+      ├──🖻image3.jpg
+      ├──🗎data.json
+      └──🗎loop.mp3
+```
+
+Policy:
+
+```yml
+content:
+  source_map:
+    essays:
+      collection: :essays
+      aggregator: :directory_merge
+      source_path: "essays"
+      match_entities: [.md]
+      match_assets: [.png, .jpg, .svg, .json, .mp3]
+```
+
+Expansion:
+
+```mermaid
+graph TD;
+  essays((Essays: essays))-->concept1(Essay: concept-1);
+  concept1-->image1(Asset: image1.png);
+  concept1-->image2(Asset: image2.png);
+  concept1-->image3(Asset: image3.jpg);
+  concept1-->data(Asset: data.json);
+  concept1-->loop(Asset: loop.mp3);
+```
+
+### Directory
+
+```
+expansion_strategy:
+  aggregator: :directory_source
+  match_source: "essays"
+  match_entities: "*.md"
+  match_assets: "*.jpg"
+```
+
+## Collection Containers
+
+Some content models may require collection types to be nested children of a top-level parent type. To define this in a policy you can manually specify a `container` which will override the default `collection` at the root level of the content hierarchy. Containers represent ‘collections of collections’, offering more modelling flexibility and applicability to wider variety of website use cases.
+
+```
+🖿 content
+└──🖿 gallery
+   ├──🖿 exhibition-a
+   │  ├──🗎1.htm
+   │  ├──🗎2.htm
+   │  ├──🗎3.htm
+   │  ├──🗎a-1.jpg
+   │  ├──🗎a-2.jpg
+   │  └──🗎a-3.jpg
+   └──🖿 exhibition-b
+      ├──🗎1.htm
+      ├──🗎2.htm
+      ├──🗎b-1.jpg
+      └──🗎b-2.jpg
+```
+
+The following policy spec allows us to model a gallery type which holds a list of exhibitions, with each exhibition in turn holding a list of artworks.
+
+```yml
+container: :gallery
+collection: :exhibitions
+entity: :artwork
+```
+
+In many cases it might be fine to just nest instances of a single collection type, but this more fine-grained content modelling can be useful when it comes to templating and editorial design.

data/README.md

@@ -38,14 +38,14 @@ Roadmap
 
 A rough sketch of the project direction.
 
-| Version | Features |
-|---------|----------|
-| `0.7`   | Content model/object mapping, template/site context |
-| `0.8`   | HTML publishing workflow |
-| `0.9`   | PDF publishing workflow |
-| `0.10`   | Media and video publishing workflow |
-| `0.11`   | Generic command line runner |
-| `1.0`   | Refactoring, performance fixes, lock down API |
+| Version | Features                                        |
+|---------|-------------------------------------------------|
+| `0.9`   | Support standard text formats and linked assets |
+| `0.10`  | Custom Markdown components                      |
+| `0.11`  | Publishing support for S3 and GitHub/Netlify    |
+| `0.12`  | Clean up local web server and watcher           |
+| `0.13`  | Content structure transformations               |
+| `1.0`   | Reintroduce generic command line runner         |
 
 License
 -------

data/lib/extensions/mementus.rb

@@ -7,6 +7,13 @@ module Mementus
       # methods. Mostly used for #map. API needs to be fixed in the gem itself.
       include Enumerable
 
+      def traverse_by(traversal)
+        case traversal
+        when :tree then depth_first
+        when :list then nodes
+        end
+      end
+
       def to
         Step.new(map { |edge| edge.to }, Pipe.new(graph), graph)
       end
@@ -36,6 +43,28 @@ module Mementus
       "<Mementus::Graph @structure=#{@structure.inspect} " +
         "nodes_count=#{nodes_count} edges_count=#{edges_count}>"
     end
+
+    def to_dot
+      statements = []
+
+      nodes.each do |node|
+        label = if node.props.key?(:type)
+          "#{node.label}: #{node.props[:type]}:#{node.props[:resource].name}"
+        elsif node.props.key?(:name)
+          "#{node.label}: #{node.props[:name]}"
+        else
+          node.label
+        end
+
+        statements << "#{node.id} [label=\"#{label}\"]"
+      end
+
+      edges.each do |edge|
+        statements << "#{edge.from.id} -> #{edge.to.id} [label=\"#{edge.label}\"];"
+      end
+
+      "digraph {\n#{statements.join("\n")}\n}"
+    end
   end
 
   class Node

data/lib/yarrow/content/expansion/aggregator.rb

@@ -0,0 +1,88 @@
+module Yarrow
+  module Content
+    module Expansion
+      class Aggregator
+        attr_reader :graph
+
+        def initialize(graph)
+          @graph = graph
+          @collections = {}
+        end
+
+        def before_traversal(policy)
+        end
+
+        def expand_container(container, policy)
+        end
+
+        def expand_collection(collection, policy)
+        end
+
+        def expand_entity(entity, policy)
+        end
+
+        def after_traversal(policy)
+        end
+
+        private
+
+        def create_collection(source_node, type, collection_const)
+          # Create a collection node with attached resource model
+          index = graph.create_node do |collection_node|
+            attributes = {
+              name: source_node.props[:name],
+              title: source_node.props[:name].capitalize,
+              body: ""
+            }
+            collection_node.label = :collection
+            collection_node.props[:type] = type
+            collection_node.props[:resource] = collection_const.new(attributes)
+          end
+
+          # Add this collection id to the lookup table for edge construction
+          @collections[source_node.props[:path]] = index
+
+          # Join the collection to its parent
+          if @collections.key?(source_node.props[:entry].parent.to_s)
+            graph.create_edge do |edge|
+              edge.label = :child
+              edge.from = @collections[source_node.props[:entry].parent.to_s].id
+              edge.to = index.id
+            end
+          end
+        end
+
+        def create_entity(source_node, parent_path, type, entity_const)
+          # Create an entity node with attached resource model
+          entity = graph.create_node do |entity_node|
+            attributes = {
+              name: source_node.props[:basename],
+              title: source_node.props[:basename].capitalize,
+              body: ""
+            }
+            entity_node.label = :entity
+            entity_node.props[:type] = type
+            entity_node.props[:resource] = entity_const.new(attributes)
+          end
+
+          graph.create_edge do |edge|
+            edge.label = :source
+            edge.from = entity.id
+            edge.to = source_node.id
+          end
+
+          if @collections.key?(parent_path)
+            graph.create_edge do |edge|
+              edge.label = :child
+              edge.from = @collections[parent_path].id
+              edge.to = entity.id
+            end
+          end
+        end
+
+        def connect_entity(entity, collection)
+        end
+      end
+    end
+  end
+end

data/lib/yarrow/content/expansion/basename_merge.rb

@@ -0,0 +1,44 @@
+module Yarrow
+  module Content
+    module Expansion
+      class BasenameMerge < Aggregator
+        def before_traversal(policy)
+          @bundles = {}
+          @container_collection = nil
+          @entity_bundles = {}
+          @entity_collections = {}
+        end
+      
+        def expand_container(container, policy)
+          puts "create_node label=:collection type=:#{policy.collection} name='#{container.props[:basename]}'"
+          @container_collection = container.props[:basename]
+        end
+      
+        def expand_collection(collection, policy)
+          if @container_collection == collection.props[:entry].parent.to_s
+            puts "create_node label=:collection type=:#{policy.entity} name='#{collection.props[:basename]}' collection='#{@container_collection}'"
+          end
+        end
+      
+        def expand_entity(entity, policy)
+          unless @entity_bundles.key?(entity.props[:basename])
+            @entity_bundles[entity.props[:basename]] = []
+          end
+      
+          @entity_bundles[entity.props[:basename]] << entity
+          @entity_collections[entity.props[:basename]] = @container_collection
+        end
+      
+        def after_traversal(policy)
+          @entity_bundles.each do |basename, bundle|
+            puts "create_node label=:resource type=:#{policy.entity} name='#{basename}' collection='#{@entity_collections[basename]}'"
+      
+            bundle.each do |asset|
+              puts "create_node label=:asset extension='#{asset.props[:ext]}' resource='#{basename}'"
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/yarrow/content/expansion/directory_map.rb

@@ -0,0 +1,21 @@
+module Yarrow
+  module Content
+    module Expansion
+      class DirectoryMap < Aggregator
+        def expand_container(container, policy)
+          puts "create_node label=:collection type=:#{policy.container} name='#{container.props[:basename]}'"
+          @current_collection = container.props[:basename]
+        end
+
+        def expand_collection(collection, policy)
+          puts "create_node label=:collection type=:#{policy.collection} name='#{collection.props[:basename]}' collection=#{@current_collection}"
+          @current_collection = collection.props[:basename]
+        end
+
+        def expand_entity(entity, policy)
+          puts "create_node label=:entity type=:#{policy.entity} name='#{entity.props[:basename]}' collection='#{@current_collection}"
+        end
+      end
+    end
+  end
+end

data/lib/yarrow/content/expansion/directory_merge.rb

@@ -0,0 +1,31 @@
+module Yarrow
+  module Content
+    module Expansion
+      class DirectoryMerge < Aggregator
+        def before_traversal(policy)
+          @bundles = {}
+          @current_collection = nil
+          @current_entity = nil
+        end
+
+        def expand_container(container, policy)
+          create_collection(container, policy.container, policy.container_const)
+          @current_collection = container.props[:path]
+        end
+
+        def expand_collection(collection, policy)
+          @current_entity = collection.props[:basename]
+        end
+
+        def expand_entity(entity, policy)
+          if entity.props[:basename] == @current_entity && entity.props[:ext] == ".md"
+            create_entity(entity, @current_collection, policy.entity, policy.entity_const)
+          else
+            # TODO: attach static assets to the entity as well
+            #puts "--> create_node label=:asset type=:asset name='#{entity.props[:basename]}' entity='#{@current_entity}'"
+          end
+        end
+      end
+    end
+  end
+end

data/lib/yarrow/content/expansion/file_list.rb

@@ -0,0 +1,15 @@
+module Yarrow
+  module Content
+    module Expansion
+      class FileList < Strategy
+        def expand(policy)
+          start_node = graph.n(:root).out(name: policy.container.to_s)
+
+          start_node.out(:files).each do |file_node|
+
+          end
+        end
+      end
+    end
+  end
+end

data/lib/yarrow/content/expansion/filename_map.rb

@@ -0,0 +1,24 @@
+module Yarrow
+  module Content
+    module Expansion
+      class FilenameMap < Aggregator
+        def expand_container(container, policy)
+          create_collection(container, policy.container, policy.container_const)
+          @current_collection = container.props[:path]
+        end
+
+        def expand_collection(collection, policy)
+          create_collection(collection, policy.collection, policy.collection_const)
+          @current_collection = collection.props[:path]
+        end
+
+        def expand_entity(entity, policy)
+          if policy.match_by_extension(entity.props[:ext])
+            parent_path = entity.incoming(:directory).first.props[:path]
+            create_entity(entity, parent_path, policy.entity, policy.entity_const)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/yarrow/content/expansion/strategy.rb

@@ -8,6 +8,36 @@ module Yarrow
 
         def initialize(graph)
           @graph = graph
+          @subcollections = {}
+          @entity_links = []
+          @index_links = []
+          @index = nil
+        end
+
+        # Expand a directory to a collection node representing a collection of entities
+        def expand_directory(policy, node)
+          index = graph.create_node do |collection_node|
+
+            collection_attrs = {
+              name: node.props[:name],
+              title: node.props[:name].capitalize,
+              body: ""
+            }
+
+            populate_collection(collection_node, policy, collection_attrs)
+          end
+
+          # Add this collection id to the lookup table for edge construction
+          @subcollections[node.props[:path]] = index
+
+          # Join the collection to its parent
+          unless node.props[:slug] == policy.collection.to_s || !@subcollections.key?(node.props[:entry].parent.to_s)
+            graph.create_edge do |edge|
+              edge.label = :child
+              edge.from = @subcollections[node.props[:entry].parent.to_s].id
+              edge.to = index.id
+            end
+          end
         end
 
         # Extract collection level configuration/metadata from the root node for
@@ -65,6 +95,23 @@ module Yarrow
           end
           # TODO: Raise error if unsupported extname reaches here
         end
+
+        def connect_expanded_entities
+          # Once all files and directories have been expanded, connect all the child
+          # edges between collections and entities
+          @entity_links.each do |entity_link|
+            graph.create_edge do |edge|
+              edge.label = :child
+              edge.from = entity_link[:parent_id].id
+              edge.to = entity_link[:child_id].id
+            end
+          end
+
+          # Merge index page body and metadata with their parent collections
+          @index_links.each do |index_link|
+            merge_collection_index(index_link[:parent_id], policy, index_link[:index_attrs])
+          end
+        end
       end
     end
   end

data/lib/yarrow/content/expansion/traversal.rb

@@ -0,0 +1,67 @@
+module Yarrow
+  module Content
+    module Expansion
+      class Traversal
+        attr_reader :graph, :policy, :aggregator
+
+        def initialize(graph, policy)
+          @graph = graph
+          @policy = policy
+          @aggregator = policy.aggregator_const.new(graph)
+        end
+
+        # If source path represents entire content dir, then include the entire
+        # content dir instead of scanning from a subfolder matching the name of
+        # the collection.
+        def source_node
+          if policy.source_path == "."
+            graph.n(:root).out(:directory)
+          else
+            graph.n(name: policy.source_path)
+          end
+        end
+
+        def on_root_visited(root_node)
+          aggregator.expand_container(root_node, policy)
+        end
+
+        def on_directory_visited(dir_node)
+          # TODO: match on potential directory extension/filter
+          aggregator.expand_collection(dir_node, policy)
+        end
+
+        def on_file_visited(file_node)
+          # TODO: dispatch underscore prefix or index files separately
+          # TODO: match on file extension
+          aggregator.expand_entity(file_node, policy)
+        end
+
+        def on_traversal_initiated
+          aggregator.before_traversal(policy)
+        end
+
+        def on_traversal_completed
+          aggregator.after_traversal(policy)
+        end
+
+        def expand
+          on_traversal_initiated
+
+          traversal = source_node.depth_first.each
+          
+          on_root_visited(traversal.next)
+
+          loop do
+            node = traversal.next
+            case node.label
+            when :directory then on_directory_visited(node)
+            when :file then on_file_visited(node)
+            end
+          end
+
+          on_traversal_completed
+        end
+      end
+    end
+  end
+end