nanoc-filesystem-i18n 0.1.0.pre1

data/LICENSE

@@ -0,0 +1,25 @@
+Copyright (c) 2010 Yann Lugrin
+
+The original source code is part of nanoc software with the same licence
+and have the following copyright notice:
+copyright (c) 2007-2010 Denis Defreyne and nanoc contributors
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.rdoc

@@ -0,0 +1,109 @@
+= nanoc-filesystem-i18n
+
+The filesystem_i18n data source is a localized data source for a nanoc
+site. It stores all data as files on the hard disk and is fully compatible
+with {Nanoc3::DataSources::FilesystemUnified} and {Nanoc3::DataSources::FilesystemVerbose}.
+
+== Resources
+
+- FilesystemI18n (http://github.com/yannlugrin/nanoc-filesystem-i18n)
+- Nanoc (http://nanoc.stoneship.org)
+- Nanoc on Github (http://github.com/ddfreyne/nanoc)
+- Ruby I18n (http://github.com/svenfuchs/i18n)
+
+== Documentation
+
+=== Installation
+
+ gem install nanoc-filesystem-i18n --pre
+
+Add following require in your `lib/default.rb` file:
+
+  require 'nanoc3/data_sources/filesystem_i18n'
+
+Edit `config.yaml` file of your nanoc site and change data_sources type
+from `filesystem_unified` or `filesystem_verbose` to `filesystem_i18n`:
+
+  data_sources:
+    -
+      type: filesystem_i18n
+
+In the data source section of `config.yaml` file, add following information:
+
+      config:
+        locale:
+          # list of availables locale, ser default to true to select default
+          # locale.
+          availables:
+            fr:
+              name: Francais
+            en:
+              name: English
+              default: true
+          # objects should be not localized
+          exclude:
+            item: ['/css*', '/js*']
+            layout: ['*']
+
+See following configuration section for more information.
+
+=== Data source specifications
+
+The filesystem_i18n data source stores its items and layouts in nested
+directories or files. Each directory represents a single item or layout,
+but an item can be a simple file in a directory. The root directory for
+items is the `content` directory; for layouts it is the `layouts`
+directory.
+
+Every object (item or layout) is represented by a meta file and one or
+more content files with a minimum of one file. The content file contains
+he actual item content or layout, while the meta file contains the item’s
+or the layout’s metadata, formatted as YAML.
+
+Both meta files and content files are named after its parent directory
+(i.e. item). For example, an item/layout named `foo` will have a directory
+named `foo`, with e.g. a `foo.markdown` or `index.markdown` content file
+and a `foo.yaml` or `index.yaml` meta file. An item/layout named `foo/bar`
+can be also created in parent directory named `foo` without dedicated
+directory, with e.g. a `foo.markdown` content file and `foo.yaml` meta
+file. Root item already named `index.markdown` for content file (extension
+can be different) and `index.yaml` for meta file.
+
+Content file extensions are not used for determining the filter that
+should be run; the meta file or configuration file defines the list of
+filters. The meta file extension must always be `.yaml`, though.
+
+An item/layout content file named `foo.markdown` contain default content
+for all locales, but you can create a content file for each locales with
+e.g. a `foo.fr.markdown` for locale `fr`. If default locale for site is
+`fr` and the `foo.fr.markdown` file is present but `foo.markdow` file not,
+the `fr` content file is used by default.
+
+The identifier is calculated by stripping the extension (part after last dot)
+and locale code.
+
+=== Configuration
+
+soon...
+
+=== Manage locale meta and content
+
+soon...
+
+=== Deployement
+
+soon...
+
+== Note on Patches/Pull Requests
+
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+== Copyright
+
+Copyright (c) 2010 Yann Lugrin. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,59 @@
+require 'rubygems'
+require 'rake'
+require File.dirname(__FILE__) + '/lib/nanoc3/data_sources/filesystem_i18n/version'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = 'nanoc-filesystem-i18n'
+    gem.version = Nanoc3::DataSources::FilesystemI18n::Version
+    gem.summary = %Q{I18n filesystem based data source for nanoc}
+    gem.description = %Q{I18n filesystem based data source for nanoc. Compatible with nanoc 3 and default filesystem based data source.}
+    gem.email = 'yann.lugrin@sans-savoir.net'
+    gem.homepage = 'http://github.com/yannlugrin/nanoc-filesystem-i18n'
+    gem.authors = ['Yann Lugrin']
+    gem.add_dependency 'nanoc', '>= 3.1.2'
+    gem.add_dependency 'i18n', '>= 0'
+    gem.add_development_dependency 'minitest', '>= 0'
+    gem.add_development_dependency 'yard', '>= 0'
+    gem.files.exclude '.gitignore', '.document', 'nanoc-filesystem-i18n.gemspec'
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/test_*.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :test => :check_dependencies
+
+task :default => :test
+
+begin
+  require 'nanoc3'
+  require 'yard'
+  YARD::Rake::YardocTask.new
+rescue LoadError
+  task :yardoc do
+    abort "YARD is not available. In order to run yardoc, you must: sudo gem install yard"
+  end
+end

data/lib/nanoc3/data_sources/filesystem_i18n.rb

@@ -0,0 +1,399 @@
+# encoding: utf-8
+
+require 'nanoc3'
+require 'nanoc3/extra/i18n'
+
+module Nanoc3::DataSources
+
+  # The filesystem_i18n data source is a localized data source for a nanoc
+  # site. It stores all data as files on the hard disk and is fully compatible
+  # with {Nanoc3::DataSources::FilesystemUnified} and {Nanoc3::DataSources::FilesystemVerbose}.
+  #
+  # None of the public api methods are documented in this file. See
+  # {Nanoc3::DataSource} for documentation on the overridden methods instead.
+  #
+  # For more information about this data source specifications and configuration,
+  # please read the Readme file.
+  class FilesystemI18n < Nanoc3::DataSource
+    identifier :filesystem_i18n
+
+    # The VCS that will be called when adding, deleting and moving files. If
+    # no VCS has been set, or if the VCS has been set to `nil`, a dummy VCS
+    # will be returned.
+    #
+    # @return [Nanoc3::Extra::VCS, nil] The VCS that will be used.
+    def vcs
+      @vcs ||= Nanoc3::Extra::VCSes::Dummy.new
+    end
+    attr_writer :vcs
+
+    # See {Nanoc3::DataSource#up}.
+    def up
+      I18n.load_config(@config ? @config[:locale] : nil)
+    end
+
+    # See {Nanoc3::DataSource#down}.
+    def down
+    end
+
+    # See {Nanoc3::DataSource#setup}.
+    def setup
+      # Create directories
+      %w( content layouts lib ).each do |dir|
+        FileUtils.mkdir_p(dir)
+        vcs.add(dir)
+      end
+    end
+
+    # See {Nanoc3::DataSource#items}.
+    def items
+      load_objects('content', 'item', Nanoc3::Item)
+    end
+
+    # See {Nanoc3::DataSource#layouts}.
+    def layouts
+      load_objects('layouts', 'layout', Nanoc3::Layout)
+    end
+
+    # See {Nanoc3::DataSource#create_item}.
+    def create_item(content, attributes, identifier, params={})
+      create_object('content', content, attributes, identifier, params)
+    end
+
+    # See {Nanoc3::DataSource#create_layout}.
+    def create_layout(content, attributes, identifier, params={})
+      create_object('layouts', content, attributes, identifier, params)
+    end
+
+  private
+
+    # Creates a new object (item or layout) on disk in dir_name according to
+    # the given identifier. The file will have its attributes taken from the
+    # attributes hash argument and its content from the content argument.
+    def create_object(dir_name, content, attributes, identifier, params={})
+      # Check for periods
+      if (@config.nil? || !@config[:allow_periods_in_identifiers]) && identifier.include?('.')
+        raise RuntimeError,
+          "Attempted to create an object in #{dir_name} with identifier #{identifier} containing a period, but allow_periods_in_identifiers is not enabled in the site configuration. (Enabling allow_periods_in_identifiers may cause the site to break, though.)"
+      end
+
+      # Get filenames
+      ext = params[:extension] || '.html'
+      meta_filename    = dir_name + (identifier == '/' ? '/index.yaml' : identifier[0..-2] + '.yaml')
+      content_filename = dir_name + (identifier == '/' ? '/index.html' : identifier[0..-2] + ext)
+      parent_path = File.dirname(meta_filename)
+
+      # Notify
+      Nanoc3::NotificationCenter.post(:file_created, meta_filename)
+      Nanoc3::NotificationCenter.post(:file_created, content_filename)
+
+      # Create files
+      FileUtils.mkdir_p(parent_path)
+      File.open(meta_filename,    'w') { |io| io.write(YAML.dump(attributes.stringify_keys)) }
+      File.open(content_filename, 'w') { |io| io.write(content) }
+    end
+
+    # Creates instances of klass corresponding to the files in dir_name. The
+    # kind attribute indicates the kind of object that is being loaded and is
+    # used solely for debugging purposes.
+    #
+    # This particular implementation loads objects from a filesystem-based
+    # data source where content and attributes can be spread over two separate
+    # files (one for metadata, with `yaml` extension, and one or more for content
+    # with same extension and locale code before extension, separated by dot. A
+    # content for default locale is used by default if file without locale code
+    # is not present.
+    #
+    # The contents and meta-file are optional (but at least one of them needs
+    # to be present, obviously. If a content file is present, file without locale
+    # code or with default locale code is needed) and the content file can start
+    # with a metadata section (if no metadata file is present).
+    def load_objects(dir_name, kind, klass)
+      all_split_files_in(dir_name).map do |base_filename, (meta_ext, content_ext, locales)|
+        I18n.locale = I18n.default_locale # Set current locale to default
+
+        # Get filenames
+        meta_filename    = filename_for(base_filename, meta_ext)
+        content_filename = filename_for(base_filename, content_ext)
+
+        # is binary content?
+        is_binary = !!(content_filename && !@site.config[:text_extensions].include?(File.extname(content_filename)[1..-1]))
+
+        # Read content and metadata
+        meta, content_or_filename = parse(content_filename, meta_filename, kind, (is_binary && klass == Nanoc3::Item))
+
+        # Is locale content?
+        # - excluded content with locale meta IS a locale content
+        # - excluded content without locale meta IS NOT locale content
+        # - included content with or without locale meta IS locale content
+        # - included content with locale meta set to `false` IS NOT locale
+        #   content
+        is_locale = !!(meta['locale'] || (meta['locale'] != false && locale_content?(content_filename || meta_filename, kind)))
+
+        # Create one item by locale, if content don't need a localized version,
+        # use default locale
+        (is_locale ? I18n.available_locales : [I18n.default_locale]).map do |locale|
+          I18n.locale = locale # Set current locale
+
+          # Read content and metadata (only if is localized, default is already
+          # loaded)
+          meta, content_or_filename = parse(content_filename, meta_filename, kind, (is_binary && klass == Nanoc3::Item)) if is_locale
+
+          # merge meta for current locale, default locale meta used by
+          # default is meta don't have key
+          if is_locale
+            meta_locale = meta.delete('locale') {|el| Hash.new }
+            meta = (meta_locale[I18n.default_locale] || Hash.new).merge(meta)
+            meta.merge!(meta_locale[locale.to_s] || Hash.new)
+          end
+
+          # Get attributes
+          attributes = {
+            :filename         => content_filename,
+            :content_filename => content_filename,
+            :meta_filename    => meta_filename,
+            :extension        => content_filename ? ext_of(content_filename)[1..-1] : nil,
+            :locale           => locale,
+            # WARNING :file is deprecated; please create a File object manually
+            # using the :content_filename or :meta_filename attributes.
+            # TODO [in nanoc 4.0] remove me
+            :file             => content_filename ? Nanoc3::Extra::FileProxy.new(content_filename) : nil
+          }.merge(meta)
+
+          # Get identifier
+          if meta_filename
+            identifier = identifier_for_filename(meta_filename[(dir_name.length+1)..-1])
+          elsif content_filename
+            identifier = identifier_for_filename(content_filename[(dir_name.length+1)..-1])
+          else
+            raise RuntimeError, "meta_filename and content_filename are both nil"
+          end
+          # Prepend locale code to identifier if content is localized
+          identifier = "/#{locale}#{identifier}" if is_locale
+
+          # Get modification times
+          meta_mtime    = meta_filename    ? File.stat(meta_filename).mtime    : nil
+          content_mtime = content_filename ? File.stat(content_filename).mtime : nil
+          if meta_mtime && content_mtime
+            mtime = meta_mtime > content_mtime ? meta_mtime : content_mtime
+          elsif meta_mtime
+            mtime = meta_mtime
+          elsif content_mtime
+            mtime = content_mtime
+          else
+            raise RuntimeError, "meta_mtime and content_mtime are both nil"
+          end
+
+          # Create layout object
+          klass.new(
+            content_or_filename, attributes, identifier,
+            :binary => is_binary, :mtime => mtime
+          )
+        end
+      end.flatten # elements is an array with all locale item, flatten in to one items list
+    end
+
+    # Finds all items/layouts/... in the given base directory. Returns a hash
+    # in which the keys are the file's dirname + basenames, and the values is
+    # an array with three elements: the metafile extension, the content file
+    # extension and an array with locales of content file. The meta file
+    # extension or the content file extension can be, but not both. Backup
+    # files are ignored. For example:
+    #
+    #   {
+    #     'content/foo' => [ 'yaml', 'html', ['en', 'fr'] ],
+    #     'content/bar' => [ 'yaml',  nil  , [] ],
+    #     'content/qux' => [ nil,    'html', ['en'] ]
+    #   }
+    def all_split_files_in(dir_name)
+      # Get all good file names
+      filenames = Dir[dir_name + '/**/*'].select { |i| File.file?(i) }
+      filenames.reject! { |fn| fn =~ /(~|\.orig|\.rej|\.bak)$/ }
+
+      # Group by identifier
+      grouped_filenames = filenames.group_by { |fn| basename_of(fn).gsub('/index', '') }
+
+      # Convert values into metafile/content file extension tuple
+      grouped_filenames.each_pair do |key, filenames|
+        # Divide
+        meta_filenames    = filenames.select { |fn| ext_of(fn) == '.yaml' }
+        content_filenames = filenames.select { |fn| ext_of(fn) != '.yaml' }
+
+        # Check number of files per type
+        if ![ 0, 1 ].include?(meta_filenames.size)
+          raise RuntimeError, "Found #{meta_filenames.size} meta files for #{key}; expected 0 or 1"
+        end
+        if !( 0 .. (I18n.available_locales.empty? ? 1 : I18n.available_locales.size) ).include?(content_filenames.size)
+          raise RuntimeError, "Found #{content_filenames.size} content files for #{key}; expected 0 to #{I18n.available_locales.size}"
+        end
+
+        # Check content file extensions and default file
+        if fn = content_filenames.find {|fn| ext_of(fn) != ext_of(content_filenames[0]) }
+          raise RuntimeError, "Found multiple content extensions for `#{basename_of(fn)}.???`"
+        end
+
+        # Reorder elements and convert to extnames
+        filenames[0] = meta_filenames[0]    ? ext_of(meta_filenames[0])[1..-1]    : nil
+        filenames[1] = content_filenames[0] ? ext_of(content_filenames[0])[1..-1] : nil
+        filenames[2] = []
+        content_filenames.each do |content_filename|
+          filenames[2] << locale_of(content_filename) if locale_of(content_filename)
+        end
+      end
+
+      # Done
+      grouped_filenames
+    end
+
+    # Returns the filename for the given base filename, current locale (or
+    # default locale) and the extension.
+    #
+    # If the extension is nil, this function should return nil as well.
+    #
+    # This implementation is compatible with simple file item and directory
+    # item (with index or named file), find order is directory with named
+    # file, directory with index file and simple file. For locale, find
+    # order is current locale file, without locale file and default locale
+    # file.
+    #
+    # Item priority order:
+    # /foo/foo.html
+    # /foo/index.html
+    # /foo.html
+    #
+    # Locale priority order:
+    # /foo/foo.{current locale}.html
+    # /foo/foo.html
+    # /foo/foo.{default locale}.html
+    #
+    def filename_for(base_filename, ext)
+      last_part = base_filename.split('/')[-1]
+      lang_part = "{.#{I18n.locale},,.#{I18n.default_locale}}"
+      base_glob = base_filename.split('/')[0..-2].join('/') + "{/,}#{last_part}{/index,}#{lang_part}."
+
+      ext ? Dir[base_glob + ext][0] : nil
+    end
+
+    # Returns the identifier that corresponds with the given filename, which
+    # can be the content filename or the meta filename.
+    def identifier_for_filename(filename)
+      # Item is a directory with an index file
+      if filename =~ /index(\.[a-z]{2})?\.[^\/]+$/
+        regex = ((@config && @config[:allow_periods_in_identifiers]) ? /index(\.[a-z]{2})?(\.[^\/\.]+)$/        : /index(\.[a-z]{2})?(\.[^\/]+)$/)
+      # Item is a directory with a named file
+      elsif basename_of(filename).split(/\//)[-1] == basename_of(filename).split(/\//)[-2]
+        regex = ((@config && @config[:allow_periods_in_identifiers]) ? /(\/[^\/]+)?(\.[a-z]{2})?(\.[^\/\.]+)$/  : /(\/[^\/]+)?(\.[a-z]{2})?(\.[^\/]+)$/)
+      # Item is a simple file
+      else
+        regex = ((@config && @config[:allow_periods_in_identifiers]) ? /(\.[a-z]{2})?(\.[^\/\.]+)$/  : /(\.[a-z]{2})?(\.[^\/]+)$/)
+      end
+
+      filename.sub(regex, '').cleaned_identifier
+    end
+
+    # Returns the base name of filename, i.e. filename with the first or all
+    # extensions stripped off. By default, all extensions are stripped off,
+    # but when allow_periods_in_identifiers is set to true in the site
+    # configuration, only the last extension will be stripped .
+    def basename_of(filename)
+      filename.sub(extension_regex, '')
+    end
+
+    # Returns the extension(s) of filename. Supports multiple extensions.
+    # Includes the leading period. Return empty string if don't found
+    # extension.
+    def ext_of(filename)
+      filename =~ extension_regex ? $2 : ''
+    end
+
+    # Returns a regex that is used for determining the extension of a file
+    # name. The first match group will be the locale code (with leading
+    # period) if exist and the second match group is entire extension,
+    # including the leading period.
+    def extension_regex
+      if @config && @config[:allow_periods_in_identifiers]
+        /(\.[a-z]{2})?(\.[^\/\.]+$)/
+      else
+        /(\.[a-z]{2})?(\.[^\/]+$)/
+      end
+    end
+
+    # Returnes the locale code of filename or nil if filename is the default
+    # file (without locale code).
+    def locale_of(filename)
+      locale = (filename =~ extension_regex ? $1 : nil)
+      locale ? locale.gsub(/^\./, '').to_sym : nil
+    end
+
+    # Returnes true if this content is localized (based on data source config)
+    def locale_content?(base_filename_or_identifier, kind)
+      base_filename_or_identifier =~ locale_exclude_regex(kind) ? false : true
+    end
+
+    # Returnes regex that is used for determing if content is excluded not
+    # localized (layouts, css, js, ...).
+    #
+    # Add to locale config follwing keys (exemple):
+    #   exclude:
+    #     item: ['/css*', '/js*']
+    #     layout: ['*']
+    #
+    def locale_exclude_regex(kind)
+      Regexp.union(I18n.exclude_list(kind.to_sym).map do |identifier|
+        if identifier.is_a? String
+          # Add leading/trailing slashes if necessary
+          new_identifier = identifier.dup
+          new_identifier[/^/] = '/' if identifier[0,1] != '/'
+          new_identifier[/$/] = '/' unless [ '*', '/' ].include?(identifier[-1,1])
+
+          /^[^\/]*#{new_identifier.gsub('*', '(.*?)').gsub('+', '(.+?)')}$/
+        else
+          identifier
+        end
+      end)
+    end
+
+    # Parses the file named `filename` and returns an array with its first
+    # element a hash with the file's metadata, and with its second element the
+    # file content itself.
+    def parse(content_filename, meta_filename, kind, is_binary)
+      # Read content and metadata from separate files
+      if meta_filename || is_binary
+        meta = (meta_filename && YAML.load_file(meta_filename)) || {}
+
+        if is_binary
+          content = content_filename
+        else
+          content = content_filename ? File.read(content_filename) : ''
+        end
+
+        return [ meta, content ]
+      end
+
+      # Read data
+      data = File.read(content_filename)
+
+      # Check presence of metadata section
+      if data !~ /^(-{5}|-{3})/
+        return [ {}, data ]
+      end
+
+      # Split data
+      pieces = data.split(/^(-{5}|-{3})/)
+      if pieces.size < 4
+        raise RuntimeError.new(
+          "The file '#{content_filename}' does not seem to be a nanoc #{kind}"
+        )
+      end
+
+      # Parse
+      meta    = YAML.load(pieces[2]) || {}
+      content = pieces[4..-1].join.strip
+
+      # Done
+      [ meta, content ]
+    end
+
+  end
+end
+