mojombo-jekyll 0.2.0 → 0.3.0

data/History.txt

@@ -1,3 +1,24 @@
+== 0.3.0 / 2008-12-24
+  * Major Enhancements
+    * Added --server option to start a simple WEBrick server on destination directory [github.com/johnreilly and github.com/mchung]
+  * Minor Enhancements
+    * Added post categories based on directories containing _posts [github.com/mreid]
+    * Added post topics based on directories underneath _posts
+    * Added new date filter that shows the full month name [github.com/mreid]
+    * Merge Post's YAML front matter into its to_liquid payload [github.com/remi]
+    * Restrict includes to regular files underneath _includes
+  * Bug Fixes
+    * Change YAML delimiter matcher so as to not chew up 2nd level markdown headers [github.com/mreid]
+    * Fix bug that meant page data (such as the date) was not available in templates [github.com/mreid]
+    * Properly reject directories in _layouts
+
+== 0.2.1 / 2008-12-15
+  * Major Changes
+    * Use Maruku (pure Ruby) for Markdown by default [github.com/mreid]
+    * Allow use of RDiscount with --rdiscount flag
+  * Minor Enhancements
+    * Don't load directory_watcher unless it's needed [github.com/pjhyett]
+
 == 0.2.0 / 2008-12-14
   * Major Changes
     * related_posts is now found in site.related_posts

data/Manifest.txt

@@ -8,7 +8,10 @@ lib/jekyll.rb
 lib/jekyll/albino.rb
 lib/jekyll/converters/csv.rb
 lib/jekyll/converters/mephisto.rb
+lib/jekyll/converters/mt.rb
+lib/jekyll/converters/wordpress.rb
 lib/jekyll/convertible.rb
+lib/jekyll/core_ext.rb
 lib/jekyll/filters.rb
 lib/jekyll/layout.rb
 lib/jekyll/page.rb
@@ -17,16 +20,17 @@ lib/jekyll/site.rb
 lib/jekyll/tags/highlight.rb
 lib/jekyll/tags/include.rb
 test/helper.rb
-test/source/_includes/sig.textile
+test/source/_includes/sig.markdown
 test/source/_layouts/default.html
 test/source/_layouts/simple.html
 test/source/_posts/2008-10-18-foo-bar.textile
 test/source/_posts/2008-11-21-complex.textile
-test/source/_posts/2008-12-13-include.textile
+test/source/_posts/2008-12-03-permalinked-post.textile
+test/source/_posts/2008-12-13-include.markdown
 test/source/css/screen.css
 test/source/index.html
-test/source/posts/2008-12-03-permalinked-post.textile
 test/suite.rb
+test/test_generated_site.rb
 test/test_jekyll.rb
 test/test_post.rb
 test/test_site.rb

data/README.textile

@@ -31,9 +31,10 @@ various data about my site. A reverse chronological list of all my blog posts
 can be found in <code>site.posts</code>. Each post, in turn, contains various
 fields such as <code>title</code> and <code>date</code>.
 
-Jekyll gets the list of blog posts by parsing the files in the
-"_posts":http://github.com/mojombo/tpw/tree/master/_posts directory. Each
-post's filename contains the publishing date and slug (what shows up in the
+Jekyll gets the list of blog posts by parsing the files in any
+"_posts":http://github.com/mojombo/tpw/tree/master/_posts directory found in
+subdirectories below the root. 
+Each post's filename contains the publishing date and slug (what shows up in the
 URL) that the final HTML file should have. Open up the file corresponding to a
 blog post:
 "2008-11-17-blogging-like-a-hacker.textile":http://github.com/mojombo/tpw/tree/master/_posts/2008-11-17-blogging-like-a-hacker.textile.
@@ -52,6 +53,12 @@ filename is used to construct the URL in the generated site. The example post,
 for instance, ends up at
 <code>http://tom.preston-werner.com/2008/11/17/blogging-like-a-hacker.html</code>.
 
+Categories for posts are derived from the directory structure the posts were
+found within. A post that appears in the directory foo/bar/_posts is placed in
+the categories 'foo' and 'bar'. By selecting posts from particular categories
+in your Liquid templates, you will be able to host multiple blogs within a
+site.
+
 Files that do not reside in directories prefixed with an underscore are
 mirrored into a corresponding directory structure in the generated site. If a
 file does not have a YAML preface, it is not run through the Liquid
@@ -79,6 +86,17 @@ The best way to install Jekyll is via RubyGems:
 
   $ sudo gem install mojombo-jekyll -s http://gems.github.com/
 
+Jekyll requires the gems `directory_watcher`, `liquid`, `open4`, 
+and `maruku` (for markdown support). These are automatically
+installed by the gem install command.
+
+Maruku comes with optional support for LaTeX to PNG rendering via
+"blahtex":http://gva.noekeon.org/blahtexml/ (Version 0.6) which must be in
+your $PATH along with `dvips`. 
+
+(NOTE: the version of maruku I am using is `remi-maruku` on GitHub as it
+does not assume a fixed location for `dvips`.)
+
 h2. Run
 
   $ cd /path/to/proto/site
@@ -112,6 +130,21 @@ during the conversion:
 
   $ jekyll --pygments
 
+By default, Jekyll uses "Maruku":http://maruku.rubyforge.org (pure Ruby) for
+Markdown support. If you'd like to use RDiscount (faster, but requires
+compilation), you must install it (gem install rdiscount) and then you can
+have it used instead:
+
+  $ jekyll --rdiscount
+  
+When previewing complex sites locally, simply opening the site in a web 
+browser (using file://) can cause problems with links that are relative to
+the site root (e.g., "/stylesheets/style.css"). To get around this, Jekyll
+can launch a simple WEBrick server (works well in conjunction with --auto).
+Default port is 4000:
+
+  $ jekyll --server [PORT]
+
 h2. Data
 
 Jekyll traverses your site looking for files to process. Any files with YAML
@@ -131,7 +164,7 @@ h3. Global
 
   content
     In layout files, this contains the content of the subview(s). In Posts or
-    pages, this is undefined.
+    Pages, this is undefined.
 
 h3. Site
 
@@ -147,6 +180,9 @@ h3. Site
     high quality but slow to compute results, run the jekyll command with the 
     --lsi (latent semantic indexing) option.
 
+  site.categories.CATEGORY
+    The list of all Posts in category CATEGORY.
+
 h3. Post
 
   post.title
@@ -163,6 +199,18 @@ h3. Post
     An identifier unique to the Post (useful in RSS feeds).
     e.g. /2008/12/14/my-post
 
+  post.categories
+    The list of categories to which this post belongs. Categories are
+    derived from the directory structure above the _posts directory. For
+    example, a post at /work/code/_posts/2008-12-24-closures.textile
+    would have this field set to ['work', 'code'].
+
+  post.topics
+    The list of topics for this Post. Topics are derived from the directory
+    structure beneath the _posts directory. For example, a post at
+    /_posts/music/metal/2008-12-24-metalocalypse.textile would have this field
+    set to ['music', 'metal'].
+
   post.content
     The content of the Post.
 
@@ -240,7 +288,8 @@ becomes
 
 h3. Include (Tag)
 
-If you have small page fragments that you wish to include in multiple places on your site, you can use the <code>include</code> tag.
+If you have small page fragments that you wish to include in multiple places
+on your site, you can use the <code>include</code> tag.
 
 <pre>{% include sig.textile %}</pre>
 
@@ -278,6 +327,22 @@ highlighting stylesheet. For an example stylesheet you can look at
 are the same styles as used by GitHub and you are free to use them for your
 own site.
 
+h2. Categories
+
+Posts are placed into categories based on the directory structure they are found
+within (see above for an example). The categories can be accessed from within
+a Liquid template as follows:
+
+<pre>
+{% for post in site.categories.foo %}
+	<li><span>{{ post.date | date_to_string }}</span> - {{ post.title }}</li>
+{% endfor %}
+</pre>
+
+This would list all the posts in the category 'foo' by date and title.
+
+The posts within each category are sorted in reverse chronological order.
+
 h2. Contribute
 
 If you'd like to hack on Jekyll, grab the source from GitHub. To get
@@ -296,6 +361,24 @@ The best way to get your changes merged back into core is as follows:
 # Push the branch up to GitHub
 # Send me (mojombo) a pull request for your branch
 
+h2. Blog migrations
+
+h3. Movable Type
+
+To migrate your MT blog into Jekyll, you'll need read access to the database.
+The lib/jekyll/converters/mt.rb module provides a simple convert to create
+.markdown files in a _posts directory based on the entries contained therein.
+
+  $ export DB=my_mtdb
+  $ export USER=dbuser
+  $ export PASS=dbpass
+  $ ruby -r './lib/jekyll/converters/mt' -e 'Jekyll::MT.process( \
+    "#{ENV["DB"]}", "#{ENV["USER"]}", "#{ENV["PASS"]}")'
+
+You may need to adjust the SQL query used to retrieve MT entries. Left alone,
+it will attempt to pull all entries across all blogs regardless of status.
+Please check the results and verify the posts before publishing.
+
 h2. License
 
 (The MIT License)

data/Rakefile

@@ -5,7 +5,7 @@ require 'lib/jekyll'
 Hoe.new('jekyll', Jekyll::VERSION) do |p|
   p.developer('Tom Preston-Werner', 'tom@mojombo.com')
   p.summary = "Jekyll is a simple, blog aware, static site generator."
-  p.extra_deps = ['RedCloth', 'liquid', 'classifier', 'rdiscount', 'directory_watcher', 'open4']
+  p.extra_deps = ['RedCloth', 'liquid', 'classifier', 'maruku', 'directory_watcher', 'open4']
 end
 
 desc "Open an irb session preloaded with this library"
@@ -18,4 +18,7 @@ namespace :convert do
   task :mephisto do
     sh %q(ruby -r './lib/jekyll/converters/mephisto' -e 'Jekyll::Mephisto.postgres(:database => "#{ENV["DB"]}")')
   end
+  task :mt do
+    sh %q(ruby -r './lib/jekyll/converters/mt' -e 'Jekyll::MT.process("#{ENV["DB"]}", "#{ENV["USER"]}", "#{ENV["PASS"]}")')
+  end
 end

data/bin/jekyll

@@ -25,6 +25,11 @@ opts = OptionParser.new do |opts|
     options[:auto] = true
   end
   
+  opts.on("--server [PORT]", "Start web server (default port 4000)") do |port|
+    options[:server] = true
+    options[:server_port] = port || 4000
+  end
+  
   opts.on("--lsi", "Use LSI for better related posts") do
     Jekyll.lsi = true
   end
@@ -32,6 +37,16 @@ opts = OptionParser.new do |opts|
   opts.on("--pygments", "Use pygments to highlight code") do
     Jekyll.pygments = true
   end
+  
+  opts.on("--rdiscount", "Use rdiscount gem for Markdown") do
+    begin
+      require 'rdiscount'
+      Jekyll.markdown_proc = Proc.new { |x| RDiscount.new(x).to_html }
+      puts 'Using rdiscount for Markdown'
+    rescue LoadError
+      puts 'You must have the rdiscount gem installed first'
+    end
+  end
 end
 
 opts.parse!
@@ -69,6 +84,8 @@ case ARGV.size
 end
 
 if options[:auto]
+  require 'directory_watcher'
+
   puts "Auto-regenerating enabled: #{source} -> #{destination}"
   
   dw = DirectoryWatcher.new(source)
@@ -83,7 +100,28 @@ if options[:auto]
   
   dw.start
   
-  loop { sleep 1000 }
+  unless options[:server]
+    loop { sleep 1000 }
+  end
 else
   Jekyll.process(source, destination)
+  puts "Successfully generated site in #{destination}"
+end
+
+if options[:server]
+  require 'webrick'
+  include WEBrick
+  
+  FileUtils.mkdir_p(destination)
+
+  s = HTTPServer.new(
+    :Port            => options[:server_port],
+    :DocumentRoot    => destination
+  )
+  t = Thread.new {
+    s.start
+  }
+  
+  trap("INT") { s.shutdown }
+  t.join()
 end

data/jekyll.gemspec

@@ -1,22 +1,22 @@
 Gem::Specification.new do |s|
   s.name = %q{jekyll}
-  s.version = "0.2.0"
+  s.version = "0.3.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Tom Preston-Werner"]
-  s.date = %q{2008-12-14}
+  s.date = %q{2008-12-24}
   s.default_executable = %q{jekyll}
   s.email = ["tom@mojombo.com"]
   s.executables = ["jekyll"]
   s.extra_rdoc_files = ["History.txt", "Manifest.txt"]
-  s.files = ["History.txt", "Manifest.txt", "README.textile", "Rakefile", "bin/jekyll", "jekyll.gemspec", "lib/jekyll.rb", "lib/jekyll/albino.rb", "lib/jekyll/converters/csv.rb", "lib/jekyll/converters/mephisto.rb", "lib/jekyll/convertible.rb", "lib/jekyll/filters.rb", "lib/jekyll/layout.rb", "lib/jekyll/page.rb", "lib/jekyll/post.rb", "lib/jekyll/site.rb", "lib/jekyll/tags/highlight.rb", "lib/jekyll/tags/include.rb", "test/helper.rb", "test/source/_includes/sig.textile", "test/source/_layouts/default.html", "test/source/_layouts/simple.html", "test/source/_posts/2008-10-18-foo-bar.textile", "test/source/_posts/2008-11-21-complex.textile", "test/source/_posts/2008-12-13-include.textile", "test/source/css/screen.css", "test/source/index.html", "test/source/posts/2008-12-03-permalinked-post.textile", "test/suite.rb", "test/test_jekyll.rb", "test/test_post.rb", "test/test_site.rb"]
+  s.files = ["History.txt", "Manifest.txt", "README.textile", "Rakefile", "bin/jekyll", "jekyll.gemspec", "lib/jekyll.rb", "lib/jekyll/albino.rb", "lib/jekyll/converters/csv.rb", "lib/jekyll/converters/mephisto.rb", "lib/jekyll/converters/mt.rb", "lib/jekyll/converters/wordpress.rb", "lib/jekyll/convertible.rb", "lib/jekyll/core_ext.rb", "lib/jekyll/filters.rb", "lib/jekyll/layout.rb", "lib/jekyll/page.rb", "lib/jekyll/post.rb", "lib/jekyll/site.rb", "lib/jekyll/tags/highlight.rb", "lib/jekyll/tags/include.rb", "test/helper.rb", "test/source/_includes/sig.markdown", "test/source/_layouts/default.html", "test/source/_layouts/simple.html", "test/source/_posts/2008-10-18-foo-bar.textile", "test/source/_posts/2008-11-21-complex.textile", "test/source/_posts/2008-12-03-permalinked-post.textile", "test/source/_posts/2008-12-13-include.markdown", "test/source/css/screen.css", "test/source/index.html", "test/suite.rb", "test/test_generated_site.rb", "test/test_jekyll.rb", "test/test_post.rb", "test/test_site.rb"]
   s.has_rdoc = true
   s.rdoc_options = ["--main", "README.txt"]
   s.require_paths = ["lib"]
   s.rubyforge_project = %q{jekyll}
   s.rubygems_version = %q{1.3.0}
   s.summary = %q{Jekyll is a simple, blog aware, static site generator.}
-  s.test_files = ["test/test_jekyll.rb", "test/test_post.rb", "test/test_site.rb"]
+  s.test_files = ["test/test_generated_site.rb", "test/test_jekyll.rb", "test/test_post.rb", "test/test_site.rb"]
 
   if s.respond_to? :specification_version then
     current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
@@ -26,7 +26,7 @@ Gem::Specification.new do |s|
       s.add_runtime_dependency(%q<RedCloth>, [">= 0"])
       s.add_runtime_dependency(%q<liquid>, [">= 0"])
       s.add_runtime_dependency(%q<classifier>, [">= 0"])
-      s.add_runtime_dependency(%q<rdiscount>, [">= 0"])
+      s.add_runtime_dependency(%q<maruku>, [">= 0"])
       s.add_runtime_dependency(%q<directory_watcher>, [">= 0"])
       s.add_runtime_dependency(%q<open4>, [">= 0"])
       s.add_development_dependency(%q<hoe>, [">= 1.8.0"])
@@ -34,7 +34,7 @@ Gem::Specification.new do |s|
       s.add_dependency(%q<RedCloth>, [">= 0"])
       s.add_dependency(%q<liquid>, [">= 0"])
       s.add_dependency(%q<classifier>, [">= 0"])
-      s.add_dependency(%q<rdiscount>, [">= 0"])
+      s.add_dependency(%q<maruku>, [">= 0"])
       s.add_dependency(%q<directory_watcher>, [">= 0"])
       s.add_dependency(%q<open4>, [">= 0"])
       s.add_dependency(%q<hoe>, [">= 1.8.0"])
@@ -43,7 +43,7 @@ Gem::Specification.new do |s|
     s.add_dependency(%q<RedCloth>, [">= 0"])
     s.add_dependency(%q<liquid>, [">= 0"])
     s.add_dependency(%q<classifier>, [">= 0"])
-    s.add_dependency(%q<rdiscount>, [">= 0"])
+    s.add_dependency(%q<maruku>, [">= 0"])
     s.add_dependency(%q<directory_watcher>, [">= 0"])
     s.add_dependency(%q<open4>, [">= 0"])
     s.add_dependency(%q<hoe>, [">= 1.8.0"])

data/lib/jekyll.rb

@@ -6,6 +6,7 @@ require 'rubygems'
 # core
 require 'fileutils'
 require 'time'
+require 'yaml'
 
 # stdlib
 
@@ -13,13 +14,24 @@ require 'time'
 require 'liquid'
 require 'redcloth'
 begin
-  require 'rdiscount'
+  require 'maruku'
+  require 'maruku/ext/math'
+  # Switch off MathML output
+  MaRuKu::Globals[:html_math_output_mathml] = false
+  MaRuKu::Globals[:html_math_engine] = 'none'
+
+  # Turn on math to PNG support with blahtex
+  # Resulting PNGs stored in `images/latex`
+  MaRuKu::Globals[:html_math_output_png] = true
+  MaRuKu::Globals[:html_png_engine] =  'blahtex'
+  MaRuKu::Globals[:html_png_dir] = 'images/latex'
+  MaRuKu::Globals[:html_png_url] = '/images/latex/'
 rescue LoadError
-  puts "The rdiscount gem is required for markdown support!"
+  puts "The maruku gem is required for markdown support!"
 end
-require 'directory_watcher'
 
 # internal requires
+require 'jekyll/core_ext'
 require 'jekyll/site'
 require 'jekyll/convertible'
 require 'jekyll/layout'
@@ -31,14 +43,15 @@ require 'jekyll/tags/include'
 require 'jekyll/albino'
 
 module Jekyll
-  VERSION = '0.2.0'
+  VERSION = '0.3.0'
   
   class << self
-    attr_accessor :source, :dest, :lsi, :pygments
+    attr_accessor :source, :dest, :lsi, :pygments, :markdown_proc
   end
   
   Jekyll.lsi = false
   Jekyll.pygments = false
+  Jekyll.markdown_proc = Proc.new { |x| Maruku.new(x).to_html }
   
   def self.process(source, dest)
     require 'classifier' if Jekyll.lsi

data/lib/jekyll/converters/mt.rb

@@ -0,0 +1,59 @@
+# Created by Nick Gerakines, open source and publically available under the
+# MIT license. Use this module at your own risk.
+# I'm an Erlang/Perl/C++ guy so please forgive my dirty ruby.
+
+require 'rubygems'
+require 'sequel'
+require 'fileutils'
+
+# NOTE: This converter requires Sequel and the MySQL gems.
+# The MySQL gem can be difficult to install on OS X. Once you have MySQL
+# installed, running the following commands should work:
+# $ sudo gem install sequel
+# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
+ 
+module Jekyll
+  module MT
+    # This query will pull blog posts from all entries across all blogs. If
+    # you've got unpublished, deleted or otherwise hidden posts please sift
+    # through the created posts to make sure nothing is accidently published.
+    QUERY = "SELECT entry_id, entry_basename, entry_text, entry_text_more, entry_created_on, entry_title FROM mt_entry"
+ 
+    def self.process(dbname, user, pass, host = 'localhost')
+      db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host)
+      
+      FileUtils.mkdir_p "_posts"
+            
+      db[QUERY].each do |post|
+        title = post[:entry_title]
+        slug = post[:entry_basename]
+        date = post[:entry_created_on]
+        content = post[:entry_text]
+        more_content = post[:entry_text_more]
+        
+        # Be sure to include the body and extended body.
+        if more_content != nil
+          conent = content + " \n" + more_content
+        end
+        
+        # Ideally, this script would determine the post format (markdown, html
+        # , etc) and create files with proper extensions. At this point it
+        # just assumes that markdown will be acceptable.
+        name = [date.year, date.month, date.day, slug].join('-') + ".markdown"
+
+        data = {
+           'layout' => 'post',
+           'title' => title.to_s,
+           'mt_id' => post[:entry_id],
+         }.delete_if { |k,v| v.nil? || v == ''}.to_yaml
+ 
+        File.open("_posts/#{name}", "w") do |f|
+          f.puts data
+          f.puts "---"
+          f.puts content
+        end
+      end
+ 
+    end
+  end
+end

data/lib/jekyll/converters/wordpress.rb

@@ -0,0 +1,54 @@
+require 'rubygems'
+require 'sequel'
+require 'fileutils'
+
+# NOTE: This converter requires Sequel and the MySQL gems.
+# The MySQL gem can be difficult to install on OS X. Once you have MySQL
+# installed, running the following commands should work:
+# $ sudo gem install sequel
+# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
+
+module Jekyll
+  module WordPress
+    
+    # Reads a MySQL database via Sequel and creates a post file for each 
+    # post in wp_posts that has post_status = 'publish'.
+    # This restriction is made because 'draft' posts are not guaranteed to
+    # have valid dates.
+    QUERY = "select * from wp_posts where post_status = 'publish' and post_type = 'post'"
+
+    def self.process(dbname, user, pass, host = 'localhost')
+      db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host)
+      
+      FileUtils.mkdir_p "_posts"
+            
+      db[QUERY].each do |post|
+        # Get required fields and construct Jekyll compatible name
+        title = post[:post_title]
+        slug = post[:post_name]
+        date = post[:post_date]
+        content = post[:post_content]
+        
+        name = [date.year, date.month, date.day, slug].join('-') + ".markdown"
+
+        # Get the relevant fields as a hash, delete empty fields and convert
+        # to YAML for the header        
+        data = {
+           'layout' => 'post',
+           'title' => title.to_s,
+           'excerpt' => post[:post_excerpt].to_s,
+           'wordpress_id' => post[:ID],
+           'wordpress_url' => post[:guid]
+         }.delete_if { |k,v| v.nil? || v == ''}.to_yaml
+
+        # Write out the data and content to file
+        File.open("_posts/#{name}", "w") do |f|
+          f.puts data
+          f.puts "---"
+          f.puts content
+        end
+      end
+
+    end
+  end
+end

data/lib/jekyll/convertible.rb

@@ -13,13 +13,13 @@ module Jekyll
     def read_yaml(base, name)
       self.content = File.read(File.join(base, name))
       
-      if self.content =~ /^(---.*\n.*?)\n---.*\n/m
+      if self.content =~ /^(---\s*\n.*?)\n---\s*\n/m
         self.content = self.content[($1.size + 5)..-1]
         
         self.data = YAML.load($1)
       end
     end
-  
+    
     # Transform the contents based on the file extension.
     #
     # Returns nothing
@@ -30,7 +30,7 @@ module Jekyll
         self.content = RedCloth.new(self.content).to_html
       when ".markdown":
         self.ext = ".html"
-        self.content = RDiscount.new(self.content).to_html
+        self.content = Jekyll.markdown_proc.call(self.content)
       end
     end
     
@@ -39,10 +39,8 @@ module Jekyll
     #   +site_payload+ is the site payload hash
     #
     # Returns nothing
-    def do_layout(payload, layouts, site_payload)
-      # construct payload
-      payload = payload.merge(site_payload)
-      # render content
+    def do_layout(payload, layouts)
+      # render and transform content (this becomes the final content of the object)
       self.content = Liquid::Template.parse(self.content).render(payload, [Jekyll::Filters])
       self.transform
       
@@ -52,7 +50,7 @@ module Jekyll
       # recursively render layouts
       layout = layouts[self.data["layout"]]
       while layout
-        payload = payload.merge({"content" => self.output, "page" => self.data})
+        payload = payload.deep_merge({"content" => self.output, "page" => layout.data})
         self.output = Liquid::Template.parse(layout.content).render(payload, [Jekyll::Filters])
         
         layout = layouts[layout.data["layout"]]

data/lib/jekyll/core_ext.rb

@@ -0,0 +1,22 @@
+class Hash
+  # Merges self with another hash, recursively.
+  # 
+  # This code was lovingly stolen from some random gem:
+  # http://gemjack.com/gems/tartan-0.1.1/classes/Hash.html
+  # 
+  # Thanks to whoever made it.
+  def deep_merge(hash)
+    target = dup
+    
+    hash.keys.each do |key|
+      if hash[key].is_a? Hash and self[key].is_a? Hash
+        target[key] = target[key].deep_merge(hash[key])
+        next
+      end
+      
+      target[key] = hash[key]
+    end
+    
+    target
+  end
+end

data/lib/jekyll/filters.rb

@@ -4,6 +4,10 @@ module Jekyll
     def date_to_string(date)
       date.strftime("%d %b %Y")
     end
+
+    def date_to_long_string(date)
+      date.strftime("%d %B %Y")
+    end
     
     def date_to_xmlschema(date)
       date.xmlschema
@@ -15,7 +19,6 @@ module Jekyll
     
     def number_of_words(input)
       input.split.length
-    end
-  end
-  
+    end    
+  end  
 end

data/lib/jekyll/layout.rb

@@ -28,21 +28,6 @@ module Jekyll
     def process(name)
       self.ext = File.extname(name)
     end
-    
-    # Add any necessary layouts to this post
-    #   +layouts+ is a Hash of {"name" => "layout"}
-    #   +site_payload+ is the site payload hash
-    #
-    # Returns nothing
-    def add_layout(layouts, site_payload)
-      payload = {"page" => self.data}.merge(site_payload)
-      self.content = Liquid::Template.parse(self.content).render(payload, [Jekyll::Filters])
-      
-      layout = layouts[self.data["layout"]] || self.content
-      payload = {"content" => self.content, "page" => self.data}
-      
-      self.content = Liquid::Template.parse(layout).render(payload, [Jekyll::Filters])
-    end
   end
 
 end

data/lib/jekyll/page.rb

@@ -37,9 +37,9 @@ module Jekyll
     #   +site_payload+ is the site payload hash
     #
     # Returns nothing
-    def add_layout(layouts, site_payload)
-      payload = {"page" => self.data}
-      do_layout(payload, layouts, site_payload)
+    def render(layouts, site_payload)
+      payload = {"page" => self.data}.deep_merge(site_payload)
+      do_layout(payload, layouts)
     end
     
     # Write the generated page file to the destination directory.

data/lib/jekyll/post.rb

@@ -3,12 +3,12 @@ module Jekyll
   class Post
     include Comparable
     include Convertible
-    
+        
     class << self
       attr_accessor :lsi
     end
     
-    MATCHER = /^(\d+-\d+-\d+)-(.*)(\.[^.]+)$/
+    MATCHER = /^(.+\/)*(\d+-\d+-\d+)-(.*)(\.[^.]+)$/
     
     # Post name validator. Post filenames must be like:
     #   2008-11-05-my-awesome-post.textile
@@ -18,22 +18,26 @@ module Jekyll
       name =~ MATCHER
     end
     
-    attr_accessor :date, :slug, :ext
+    attr_accessor :date, :slug, :ext, :categories, :topics
     attr_accessor :data, :content, :output
     
     # Initialize this Post instance.
     #   +base+ is the String path to the dir containing the post file
     #   +name+ is the String filename of the post file
+    #   +categories+ is an Array of Strings for the categories for this post
     #
     # Returns <Post>
-    def initialize(base, name)
-      @base = base
+    def initialize(source, dir, name)
+      @base = File.join(source, dir, '_posts')
       @name = name
       
+      self.categories = dir.split('/').reject { |x| x.empty? }
+      
+      parts = name.split('/')
+      self.topics = parts.size > 1 ? parts[0..-2] : []
+      
       self.process(name)
-      self.read_yaml(base, name)
-      #Removed to avoid munging of liquid tags, replaced in convertible.rb#48
-      #self.transform
+      self.read_yaml(@base, name)
     end
     
     # Spaceship is based on Post#date
@@ -48,7 +52,7 @@ module Jekyll
     #
     # Returns nothing
     def process(name)
-      m, date, slug, ext = *name.match(MATCHER)
+      m, cats, date, slug, ext = *name.match(MATCHER)
       self.date = Time.parse(date)
       self.slug = slug
       self.ext = ext
@@ -61,9 +65,12 @@ module Jekyll
     #
     # Returns <String>
     def dir
-      permalink ?
-        permalink.to_s.split("/")[0..-2].join("/") :
-        date.strftime("/%Y/%m/%d/")
+      if permalink
+        permalink.to_s.split("/")[0..-2].join("/")
+      else
+        prefix = self.categories.empty? ? '' : '/' + self.categories.join('/')
+        prefix + date.strftime("/%Y/%m/%d/")
+      end
     end
     
     # The full path and filename of the post.
@@ -90,7 +97,7 @@ module Jekyll
     def id
       self.dir + self.slug
     end
-    
+        
     # Calculate related posts.
     #
     # Returns [<Post>]
@@ -118,11 +125,16 @@ module Jekyll
     #   +site_payload+ is the site payload hash
     #
     # Returns nothing
-    def add_layout(layouts, site_payload)
-      # construct post payload
-      related = related_posts(site_payload["site"]["posts"])
-      payload = {"page" => self.to_liquid.merge(self.data)}
-      do_layout(payload, layouts, site_payload.merge({"site" => {"related_posts" => related}}))
+    def render(layouts, site_payload)
+      # construct payload
+      payload =
+      {
+        "site" => { "related_posts" => related_posts(site_payload["site"]["posts"]) },
+        "page" => self.to_liquid
+      }
+      payload = payload.deep_merge(site_payload)
+      
+      do_layout(payload, layouts)
     end
     
     # Write the generated post file to the destination directory.
@@ -146,7 +158,12 @@ module Jekyll
         "url" => self.url,
         "date" => self.date,
         "id" => self.id,
-        "content" => self.content }
+        "topics" => self.topics,
+        "content" => self.content }.deep_merge(self.data)
+    end
+    
+    def inspect
+      "<Post: #{self.id}>"
     end
   end
 

data/lib/jekyll/site.rb

@@ -24,19 +24,19 @@ module Jekyll
     # Returns nothing
     def process
       self.read_layouts
-      self.read_posts
-      self.write_posts
       self.transform_pages
+      self.write_posts
     end
     
-    # Read all the files in <source>/_layouts into memory for
-    # later use.
+    # Read all the files in <source>/_layouts except backup files
+    # (end with "~") into memory for later use.
     #
     # Returns nothing
     def read_layouts
       base = File.join(self.source, "_layouts")
       entries = Dir.entries(base)
-      entries = entries.reject { |e| File.directory?(e) }
+      entries = entries.reject { |e| e[-1..-1] == '~' }
+      entries = entries.reject { |e| File.directory?(File.join(base, e)) }
       
       entries.each do |f|
         name = f.split(".")[0..-2].join(".")
@@ -46,17 +46,29 @@ module Jekyll
       # ignore missing layout dir
     end
     
-    # Read all the files in <source>/posts and create a new Post
-    # object with each one.
+    # Read all the files in <base>/_posts except backup files (end with "~")
+    # and create a new Post object with each one.
     #
     # Returns nothing
-    def read_posts
-      base = File.join(self.source, "_posts")
-      entries = Dir.entries(base)
-      entries = entries.reject { |e| File.directory?(e) }
+    def read_posts(dir)
+      base = File.join(self.source, dir, '_posts')
+      
+      entries = []
+      Dir.chdir(base) { entries = Dir['**/*'] }
+      entries = entries.reject { |e| e[-1..-1] == '~' }
+      entries = entries.reject { |e| File.directory?(File.join(base, e)) }
 
+      # first pass processes, but does not yet render post content
       entries.each do |f|
-        self.posts << Post.new(base, f) if Post.valid?(f)
+        if Post.valid?(f)
+          post = Post.new(self.source, dir, f)
+          self.posts << post
+        end
+      end
+      
+      # second pass renders each post now that full site payload is available
+      self.posts.each do |post|
+        post.render(self.layouts, site_payload)
       end
       
       self.posts.sort!
@@ -69,14 +81,14 @@ module Jekyll
     # Returns nothing
     def write_posts
       self.posts.each do |post|
-        post.add_layout(self.layouts, site_payload)
         post.write(self.dest)
       end
     end
     
     # Copy all regular files from <source> to <dest>/ ignoring
-    # any files/directories that are hidden (start with ".") or contain
-    # site content (start with "_")
+    # any files/directories that are hidden or backup files (start
+    # with "." or end with "~") or contain site content (start with "_")
+    # unless they are "_posts" directories
     #   The +dir+ String is a relative path used to call this method
     #            recursively as it descends through directories
     #
@@ -84,7 +96,17 @@ module Jekyll
     def transform_pages(dir = '')
       base = File.join(self.source, dir)
       entries = Dir.entries(base)
-      entries = entries.reject { |e| ['.', '_'].include?(e[0..0]) }
+      entries = entries.reject { |e| e[-1..-1] == '~' }
+      entries = entries.reject do |e|
+        (e != '_posts') and ['.', '_'].include?(e[0..0])
+      end
+
+      # we need to make sure to process _posts *first* otherwise they 
+      # might not be available yet to other templates as {{ site.posts }}
+      if entries.include?('_posts')
+        entries.delete('_posts')
+        read_posts(dir)
+      end
 
       entries.each do |f|
         if File.directory?(File.join(base, f))
@@ -93,13 +115,13 @@ module Jekyll
         else
           first3 = File.open(File.join(self.source, dir, f)) { |fd| fd.read(3) }
           
-          # if the file appears to have a YAML header then process it as a page
           if first3 == "---"
+            # file appears to have a YAML header so process it as a page
             page = Page.new(self.source, dir, f)
-            page.add_layout(self.layouts, site_payload)
+            page.render(self.layouts, site_payload)
             page.write(self.dest)
-          # otherwise copy the file without transforming it
           else
+            # otherwise copy the file without transforming it
             FileUtils.mkdir_p(File.join(self.dest, dir))
             FileUtils.cp(File.join(self.source, dir, f), File.join(self.dest, dir, f))
           end
@@ -111,7 +133,17 @@ module Jekyll
     #
     # Returns {"site" => {"time" => <Time>, "posts" => [<Post>]}}
     def site_payload
-      {"site" => {"time" => Time.now, "posts" => self.posts.sort.reverse}}
+      # Build the category hash map of category ( names => arrays of posts )
+      # then sort each array in reverse order
+      categories = Hash.new { |hash, key| hash[key] = Array.new }
+      self.posts.each { |p| p.categories.each { |c| categories[c] << p } }
+      categories.values.map { |cats| cats.sort! { |a, b| b <=> a} }
+      
+      {"site" => {
+        "time" => Time.now, 
+        "posts" => self.posts.sort { |a,b| b <=> a },
+        "categories" => categories
+      }}
     end
   end
 

data/lib/jekyll/tags/include.rb

@@ -7,7 +7,18 @@ module Jekyll
     end
     
     def render(context)
-      File.read(File.join(Jekyll.source, '_includes', @file))
+      if @file !~ /^[a-zA-Z0-9_\/\.-]+$/ || @file =~ /\.\// || @file =~ /\/\./
+        return "Include file '#{@file}' contains invalid characters or sequences"
+      end
+      
+      Dir.chdir(File.join(Jekyll.source, '_includes')) do
+        choices = Dir['**/*'].reject { |x| File.symlink?(x) }
+        if choices.include?(@file)
+          File.read(@file)
+        else
+          "Included file '#{@file}' not found in _includes directory"
+        end
+      end
     end
   end
   

data/test/source/_posts/2008-12-13-include.markdown

@@ -0,0 +1,8 @@
+---
+layout: default
+title: Include
+---
+
+{% include sig.markdown %}
+
+This _is_ cool

data/test/source/index.html

@@ -5,8 +5,18 @@ title: Tom Preston-Werner
 
 h1. Welcome to my site
 
+h2. Please read our {{ site.posts | size }} Posts
+
 <ul>
   {% for post in site.posts %}
     <li>{{ post.date }} <a href="{{ post.url }}">{{ post.title }}</a></li>
   {% endfor %}
-</ul>
+</ul>
+
+{% assign first_post = site.posts.first %}
+<div id="first_post">
+  <h1>{{ first_post.title }}</h1>
+  <div>
+    {{ first_post.content }}
+  </div>
+</div>

data/test/test_generated_site.rb

@@ -0,0 +1,21 @@
+require File.dirname(__FILE__) + '/helper'
+
+class TestGeneratedSite < Test::Unit::TestCase
+  def setup
+    clear_dest
+    source = File.join(File.dirname(__FILE__), *%w[source])
+    @s = Site.new(source, dest_dir)
+    @s.process
+    @index = File.read(File.join(dest_dir, 'index.html'))
+  end
+  
+  def test_site_posts_in_index
+    # confirm that {{ site.posts }} is working
+    assert @index.include?("#{@s.posts.size} Posts")
+  end
+
+  def test_post_content_in_index
+    # confirm that the {{ post.content }} is rendered OK
+    assert @index.include?('<p>This <em>is</em> cool</p>')
+  end
+end

data/test/test_post.rb

@@ -7,6 +7,9 @@ class TestPost < Test::Unit::TestCase
   
   def test_valid
     assert Post.valid?("2008-10-19-foo-bar.textile")
+    assert Post.valid?("foo/bar/2008-10-19-foo-bar.textile")
+    
+    assert !Post.valid?("lol2008-10-19-foo-bar.textile")
     assert !Post.valid?("blah")
   end
   
@@ -21,6 +24,7 @@ class TestPost < Test::Unit::TestCase
   
   def test_url
     p = Post.allocate
+    p.categories = []
     p.process("2008-10-19-foo-bar.textile")
     
     assert_equal "/2008/10/19/foo-bar.html", p.url
@@ -29,7 +33,7 @@ class TestPost < Test::Unit::TestCase
   def test_permalink
     p = Post.allocate
     p.process("2008-12-03-permalinked-post.textile")
-    p.read_yaml(File.join(File.dirname(__FILE__), *%w[source posts]), "2008-12-03-permalinked-post.textile")
+    p.read_yaml(File.join(File.dirname(__FILE__), *%w[source _posts]), "2008-12-03-permalinked-post.textile")
 
     assert_equal "my_category/permalinked-post", p.permalink
   end
@@ -37,7 +41,7 @@ class TestPost < Test::Unit::TestCase
   def test_dir_respects_permalink
     p = Post.allocate
     p.process("2008-12-03-permalinked-post.textile")
-    p.read_yaml(File.join(File.dirname(__FILE__), *%w[source posts]), "2008-12-03-permalinked-post.textile")
+    p.read_yaml(File.join(File.dirname(__FILE__), *%w[source _posts]), "2008-12-03-permalinked-post.textile")
 
     assert_equal "my_category", p.dir
   end
@@ -59,10 +63,10 @@ class TestPost < Test::Unit::TestCase
     assert_equal "<h1>{{ page.title }}</h1>\n<p>Best <strong>post</strong> ever</p>", p.content
   end
   
-  def test_add_layout
-    p = Post.new(File.join(File.dirname(__FILE__), *%w[source _posts]), "2008-10-18-foo-bar.textile")
+  def test_render
+    p = Post.new(File.join(File.dirname(__FILE__), *%w[source]), '',  "2008-10-18-foo-bar.textile")
     layouts = {"default" => Layout.new(File.join(File.dirname(__FILE__), *%w[source _layouts]), "simple.html")}
-    p.add_layout(layouts, {"site" => {"posts" => []}})
+    p.render(layouts, {"site" => {"posts" => []}})
     
     assert_equal "<<< <h1>Foo Bar</h1>\n<p>Best <strong>post</strong> ever</p> >>>", p.output
   end
@@ -70,26 +74,26 @@ class TestPost < Test::Unit::TestCase
   def test_write
     clear_dest
     
-    p = Post.new(File.join(File.dirname(__FILE__), *%w[source _posts]), "2008-10-18-foo-bar.textile")
+    p = Post.new(File.join(File.dirname(__FILE__), *%w[source]), '', "2008-10-18-foo-bar.textile")
     layouts = {"default" => Layout.new(File.join(File.dirname(__FILE__), *%w[source _layouts]), "simple.html")}
-    p.add_layout(layouts, {"site" => {"posts" => []}})
+    p.render(layouts, {"site" => {"posts" => []}})
     p.write(dest_dir)
   end
   
   def test_data
-    p = Post.new(File.join(File.dirname(__FILE__), *%w[source _posts]), "2008-11-21-complex.textile")
+    p = Post.new(File.join(File.dirname(__FILE__), *%w[source]), '', "2008-11-21-complex.textile")
     layouts = {"default" => Layout.new(File.join(File.dirname(__FILE__), *%w[source _layouts]), "simple.html")}
-    p.add_layout(layouts, {"site" => {"posts" => []}})
+    p.render(layouts, {"site" => {"posts" => []}})
     
     assert_equal "<<< <p>url: /2008/11/21/complex.html<br />\ndate: #{Time.parse("2008-11-21")}<br />\nid: /2008/11/21/complex</p> >>>", p.output
   end
   
   def test_include
     Jekyll.source = File.join(File.dirname(__FILE__), *%w[source])
-    p = Post.new(File.join(File.dirname(__FILE__), *%w[source _posts]), "2008-12-13-include.textile")
+    p = Post.new(File.join(File.dirname(__FILE__), *%w[source]), '', "2008-12-13-include.markdown")
     layouts = {"default" => Layout.new(File.join(File.dirname(__FILE__), *%w[source _layouts]), "simple.html")}
-    p.add_layout(layouts, {"site" => {"posts" => []}})
+    p.render(layouts, {"site" => {"posts" => []}})
     
-    assert_equal "<<< <p>&#8212;<br />\nTom Preston-Werner<br />\ngithub.com/mojombo</p> >>>", p.output
+    assert_equal "<<< <hr />\n<p>Tom Preston-Werner github.com/mojombo</p>\n\n<p>This <em>is</em> cool</p> >>>", p.output
   end
 end

data/test/test_site.rb

@@ -15,11 +15,11 @@ class TestSite < Test::Unit::TestCase
     
     assert_equal ["default", "simple"].sort, @s.layouts.keys.sort
   end
-  
+ 
   def test_read_posts
-    @s.read_posts
+    @s.read_posts('')
     
-    assert_equal 3, @s.posts.size
+    assert_equal 4, @s.posts.size
   end
   
   def test_write_posts
@@ -27,4 +27,4 @@ class TestSite < Test::Unit::TestCase
     
     @s.process
   end
-end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: mojombo-jekyll
 version: !ruby/object:Gem::Version 
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors: 
 - Tom Preston-Werner
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2008-12-14 00:00:00 -08:00
+date: 2008-12-24 00:00:00 -08:00
 default_executable: jekyll
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -40,7 +40,7 @@ dependencies:
         version: "0"
     version: 
 - !ruby/object:Gem::Dependency 
-  name: rdiscount
+  name: maruku
   version_requirement: 
   version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
@@ -96,7 +96,10 @@ files:
 - lib/jekyll/albino.rb
 - lib/jekyll/converters/csv.rb
 - lib/jekyll/converters/mephisto.rb
+- lib/jekyll/converters/mt.rb
+- lib/jekyll/converters/wordpress.rb
 - lib/jekyll/convertible.rb
+- lib/jekyll/core_ext.rb
 - lib/jekyll/filters.rb
 - lib/jekyll/layout.rb
 - lib/jekyll/page.rb
@@ -105,16 +108,17 @@ files:
 - lib/jekyll/tags/highlight.rb
 - lib/jekyll/tags/include.rb
 - test/helper.rb
-- test/source/_includes/sig.textile
+- test/source/_includes/sig.markdown
 - test/source/_layouts/default.html
 - test/source/_layouts/simple.html
 - test/source/_posts/2008-10-18-foo-bar.textile
 - test/source/_posts/2008-11-21-complex.textile
-- test/source/_posts/2008-12-13-include.textile
+- test/source/_posts/2008-12-03-permalinked-post.textile
+- test/source/_posts/2008-12-13-include.markdown
 - test/source/css/screen.css
 - test/source/index.html
-- test/source/posts/2008-12-03-permalinked-post.textile
 - test/suite.rb
+- test/test_generated_site.rb
 - test/test_jekyll.rb
 - test/test_post.rb
 - test/test_site.rb
@@ -146,6 +150,7 @@ signing_key:
 specification_version: 2
 summary: Jekyll is a simple, blog aware, static site generator.
 test_files: 
+- test/test_generated_site.rb
 - test/test_jekyll.rb
 - test/test_post.rb
 - test/test_site.rb

data/test/source/_posts/2008-12-13-include.textile

@@ -1,6 +0,0 @@
----
-layout: default
-title: Include
----
-
-{% include sig.textile %}