yard 0.4.0 → 0.5.0

data/README.md

@@ -1,7 +1,7 @@
-YARD Release 0.4.0 "The Whole Nine" (Nov 15th 2009)
+YARD Release 0.5.0 "The Longest" (Dec 14th 2009)
 ===================================================
 
-**Homepage**:  [http://yard.rubyforge.org](http://yard.rubyforge.org)   
+**Homepage**:  [http://yardoc.org](http://yardoc.org)   
 **IRC**:       **Join us on IRC in #yard on irc.freenode.net!**   
 **Git**:       [http://github.com/lsegal/yard](http://github.com/lsegal/yard)   
 **Author**:    Loren Segal   
@@ -197,6 +197,14 @@ work with the stdlib or core Ruby libraries, only the active project. Example:
 Note that class methods must not be referred to with the "::" namespace 
 separator. Only modules, classes and constants should use "::".
 
+You can also do lookups on any installed gems. Just make sure to build the
+.yardoc databases for installed gems with:
+
+    $ sudo yardoc --build-gems
+    
+If you don't have sudo access, it will write these files to your `~/.yard`
+directory. `yri` will also cache lookups there.
+
 **4. `yard-graph` Graphviz Generator**
 
 You can use `yard-graph` to generate dot graphs of your code. This, of course,

data/benchmarks/yri_cache.rb

@@ -0,0 +1,19 @@
+require File.dirname(__FILE__) + "/../lib/yard"
+require "benchmark"
+include YARD::CLI
+
+class YARD::CLI::YRI
+  def print_object(object) end
+end
+
+def remove_cache
+  File.unlink(YRI::CACHE_FILE)
+end
+
+TIMES = 10
+NAME = 'YARD'
+remove_cache; YRI.run(NAME)
+Benchmark.bmbm do |x|
+  x.report("cache   ") { TIMES.times { YRI.run(NAME) } }
+  x.report("no-cache") { TIMES.times { remove_cache; YRI.run(NAME) } }
+end

data/bin/yri

@@ -1,29 +1,4 @@
 #!/usr/bin/env ruby
 require File.dirname(__FILE__) + '/../lib/yard'
 
-YARD::Registry.load
-
-if ARGV[0] == '-T' || ARGV[0] == '--no-pager'
-  output = YARD::Serializers::StdoutSerializer.new
-  ARGV.shift
-else
-  output = YARD::Serializers::ProcessSerializer.new('less')
-end
-
-if ARGV[0].nil? || ARGV[0].empty?
-  puts "Missing object argument"
-  exit(1)
-end
-
-obj = YARD::Registry.at(ARGV[0])
-if obj.nil?
-  puts "No documentation for #{ARGV[0]}"
-  exit(1)
-end
-
-if obj.type == :method && obj.is_alias?
-  tmp = P(obj.namespace, (obj.scope == :instance ? "#" : "") + 
-    obj.namespace.aliases[obj].to_s) 
-  obj = tmp unless YARD::CodeObjects::Proxy === tmp
-end
-obj.format(:serializer => output)
+YARD::CLI::YRI.run(*ARGV)

data/docs/WhatsNew.md

@@ -1,3 +1,102 @@
+What's New in 0.5.x?
+====================
+
+1. **Support for documenting native Ruby C code**
+2. **Incremental parsing and output generation with `yardoc -c`**
+2. **Improved `yri` support to perform lookups on installed Gems**
+3. **Added `yardoc --default-return` and `yardoc --hide-void-return`**
+4. **Multiple syntax highlighting language support**
+5. **New .yardoc format**
+
+Support for documenting native Ruby C code
+------------------------------------------
+
+It is now possible to document native Ruby extensions with YARD with a new
+C parser mostly borrowed from RDoc. This enables the ability to document
+Ruby's core and stdlibs which will be hosted on http://yardoc.org/docs. In
+addition, the .yardoc dump for the Ruby-core classes will become available
+as an installable gem for yri support (see #3).
+
+Incremental parsing and output generation with `yardoc -c`
+----------------------------------------------------------
+
+YARD now compares file checksums before parsing when using `yardoc -c` 
+(aka `yardoc --use-cache`) to do incremental parsing of only the files that
+have changed. HTML (or other output format) generation will also only be
+done on the objects that were parsed from changed files. This makes doing
+a documentation development cycle much faster for quick HTML previews. Just
+remember that when using incremental output generation, the index will not
+be rebuilt and inter-file links might not hook up right, so it is best to
+perform a full rebuild at the end of such previews.
+
+Improved `yri` support to perform lookups on installed Gems
+-----------------------------------------------------------
+
+The `yri` executable can now perform lookups on gems that have been parsed
+by yard. Therefore, to use this command you must first parse all gems with
+YARD. To parse all gems, use the following command:
+
+    $ sudo yardoc --build-gems
+    
+The above command builds a .yardoc file for all installed gems in the
+respective gem directory. If you do not have write access to the gem path,
+YARD will write the yardoc file to `~/.yard/gem_index/NAME-VERSION.yardoc`.
+
+Note: you can also use `--re-build-gems` to force re-parsing of all gems.
+
+You can now do lookups with yri:
+
+    $ yri JSON
+    
+All lookups are cached to `~/.yard/yri_cache` for quicker lookups the second
+time onward.
+
+Added `yardoc --default-return` and `yardoc --hide-void-return`
+---------------------------------------------------------------
+
+YARD defaults to displaying (Object) as the default return type of any
+method that has not declared a @return tag. To customize the default
+return type, you can specify:
+
+    $ yardoc --default-return 'MyDefaultType'
+    
+You can also use the empty string to list no return type.
+
+In addition, you can use --hide-void-return to ignore any method that
+defines itself as a void type by: `@return [void]`
+
+Multiple syntax highlighting language support
+---------------------------------------------
+
+YARD now supports the ability to specify a language type for code blocks in 
+docstrings. Although no actual highlighting support is added for any language
+but Ruby, you can add your own support by writing your own helper method:
+
+    # Where LANGNAME is the language:
+    def html_syntax_highlight_LANGNAME(source)
+      # return highlighted HTML
+    end
+    
+To use this language in code blocks, prefix the block with `!!!LANGNAME`:
+
+    !!!plain
+    !!!python
+    def python_code(self):
+      return self
+
+By the same token. you can now use `!!!plain` to ignore highlighting for
+a specific code block.
+
+New .yardoc format
+------------------
+
+To make the above yri support possible, the .yardoc format was redesigned
+to be a directory instead of a file. YARD can still load old .yardoc files,
+but they will be automatically upgraded if re-saved. The new .yardoc format
+does have a larger memory footprint, but this will hopefully be optimized
+downward.
+
+
 What's New in 0.4.x?
 ====================
 

data/lib/rubygems_plugin.rb

@@ -14,6 +14,7 @@ class Gem::Specification
     @has_rdoc == 'yard'
   end
   
+  undef has_rdoc?
   def has_rdoc?
     @has_rdoc && @has_rdoc != 'yard'
   end
@@ -52,6 +53,7 @@ class Gem::DocManager
     Dir.chdir(old_pwd)
   end
 
+  undef setup_rdoc
   def setup_rdoc
     if File.exist?(@doc_dir) && !File.writable?(@doc_dir) then
       raise Gem::FilePermissionError.new(@doc_dir)

data/lib/yard.rb

@@ -1,5 +1,5 @@
 module YARD
-  VERSION = "0.4.0"
+  VERSION = "0.5.0"
   ROOT = File.dirname(__FILE__)
   TEMPLATE_ROOT = File.join(File.dirname(__FILE__), '..', 'templates')
   CONFIG_DIR = File.expand_path('~/.yard')
@@ -31,7 +31,7 @@ module YARD
       ignored_plugins = []
     end
     
-    Gem.source_index.entries.each do |pkgname, gem|
+    Gem.source_index.find_name('').each do |gem|
       begin
         if gem.name =~ /^yard[-_]/ && !ignored_plugins.include?(gem.name)
           log.debug "Loading plugin '#{gem.name}'..."

data/lib/yard/autoload.rb

@@ -3,8 +3,10 @@ def __p(*path) File.join(YARD::ROOT, 'yard', *path) end
 
 module YARD
   module CLI # Namespace for command-line interface components
+    autoload :Base,       __p('cli/base')
     autoload :YardGraph,  __p('cli/yard_graph')
     autoload :Yardoc,     __p('cli/yardoc')
+    autoload :YRI,        __p('cli/yri')
   end
   
   # A "code object" is defined as any entity in the Ruby language.
@@ -100,6 +102,7 @@ module YARD
       autoload :RubyParser,        __p('parser/ruby/ruby_parser')
     end
 
+    autoload :CParser,             __p('parser/c_parser')
     autoload :ParserSyntaxError,   __p('parser/source_parser')
     autoload :SourceParser,        __p('parser/source_parser')
     autoload :UndocumentableError, __p('parser/source_parser')
@@ -114,6 +117,7 @@ module YARD
     autoload :FileSystemSerializer, __p('serializers/file_system_serializer')
     autoload :ProcessSerializer,    __p('serializers/process_serializer')
     autoload :StdoutSerializer,     __p('serializers/stdout_serializer')
+    autoload :YardocSerializer,     __p('serializers/yardoc_serializer')
   end
   
   module Tags # Namespace for Tag components
@@ -145,10 +149,11 @@ module YARD
     autoload :Template, __p('templates/template')
   end
 
-  autoload :Docstring, __p('docstring')
-  autoload :Logger,    __p('logging')
-  autoload :Registry,  __p('registry')
-  autoload :Verifier,  __p('verifier')
+  autoload :Docstring,      __p('docstring')
+  autoload :Logger,         __p('logging')
+  autoload :Registry,       __p('registry')
+  autoload :RegistryStore,  __p('registry_store')
+  autoload :Verifier,       __p('verifier')
 end
 
 undef __p

data/lib/yard/cli/base.rb

@@ -0,0 +1,26 @@
+require 'optparse'
+
+module YARD
+  module CLI
+    # Abstract base class for CLI utilities. Provides some helper methods for
+    # the option parser
+    # 
+    # @abstract
+    class Base
+      # Adds a set of common options to the tail of the OptionParser
+      # 
+      # @param [OptionParser] opts the option parser object
+      # @return [void]
+      def common_options(opts)
+        opts.separator ""
+        opts.separator "Other options:"
+        opts.on_tail('-q', '--quiet', 'Show no warnings.') { log.level = Logger::ERROR }
+        opts.on_tail('--verbose', 'Show more information.') { log.level = Logger::INFO }
+        opts.on_tail('--debug', 'Show debugging information.') { log.level = Logger::DEBUG }
+        opts.on_tail('--backtrace', 'Show stack traces') { log.show_backtraces = true }
+        opts.on_tail('-v', '--version', 'Show version.') { puts "yard #{YARD::VERSION}"; exit }
+        opts.on_tail('-h', '--help', 'Show this help.')  { puts opts; exit }
+      end
+    end
+  end
+end

data/lib/yard/cli/yard_graph.rb

@@ -1,12 +1,10 @@
-require 'optparse'
-
 module YARD
   module CLI
     # A command-line utility to generate Graphviz graphs from
     # a set of objects
     # 
     # @see YardGraph#run
-    class YardGraph
+    class YardGraph < Base
       # The options parsed out of the commandline.
       # Default options are:
       #   :format => :dot,
@@ -90,12 +88,7 @@ module YARD
           @serializer.instance_eval "def serialized_path(object) #{file.inspect} end"
         end
         
-        opts.separator ""
-        opts.separator "Other options:"
-        opts.on_tail('-q', '--quiet', 'Show no warnings') { log.level = Logger::ERROR }
-        opts.on_tail('--verbose', 'Show debugging information') { log.level = Logger::DEBUG }
-        opts.on_tail('-v', '--version', 'Show version.') { puts "yard #{YARD::VERSION}"; exit }
-        opts.on_tail('-h', '--help', 'Show this help.')  { puts opts; exit }
+        common_options(opts)
 
         begin
           opts.parse!(args)

data/lib/yard/cli/yardoc.rb

@@ -1,8 +1,9 @@
-require 'optparse'
+require 'digest/sha1'
+require 'fileutils'
 
 module YARD
   module CLI
-    class Yardoc
+    class Yardoc < Base
       # The configuration filename to load extra options from
       DEFAULT_YARDOPTS_FILE = ".yardopts"
       
@@ -13,9 +14,10 @@ module YARD
       # @return [Array<String>] list of Ruby source files to process
       attr_accessor :files
       
-      # @return [Boolean] whether to reparse the source files even if the 
-      #   .yardoc already exists.
-      attr_accessor :reload
+      # @return [Boolean] whether to use the existing yardoc db if the 
+      #   .yardoc already exists. Also makes use of file checksums to
+      #   parse only changed files.
+      attr_accessor :use_cache
       
       # @return [Boolean] whether to generate output
       attr_accessor :generate
@@ -30,17 +32,21 @@ module YARD
         
       # Creates a new instance of the commandline utility
       def initialize
-        @options = SymbolHash[
+        @options = SymbolHash.new(false)
+        @options.update(
           :format => :html, 
           :template => :default, 
           :markup => :rdoc,
-          :serializer => YARD::Serializers::FileSystemSerializer.new, 
+          :serializer => YARD::Serializers::FileSystemSerializer.new,
+          :default_return => "Object",
+          :hide_void_return => false,
+          :no_highlight => false, 
           :files => [],
           :visibilities => [:public],
           :verifier => nil
-        ]
+        )
         @files = []
-        @reload = true
+        @use_cache = false
         @generate = true
         @options_file = DEFAULT_YARDOPTS_FILE
       end
@@ -49,15 +55,27 @@ module YARD
       # output if set.
       # 
       # @param [Array<String>] args the list of arguments
-      # @return [nil] 
+      # @return [void] 
       def run(*args)
         args += support_rdoc_document_file!
         optparse(*yardopts)
         optparse(*args)
-        Registry.load(files, reload)
+        
+        Registry.load if use_cache
+        YARD.parse(files)
+        Registry.save(use_cache)
         
         if generate
-          Templates::Engine.generate(all_objects, options)
+          if use_cache
+            objects = all_objects
+            Registry.all # load all
+            objects.each do |object|
+              opts = options.merge(:object => object, :type => :layout)
+              Templates::Engine.render(opts)
+            end
+          else
+            Templates::Engine.generate(all_objects, options)
+          end
         end
         
         true
@@ -68,13 +86,25 @@ module YARD
       # 
       # @return [Array<CodeObjects::Base>] a list of code objects to process
       def all_objects
-        Registry.all(:root, :module, :class)
+        types = [:root, :module, :class]
+        if use_cache
+          Registry.paths(false).map do |path|
+            obj = Registry.at(path)
+            if types.include?(obj.type)
+              obj
+            else
+              nil
+            end
+          end.compact
+        else
+          Registry.all(*types)
+        end
       end
       
       # Parses the .yardopts file for default yard options
-      # @return [nil] 
+      # @return [void] 
       def yardopts
-        IO.read(options_file).split(/\s+/)
+        IO.read(options_file).shell_split
       rescue Errno::ENOENT
         []
       end
@@ -82,9 +112,9 @@ module YARD
       private
       
       # Reads a .document file in the directory to get source file globs
-      # @return [nil] 
+      # @return [void] 
       def support_rdoc_document_file!
-        IO.read(".document").split(/\s+/)
+        IO.read(".document").gsub(/^[ \t]*#.+/m, '').split(/\s+/)
       rescue Errno::ENOENT
         []
       end
@@ -107,7 +137,7 @@ module YARD
       # @example Parses a set of Ruby files with a separator and extra files
       #   parse_files %w(file1 file2 - extrafile1 extrafile2)
       # @param [Array<String>] files the list of files to parse
-      # @return [nil] 
+      # @return [void] 
       def parse_files(*files)
         self.files = []
         seen_extra_files_marker = false
@@ -126,22 +156,39 @@ module YARD
         end
       end
       
-      # @param [Array<String>] expressions a list of query expressions to
-      #   turn into a proc
-      def add_verifier(*expressions)
-        
+      # Builds .yardoc files for all non-existing gems
+      # @param [Boolean] rebuild Forces rebuild of all gems
+      def build_gems(rebuild = false)
+        require 'rubygems'
+        Gem.source_index.find_name('').each do |spec|
+          reload = true
+          dir = Registry.yardoc_file_for_gem(spec.name)
+          next unless dir
+          if File.directory?(dir) && !rebuild
+            log.debug "#{spec.name} index already exists at '#{dir}'"
+          else
+            Dir.chdir(spec.full_gem_path)
+            log.info "Building yardoc index for gem: #{spec.full_name}"
+            yfile = Registry.yardoc_file_for_gem(spec.name, ">= 0", true)
+            Registry.clear
+            Yardoc.run('-n', '-b', yfile)
+            reload = false
+          end
+        end
+        exit(0)
       end
       
       # Parses commandline options.
       # @param [Array<String>] args each tokenized argument
       def optparse(*args)
         query_expressions = []
+        do_build_gems, do_rebuild_gems = false, false
         serialopts = SymbolHash.new
         
         opts = OptionParser.new
         opts.banner = "Usage: yardoc [options] [source_files [- extra_files]]"
 
-        opts.separator "(if a list of source files is omitted, lib/**/*.rb is used.)"
+        opts.separator "(if a list of source files is omitted, lib/**/*.rb ext/**/*.c is used.)"
         opts.separator ""
         opts.separator "Example: yardoc -o documentation/ - FAQ LICENSE"
         opts.separator "  The above example outputs documentation for files in"
@@ -155,9 +202,9 @@ module YARD
         opts.separator "General Options:"
 
         opts.on('-c', '--use-cache [FILE]', 
-                'Use the cached .yardoc db to generate documentation. (defaults to no cache)') do |file|
+                "Use the cached .yardoc db to generate documentation. (defaults to no cache)") do |file|
           YARD::Registry.yardoc_file = file if file
-          self.reload = false
+          self.use_cache = true
         end
         
         opts.on('-b', '--db FILE', 'Use a specified .yardoc db to load from or save to. (defaults to .yardoc)') do |yfile|
@@ -178,6 +225,15 @@ module YARD
         opts.on('--legacy', 'Use old style parser and handlers. Unavailable under Ruby 1.8.x') do
           YARD::Parser::SourceParser.parser_type = :ruby18
         end
+        
+        opts.on('--build-gems', 'Builds .yardoc files for all gems (implies -n)') do
+          do_build_gems = true
+        end
+
+        opts.on('--re-build-gems', 'Forces building .yardoc files for all gems (implies -n)') do
+          do_build_gems = true
+          do_rebuild_gems = true
+        end
 
         opts.separator ""
         opts.separator "Output options:"
@@ -202,6 +258,14 @@ module YARD
           options[:no_highlight] = true
         end
         
+        opts.on('--default-return TYPE', "Shown if method has no return type. Defaults to 'Object'") do |type|
+          options[:default_return] = type
+        end
+        
+        opts.on('--hide-void-return', "Hides return types specified as 'void'. Default is shown.") do
+          options[:hide_void_return] = true
+        end
+        
         opts.on('--query QUERY', "Only show objects that match a specific query") do |query|
           query_expressions << query.taint
         end
@@ -247,15 +311,10 @@ module YARD
         
         opts.on('-f', '--format FORMAT', 
                 'The output format for the template. (defaults to html)') do |format|
-          options[:format] = format
+          options[:format] = format.to_sym
         end
 
-        opts.separator ""
-        opts.separator "Other options:"
-        opts.on_tail('-q', '--quiet', 'Show no warnings.') { log.level = Logger::ERROR }
-        opts.on_tail('--verbose', 'Show debugging information.') { log.level = Logger::DEBUG }
-        opts.on_tail('-v', '--version', 'Show version.') { puts "yard #{YARD::VERSION}"; exit }
-        opts.on_tail('-h', '--help', 'Show this help.')  { puts opts; exit }
+        common_options(opts)
         
         begin
           opts.parse!(args)
@@ -266,8 +325,9 @@ module YARD
         end
         
         # Last minute modifications
+        build_gems(do_rebuild_gems) if do_build_gems
         parse_files(*args) unless args.empty?
-        self.files = ['lib/**/*.rb'] if self.files.empty?
+        self.files = ['lib/**/*.rb', 'ext/**/*.c'] if self.files.empty?
         options[:verifier] = Verifier.new(*query_expressions) unless query_expressions.empty?
         options[:visibilities].uniq!
         options[:serializer] ||= Serializers::FileSystemSerializer.new(serialopts)