yard 0.2.3.5 → 0.4.0

data/docs/{HANDLERS.markdown → Handlers.md}

@@ -4,14 +4,14 @@ Handlers Architecture
 Handlers allow the processing of parsed source code. Handling is done after
 parsing to abstract away the implementation details of lexical and semantic
 analysis on source and to only deal with the logic regarding recognizing
-source statements as {file:CODE_OBJECTS.markdown code objects}.
+source statements as {file:CodeObjects.md code objects}.
 
 ![Handlers Architecture Class Diagram](images/handlers-class-diagram.png)
 
 The Pipeline
 ------------
 
-After the {file:PARSER.markdown parser component} finishes analyzing the
+After the {file:Parser.md parser component} finishes analyzing the
 source, it is handed off for post-processing to the {YARD::Handlers::Processor}
 class, which is responsible for traversing the set of statements given by
 the parser and delegating them to matching handlers. Handlers match when the
@@ -51,7 +51,7 @@ class method. A very simple handler that handles a module definition would be:
     end
     
 For details on what nodes are, and what node types are, see the 
-{file:PARSER.markdown parser architecture document}.
+{file:Parser.md parser architecture document}.
 
 In this case the node type being handled is the `:module` type. More than one
 node type or `handles` declarations may describe a single handler, for instance,
@@ -89,7 +89,7 @@ Creating a new Code Object
 --------------------------
 
 Usually (but not always) handling is performed to create new code objects to add
-to the registry (for information about code objects, see {file:CODE_OBJECTS.markdown this document}).
+to the registry (for information about code objects, see {file:CodeObjects.md this document}).
 Code objects should simply be created and added to the existing `namespace`. This
 will be enough to add them to the registry. There is also a convenience 
 {YARD::Handlers::Base#register register} method which quickly sets standard attributed
@@ -132,7 +132,7 @@ Because the legacy handler uses the legacy parser and therefore a different kind
 of AST, there are subtle differences in the handler API. Most importantly, the
 `handles` method usually deals with either lexical tokens or source code as a string
 or RegExp object. The statement object, similarly, is made up of lexical tokens instead
-of semantically parsed nodes (this is described in the {file:PARSER.markdown parser document}). 
+of semantically parsed nodes (this is described in the {file:Parser.md parser document}). 
 
 The module example above can be rewritten as a legacy handler as follows:
 

data/docs/{OVERVIEW.markdown → Overview.md}

@@ -8,7 +8,7 @@ that tools like RDoc do not do. These components are:
 
 * [Code Parsing & Processing Component](#parsing)
 * [Data Storage Component](#storage)
-* [Post Processing & Output Generation Component](#generators)
+* [Post Processing & Templating System](#templates)
 
 This separation is a major goal of the project, and means that YARD is not *just* 
 a tool to generate HTML output. The expectation is that any subset of YARD's 
@@ -30,10 +30,10 @@ This component is made up of four sub-components, each of which have separate
 tasks during the data gathering process (*note: the tag architecture is not*
 *shown in the class diagram*). These sub-components are:
 
-  * {file:PARSER.markdown Parser Architecture}
-  * {file:HANDLERS.markdown Handler Architecture}
-  * {file:CODE_OBJECTS.markdown Code Object Architecture}
-  * {file:TAGS.markdown Tag Architecture}
+  * {file:Parser.md Parser Architecture}
+  * {file:Handlers.md Handler Architecture}
+  * {file:CodeObjects.md Code Object Architecture}
+  * {file:Tags.md Tag Architecture}
 
 The parser component reads source files and converts it into a set of statements
 which the handlers then process, creating code objects which in turn create tags 
@@ -50,15 +50,11 @@ is the centralized repository for all data being parsed, stored and accessed. Th
 are future plans to improve this storage mechanism to be backend agnostic and allow
 for more robust storage.
 
-<a name="generators"></a>
-Post Processing & Output Generation Component
----------------------------------------------
+<a name="templates"></a>
+Post Processing & Templating System
+-----------------------------------
 
 This component handles processing of objects from the registry through a templating
 engine that allows output to a variety of formats. Practically speaking, this is
 where templates can be implemented to change the design, output or structure of
-the data. The next release, 0.2.4, will see a significant change in the design of 
-this component, so documentation might be sparse for now.
-
-The current design is documented in the {file:GENERATORS.markdown Generators Architecture} 
-document.
+the data. See {file:Templates.md Templates Architecture} for a complete overview.

data/docs/{PARSER.markdown → Parser.md}

@@ -3,7 +3,7 @@ Parser Architecture
 
 The parser component of YARD is the first component in the data processing pipeline
 that runs before any handling is done on the source. The parser is meant to translate
-the source into a set of statements that can be understood by the {file:HANDLERS.markdown Handlers}
+the source into a set of statements that can be understood by the {file:Handlers.md Handlers}
 that run immediately afterwards.
 
 The important classes are described in the class diagram of the entire parser 

data/docs/{TAGS.markdown → Tags.md}

@@ -19,7 +19,7 @@ class.
 Accessing Tag Information
 -------------------------
 
-Tag metadata is added when a {YARD::Docstring} is added to a {file:CODE_OBJECTS.markdown code object}
+Tag metadata is added when a {YARD::Docstring} is added to a {file:CodeObjects.md code object}
 using the {YARD::CodeObjects::Base#docstring=} attribute. In addition to adding
 conventional comments, tags are parsed and associated with the object. The easiest
 way to access tags on an object is to use the {YARD::CodeObjects::Base#tag} and `#tags`

data/docs/Templates.md

@@ -0,0 +1,286 @@
+Templates Architecture
+======================
+
+Templates are the main component in the output rendering process of YARD,
+which is invoked when conventional HTML/text output needs to be rendered
+for a set of code objects.
+
+Design Goals
+------------
+
+The general design attempts to be as abstracted from actual content and templates
+as possible. Unlike RDoc which uses one file to describe the entire template,
+YARD splits up the rendering of code objects into small components, allowing
+template modification for smaller subsets of a full template without having to
+duplicate the entire template itself. This is necessary because of YARD's support
+for plugins. YARD is designed for extensibility by external plugins, and because
+of this, no one plugin can be responsible for the entire template because no
+one plugin knows about the other plugins being used. For instance, if an RSpec
+plugin was added to support and document specifications in class templates,
+this information would need to be transparently added to the template to work
+in conjunction with any other plugin that performed similar template modifications.
+The design goals can be summarized as follows:
+
+  1. Output should be able to be rendered for any arbitrary format with little
+     modification to YARD's source code. The addition of extra templates should
+     be sufficient.
+  2. The output rendered for an object should independently rendered data
+     from arbitrary sources. These independent components are called "sections".
+  3. Sections should be able to be inserted into any object without affecting
+     any existing sections in the document. This allows for easy modification
+     of templates by plugins.
+     
+Templates
+---------
+
+Template modules are the objects used to orchestrate the design goals listed 
+above. Specifically, they organize the sections and render the template contents
+depending on the format. 
+
+Engine
+------
+
+The Engine class orchestrates the creation and rendering of Template modules and 
+handles serialization or specific rendering scenarios (like HTML). To create
+a template, use the {YARD::Templates::Engine.template template} method. The two most 
+common methods used to initiate output are the {YARD::Templates::Engine.render render} 
+and {YARD::Templates::Engine.generate generate} methods which generate and 
+optionally serialize output to a file. The latter, `#generate`, is used 
+specially to generate HTML documentation and copy over assets that may be 
+needed. For instance, an object may be rendered with:
+
+    YARD::Templates::Engine.render(:object => myobject)
+    
+A set of objects may be rendered into HTML documentation by using:
+
+    # all_objects is an array of module and class objects
+    # options includes a :serializer key to copy output to the file system
+    YARD::Templates::Engine.generate(all_objects, options)
+    
+Note that these methods should not be called directly. The {YARD::CodeObjects::Base}
+class has a {YARD::CodeObjects::Base#format #format} helper method to render an
+object. For instance, the above render example is equivalent to the simple
+call `myobject.format`. The `generate` method is a special kind of render
+and is called from the {YARD::CLI::Yardoc} command line utility.
+
+Template Options
+----------------
+
+A template keeps state when it is rendering output. This state is kept in
+an options hash which is initially passed to it during instantiation. Some
+default options set the template style (`:template`), the output format (`:format`),
+and the serializer to use (`:serializer`). This options hash is modifiable
+from all methods seen above. For example, initializing a template to output as 
+HTML instead of text can be done as follows:
+
+    myobject.format(:format => :html)
+    
+Serializer
+----------
+
+This class abstracts the logic involved in deciding how to serialize data to
+the expected endpoint. For instance, there is both a {YARD::Serializers::StdoutSerializer StdoutSerializer}
+and {YARD::Serializers::FileSystemSerializer FileSystemSerializer} class for
+outputting to console or to a file respectively. When endpoints with locations
+are used (like files or URLs), the serializer implements the {YARD::Serializers::Base#serialized_path #serialized_path}
+method. This allows the translation from a code object to its path at the endpoint,
+which enables inter-document linking.
+
+Rendered objects are automatically serialized using the object if present, 
+otherwise the rendered object is returned as a string to its parent. Nested
+Templates automatically set the serializer to nil so that they return
+as a String to their parent.
+
+Creating a Template
+-------------------
+
+Templates are represented by a directory inside the {YARD::Templates::Engine.template_paths}
+on disk. A standard template directory looks like the following tree:
+
+    (Assuming templates/ is a template path)
+    templates
+    `-- default
+        |-- class
+        |   |-- dot
+        |   |   |-- setup.rb
+        |   |   `-- superklass.erb
+        |   |-- html
+        |   |   |-- constructor_details.erb
+        |   |   |-- setup.rb
+        |   |   `-- subclasses.erb
+        |   |-- setup.rb
+        |   `-- text
+        |       |-- setup.rb
+        |       `-- subclasses.erb
+        |-- docstring
+        |   |-- html
+        |   |   |-- abstract.erb
+        |   |   |-- deprecated.erb
+        |   |   |-- index.erb
+        |   |   `-- text.erb
+        |   |-- setup.rb
+        |   `-- text
+        |       |-- abstract.erb
+        |       |-- deprecated.erb
+        |       |-- index.erb
+        |       `-- text.erb
+
+The path `default` refers to the template style (:template key in options hash) 
+and the directories at the next level (such as `class`) refer to template
+`:type` (options hash key) for a template. The next directory refers to the 
+output format being used defined by the `:format` template option. 
+
+As we saw in the above example, the format option can be set to `:html`, which
+would use the `html/` directory instead of `text/`. Finally, the individual .erb 
+files are the sections that make up the template.
+
+Note that the subdirectory `html/` is also its own "template" that inherits
+from the parent directory. We will see more on this later.
+
+setup.rb
+--------
+
+Every template should have at least one `setup.rb` file that defines the 
+{YARD::Templates::Template#init #init} method to set the 
+{YARD::Templates::Template#sections #sections} used by the template. If
+a setup.rb is not defined in the template itself, there should be a template
+that is inherited (via parent directory or explcitly) that sets the sections
+on a newly created template.
+
+A standard setup.rb file looks like:
+
+    def init
+      sections :section1, :section2, :section3
+    end
+
+Sections
+--------
+
+Sections are smaller components that correlate to template
+fragments. Practically speaking, a section can either be a template fragment 
+(a conventional .erb file or other supported templating language), a method 
+(which returns a String) or another {YARD::Templates::Template} (which in turn has its own 
+list of sections).
+
+Nested Sections
+---------------
+
+Sections often require the ability to encapsulate a set of sub-sections in markup
+(HTML, for instance). Rather than use heavier Template subclass objects, a more
+lightweight solution is to nest a set of sub-sections as a list that follows
+a section, for example:
+
+    def init
+      sections :header, [:section_a, :section_b]
+    end
+    
+The above example nests `section_a` and `section_b` within the `header` section.
+Practically speaking, these sections can be placed in the result by `yield`ing
+to them. A sample header.erb template might contain:
+
+    <h2>Header</h2>
+    <div id="contents">
+      <%= yieldall %>
+    </div>
+    
+This template code would place the output of `section_a` and `section_b` within
+the above div element. Using `yieldall`, we can also change the object that is being
+rendered. For example, we may want to yield the first method of the class.
+We can do this like so:
+
+    <h2>First method</h2>
+    <%= yieldall :object => object.meths.first %>
+
+This would run the nested sections for the method object instead of the class.
+
+Note that `yieldall` yields to all subsections, whereas `yield` will yield
+to each individually (in order) until there are no more left to yield to.
+In the vast majority of cases, you'd want to use `yieldall`, since `yield`
+makes it hard for users to override your template.
+
+Inheriting Templates
+--------------------
+
+Parent directory templates are automatically inherited (or mixed in, to be
+more accurate) by the current template. This means that the 'default/class/html'
+template automatically inherits from 'default/class'. This also means that anything
+defined in 'default/class/setup.rb' can be overridden by 'default/class/html/setup.rb'.
+
+Since the Template module is a module, and not a class, they can be mixed in 
+explicitly (via include/extend) from other templates, which allows templates
+to share erb files or helper logic. The 'default/class' template explicitly
+mixes in the 'default/module' template, since it uses much of the same sections.
+This is done with the helper {YARD::Templates::Template::ClassMethods#T T} method, which
+is simply a shorthand for {YARD::Templates::Engine.template Engine.template}.
+It can then override (using standard inheritance) the sections from the module
+template and insert sections pertaining to classes. This is one of the design
+goals described above.
+
+For instance, the first line in `default/class/html/setup.rb` is:
+
+    include T('default/module/html')
+
+This includes the 'default/module/html', which means it also includes 'default/module'
+by extension. This allows class to make use of any of module's erb files.
+
+Inserting Sections
+------------------
+
+The ability to insert sections was mentioned above. The class template, for
+instance, will modify the #init method to insert class specific sections:
+
+    def init
+      super
+      sections.place(:subclasses).before(:children)
+      sections.delete(:children)
+      sections.place([:constructor_details, [T('method_details')]]).before(:methodmissing)
+    end
+    
+Observe how sections has been modified after the super method was called (the
+super method would have been defined in `default/module/setup.rb`). The
+custom method {Array#place} is added by YARD to allow sections to be inserted
+before or after another section by it's given name rather than index. This
+allows the overriding of templates in a way that does not depend on where
+the section is located (since it may have been overriden by another module).
+
+Overriding Templates by Registering a Template Path
+---------------------------------------------------
+
+Inheriting templates explicitly is useful when creating a customized template
+that wants to take advantage of code re-use. However, most users who want
+to customize YARD templates will want to override existing behaviour without
+creating a template from scratch.
+
+YARD solves this problem by allowing other template paths to be registered.
+Because template modules are represented by a relative path such as 'default/class',
+they can be found within any of the registered template paths. A new template
+path is registered as:
+  
+    YARD::Templates::Engine.register_template_path '/path/to/mytemplates'
+    
+At this point, any time the 'default/class' template is loaded, the template
+will first be looked for inside the newly registered template path. If found,
+it will be used as the template module, with the modules from the other
+template paths implicitly mixed in. 
+
+Therefore, by using the same directory structure as a builtin YARD template,
+a user can customize or override individual templates as if the old ones were 
+inherited. A real world example would further modify the 'default/class' template
+seen above by creating such a path in our '/path/to/mytemplates' custom template
+path:
+
+    /path/to/mytemplates/:
+    |-- class
+    |   |-- html
+    |   |   |-- customsection.erb
+    |   |-- setup.rb
+
+The `setup.rb` file would look like:
+
+    def init
+      super
+      sections.push :customsection
+    end
+
+Now, when a class object is formatted as HTML, our customsection.erb will be 
+appended to the rendered data.

data/docs/{WHATSNEW.markdown → WhatsNew.md}

@@ -1,5 +1,85 @@
-YARD: What's New in 0.2.3?
-==========================
+What's New in 0.4.x?
+====================
+
+1. **New templating engine and templates**
+2. **yardoc `--query` argument**
+3. **Greatly expanded API documentation**
+4. **New plugin support**
+5. **New tags (@abstract, @private)**
+6. **Default rake task is now `rake yard`**
+
+New templating engine and templates
+-----------------------------------
+
+The templates were redesigned, most notably removing the ugly frameset, adding
+search to the class/method lists, simplifying the layout and making things 
+generally prettier. You should also notice that more tags are now visible in
+the templates such as @todo, the new @abstract and @note tags and some others
+that existed but were previously omitted from the generated documentation.
+
+There is also a new templating engine (based on the tadpole templating library) 
+to allow for much more user customization. You can read about it in 
+{file:Templates.md}.
+
+yardoc `--query` argument
+-------------------------
+
+The yardoc command-line tool now supports queries to select which classes,
+modules or methods to include in documentation based on their data or meta-data.
+For instance, you can now generate documentation for your "public" API only by
+adding "@api public" to each of your public API methods/classes and using
+the following argument:
+
+    --query '@api.text == "public"'
+    
+More information on queries is in the {file:README.md}.
+
+Greatly expanded API documentation
+----------------------------------
+
+Last release focused on many how-to and architecture documents to explain
+the design of YARD, but many of the actual API classes/methods were still
+left undocumented. This release marks a focus on getting YARD's own documentation
+up to par so that it can serve as an official reference on the recommended
+conventions to use when documenting code.
+
+New plugin support
+------------------
+
+YARD now supports loading of plugins via RubyGems. Any gem named `yard-*` or
+`yard_*` will now be loaded when YARD starts up. Note that the '-' separator 
+is the recommended naming scheme.
+
+To ignore plugins, add the gem names to `~/.yard/ignored_plugins` on separate
+lines (or separated by whitespace).
+
+New tags (@abstract, @private)
+------------------------------
+
+Two new tags were added to the list of builtin meta-tags in YARD. `@abstract`
+marks a class/module/method as abstract while `@private` marks an object
+as "private". The latter tag is unsed in situations where an object is public
+due to Ruby's own visibility limitations (constants, classes and modules
+can never be private) but not actually part of your public API. You should
+use this tag sparingly, as it is not meant to be an equivalent to RDoc's
+`:nodoc:` tag. Remember, YARD recommends documenting private objects too.
+This tag exists so that you can create a query (`--query !@private`) to
+ignore all of these private objects in your documentation. You can also
+use the new `--no-private` switch, which is a shortcut to the afformentioned
+query. You can read more about the new tags in the {file:GettingStarted.md} 
+guide.
+
+Default rake task is now `rake yard`
+------------------------------------
+
+Not a big change, but anyone using the default "rake yardoc" task should
+update their scripts: 
+
+[http://github.com/lsegal/yard/commit/ad38a68dd73898b06bd5d0a1912b7d815878fae0](http://github.com/lsegal/yard/commit/ad38a68dd73898b06bd5d0a1912b7d815878fae0)
+
+
+What's New in 0.2.3.x?
+======================
 
 1. **Full Ruby 1.9 support**
 2. **New parser code and handler API for 1.9**