yard 0.2.2 → 0.2.3

data/docs/GLOSSARY.markdown

@@ -0,0 +1,13 @@
+Glossary
+========
+
+* **Code Object**: Any explicitly defined Ruby source that describes a feature
+  of the code. By default, this refers to classes, modules, methods, constants
+  and class variables, though it can be extended to refer to custom functionality
+  defined by a DSL (like a spec, for instance).
+
+* **Domain Specific Language (DSL)**: In the context of Ruby, a DSL is a languge
+  optimized for a specific domain (problem) but represented using Ruby syntax.
+
+* **Docstring (Documentation String)**: Comments associated with a code object
+  used for documentation purposes.

data/docs/HANDLERS.markdown

@@ -0,0 +1,158 @@
+Handlers Architecture
+=====================
+
+Handlers allow the processing of parsed source code. Handling is done after
+parsing to abstract away the implementation details of lexical and semantic
+analysis on source and to only deal with the logic regarding recognizing
+source statements as {file:CODE_OBJECTS.markdown code objects}.
+
+![Handlers Architecture Class Diagram](images/handlers-class-diagram.png)
+
+The Pipeline
+------------
+
+After the {file:PARSER.markdown parser component} finishes analyzing the
+source, it is handed off for post-processing to the {YARD::Handlers::Processor}
+class, which is responsible for traversing the set of statements given by
+the parser and delegating them to matching handlers. Handlers match when the
+{YARD::Handlers::Base.handles?} method returns true for a given statement.
+The handler can then perform any action after being invoked by the `process`
+method.
+
+The Processor Class
+-------------------
+
+The main purpose of the processor, as mentioned above, is to traverse through
+the list of statements given to it by the parser. The processor also keeps
+state about what is being processed. For instance, the processor is what keeps
+track of the current namespace (the module or class an object is being defined
+in), scope (class or instance), file and owner. The owner refers to the object
+that is most directly responsible for the source statement being processed. This
+is most often the same as the namespace, except when parsing the body of a method, 
+where the namespace would be the class/module the method is defined in and the 
+owner would be the method object itself.
+
+Implementing a Handler
+======================
+
+This section covers the basics of implementing a *new-style* Ruby handler. For
+details on implementing a legacy handler, see the "API Differences" section below.
+
+a Ruby handler can be implemented simply by subclassing the {YARD::Handlers::Ruby::Base}
+class and declaring what node types or source to process with the {YARD::Handlers::Base.handles handles}
+class method. A very simple handler that handles a module definition would be:
+
+    class MyModuleHandler < YARD::Handlers::Ruby::Base
+      handles :module
+      
+      def process
+        puts "Handling a module named #{statement[0].source}"
+      end
+    end
+    
+For details on what nodes are, and what node types are, see the 
+{file:PARSER.markdown parser architecture document}.
+
+In this case the node type being handled is the `:module` type. More than one
+node type or `handles` declarations may describe a single handler, for instance,
+a handler that handles class definitions should handle the `:class` and `:sclass`
+node types respectively (the latter refers to classes defined as `class << Something`).
+The {YARD::Handlers::Base#statement statement} attribute refers to the current
+node (or statement) that is being handled by the handler.
+
+Handling a Method Call
+----------------------
+
+In some cases, a developer might need to handle a method call. The parser can
+express a method call in many AST forms, so to simplify this process, a method
+call can be handled by declaring the following in a `handles` statement:
+
+    class MyHandler < YARD::Handlers::Ruby::Base
+      handles method_call(:describe)
+      
+      def process
+        # Process the method call
+      end
+    end
+    
+In this case we handle any of the method calls to method name `describe` with 
+the following syntaxes:
+    
+    describe(something)
+    describe arg1, arg2, arg3
+    describe(something) { perform_a_block }
+    describe "Something" do
+      a_block
+    end
+    
+Creating a new Code Object
+--------------------------
+
+Usually (but not always) handling is performed to create new code objects to add
+to the registry (for information about code objects, see {file:CODE_OBJECTS.markdown this document}).
+Code objects should simply be created and added to the existing `namespace`. This
+will be enough to add them to the registry. There is also a convenience 
+{YARD::Handlers::Base#register register} method which quickly sets standard attributed
+on the newly created object, such as the file, line, source and docstring of the
+object. This method will be seen in the next example.
+    
+Handling an Inner Block
+-----------------------
+
+By default, the parser gives the processor class a list of all the top level
+statements and the processor parses only those top level statements. If an inner
+block of a module, class, method declaration or even a block passed to a method call
+needs to be handled, the {YARD::Handlers::Base#parse_block parse_block} method must be called on the list of statements
+to parse. This will send the list to the processor to continue processing on that
+statement list. The source tree can be selectively parsed in this manner by parsing
+only the inner blocks that are relevant to documentation.
+
+For example, the module handler parses the inner body of a module by performing
+the following commands:
+
+    class YARD::Handlers::Ruby::ModuleHandler < YARD::Handlers::Ruby::Base
+      handles :module
+  
+      def process
+        modname = statement[0].source
+        mod = register ModuleObject.new(namespace, modname)
+        parse_block(statement[1], :namespace => mod)
+      end
+    end
+
+In this case `statement[1]` refers to a list of extra statements, the block we
+wish to parse. Note here that when parsing objects like modules and classes,
+we set the namespace for the duration of the block parsing by setting options
+on the `parse_block` method.
+
+API Differences for Legacy Handler
+----------------------------------
+
+Because the legacy handler uses the legacy parser and therefore a different kind
+of AST, there are subtle differences in the handler API. Most importantly, the
+`handles` method usually deals with either lexical tokens or source code as a string
+or RegExp object. The statement object, similarly, is made up of lexical tokens instead
+of semantically parsed nodes (this is described in the {file:PARSER.markdown parser document}). 
+
+The module example above can be rewritten as a legacy handler as follows:
+
+    class YARD::Handlers::Ruby::Legacy::ModuleHandler < YARD::Handlers::Ruby::Legacy::Base
+      handles TkMODULE
+  
+      def process
+        modname = statement.tokens.to_s[/^module\s+(#{NAMESPACEMATCH})/, 1]
+        mod = register ModuleObject.new(namespace, modname)
+        parse_block(:namespace => mod)
+      end
+    end
+    
+A few notes on the differences:
+
+  * We inherit from `Legacy::Base` instead of the standard Ruby Base handler class.
+  * We exchange node type `:module` for `TkMODULE`, which represents the
+    first token in the statement. 
+  * We perform direct string manipulation to get the module name. 
+  * `parse_block` does not take a list of statements. In the old parser API, 
+    each statement has a `block` attribute which defines the list of 
+    statements within that statement, if any. Therefore, `parse_block` will 
+    always parse the `statement.block` if it exists.

data/docs/OVERVIEW.markdown

@@ -0,0 +1,64 @@
+Architecture Overview
+=====================
+
+YARD is separated in three major components, each of which allows YARD to be
+extended for a separate purpose. The split also emphasizes YARD's design choice
+to explicitly separate data gathering from HTML document generation, something
+that tools like RDoc do not do. These components are:
+
+* [Code Parsing & Processing Component](#parsing)
+* [Data Storage Component](#storage)
+* [Post Processing & Output Generation Component](#generators)
+
+This separation is a major goal of the project, and means that YARD is not *just* 
+a tool to generate HTML output. The expectation is that any subset of YARD's 
+major components may be used, extended or modified independently. YARD may be 
+used just as a data gathering tool (to parse and audit code), just as as a data 
+source (a webserver containing raw unformatted data about code), or just as a 
+conventional HTML documentation generation tool (like RDoc).
+
+The important classes and dependencies of these components are shown in the 
+following class diagram:
+
+![Overview Class Diagram](images/overview-class-diagram.png)
+
+<a name="parsing"></a>
+Code Parsing & Processing Component
+-----------------------------------
+
+This component is made up of four sub-components, each of which have separate
+tasks during the data gathering process (*note: the tag architecture is not*
+*shown in the class diagram*). These sub-components are:
+
+  * {file:PARSER.markdown Parser Architecture}
+  * {file:HANDLERS.markdown Handler Architecture}
+  * {file:CODE_OBJECTS.markdown Code Object Architecture}
+  * {file:TAGS.markdown Tag Architecture}
+
+The parser component reads source files and converts it into a set of statements
+which the handlers then process, creating code objects which in turn create tags 
+(meta-data) attached to the objects. These objects are all added to the {YARD::Registry},
+the data store component.
+
+<a name="storage"></a>
+Data Storage Component
+----------------------
+
+This component is currently implemented as a simple Ruby marshalled flat namespace
+of object. The implementation is found in the single class {YARD::Registry}, which
+is the centralized repository for all data being parsed, stored and accessed. There
+are future plans to improve this storage mechanism to be backend agnostic and allow
+for more robust storage.
+
+<a name="generators"></a>
+Post Processing & Output Generation Component
+---------------------------------------------
+
+This component handles processing of objects from the registry through a templating
+engine that allows output to a variety of formats. Practically speaking, this is
+where templates can be implemented to change the design, output or structure of
+the data. The next release, 0.2.4, will see a significant change in the design of 
+this component, so documentation might be sparse for now.
+
+The current design is documented in the {file:GENERATORS.markdown Generators Architecture} 
+document.

data/docs/PARSER.markdown

@@ -0,0 +1,180 @@
+Parser Architecture
+===================
+
+The parser component of YARD is the first component in the data processing pipeline
+that runs before any handling is done on the source. The parser is meant to translate
+the source into a set of statements that can be understood by the {file:HANDLERS.markdown Handlers}
+that run immediately afterwards.
+
+The important classes are described in the class diagram of the entire parser 
+system below:
+
+![Parser Class Diagram](images/parser-class-diagram.png)
+
+(Note: the RubyToken classes are omitted from the diagram)
+
+SourceParser
+------------
+
+The main class {YARD::Parser::SourceParser} acts as a factory class, instantiating
+the correct parser class given the source type being parsed. This usually involves
+a file extension check, though this can be overriden. Currently, only a Ruby source
+parser is implemented, though future plans include a C parser for Ruby extensions.
+
+This factory class should always be used when parsing source files rather than 
+the individual parser classes since it initiates the pipeline that runs the
+handlers on the parsed source. The parser used must also match the handlers,
+and this is coordinated by the `SourceParser` class as well.
+
+Using the SourceParser Class
+----------------------------
+
+The `SourceParser` class API is optimized for parsing globs of files. As such,
+the main method to use the class is the `parse` class method, which takes an
+array of file globs or a single file glob.
+
+    YARD::Parser::SourceParser.parse('spec_*.rb')
+    YARD::Parser::SourceParser.parse(['spec_*.rb', '*_helper.rb'])
+    
+This is equivalent to the convenience method {YARD.parse}:
+
+    YARD.parse('lib/**/*.rb')
+    
+In some cases (ie. for testing), it may be more helpful to parse a string of input
+directly. In such a case, the method {YARD::Parser::SourceParser.parse_string} should be
+used:
+
+    YARD::Parser::SourceParser.parse_string("def method(a, b) end")
+    
+Because no filename information is given, this method allows the setting of the
+parser type as an argument:
+
+    # Parses a string of C (not implemented)
+    YARD::Parser::SourceParser.parse_string("int main() { }", :c)
+
+The Two Ruby Parser Types
+-------------------------
+
+When parsing Ruby, the SourceParser can either instantiate the new {YARD::Parser::Ruby::RubyParser}
+class or the {YARD::Parser::Ruby::Legacy::StatementList} class. The first of the 
+two, although faster, more robust and more efficient, is only available for 
+Ruby 1.9. The legacy parser parser is available in both 1.8.x and 1.9, if 
+compatibility is required. The choice of parser will affect which handlers 
+ultimately get used, since new handlers can only use the new parser and the 
+same requirement applies to the legacy parser & handlers.
+
+Switching to Legacy Parser
+--------------------------
+
+By default, running YARD under Ruby 1.9 will automatically select the new parser
+and new handlers by extension. Although YARD supports both handler styles, plugins
+may choose to only implement one of the two (though this is not recommended). If
+only the legacy handlers are implemented, the `SourceParser` class should force
+the use of the legacy parser by setting the `parser_type` attribute as such:
+
+    YARD::Parser::SourceParser.parser_type = :ruby18
+    
+The default value is `:ruby`. Note that this cannot be forced the other way around,
+a parser type of `:ruby` cannot be set under Ruby 1.8.x as the new parser is not
+supported under 1.8.
+
+RubyParser (the New Parser)
+===========================
+
+The new Ruby parser uses the Ripper library that is packaged as part of stdlib
+in Ruby 1.9. Because of this, it can generate an AST from a string of Ruby input
+that is similar to the style of other sexp libraries (such as ParseTree). Each
+node generated in the tree is of the base type {YARD::Parser::Ruby::AstNode},
+which has some subclasses for common node types.
+
+AstNode Basics
+--------------
+
+The `AstNode` class behaves like a standard Array class in which all of its data
+make up the list of elements in the array. Unlike other sexp style libraries, however,
+the node type is not the first element of the list. Instead, the node type is defined
+by the `#type` method. The following examples show some of the basic uses of `AstNode`:
+
+    # The sexp defines the statement `hello if 1`
+    node = s(:if_mod, s(:int, "1"), s(:var_ref, s(:ident, "hello")))
+    node.type  #=> :if_mod
+    node[0]    #=> s(:int, "1")
+    node[0][0] #=> "1"
+    
+(Note the `s()` syntax is shorthand for `AstNode.new(...)`. `s()` with no type
+is shorthand for a node of type `:list`)
+
+As shown, not all of the elements are AstNodes in themselves, some are String 
+objects containing values. A list of only the AstNodes within a node can be 
+accessed via the {YARD::Parser::Ruby::AstNode#children #children} method. Using
+the sexp declared above, we can do:
+
+    node.children #=> [s(:int, "1"), s(:var_ref, s(:ident, "hello"))]
+
+AstNode#source and #line
+------------------------
+
+Every node defines the `#source` method which returns the source code that the 
+node represents. One of the most common things to do with a node is to grab its 
+source. The following example shows how this can be done:
+
+    source = "if 1 == 1 then\n  raise Exception\n end"
+    ast = YARD::Parser::Ruby::RubyParser.parse(source).root
+    ast[0].condition.source  #=> "1 == 1"
+    ast[0].then_block.source #=> "raise Exception"
+    
+Note that this only works on source parsed from the RubyParser, not sexps
+declared using the `s()` syntax. This is because no source code is generated
+or stored by nodes. Instead, only the character ranges are stored, which are
+then looked up in the original full source string object. For example:
+
+    # Following the code snippet above
+    ast[0].then_block.source_range #=> 17..31 
+    
+We can also get the line and line ranges in a similar fashion:
+
+    ast[0].type       #=> :if
+    ast[0].line       #=> 1
+    ast[0].line_range #=> 1..3 (note the newlines in the source)
+
+AstNode#jump
+------------
+
+Often the AST will be such that the node we care about might be buried arbitrarily
+deep in a node's hierarchy. The {YARD::Parser::Ruby::AstNode#jump} method exists
+to quickly get at a node of a specific type in such a situation:
+
+    # Get the first identifier in the statement
+    ast = s(s(:int, "1"), s(s(:var_ref, s(:ident, "hello"))))
+    ast.jump(:ident)[0] #=> "hello"
+    
+Multiple types can be searched for at once. If none are found, the original root
+node is returned so that it may be chained.
+
+The Legacy Parser
+=================
+
+The goal of the legacy parser is much the same as the new parser, but it is far
+more simplistic. Instead of a full-blown AST, the legacy parser simply groups
+together lists of "statements" called a {YARD::Parser::Ruby::Legacy::StatementList}. 
+These statement lists are made up of {YARD::Parser::Ruby::Legacy::Statement} objects.
+A statement is any method call condition, loop, or declaration. Each statement 
+may or may not have a block. In the case of a condition or loop, the block is 
+the inner list of statements; in the case of a method call, the block is a do 
+block (if provided). The statements themselves are made up of tokens, so instead
+of being semantic in nature like the new parser, statements are tied directly
+to the lexical tokens that make them up. To convert a statement into source, you
+simply join all the tokens together (this is done through the use of `#to_s`).
+
+Note that because there is little semantic parsing, the legacy parser is less
+able to deal with certain Ruby syntaxes. Specifically, the `:if_mod` syntax
+seen above ("hello if 1") would be considered two statements with the new parser,
+but using the legacy parser it is only one statement:
+
+    stmts = ARD::Parser::Ruby::Legacy::StatementList.new("hello if 1")
+    stmts[0].block       #=> nil
+    stmts[0].tokens.to_s #=> "hello if 1"
+    
+In addition, this means that most handling still needs to be done via string
+manipulation and regular expression matching, making it considerably more
+difficult to use in edge case scenarios.

data/docs/TAGS.markdown

@@ -0,0 +1,181 @@
+Tags Architecture
+=================
+
+Tags represent the metadata that can be added to documentation through the `@tag`
+style syntax:
+
+    # @tagname some data
+    class Foo
+    end
+
+The above example adds metadata under the name `tagname` to the Foo class object.
+
+Tags are the best way to add arbitrary metadata when documenting an object in a
+way to access it later without having to parse the entire comment string. The
+rest of the document will describe how to access the tag metadata and how to extend
+YARD to support custom tags or override existing tags to the {YARD::Tags::Library}
+class.
+
+Accessing Tag Information
+-------------------------
+
+Tag metadata is added when a {YARD::Docstring} is added to a {file:CODE_OBJECTS.markdown code object}
+using the {YARD::CodeObjects::Base#docstring=} attribute. In addition to adding
+conventional comments, tags are parsed and associated with the object. The easiest
+way to access tags on an object is to use the {YARD::CodeObjects::Base#tag} and `#tags`
+methods, for example:
+
+    # Using the Foo class object from above
+    obj.tags(:tagname).first.text #=> "some data"
+    
+Because multiple tags can be stored with the same name, they are stored as a list
+of tags. The `#tag` method is an alias for the first item in the list of tags.
+Also note that the `#tag`, `#tags` and `#has_tag?` methods are all convenience
+methods that delegate to the {YARD::Docstring} object described above.
+
+Adding Custom Tags
+------------------
+
+The `@tagname` tag used in the above examples is clearly not part of the tags
+that come with YARD. If such a tag would actually be part of documentation under
+a default install, YARD would raise a warning that the tag does not exist. It is,
+however, trivial to add this tag to be recognized by YARD.
+
+All tags in YARD are added to the {YARD::Tags::Library tag library} which makes
+use of a tag factory class to parse the data inside the tags. To simply add a
+tag that stores simple text like our `@tagname` tag above, use:
+
+    YARD::Tags::Library.define_tag("A Sample Tag", :tagname)
+
+This will now allow YARD to add the metadata from `@tagname` to the docstring.
+
+The Tag Factory Architecture
+============================
+
+Recognizing a tag is one part of the process. Parsing the tag contents is the
+second step. YARD has a tag architecture that allows developers to add or completely
+change the way tags contents can be parsed.
+
+The separation of registration and tag creation can be seen in the following
+class diagram:
+
+![Tags Architecture Class Diagram](images/tags-class-diagram.png)
+
+DefaultFactory
+--------------
+
+By default, YARD has a few standard syntaxes that can be parsed for tags. These
+are all implemented by the {YARD::Tags::DefaultFactory} class. These syntaxes
+are:
+
+  * Standard text: no parsing is done, but text is stripped of newlines and
+    multiple spaces.
+    
+  * Raw text: does no parsing at all, no stripping of newlines or spaces. This
+    is best used for code snippets.
+
+  * Raw text with title: does no parsing on the text but extracts the first line
+    of the metadata as the "title", useful for tags such as `@example`:
+        
+        # @example Inspect an element
+        #   myobj.inspect #=> #<Object:0x123525>
+
+  * Text with types: parses a list of types at the beginning of the text. Types
+    are optional. The standard syntax is in the form `[type1, type2, ...]`, 
+    for example:
+        
+        # @return [String, Symbol] a description here
+        # @return description here with no types
+    
+  * Text with types and a name: parses a list of types at the beginning of text
+    followed by a name and extra descriptive text. For example:
+        
+        # @param [String] str the string to reverse
+        def reverse(str) '...' end
+
+As mentioned above, this syntax is implemented by the `DefaultFactory` which can
+be swapped out for any factory. In some cases, a developer may want to change
+the type declaration syntax to be in the form:
+
+    # @tagname name <Types, here> description
+    
+This can be done by simply implementing a new factory that parses the data in
+this form.
+
+Implementing a Factory
+----------------------
+
+Factories should implement the method `parse_tag` as well as any `parse_tag_SUFFIX`
+method where SUFFIX refers to the suffix added when declaring the tag. For example,
+a tag can also be declared as follows:
+
+    YARD::Tags::Library.define_tag "Parameter", :param, :with_types
+    
+In such a case, the factory will be called with method `parse_tag_with_types`. In
+all cases, the method should return a new {YARD::Tags::Tag} object. Generally,
+the `parse_tag` methods take 2 or 3 parameters. A simple tag can be implemented
+as:
+
+    def parse_tag(tag_name, text)
+      Tag.new(tag_name, text)
+    end
+
+The text parameter contains pre-parsed text with extra spaces and newlines removed.
+If required, the method could also be declared with a third parameter containing
+unmodified raw text:
+
+    def parse_tag_with_raw_text(tag_name, text, raw_text)
+      Tag.new(tag_name, raw_text)
+    end
+
+Note that this method would be invoked for a tag declared with the `:with_raw_text`
+suffix.
+
+Changing the Factory
+--------------------
+
+To change the factory, set the {YARD::Tags::Library.default_factory} attribute:
+
+    YARD::Tags::Library.default_factory = MyFactory
+
+This must be done before any parsing is done, or the factory will not be used.
+
+<a name="reftags"></a>
+Reference Tags
+--------------
+
+Although attempt is made in YARD to leave as many of the syntax details as
+possible to the factory provider, there is a special tag syntax for referencing
+tags created in other objects so that they can be reused again. This is common
+when an object describes a return type or parameters that are passed through to
+other methods. In such a case, it is more manageable to use the reference tag
+syntax. Consider the following example:
+
+    class User
+      # @param [String] username the nam of the user to add
+      # @param [Number] uid the user ID
+      # @param [Number] gid the group ID
+      def initialize(username, uid, gid)
+      end
+    end
+
+    module UserHelper
+      # @param (see User#initialize)
+      def add_user(username, uid, gid)
+        User.new(username, uid, gid)
+      end
+      
+      # @param username (see User#initialize)
+      def add_root_user(username)
+        User.new(username, 0, 0)
+      end
+    end
+    
+Because the UserHelper module methods delegate directly to `User.new`, copying
+the documentation details would be unmaintainable. In this case, the (see METHODNAME)
+syntax is used to reference the tags from the User constructor to the helper methods.
+For the first method, all `@param` tags are referenced in one shot, but the second
+method only references one of the tags by adding `username` before the reference.
+
+Reference tags are represented by the {YARD::Tags::RefTag} class and are created
+directly during parsing by {YARD::Docstring}.