yard 0.9.29 → 0.9.30

data/docs/Handlers.md

@@ -0,0 +1,152 @@
+# @title Handlers Architecture
+
+# Handlers Architecture
+
+Handlers allow the processing of parsed source code. Handling is done after
+parsing to abstract away the implementation details of lexical and semantic
+analysis on source and to only deal with the logic regarding recognizing
+source statements as {file:docs/CodeObjects.md code objects}.
+
+![Handlers Architecture Class Diagram](images/handlers-class-diagram.png)
+
+## The Pipeline
+
+After the {file:docs/Parser.md parser component} finishes analyzing the
+source, it is handed off for post-processing to the {YARD::Handlers::Processor}
+class, which is responsible for traversing the set of statements given by
+the parser and delegating them to matching handlers. Handlers match when the
+{YARD::Handlers::Base.handles?} method returns true for a given statement.
+The handler can then perform any action after being invoked by the `process`
+method.
+
+## The Processor Class
+
+The main purpose of the processor, as mentioned above, is to traverse through
+the list of statements given to it by the parser. The processor also keeps
+state about what is being processed. For instance, the processor is what keeps
+track of the current namespace (the module or class an object is being defined
+in), scope (class or instance), file and owner. The owner refers to the object
+that is most directly responsible for the source statement being processed. This
+is most often the same as the namespace, except when parsing the body of a method,
+where the namespace would be the class/module the method is defined in and the
+owner would be the method object itself.
+
+## Implementing a Handler
+
+This section covers the basics of implementing a *new-style* Ruby handler. For
+details on implementing a legacy handler, see the "API Differences" section below.
+
+a Ruby handler can be implemented simply by subclassing the {YARD::Handlers::Ruby::Base}
+class and declaring what node types or source to process with the {YARD::Handlers::Base.handles handles}
+class method. A very simple handler that handles a module definition would be:
+
+    class MyModuleHandler < YARD::Handlers::Ruby::Base
+      handles :module
+
+      def process
+        puts "Handling a module named #{statement[0].source}"
+      end
+    end
+
+For details on what nodes are, and what node types are, see the
+{file:docs/Parser.md parser architecture document}.
+
+In this case the node type being handled is the `:module` type. More than one
+node type or `handles` declarations may describe a single handler, for instance,
+a handler that handles class definitions should handle the `:class` and `:sclass`
+node types respectively (the latter refers to classes defined as `class << Something`).
+The {YARD::Handlers::Base#statement statement} attribute refers to the current
+node (or statement) that is being handled by the handler.
+
+### Handling a Method Call
+
+In some cases, a developer might need to handle a method call. The parser can
+express a method call in many AST forms, so to simplify this process, a method
+call can be handled by declaring the following in a `handles` statement:
+
+    class MyHandler < YARD::Handlers::Ruby::Base
+      handles method_call(:describe)
+
+      def process
+        # Process the method call
+      end
+    end
+
+In this case we handle any of the method calls to method name `describe` with
+the following syntaxes:
+
+    describe(something)
+    describe arg1, arg2, arg3
+    describe(something) { perform_a_block }
+    describe "Something" do
+      a_block
+    end
+
+### Creating a new Code Object
+
+Usually (but not always) handling is performed to create new code objects to add
+to the registry (for information about code objects, see {file:docs/CodeObjects.md this document}).
+Code objects should simply be created and added to the existing `namespace`. This
+will be enough to add them to the registry. There is also a convenience
+{YARD::Handlers::Base#register register} method which quickly sets standard attributed
+on the newly created object, such as the file, line, source and docstring of the
+object. This method will be seen in the next example.
+
+### Handling an Inner Block
+
+By default, the parser gives the processor class a list of all the top level
+statements and the processor parses only those top level statements. If an inner
+block of a module, class, method declaration or even a block passed to a method call
+needs to be handled, the {YARD::Handlers::Base#parse_block parse_block} method must be called on the list of statements
+to parse. This will send the list to the processor to continue processing on that
+statement list. The source tree can be selectively parsed in this manner by parsing
+only the inner blocks that are relevant to documentation.
+
+For example, the module handler parses the inner body of a module by performing
+the following commands:
+
+    class YARD::Handlers::Ruby::ModuleHandler < YARD::Handlers::Ruby::Base
+      handles :module
+
+      def process
+        modname = statement[0].source
+        mod = register ModuleObject.new(namespace, modname)
+        parse_block(statement[1], :namespace => mod)
+      end
+    end
+
+In this case `statement[1]` refers to a list of extra statements, the block we
+wish to parse. Note here that when parsing objects like modules and classes,
+we set the namespace for the duration of the block parsing by setting options
+on the `parse_block` method.
+
+### API Differences for Legacy Handler
+
+Because the legacy handler uses the legacy parser and therefore a different kind
+of AST, there are subtle differences in the handler API. Most importantly, the
+`handles` method usually deals with either lexical tokens or source code as a string
+or RegExp object. The statement object, similarly, is made up of lexical tokens instead
+of semantically parsed nodes (this is described in the {file:docs/Parser.md parser document}).
+
+The module example above can be rewritten as a legacy handler as follows:
+
+    class YARD::Handlers::Ruby::Legacy::ModuleHandler < YARD::Handlers::Ruby::Legacy::Base
+      handles TkMODULE
+
+      def process
+        modname = statement.tokens.to_s[/^module\s+(#{NAMESPACEMATCH})/, 1]
+        mod = register ModuleObject.new(namespace, modname)
+        parse_block(:namespace => mod)
+      end
+    end
+
+A few notes on the differences:
+
+  * We inherit from `Legacy::Base` instead of the standard Ruby Base handler class.
+  * We exchange node type `:module` for `TkMODULE`, which represents the
+    first token in the statement.
+  * We perform direct string manipulation to get the module name.
+  * `parse_block` does not take a list of statements. In the old parser API,
+    each statement has a `block` attribute which defines the list of
+    statements within that statement, if any. Therefore, `parse_block` will
+    always parse the `statement.block` if it exists.

data/docs/Overview.md

@@ -0,0 +1,61 @@
+# @title Architecture Overview
+
+# Architecture Overview
+
+YARD is separated in three major components, each of which allows YARD to be
+extended for a separate purpose. The split also emphasizes YARD's design choice
+to explicitly separate data gathering from HTML document generation, something
+that tools like RDoc do not do. These components are:
+
+* [Code Parsing & Processing Component](#parsing)
+* [Data Storage Component](#storage)
+* [Post Processing & Templating System](#templates)
+
+This separation is a major goal of the project, and means that YARD is not *just*
+a tool to generate HTML output. The expectation is that any subset of YARD's
+major components may be used, extended or modified independently. YARD may be
+used just as a data gathering tool (to parse and audit code), just as a data
+source (a webserver containing raw unformatted data about code), or just as a
+conventional HTML documentation generation tool (like RDoc).
+
+The important classes and dependencies of these components are shown in the
+following class diagram:
+
+![Overview Class Diagram](images/overview-class-diagram.png)
+
+<a name="parsing"></a>
+
+## Code Parsing & Processing Component
+
+This component is made up of four sub-components, each of which have separate
+tasks during the data gathering process (*note: the tag architecture is not*
+*shown in the class diagram*). These sub-components are:
+
+  * {file:docs/Parser.md}
+  * {file:docs/Handlers.md}
+  * {file:docs/CodeObjects.md}
+  * {file:docs/Tags.md}
+
+The parser component reads source files and converts it into a set of statements
+which the handlers then process, creating code objects which in turn create tags
+(meta-data) attached to the objects. These objects are all added to the {YARD::Registry},
+the data store component.
+
+<a name="storage"></a>
+
+## Data Storage Component
+
+This component is currently implemented as a simple Ruby marshalled flat namespace
+of object. The implementation is found in the single class {YARD::Registry}, which
+is the centralized repository for all data being parsed, stored and accessed. There
+are future plans to improve this storage mechanism to be backend agnostic and allow
+for more robust storage.
+
+<a name="templates"></a>
+
+## Post Processing & Templating System
+
+This component handles processing of objects from the registry through a templating
+engine that allows output to a variety of formats. Practically speaking, this is
+where templates can be implemented to change the design, output or structure of
+the data. See {file:docs/Templates.md Templates Architecture} for a complete overview.

data/docs/Parser.md

@@ -0,0 +1,191 @@
+# @title Parser Architecture
+
+# Parser Architecture
+
+The parser component of YARD is the first component in the data processing pipeline
+that runs before any handling is done on the source. The parser is meant to translate
+the source into a set of statements that can be understood by the {file:docs/Handlers.md Handlers}
+that run immediately afterwards.
+
+The important classes are described in the class diagram of the entire parser
+system below:
+
+![Parser Class Diagram](images/parser-class-diagram.png)
+
+(Note: the RubyToken classes are omitted from the diagram)
+
+## SourceParser
+
+The main class {YARD::Parser::SourceParser} acts as a factory class, instantiating
+the correct parser class, an implementation of {YARD::Parser::Base}. The selected parser
+is chosen based on either the file extension or by selecting it explicitly (as an argument
+to parsing methods). YARD supports Ruby and C source files, but custom parsers can
+be implemented and registered for various other languages by subclassing `Parser::Base`
+and registering the parser with {YARD::Parser::SourceParser.register_parser_type}.
+
+This factory class should always be used when parsing source files rather than
+the individual parser classes since it initiates the pipeline that runs the
+handlers on the parsed source. The parser used must also match the handlers,
+and this is coordinated by the `SourceParser` class as well.
+
+## Using the SourceParser Class
+
+The `SourceParser` class API is optimized for parsing globs of files. As such,
+the main method to use the class is the `parse` class method, which takes an
+array of file globs or a single file glob.
+
+    YARD::Parser::SourceParser.parse('spec_*.rb')
+    YARD::Parser::SourceParser.parse(['spec_*.rb', '*_helper.rb'])
+
+This is equivalent to the convenience method {YARD.parse}:
+
+    YARD.parse('lib/**/*.rb')
+
+In some cases (ie. for testing), it may be more helpful to parse a string of input
+directly. In such a case, the method {YARD::Parser::SourceParser.parse_string} should be
+used:
+
+    YARD::Parser::SourceParser.parse_string("def method(a, b) end")
+
+You can also provide the parser type explicitly as the second argument:
+
+    # Parses a string of C
+    YARD::Parser::SourceParser.parse_string("int main() { }", :c)
+
+Note that these two methods are aliased as {YARD.parse} and {YARD.parse_string} for
+convenience.
+
+## Implementing and Registering a Custom Parser
+
+To implement a custom parser, subclass {YARD::Parser::Base}. Documentation on which
+abstract methods should be implemented are documented in that class. After the class
+is implemented, it is registered with the {YARD::Parser::SourceParser} factory class
+to be called when a file of the right extension needs to be parsed, or when a user
+selects that parser type explicitly. To register your new parser class, call the
+method {YARD::Parser::SourceParser.register_parser_type}:
+
+    SourceParser.register_parser_type(:my_parser, MyParser, 'my_parser_ext')
+
+The last argument can be a single extension, a list of extensions (Array), a single Regexp, or a
+list of Regexps. Do not include the '.' in the extension.
+
+
+## The Two Ruby Parser Types
+
+When parsing Ruby, the SourceParser can either instantiate the new {YARD::Parser::Ruby::RubyParser}
+class or the {YARD::Parser::Ruby::Legacy::StatementList} class. The first of the
+two, although faster, more robust and more efficient, is only available for
+Ruby 1.9. The legacy parser parser is available in both 1.8.x and 1.9, if
+compatibility is required. The choice of parser will affect which handlers
+ultimately get used, since new handlers can only use the new parser and the
+same requirement applies to the legacy parser & handlers.
+
+## Switching to Legacy Parser
+
+By default, running YARD under Ruby 1.9 will automatically select the new parser
+and new handlers by extension. Although YARD supports both handler styles, plugins
+may choose to only implement one of the two (though this is not recommended). If
+only the legacy handlers are implemented, the `SourceParser` class should force
+the use of the legacy parser by setting the `parser_type` attribute as such:
+
+    YARD::Parser::SourceParser.parser_type = :ruby18
+
+The default value is `:ruby`. Note that this cannot be forced the other way around,
+a parser type of `:ruby` cannot be set under Ruby 1.8.x as the new parser is not
+supported under 1.8.
+
+## RubyParser (the New Parser)
+
+The new Ruby parser uses the Ripper library that is packaged as part of stdlib
+in Ruby 1.9. Because of this, it can generate an AST from a string of Ruby input
+that is similar to the style of other sexp libraries (such as ParseTree). Each
+node generated in the tree is of the base type {YARD::Parser::Ruby::AstNode},
+which has some subclasses for common node types.
+
+### AstNode Basics
+
+The `AstNode` class behaves like a standard Array class in which all of its data
+make up the list of elements in the array. Unlike other sexp style libraries, however,
+the node type is not the first element of the list. Instead, the node type is defined
+by the `#type` method. The following examples show some of the basic uses of `AstNode`:
+
+    # The sexp defines the statement `hello if 1`
+    node = s(:if_mod, s(:int, "1"), s(:var_ref, s(:ident, "hello")))
+    node.type  #=> :if_mod
+    node[0]    #=> s(:int, "1")
+    node[0][0] #=> "1"
+
+(Note the `s()` syntax is shorthand for `AstNode.new(...)`. `s()` with no type
+is shorthand for a node of type `:list`)
+
+As shown, not all of the elements are AstNodes in themselves, some are String
+objects containing values. A list of only the AstNodes within a node can be
+accessed via the {YARD::Parser::Ruby::AstNode#children #children} method. Using
+the sexp declared above, we can do:
+
+    node.children #=> [s(:int, "1"), s(:var_ref, s(:ident, "hello"))]
+
+### AstNode#source and #line
+
+Every node defines the `#source` method which returns the source code that the
+node represents. One of the most common things to do with a node is to grab its
+source. The following example shows how this can be done:
+
+    source = "if 1 == 1 then\n  raise Exception\n end"
+    ast = YARD::Parser::Ruby::RubyParser.parse(source).root
+    ast[0].condition.source  #=> "1 == 1"
+    ast[0].then_block.source #=> "raise Exception"
+
+Note that this only works on source parsed from the RubyParser, not sexps
+declared using the `s()` syntax. This is because no source code is generated
+or stored by nodes. Instead, only the character ranges are stored, which are
+then looked up in the original full source string object. For example:
+
+    # Following the code snippet above
+    ast[0].then_block.source_range #=> 17..31
+
+We can also get the line and line ranges in a similar fashion:
+
+    ast[0].type       #=> :if
+    ast[0].line       #=> 1
+    ast[0].line_range #=> 1..3 (note the newlines in the source)
+
+### AstNode#jump
+
+Often the AST will be such that the node we care about might be buried arbitrarily
+deep in a node's hierarchy. The {YARD::Parser::Ruby::AstNode#jump} method exists
+to quickly get at a node of a specific type in such a situation:
+
+    # Get the first identifier in the statement
+    ast = s(s(:int, "1"), s(s(:var_ref, s(:ident, "hello"))))
+    ast.jump(:ident)[0] #=> "hello"
+
+Multiple types can be searched for at once. If none are found, the original root
+node is returned so that it may be chained.
+
+## The Legacy Parser
+
+The goal of the legacy parser is much the same as the new parser, but it is far
+more simplistic. Instead of a full-blown AST, the legacy parser simply groups
+together lists of "statements" called a {YARD::Parser::Ruby::Legacy::StatementList}.
+These statement lists are made up of {YARD::Parser::Ruby::Legacy::Statement} objects.
+A statement is any method call condition, loop, or declaration. Each statement
+may or may not have a block. In the case of a condition or loop, the block is
+the inner list of statements; in the case of a method call, the block is a do
+block (if provided). The statements themselves are made up of tokens, so instead
+of being semantic in nature like the new parser, statements are tied directly
+to the lexical tokens that make them up. To convert a statement into source, you
+simply join all the tokens together (this is done through the use of `#to_s`).
+
+Note that because there is little semantic parsing, the legacy parser is less
+able to deal with certain Ruby syntaxes. Specifically, the `:if_mod` syntax
+seen above ("hello if 1") would be considered two statements with the new parser,
+but using the legacy parser it is only one statement:
+
+    stmts = ARD::Parser::Ruby::Legacy::StatementList.new("hello if 1")
+    stmts[0].block       #=> nil
+    stmts[0].tokens.to_s #=> "hello if 1"
+
+In addition, this means that most handling still needs to be done via string
+manipulation and regular expression matching, making it considerably more
+difficult to use in edge case scenarios.