thewoolleyman-webgen 0.5.8.20090419

data/doc/manual.page

@@ -0,0 +1,589 @@
+---
+title: Manual
+---
+# Introduction
+
+This manual describes all of webgen in as much detail as possible. If you still find something
+missing, don't hesistate to write a mail to the mailing list!
+
+The manual is structured in such a way that it can be read from top to bottom while creating a
+webgen website. This means that first the information on how to use the webgen CLI comes, then how a
+webgen website is structured and how content can be added and so on.
+
+
+
+# The `webgen` command {#cli}
+
+The executable for webgen is called... webgen ;-) It uses a command style syntax (like Subversion's
+`svn` or Rubygem's `gem` commands) through the [cmdparse] library. To get an overview of the
+possible commands run `webgen help`.
+
+The main command is the `render` command which does the actual website generation. This command uses
+either the environment variable `WEBGEN_WEBSITE` (if it is set and non-empty) or the current working
+directory as website directory if none was specified via the gloabl `-d` option. All other commands
+that need a valid webgen website directory specified work in the same way.
+
+You can invoke a command by specifying its name after the executable name. Also counting the
+executable `webgen` as a command, the options for a command are specified directly after the command
+name and before the next command or any arguments. For example, all the following command lines are
+valid:
+
+    $ webgen
+    $ webgen render
+    $ webgen -d doc render
+    $ webgen -v create -t project new_site
+    $ webgen help create
+
+Following is a short overview of the available commands:
+
+*   `apply [-f] STYLE`
+
+    Applies the style `STYLE` to the given webgen website. The files that will be written to website
+    directory are shown and the user is asked to confirm that the shown files should indeed be
+    written (and sometimes overwritten). If the `-f` option is used, the files are always written.
+
+*   `create [-t TEMPLATE] [-s STYLE] SITE_DIR`
+
+    Creates a basic webgen website in `SITE_DIR` using the optional template and styles. All
+    available templates and styles are listed in the help output for the command.
+
+*   `help`
+
+    Displays usage information. Can be used to show information about a command by using the command
+    name as argument, eg. `webgen help create`.
+
+*   `render`
+
+    Renders the given webgen website.
+
+*   `version`
+
+    Displays the version of webgen.
+
+*   `webgui`
+
+    Starts the webgen webgui, a browser based graphical user interface for managing webgen
+    websites. First the webgui web application is started and then the webgui is opened in the
+    default browser.
+
+[cmdparse]: http://cmdparse.rubyforge.org
+
+
+
+# A webgen Website {#website}
+
+webgen needs a special directory structure so that it works out of the box. This directory structure
+is automatically created by the `webgen create` command. You can alternatively use the webgui to
+create the website directory.
+
+The root directory of webgen website is called the website directory. You can have the following
+files and directories under this directory:
+
+* `src`: The source directory in which all the source files for the website are. If this directory
+  should not be called `src` or you want to include additional source directories, you need to
+  change the [`sources` configuration option]({relocatable: reference_configuration.html#sources}).
+
+* `out`: This directory is created, if it does not exist, when webgen generates the HTML files. All
+  the output files are put into this directory. The name of this directory can be changed by setting
+  the [`output` configuration option]({relocatable: reference_configuration.html#output}).
+
+* `ext`: The extension directory (optional). You can put self-written extensions into this directory
+  so that they are used by webgen. See the [extension directory]({relocatable: '#website-ext'})
+  section for more information.
+
+* `config.yaml`: This file can be used to set configuration options for the website. See the
+  [Configuration File]({relocatable: '#website-configfile'}) section for more information.
+
+* `Rakefile`: This file is provided for your convenience to execute tasks via `rake` and provides
+  some useful tasks out of the box. See the [Rakefile]({relocatable: '#website-rakefile'}) section
+  for more information.
+
+## Extension Directory {#website-ext}
+
+The extension directory is used for modifying core behaviour of webgen or adding extensions. It is
+called `ext` and has to reside directly under the website directory. All files called `init.rb` in
+this directory or any of its subdirectories are loaded when webgen renders the website. So you need
+to make sure to either place all extensions in `init.rb` or load them from this file. The latter
+approach is better since you can use the lazy loading feature that webgen uses internally,
+ie. extensions are loaded only when they are needed.
+
+## Configuration File {#website-configfile}
+
+Many user will want to change some configuration options, for example, the default language of the
+website since not all people will want to write websites in English. This is primarily done via the
+configuration file.
+
+The configuration file is called `config.yaml` and has to be placed directly under the website
+directory. It uses [YAML](http://www.yaml.org) as file format. You can find a list of all available
+configuration options that can be set in the [Configuration Options Reference]({relocatable:
+reference_configuration.html}).
+
+Each configuration option can be set in the configuration file by specifing the configuration option
+name and the new value as a key/value pair. A sample configuration file looks like this:
+
+    website.lang: de
+    website.link_to_current_page: true
+
+Since some configuration options have a complex structure, there exist several special configuration
+keys to help setting these configuration options:
+
+*   `default_meta_info`: Set the default meta information for one or more source handlers.
+
+    This key needs needs a Hash as value which should be of the following form:
+
+        SOURCE_HANDLER_NAME:
+          mi_key: mi_value
+          other_key: other_value
+          :action: replace
+
+    So each entry has to specify the name of a source handler (or the special value `:all` which
+    sets the default meta information for all source handlers) and the meta information items that
+    should be set or modified. If you don't want to update the meta information hash but want to
+    replace it, you need to add `:action: replace` as meta information entry.
+
+    If a source handler called `Webgen::SourceHandler::SOURCE_HANDLER_NAME` exists, the meta
+    information is set for this source handler instead.
+
+    For example, the following snippet of a configuration file sets the meta information item
+    `in_menu` to `true` for `Webgen::SourceHandler::Page`:
+
+        default_meta_info:
+          Page:
+            in_menu: true
+
+*   `default_processing_pipeline`: Set the default processing pipeline for one or more source
+    handlers.
+
+    This key needs a Hash as value which should be of the following form:
+
+        SOURCE_HANDLER_NAME: PIPELINE
+
+    Each key-value pair specifies the name of a source handler and the new default processing
+    pipeline. The value for the processing pipeline has to consist of the short names of content
+    processors separated by commas and normally includes `erb`, `tags`, `blocks` and the name of a
+    content processor for a markup language. The short name(s) of a content processor is (are)
+    stated on its documentation page. Be aware that the content processors are executed in the order
+    in which they are specified!
+
+    If a source handler called `Webgen::SourceHandler::SOURCE_HANDLER_NAME` exists, the meta
+    information is set for this source handler instead.
+
+    For example, the following snippet of a configuration file sets the default processing pipeline
+    for 'Webgen::SourceHandler::Page':
+
+        default_processing_pipeline:
+          Page: erb,tags,textile,blocks
+
+*   `patterns`: Set the used path patterns for one or more source handlers.
+
+    This key needs a Hash value and provides two different ways of setting the path patterns:
+
+        SOURCE_HANDLER_NAME: [**/*.jpg, **/*.css]
+
+        SOURCE_HANDLER_NAME:
+          add: [**/*.jpg]
+          del: [**/*.js]
+
+    The first form replaces the path patterns for the source handler with the given patterns. The
+    second form allows to add or delete individual patterns.
+
+    If a source handler called `Webgen::SourceHandler::SOURCE_HANDLER_NAME` exists, the meta
+    information is set for this source handler instead.
+
+    For example, the following snippet of a configuration file adds the pattern `**/*.pdf` to the
+    patterns of `Webgen::SourceHandler::Copy`:
+
+        patterns:
+          Copy:
+            add: [**/*.pdf]
+
+## Rakefile {#website-rakefile}
+
+The Rakefile that is automatically created upon website creation provides a place to specify
+recurring task for your website, for example, for deploying the website to a server. It contains
+some useful tasks out of the box:
+
+* `webgen`: Renders the webgen website once.
+* `auto_webgen`: Automatically renders the website when a source file has changed. This is very
+  useful when developing a website because you don't need to change back and forth between your
+  website code and the command line to rebuild the site.
+* `clobber_webgen`: Removes all webgen generated products (the output files and the cache file).
+
+
+
+# All About Paths and Sources   {#source}
+
+A source provides paths that identity files or directories. webgen can use paths from many sources.
+The most commonly used source is the file system source which provides paths and information on them
+from the file system.
+
+
+## Path Types {#source-types}
+
+webgen can handle many different types of files through the different source handler classes.
+
+The most important files are the page and template files as they are used to define the content and
+the layout of a website. Have a look at the [Webgen Page Format documentation]({relocatable:
+webgen_page_format.html}) to see how these files look like and how they are structured. After that
+have a look at the documentation of the source handler class SourceHandler::Page and
+SourceHandler::Template as they are responsible for handling these page and template files!
+
+You can naturally use any other type of file. However, be aware that some files may not be processed
+by webgen when no source handler class for them exists. For example, there is currently no source
+handler class for `.svg` files, so those files would be ignored. If you just want to have files
+copied from a source to the output directory (like images or CSS files), the
+[SourceHandler::Copy]({relocatable: sourcehandler/copy.html}) class is what you need! Look through
+the documentation of the [availabe source handler classes]({relocatable: extensions.html}) to get a
+feeling of what files are handled by webgen.
+
+
+## Source Paths Naming Convention {#source-naming}
+
+webgen assumes that the paths provided by the sources follow a special naming convention sothat meta
+information can be extracted correctly from the path name:
+
+    [sort_info.]basename[.lang].extension
+
+*   `sort_info`
+
+    This part is optional and has to consist of the digits 0 to 9. Its value is used for the meta
+    information `sort_info`. If not specified, it defaults to the value zero.
+
+*   `basename`
+
+    This part is used on the one hand to generate the `title` meta information (but with `_` and `-`
+    replaced by spaces). And on the other hand, the canonical name is derived from it. `basename`
+    must not contain any dots, spaces or any character from the following list: ``; / ? * : ` & = +
+    $ ,``. If you do use one of them webgen may not work correctly!
+
+    > If two paths have the same `basename` and `extension` part, they should define the same
+    > content but for different languages. This allows webgen to automatically deliver the right
+    > language version of the content
+    {.important}
+
+*   `lang`
+
+    This part is optional and has to be an [ISO-639-1/2](http://www.loc.gov/standards/iso639-2/)
+    language identifier (two or three characters (a-z) long). If not specified, it is assumed that
+    the path is language independent (for example, images are normally not specific for a specific
+    language). However, this behaviour may be different for some source handler classes (for
+    example, all paths handled by SourceHandler::Page are assigned the default language if none is
+    set).
+
+    If the language identifier can't be matched to a valid language, it is assumed that this part
+    isn't actually a language identifier but a part of the extension. This also means that in the
+    special case where the first part of an extension is also a valid language identifier, the first
+    part is interpreted as language identifier and not as part of the extension.
+
+*   `extension`
+
+    The file extension can be anything and can include dots.
+
+Following are some examples of source path names:
+
+|Path name                 | Parsed meta information
+|--------------------------|------------------------------------------------
+|`name.png`                | title: Name, language: none, sort\_info: 0, basename: name, cn: name.png
+|`name.de.png`             | title: Name, language: de, sort\_info: 0, basename: name, cn: name.png
+|`01.name_of-file.eo.page` | title: Name of file, language: eo, sort\_info: 1, basename: name_of-file, cn: name_of-file.page
+|`name.tar.bz2`            | title: Name, language: none, sort\_info: 0, basename: name, cn: name.tar.bz2
+|`name.de.tar.bz2`         | title: Name, language: de, sort\_info: 0, basename: name, cn: name.tar.bz2
+
+Notice: The first two and the last two examples define the same content for two different languages
+(or more exactly: the first one is unlocalized and the second one localized to German) as they have
+the same canonical name.
+
+
+## Canonical Name of a File ### {#source-cn}
+
+webgen provides the functionality to define the same content in more than one language, ie. to
+localize content. This is achieved with the _canonical name_ of a path, short CN.
+
+When multiple paths share the same canonical name, webgen assumes that they have the same content
+but in different languages. It is also possible to specify a _language independent_ path (i.e. one
+without a language) which is used as a fallback. Therefore when a path should be resolved using a
+canonical name and a given language, it is first tried to get the path in the requested language. If
+this is not possible (ie. no such localization exists), the unlocalized path is returned if it
+exists.
+
+For example, assume that we have the following `src` directory:
+
+    /test.jpg
+    /images/my.jpg
+    /images/my.de.jpg
+    /index.page
+    /index.fr.page
+    /index.de.page
+
+Since the path `images/my.jpg` has no language part, it is assumed to be unlocalized. In contrast,
+`images/my.de.jpg` has the "same" picture but in a German version. So, when specifying
+`images/my.jpg` in `index.de.page`, the correct localized version will be returned, i.e.
+`images/my.de.jpg`. When specifying `images/my.jpg` in `index.page` or in `index.fr.page`, the
+unlocalized version will be returned since no localized version exists.
+
+> Directories and fragments are never localized, only files are!
+{.important}
+
+It is also possible to use the _localized canonical name_ of a path (short: LCN) to resolve it. The
+localized canonical name is the same as the canonical name but with a language code inserted before
+the extension. If the localized canonical name is used to resolve a path, a possibly additionally
+specified language is ignored as it is assumed that the user really only wants the path in the
+specified language!
+
+Continuing the above example, this means that `images/my.de.jpg` is always returned when one
+specifies `images/my.de.jpg` in any of the index page files, e.g. in `index.fr.page`.
+
+Both canonical and localized canonical names can be absolute (ACN and ALCN) which is done by using
+the full path starting with a slash. This is especially useful when you don't exactly know in which
+hierarchy level a certain canoncial name gets evaluated.
+
+Everything said above also means that all paths are not resolved using their real source or output
+names but using the (localized) canonical name! So when specifying paths for extensions, for example
+when using the [relocatable tag]({relocatable: tag/relocatable.html}), you always have to use the
+(absolute) (localized) canonical names. This is different from previous webgen versions!
+
+
+## Output Path Name Construction ### {#source-output}
+
+There is currently only one method, called `standard`, for creating output path names which is
+described here. However, webgen allows developers to create other output path name creation
+methods. More information on this can be found in the [API documentation]({relocatable: api.html}).
+
+The output path for a given source path is constructed using the meta information
+`output_path_style`. This meta information is set to a default value and can be overwritten by
+setting it for a specific source handler or for a path individually. The value for this meta
+information key is an array which can have the following values:
+
+* strings (for inserting arbitrary text into output names)
+* arrays (for grouping values - only interesting for the language part)
+* symbols for inserting special values:
+  * `:cnbase`: The basename of the path.
+  * `:parent`: The parent path.
+  * `:lang`: The language.
+  * `:ext`: The file extension including the leading dot.
+  * `:year`, `:month`, `:day`: The creation year, month and day, respectively, of the source path.
+    An error is raised if the needed meta information `created_at` is not set on the path.
+
+> The constructed output path must always be an absolute one, ie. it has to start at the root of the
+> output directory. Therefore, the `:parent` part should always be included!
+{.important}
+
+Following are some examples of output path names for given source path names (assuming that `en` is
+the default language and that the path is under a directory called `/img/`):
+
+*   `output_path_style=[:parent, :cnbase, [., :lang], :ext]` (the default)
+
+    *   `index.jpg` --> `/img/index.jpg`
+
+        Since the source path is unlocalized, no language part is used and the whole sub array with
+        the `:lang` symbol is dropped.
+
+    *   `index.en.jpg` --> `/img/index.jpg`
+
+        This happens if the configuration option `sourcehandler.default_lang_in_output_path` is
+        `false` and no unlocalized version of this path exists.
+
+    *   `index.en.jpg` --> `/img/index.en.jpg`
+
+        Similar to the last example but this result occurs when there is an unlocalized version of
+        the path which is naturally named `index.jpg`!
+
+    *   `index.de.jpg` --> `/img/index.de.jpg`
+
+        Since `de` is not the default language, the language part is always used!
+
+*   `output_path_style=[:parent, :cnbase, :ext, ., :lang]`
+
+    *   `index.jpg` --> `/img/index.jpg.`
+
+        Be aware of the trailing dot since the `:lang` value is not defined in a sub array!
+
+*   `output_path_style=[:parent, :year, /, :month, /, :cnbase, [., :lang], :ext]`
+
+    *   `index.jpg` --> `/img/2008/09/index.jpg`
+
+        This output path style can be used to create blog archive style output names.
+
+
+## Path Patterns {#source-pathpattern}
+
+Each source handler specifies path patterns which are used to locate the files that the class can
+handle. Normally these patterns are used to match file extensions, however, they are much more
+powerful. For detailed information on the structure of path patterns have a look at the
+[Dir.glob](http://ruby-doc.org/core/classes/Dir.html#M002375) API documentation.
+
+The path patterns that are handled by a particular source handler are stated on its documentation
+page. These patterns can be changed by modfying the configuration option `sourcehandler.patterns`
+although that is not recommended except in some few cases (for example, it is useful to add some
+patterns for SourceHandler::Copy). The information about how these path patterns work are useful for
+the usage of webgen because of two reasons:
+
+* so that you know which files will be processed by a specific source handler class
+
+* so that you can specify additional path patterns for some source handlers like the
+  SourceHandler::Copy
+
+Here are some example path patterns:
+
+<table class="examples" markdown='1'>
+<tr><th>Path Pattern</th><th>Result</th></tr>
+<tr>
+  <td>`*/*.html`</td>
+  <td>All files with the extension `html` in the subdirectories of the source directory</td>
+</tr>
+<tr>
+  <td>`**/*.html`</td>
+  <td>All files with the extension `html` in all directories</td>
+</tr>
+<tr>
+  <td>`**/{foo,bar}*`</td>
+  <td>All files in all directories which start with `foo` or `bar`</td>
+</tr>
+<tr>
+  <td>`**/???`</td>
+  <td>All files in all directories whose file name is exactly three characters long</td>
+</tr>
+</table>
+
+
+## Handling of files in the source directory {#source-handling}
+
+Following is the list of rules how source files are handled by webgen:
+
+* All path names of all sources specified in the configuration option [`sources`]({relocatable:
+  reference_configuration.html#sources}) are fetched. Prior listed sources have priority over later
+  listed sources if both specify the same path.
+
+* Those paths which match a pattern of the configuration option
+  [`sourcehandler.ignore`]({relocatable: reference_configuration.html#sourcehandlerignore}) are excluded.
+
+* The source handler classes are invoked according to the invocation order specified in
+  [`sourcehandler.invoke`]({relocatable: reference_configuration.html#sourcehandlerinvoke}) and
+  they use only those paths that match one of their path patterns specified in
+  [`sourcehandler.patterns`]({relocatable: reference_configuration.html#sourcehandlerpatterns}).
+
+As you might have deduced from the processing list above, it is possible that one path is handled by
+multiple source handlers. This can be used, for example, to render an XML file as HTML and copy it
+verbatim.
+
+
+## Path Meta Information {#source-metainfo}
+
+Each path can have meta information, i.e. information about the path itself, associated with it, for
+example the title of the path, if it should appear in a menu and so on. This meta information can be
+specified in several ways, including:
+
+* Source handlers can provide default meta information for their handled paths (set using the
+  configuration option [`sourcehandler.default_meta_info`]({relocatable:
+  reference_configuration.html#sourcehandlerdefault_meta_info}).
+
+* Some file types allow meta information to be specified directly in the file, for example page
+  files in [Webgen Page Format]({relocatable: webgen_page_format.html}).
+
+* Meta information can also be specified in special meta information backing files. These files are
+  handled by the source handler [SourceHandler::Metainfo]({relocatable:
+  sourcehandler/metainfo.html}).
+
+There is clearly defined order in which meta information is applied to a node for a path:
+
+1. The path gets read by a source and some meta information is extracted from the path name.
+
+2. This meta information is overwritten with meta information specified for all source handlers and
+   then with meta information specified for the source handler that handles the path.
+
+3. Extensions can now overwrite meta information before the path gets handled by a source
+   handler. For example, the SourceHandler::Metainfo assigns meta information specified for paths at
+   this stage.
+
+4. Before the node for the path gets created, meta information from the content of the path itself
+   may overwrite the current meta information (this is the case, for example, with files in Webgen
+   Page Format).
+
+5. The node gets created with the provided meta information.
+
+6. After the node has been created, extensions may overwrite meta information again. For example,
+   the SourceHandler::Metainfo assigns meta information specified for nodes at this stage.
+
+
+
+# webgen Extensions
+
+webgen comes with many extensions that allow for rapid creation of static websites. The variety of
+the extensions allows you to use your tools of choice, for example, your preferred markup
+language. And if your choice is not available, you can write the extension for it yourself or make a
+feature request!
+
+Extensions are only loaded when they are needed. For example, if you don't use `.feed` files for
+automatically generating atom/RSS feeds for your website, the source handler that handles these
+files will never be loaded. Therefore you don't need to explicitly enable extensions, they just sit
+in the background and wait till the webgen core needs them.
+
+There are several types of extensions:
+
+*   **Content Processors**: These extensions process content, for example, the content of files in
+    Webgen Page Format. It is not specified how they have to process the content but this type of
+    extension can basically be divided into two groups:
+
+    * Markup processors: Processors like Maruku or RedCloth belong to this group and they convert
+      markup text that is easy to read and write to a more structure format like HTML. This allows
+      you to write an HTML page without knowing HTML.
+
+    * String replacers: These processors normally process special strings and substitute them with
+      other content. For example, the `erb` processors replaces delimited strings interpreted as
+      Ruby code with the interpreted value. Another example would be webgen's `tags` processor which
+      replaces strings like `\{relocatable: ../index.html}` with a processed value.
+
+*   **Source Handler**: These extensions are used for handling source paths. They read the content of
+    a path and produce one or more nodes (the internal representation of an output path, see [source
+    handling]({relocatable: '#source-handling'}) for more information on nodes). So a source handler
+    can do something simple like just copying a source path to the output directory but they can
+    also do complex things like creating a whole image gallery from just one source path.
+
+*   **Tags**: This type of extension allows the easy inclusion of dynamic content in, for example,
+    page and template files. Each tag extension is used for a specific task like creating a menu or
+    a breadcrumb trail.
+
+
+
+# A webgen Run
+
+When webgen is started, it first looks for a cache file and uses it if it exists. The cache file
+provides information on the previous run and allows webgen to render only those paths that have
+changed since the last time.
+
+Each webgen run consists of one or more update/write cycles. During the update phase the internal
+tree structure is updated to reflect the current situation:
+
+* Nodes are created from newly found source paths.
+* If a source path was deleted since the last webgen run, the nodes created from it will be deleted.
+* All nodes in the tree are checked if they have changed and, if so, are marked for regeneration.
+
+> The name used for describing a directory or file once it is placed in the internal tree structure
+> is 'node'.
+{.information}
+
+After the update cycle has finished, the internal tree representation is up-to-date and contains all
+necessary information to write its nodes. This is done in the write phase which writes out all
+changed nodes.
+
+It is possible that the internal tree structure changes during the write phase. For example, after
+writing a page file fragments node for it may have been generated. After the write phase it is
+checked if something like this has happened. If webgen finds such a condition, a new update/write
+cycle is initiated. This is done as long as needed.
+
+Since it is possible that multiple update/write cycles are used in one webgen run, it is shown in
+the log messages when a new cycle started. This is necessary because sometimes some warning or error
+log messages may be generated during an earlier cycle but the error conditions are automatically
+solved in later cycles. By marking where the cycles start a user can compare the log messages of the
+different cycles and see what he needs to solve himself.
+
+
+# Extending webgen
+
+If you know the programming language Ruby a little bit, you can easily extend webgen and add new
+features that you need. All the needed developer documentation is available in the [API
+documentation](rdoc/index.html), along with sample implementations for all major types of extensions
+(source class, output classes, source handler class, content processors classes, tag classes, ...)
+and detailed information about the inner workings of webgen. Have a look at the [extension
+directory]({relocatable: '#website-ext'}) section for more information on naming files and where to
+put the extensions.