pygments.rb 1.2.0 → 2.1.0

data/vendor/pygments-main/doc/docs/tokens.rst

@@ -1,372 +0,0 @@
-.. -*- mode: rst -*-
-
-==============
-Builtin Tokens
-==============
-
-.. module:: pygments.token
-
-In the :mod:`pygments.token` module, there is a special object called `Token`
-that is used to create token types.
-
-You can create a new token type by accessing an attribute of `Token`:
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import Token
-    >>> Token.String
-    Token.String
-    >>> Token.String is Token.String
-    True
-
-Note that tokens are singletons so you can use the ``is`` operator for comparing
-token types.
-
-As of Pygments 0.7 you can also use the ``in`` operator to perform set tests:
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import Comment
-    >>> Comment.Single in Comment
-    True
-    >>> Comment in Comment.Multi
-    False
-
-This can be useful in :doc:`filters <filters>` and if you write lexers on your
-own without using the base lexers.
-
-You can also split a token type into a hierarchy, and get the parent of it:
-
-.. sourcecode:: pycon
-
-    >>> String.split()
-    [Token, Token.Literal, Token.Literal.String]
-    >>> String.parent
-    Token.Literal
-
-In principle, you can create an unlimited number of token types but nobody can
-guarantee that a style would define style rules for a token type. Because of
-that, Pygments proposes some global token types defined in the
-`pygments.token.STANDARD_TYPES` dict.
-
-For some tokens aliases are already defined:
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import String
-    >>> String
-    Token.Literal.String
-
-Inside the :mod:`pygments.token` module the following aliases are defined:
-
-============= ============================ ====================================
-`Text`        `Token.Text`                 for any type of text data
-`Whitespace`  `Token.Text.Whitespace`      for specially highlighted whitespace
-`Error`       `Token.Error`                represents lexer errors
-`Other`       `Token.Other`                special token for data not
-                                           matched by a parser (e.g. HTML
-                                           markup in PHP code)
-`Keyword`     `Token.Keyword`              any kind of keywords
-`Name`        `Token.Name`                 variable/function names
-`Literal`     `Token.Literal`              Any literals
-`String`      `Token.Literal.String`       string literals
-`Number`      `Token.Literal.Number`       number literals
-`Operator`    `Token.Operator`             operators (``+``, ``not``...)
-`Punctuation` `Token.Punctuation`          punctuation (``[``, ``(``...)
-`Comment`     `Token.Comment`              any kind of comments
-`Generic`     `Token.Generic`              generic tokens (have a look at
-                                           the explanation below)
-============= ============================ ====================================
-
-The `Whitespace` token type is new in Pygments 0.8. It is used only by the
-`VisibleWhitespaceFilter` currently.
-
-Normally you just create token types using the already defined aliases. For each
-of those token aliases, a number of subtypes exists (excluding the special tokens
-`Token.Text`, `Token.Error` and `Token.Other`)
-
-The `is_token_subtype()` function in the `pygments.token` module can be used to
-test if a token type is a subtype of another (such as `Name.Tag` and `Name`).
-(This is the same as ``Name.Tag in Name``. The overloaded `in` operator was newly
-introduced in Pygments 0.7, the function still exists for backwards
-compatibility.)
-
-With Pygments 0.7, it's also possible to convert strings to token types (for example
-if you want to supply a token from the command line):
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import String, string_to_tokentype
-    >>> string_to_tokentype("String")
-    Token.Literal.String
-    >>> string_to_tokentype("Token.Literal.String")
-    Token.Literal.String
-    >>> string_to_tokentype(String)
-    Token.Literal.String
-
-
-Keyword Tokens
-==============
-
-`Keyword`
-    For any kind of keyword (especially if it doesn't match any of the
-    subtypes of course).
-
-`Keyword.Constant`
-    For keywords that are constants (e.g. ``None`` in future Python versions).
-
-`Keyword.Declaration`
-    For keywords used for variable declaration (e.g. ``var`` in some programming
-    languages like JavaScript).
-
-`Keyword.Namespace`
-    For keywords used for namespace declarations (e.g. ``import`` in Python and
-    Java and ``package`` in Java).
-
-`Keyword.Pseudo`
-    For keywords that aren't really keywords (e.g. ``None`` in old Python
-    versions).
-
-`Keyword.Reserved`
-    For reserved keywords.
-
-`Keyword.Type`
-    For builtin types that can't be used as identifiers (e.g. ``int``,
-    ``char`` etc. in C).
-
-
-Name Tokens
-===========
-
-`Name`
-    For any name (variable names, function names, classes).
-
-`Name.Attribute`
-    For all attributes (e.g. in HTML tags).
-
-`Name.Builtin`
-    Builtin names; names that are available in the global namespace.
-
-`Name.Builtin.Pseudo`
-    Builtin names that are implicit (e.g. ``self`` in Ruby, ``this`` in Java).
-
-`Name.Class`
-    Class names. Because no lexer can know if a name is a class or a function
-    or something else this token is meant for class declarations.
-
-`Name.Constant`
-    Token type for constants. In some languages you can recognise a token by the
-    way it's defined (the value after a ``const`` keyword for example). In
-    other languages constants are uppercase by definition (Ruby).
-
-`Name.Decorator`
-    Token type for decorators. Decorators are syntactic elements in the Python
-    language. Similar syntax elements exist in C# and Java.
-
-`Name.Entity`
-    Token type for special entities. (e.g. ``&nbsp;`` in HTML).
-
-`Name.Exception`
-    Token type for exception names (e.g. ``RuntimeError`` in Python). Some languages
-    define exceptions in the function signature (Java). You can highlight
-    the name of that exception using this token then.
-
-`Name.Function`
-    Token type for function names.
-
-`Name.Function.Magic`
-    same as `Name.Function` but for special function names that have an implicit use
-    in a language (e.g. ``__init__`` method in Python).
-
-`Name.Label`
-    Token type for label names (e.g. in languages that support ``goto``).
-
-`Name.Namespace`
-    Token type for namespaces. (e.g. import paths in Java/Python), names following
-    the ``module``/``namespace`` keyword in other languages.
-
-`Name.Other`
-    Other names. Normally unused.
-
-`Name.Tag`
-    Tag names (in HTML/XML markup or configuration files).
-
-`Name.Variable`
-    Token type for variables. Some languages have prefixes for variable names
-    (PHP, Ruby, Perl). You can highlight them using this token.
-
-`Name.Variable.Class`
-    same as `Name.Variable` but for class variables (also static variables).
-
-`Name.Variable.Global`
-    same as `Name.Variable` but for global variables (used in Ruby, for
-    example).
-
-`Name.Variable.Instance`
-    same as `Name.Variable` but for instance variables.
-
-`Name.Variable.Magic`
-    same as `Name.Variable` but for special variable names that have an implicit use
-    in a language (e.g. ``__doc__`` in Python).
-
-
-Literals
-========
-
-`Literal`
-    For any literal (if not further defined).
-
-`Literal.Date`
-    for date literals (e.g. ``42d`` in Boo).
-
-
-`String`
-    For any string literal.
-
-`String.Affix`
-    Token type for affixes that further specify the type of the string they're
-    attached to (e.g. the prefixes ``r`` and ``u8`` in ``r"foo"`` and ``u8"foo"``). 
-
-`String.Backtick`
-    Token type for strings enclosed in backticks.
-
-`String.Char`
-    Token type for single characters (e.g. Java, C).
-
-`String.Delimiter`
-    Token type for delimiting identifiers in "heredoc", raw and other similar
-    strings (e.g. the word ``END`` in Perl code ``print <<'END';``).
-
-`String.Doc`
-    Token type for documentation strings (for example Python).
-
-`String.Double`
-    Double quoted strings.
-
-`String.Escape`
-    Token type for escape sequences in strings.
-
-`String.Heredoc`
-    Token type for "heredoc" strings (e.g. in Ruby or Perl).
-
-`String.Interpol`
-    Token type for interpolated parts in strings (e.g. ``#{foo}`` in Ruby).
-
-`String.Other`
-    Token type for any other strings (for example ``%q{foo}`` string constructs
-    in Ruby).
-
-`String.Regex`
-    Token type for regular expression literals (e.g. ``/foo/`` in JavaScript).
-
-`String.Single`
-    Token type for single quoted strings.
-
-`String.Symbol`
-    Token type for symbols (e.g. ``:foo`` in LISP or Ruby).
-
-
-`Number`
-    Token type for any number literal.
-
-`Number.Bin`
-    Token type for binary literals (e.g. ``0b101010``).
-
-`Number.Float`
-    Token type for float literals (e.g. ``42.0``).
-
-`Number.Hex`
-    Token type for hexadecimal number literals (e.g. ``0xdeadbeef``).
-
-`Number.Integer`
-    Token type for integer literals (e.g. ``42``).
-
-`Number.Integer.Long`
-    Token type for long integer literals (e.g. ``42L`` in Python).
-
-`Number.Oct`
-    Token type for octal literals.
-
-
-Operators
-=========
-
-`Operator`
-    For any punctuation operator (e.g. ``+``, ``-``).
-
-`Operator.Word`
-    For any operator that is a word (e.g. ``not``).
-
-
-Punctuation
-===========
-
-.. versionadded:: 0.7
-
-`Punctuation`
-    For any punctuation which is not an operator (e.g. ``[``, ``(``...)
-
-
-Comments
-========
-
-`Comment`
-    Token type for any comment.
-
-`Comment.Hashbang`
-    Token type for hashbang comments (i.e. first lines of files that start with
-     ``#!``).
-
-`Comment.Multiline`
-    Token type for multiline comments.
-
-`Comment.Preproc`
-    Token type for preprocessor comments (also ``<?php``/``<%`` constructs).
-
-`Comment.Single`
-    Token type for comments that end at the end of a line (e.g. ``# foo``).
-
-`Comment.Special`
-    Special data in comments. For example code tags, author and license
-    information, etc.
-
-
-Generic Tokens
-==============
-
-Generic tokens are for special lexers like the `DiffLexer` that doesn't really
-highlight a programming language but a patch file.
-
-
-`Generic`
-    A generic, unstyled token. Normally you don't use this token type.
-
-`Generic.Deleted`
-    Marks the token value as deleted.
-
-`Generic.Emph`
-    Marks the token value as emphasized.
-
-`Generic.Error`
-    Marks the token value as an error message.
-
-`Generic.Heading`
-    Marks the token value as headline.
-
-`Generic.Inserted`
-    Marks the token value as inserted.
-
-`Generic.Output`
-    Marks the token value as program output (e.g. for python cli lexer).
-
-`Generic.Prompt`
-    Marks the token value as command prompt (e.g. bash lexer).
-
-`Generic.Strong`
-    Marks the token value as bold (e.g. for rst lexer).
-
-`Generic.Subheading`
-    Marks the token value as subheadline.
-
-`Generic.Traceback`
-    Marks the token value as a part of an error traceback.

data/vendor/pygments-main/doc/docs/unicode.rst

@@ -1,58 +0,0 @@
-=====================
-Unicode and Encodings
-=====================
-
-Since Pygments 0.6, all lexers use unicode strings internally. Because of that
-you might encounter the occasional :exc:`UnicodeDecodeError` if you pass strings
-with the wrong encoding.
-
-Per default all lexers have their input encoding set to `guess`.  This means
-that the following encodings are tried:
-
-* UTF-8 (including BOM handling)
-* The locale encoding (i.e. the result of `locale.getpreferredencoding()`)
-* As a last resort, `latin1`
-
-If you pass a lexer a byte string object (not unicode), it tries to decode the
-data using this encoding.
-
-You can override the encoding using the `encoding` or `inencoding` lexer
-options.  If you have the `chardet`_ library installed and set the encoding to
-``chardet`` if will analyse the text and use the encoding it thinks is the
-right one automatically:
-
-.. sourcecode:: python
-
-    from pygments.lexers import PythonLexer
-    lexer = PythonLexer(encoding='chardet')
-
-The best way is to pass Pygments unicode objects. In that case you can't get
-unexpected output.
-
-The formatters now send Unicode objects to the stream if you don't set the
-output encoding. You can do so by passing the formatters an `encoding` option:
-
-.. sourcecode:: python
-
-    from pygments.formatters import HtmlFormatter
-    f = HtmlFormatter(encoding='utf-8')
-
-**You will have to set this option if you have non-ASCII characters in the
-source and the output stream does not accept Unicode written to it!**
-This is the case for all regular files and for terminals.
-
-Note: The Terminal formatter tries to be smart: if its output stream has an
-`encoding` attribute, and you haven't set the option, it will encode any
-Unicode string with this encoding before writing it. This is the case for
-`sys.stdout`, for example. The other formatters don't have that behavior.
-
-Another note: If you call Pygments via the command line (`pygmentize`),
-encoding is handled differently, see :doc:`the command line docs <cmdline>`.
-
-.. versionadded:: 0.7
-   The formatters now also accept an `outencoding` option which will override
-   the `encoding` option if given. This makes it possible to use a single
-   options dict with lexers and formatters, and still have different input and
-   output encodings.
-
-.. _chardet: https://chardet.github.io/

data/vendor/pygments-main/doc/download.rst

@@ -1,41 +0,0 @@
-Download and installation
-=========================
-
-The current release is version |version|.
-
-Packaged versions
------------------
-
-You can download it `from the Python Package Index
-<http://pypi.python.org/pypi/Pygments>`_.  For installation of packages from
-PyPI, we recommend `Pip <http://www.pip-installer.org>`_, which works on all
-major platforms.
-
-Under Linux, most distributions include a package for Pygments, usually called
-``pygments`` or ``python-pygments``.  You can install it with the package
-manager as usual.
-
-Development sources
--------------------
-
-We're using the `Mercurial <http://selenic.com/mercurial>`_ version control
-system.  You can get the development source using this command::
-
-    hg clone http://bitbucket.org/birkenfeld/pygments-main pygments
-
-Development takes place at `Bitbucket
-<http://bitbucket.org/birkenfeld/pygments-main>`_, you can browse the source
-online `here <http://bitbucket.org/birkenfeld/pygments-main/src>`_.
-
-The latest changes in the development source code are listed in the `changelog
-<http://bitbucket.org/birkenfeld/pygments-main/src/tip/CHANGES>`_.
-
-.. Documentation
-   -------------
-
-.. XXX todo
-
-   You can download the <a href="/docs/">documentation</a> either as
-      a bunch of rst files from the Mercurial repository, see above, or
-      as a tar.gz containing rendered HTML files:</p>
-      <p><a href="/docs/download/pygmentsdocs.tar.gz">pygmentsdocs.tar.gz</a></p>