pygments.rb 1.2.1 → 2.0.0

data/vendor/pygments-main/doc/docs/tokens.rst

@@ -1,372 +0,0 @@
-.. -*- mode: rst -*-
-
-==============
-Builtin Tokens
-==============
-
-.. module:: pygments.token
-
-In the :mod:`pygments.token` module, there is a special object called `Token`
-that is used to create token types.
-
-You can create a new token type by accessing an attribute of `Token`:
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import Token
-    >>> Token.String
-    Token.String
-    >>> Token.String is Token.String
-    True
-
-Note that tokens are singletons so you can use the ``is`` operator for comparing
-token types.
-
-As of Pygments 0.7 you can also use the ``in`` operator to perform set tests:
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import Comment
-    >>> Comment.Single in Comment
-    True
-    >>> Comment in Comment.Multi
-    False
-
-This can be useful in :doc:`filters <filters>` and if you write lexers on your
-own without using the base lexers.
-
-You can also split a token type into a hierarchy, and get the parent of it:
-
-.. sourcecode:: pycon
-
-    >>> String.split()
-    [Token, Token.Literal, Token.Literal.String]
-    >>> String.parent
-    Token.Literal
-
-In principle, you can create an unlimited number of token types but nobody can
-guarantee that a style would define style rules for a token type. Because of
-that, Pygments proposes some global token types defined in the
-`pygments.token.STANDARD_TYPES` dict.
-
-For some tokens aliases are already defined:
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import String
-    >>> String
-    Token.Literal.String
-
-Inside the :mod:`pygments.token` module the following aliases are defined:
-
-============= ============================ ====================================
-`Text`        `Token.Text`                 for any type of text data
-`Whitespace`  `Token.Text.Whitespace`      for specially highlighted whitespace
-`Error`       `Token.Error`                represents lexer errors
-`Other`       `Token.Other`                special token for data not
-                                           matched by a parser (e.g. HTML
-                                           markup in PHP code)
-`Keyword`     `Token.Keyword`              any kind of keywords
-`Name`        `Token.Name`                 variable/function names
-`Literal`     `Token.Literal`              Any literals
-`String`      `Token.Literal.String`       string literals
-`Number`      `Token.Literal.Number`       number literals
-`Operator`    `Token.Operator`             operators (``+``, ``not``...)
-`Punctuation` `Token.Punctuation`          punctuation (``[``, ``(``...)
-`Comment`     `Token.Comment`              any kind of comments
-`Generic`     `Token.Generic`              generic tokens (have a look at
-                                           the explanation below)
-============= ============================ ====================================
-
-The `Whitespace` token type is new in Pygments 0.8. It is used only by the
-`VisibleWhitespaceFilter` currently.
-
-Normally you just create token types using the already defined aliases. For each
-of those token aliases, a number of subtypes exists (excluding the special tokens
-`Token.Text`, `Token.Error` and `Token.Other`)
-
-The `is_token_subtype()` function in the `pygments.token` module can be used to
-test if a token type is a subtype of another (such as `Name.Tag` and `Name`).
-(This is the same as ``Name.Tag in Name``. The overloaded `in` operator was newly
-introduced in Pygments 0.7, the function still exists for backwards
-compatibility.)
-
-With Pygments 0.7, it's also possible to convert strings to token types (for example
-if you want to supply a token from the command line):
-
-.. sourcecode:: pycon
-
-    >>> from pygments.token import String, string_to_tokentype
-    >>> string_to_tokentype("String")
-    Token.Literal.String
-    >>> string_to_tokentype("Token.Literal.String")
-    Token.Literal.String
-    >>> string_to_tokentype(String)
-    Token.Literal.String
-
-
-Keyword Tokens
-==============
-
-`Keyword`
-    For any kind of keyword (especially if it doesn't match any of the
-    subtypes of course).
-
-`Keyword.Constant`
-    For keywords that are constants (e.g. ``None`` in future Python versions).
-
-`Keyword.Declaration`
-    For keywords used for variable declaration (e.g. ``var`` in some programming
-    languages like JavaScript).
-
-`Keyword.Namespace`
-    For keywords used for namespace declarations (e.g. ``import`` in Python and
-    Java and ``package`` in Java).
-
-`Keyword.Pseudo`
-    For keywords that aren't really keywords (e.g. ``None`` in old Python
-    versions).
-
-`Keyword.Reserved`
-    For reserved keywords.
-
-`Keyword.Type`
-    For builtin types that can't be used as identifiers (e.g. ``int``,
-    ``char`` etc. in C).
-
-
-Name Tokens
-===========
-
-`Name`
-    For any name (variable names, function names, classes).
-
-`Name.Attribute`
-    For all attributes (e.g. in HTML tags).
-
-`Name.Builtin`
-    Builtin names; names that are available in the global namespace.
-
-`Name.Builtin.Pseudo`
-    Builtin names that are implicit (e.g. ``self`` in Ruby, ``this`` in Java).
-
-`Name.Class`
-    Class names. Because no lexer can know if a name is a class or a function
-    or something else this token is meant for class declarations.
-
-`Name.Constant`
-    Token type for constants. In some languages you can recognise a token by the
-    way it's defined (the value after a ``const`` keyword for example). In
-    other languages constants are uppercase by definition (Ruby).
-
-`Name.Decorator`
-    Token type for decorators. Decorators are syntactic elements in the Python
-    language. Similar syntax elements exist in C# and Java.
-
-`Name.Entity`
-    Token type for special entities. (e.g. ``&nbsp;`` in HTML).
-
-`Name.Exception`
-    Token type for exception names (e.g. ``RuntimeError`` in Python). Some languages
-    define exceptions in the function signature (Java). You can highlight
-    the name of that exception using this token then.
-
-`Name.Function`
-    Token type for function names.
-
-`Name.Function.Magic`
-    same as `Name.Function` but for special function names that have an implicit use
-    in a language (e.g. ``__init__`` method in Python).
-
-`Name.Label`
-    Token type for label names (e.g. in languages that support ``goto``).
-
-`Name.Namespace`
-    Token type for namespaces. (e.g. import paths in Java/Python), names following
-    the ``module``/``namespace`` keyword in other languages.
-
-`Name.Other`
-    Other names. Normally unused.
-
-`Name.Tag`
-    Tag names (in HTML/XML markup or configuration files).
-
-`Name.Variable`
-    Token type for variables. Some languages have prefixes for variable names
-    (PHP, Ruby, Perl). You can highlight them using this token.
-
-`Name.Variable.Class`
-    same as `Name.Variable` but for class variables (also static variables).
-
-`Name.Variable.Global`
-    same as `Name.Variable` but for global variables (used in Ruby, for
-    example).
-
-`Name.Variable.Instance`
-    same as `Name.Variable` but for instance variables.
-
-`Name.Variable.Magic`
-    same as `Name.Variable` but for special variable names that have an implicit use
-    in a language (e.g. ``__doc__`` in Python).
-
-
-Literals
-========
-
-`Literal`
-    For any literal (if not further defined).
-
-`Literal.Date`
-    for date literals (e.g. ``42d`` in Boo).
-
-
-`String`
-    For any string literal.
-
-`String.Affix`
-    Token type for affixes that further specify the type of the string they're
-    attached to (e.g. the prefixes ``r`` and ``u8`` in ``r"foo"`` and ``u8"foo"``). 
-
-`String.Backtick`
-    Token type for strings enclosed in backticks.
-
-`String.Char`
-    Token type for single characters (e.g. Java, C).
-
-`String.Delimiter`
-    Token type for delimiting identifiers in "heredoc", raw and other similar
-    strings (e.g. the word ``END`` in Perl code ``print <<'END';``).
-
-`String.Doc`
-    Token type for documentation strings (for example Python).
-
-`String.Double`
-    Double quoted strings.
-
-`String.Escape`
-    Token type for escape sequences in strings.
-
-`String.Heredoc`
-    Token type for "heredoc" strings (e.g. in Ruby or Perl).
-
-`String.Interpol`
-    Token type for interpolated parts in strings (e.g. ``#{foo}`` in Ruby).
-
-`String.Other`
-    Token type for any other strings (for example ``%q{foo}`` string constructs
-    in Ruby).
-
-`String.Regex`
-    Token type for regular expression literals (e.g. ``/foo/`` in JavaScript).
-
-`String.Single`
-    Token type for single quoted strings.
-
-`String.Symbol`
-    Token type for symbols (e.g. ``:foo`` in LISP or Ruby).
-
-
-`Number`
-    Token type for any number literal.
-
-`Number.Bin`
-    Token type for binary literals (e.g. ``0b101010``).
-
-`Number.Float`
-    Token type for float literals (e.g. ``42.0``).
-
-`Number.Hex`
-    Token type for hexadecimal number literals (e.g. ``0xdeadbeef``).
-
-`Number.Integer`
-    Token type for integer literals (e.g. ``42``).
-
-`Number.Integer.Long`
-    Token type for long integer literals (e.g. ``42L`` in Python).
-
-`Number.Oct`
-    Token type for octal literals.
-
-
-Operators
-=========
-
-`Operator`
-    For any punctuation operator (e.g. ``+``, ``-``).
-
-`Operator.Word`
-    For any operator that is a word (e.g. ``not``).
-
-
-Punctuation
-===========
-
-.. versionadded:: 0.7
-
-`Punctuation`
-    For any punctuation which is not an operator (e.g. ``[``, ``(``...)
-
-
-Comments
-========
-
-`Comment`
-    Token type for any comment.
-
-`Comment.Hashbang`
-    Token type for hashbang comments (i.e. first lines of files that start with
-     ``#!``).
-
-`Comment.Multiline`
-    Token type for multiline comments.
-
-`Comment.Preproc`
-    Token type for preprocessor comments (also ``<?php``/``<%`` constructs).
-
-`Comment.Single`
-    Token type for comments that end at the end of a line (e.g. ``# foo``).
-
-`Comment.Special`
-    Special data in comments. For example code tags, author and license
-    information, etc.
-
-
-Generic Tokens
-==============
-
-Generic tokens are for special lexers like the `DiffLexer` that doesn't really
-highlight a programming language but a patch file.
-
-
-`Generic`
-    A generic, unstyled token. Normally you don't use this token type.
-
-`Generic.Deleted`
-    Marks the token value as deleted.
-
-`Generic.Emph`
-    Marks the token value as emphasized.
-
-`Generic.Error`
-    Marks the token value as an error message.
-
-`Generic.Heading`
-    Marks the token value as headline.
-
-`Generic.Inserted`
-    Marks the token value as inserted.
-
-`Generic.Output`
-    Marks the token value as program output (e.g. for python cli lexer).
-
-`Generic.Prompt`
-    Marks the token value as command prompt (e.g. bash lexer).
-
-`Generic.Strong`
-    Marks the token value as bold (e.g. for rst lexer).
-
-`Generic.Subheading`
-    Marks the token value as subheadline.
-
-`Generic.Traceback`
-    Marks the token value as a part of an error traceback.

data/vendor/pygments-main/doc/docs/unicode.rst

@@ -1,58 +0,0 @@
-=====================
-Unicode and Encodings
-=====================
-
-Since Pygments 0.6, all lexers use unicode strings internally. Because of that
-you might encounter the occasional :exc:`UnicodeDecodeError` if you pass strings
-with the wrong encoding.
-
-Per default all lexers have their input encoding set to `guess`.  This means
-that the following encodings are tried:
-
-* UTF-8 (including BOM handling)
-* The locale encoding (i.e. the result of `locale.getpreferredencoding()`)
-* As a last resort, `latin1`
-
-If you pass a lexer a byte string object (not unicode), it tries to decode the
-data using this encoding.
-
-You can override the encoding using the `encoding` or `inencoding` lexer
-options.  If you have the `chardet`_ library installed and set the encoding to
-``chardet`` if will analyse the text and use the encoding it thinks is the
-right one automatically:
-
-.. sourcecode:: python
-
-    from pygments.lexers import PythonLexer
-    lexer = PythonLexer(encoding='chardet')
-
-The best way is to pass Pygments unicode objects. In that case you can't get
-unexpected output.
-
-The formatters now send Unicode objects to the stream if you don't set the
-output encoding. You can do so by passing the formatters an `encoding` option:
-
-.. sourcecode:: python
-
-    from pygments.formatters import HtmlFormatter
-    f = HtmlFormatter(encoding='utf-8')
-
-**You will have to set this option if you have non-ASCII characters in the
-source and the output stream does not accept Unicode written to it!**
-This is the case for all regular files and for terminals.
-
-Note: The Terminal formatter tries to be smart: if its output stream has an
-`encoding` attribute, and you haven't set the option, it will encode any
-Unicode string with this encoding before writing it. This is the case for
-`sys.stdout`, for example. The other formatters don't have that behavior.
-
-Another note: If you call Pygments via the command line (`pygmentize`),
-encoding is handled differently, see :doc:`the command line docs <cmdline>`.
-
-.. versionadded:: 0.7
-   The formatters now also accept an `outencoding` option which will override
-   the `encoding` option if given. This makes it possible to use a single
-   options dict with lexers and formatters, and still have different input and
-   output encodings.
-
-.. _chardet: https://chardet.github.io/

data/vendor/pygments-main/doc/download.rst

@@ -1,41 +0,0 @@
-Download and installation
-=========================
-
-The current release is version |version|.
-
-Packaged versions
------------------
-
-You can download it `from the Python Package Index
-<http://pypi.python.org/pypi/Pygments>`_.  For installation of packages from
-PyPI, we recommend `Pip <http://www.pip-installer.org>`_, which works on all
-major platforms.
-
-Under Linux, most distributions include a package for Pygments, usually called
-``pygments`` or ``python-pygments``.  You can install it with the package
-manager as usual.
-
-Development sources
--------------------
-
-We're using the `Mercurial <http://selenic.com/mercurial>`_ version control
-system.  You can get the development source using this command::
-
-    hg clone http://bitbucket.org/birkenfeld/pygments-main pygments
-
-Development takes place at `Bitbucket
-<http://bitbucket.org/birkenfeld/pygments-main>`_, you can browse the source
-online `here <http://bitbucket.org/birkenfeld/pygments-main/src>`_.
-
-The latest changes in the development source code are listed in the `changelog
-<http://bitbucket.org/birkenfeld/pygments-main/src/tip/CHANGES>`_.
-
-.. Documentation
-   -------------
-
-.. XXX todo
-
-   You can download the <a href="/docs/">documentation</a> either as
-      a bunch of rst files from the Mercurial repository, see above, or
-      as a tar.gz containing rendered HTML files:</p>
-      <p><a href="/docs/download/pygmentsdocs.tar.gz">pygmentsdocs.tar.gz</a></p>

data/vendor/pygments-main/doc/faq.rst

@@ -1,139 +0,0 @@
-:orphan:
-
-Pygments FAQ
-=============
-
-What is Pygments?
------------------
-
-Pygments is a syntax highlighting engine written in Python. That means, it will
-take source code (or other markup) in a supported language and output a
-processed version (in different formats) containing syntax highlighting markup.
-
-Its features include:
-
-* a wide range of common :doc:`languages and markup formats <languages>` is supported
-* new languages and formats are added easily
-* a number of output formats is available, including:
-
-  - HTML
-  - ANSI sequences (console output)
-  - LaTeX
-  - RTF
-
-* it is usable as a command-line tool and as a library
-* parsing and formatting is fast
-
-Pygments is licensed under the BSD license.
-
-Where does the name Pygments come from?
----------------------------------------
-
-*Py* of course stands for Python, while *pigments* are used for coloring paint,
-and in this case, source code!
-
-What are the system requirements?
----------------------------------
-
-Pygments only needs a standard Python install, version 2.6 or higher or version
-3.3 or higher for Python 3. No additional libraries are needed.
-
-How can I use Pygments?
------------------------
-
-Pygments is usable as a command-line tool as well as a library.
-
-From the command-line, usage looks like this (assuming the pygmentize script is
-properly installed)::
-
-    pygmentize -f html /path/to/file.py
-
-This will print a HTML-highlighted version of /path/to/file.py to standard output.
-
-For a complete help, please run ``pygmentize -h``.
-
-Usage as a library is thoroughly demonstrated in the Documentation section.
-
-How do I make a new style?
---------------------------
-
-Please see the :doc:`documentation on styles <docs/styles>`.
-
-How can I report a bug or suggest a feature?
---------------------------------------------
-
-Please report bugs and feature wishes in the tracker at Bitbucket.
-
-You can also e-mail the author or use IRC, see the contact details.
-
-I want this support for this language!
---------------------------------------
-
-Instead of waiting for others to include language support, why not write it
-yourself? All you have to know is :doc:`outlined in the docs
-<docs/lexerdevelopment>`.
-
-Can I use Pygments for programming language processing?
--------------------------------------------------------
-
-The Pygments lexing machinery is quite powerful can be used to build lexers for
-basically all languages. However, parsing them is not possible, though some
-lexers go some steps in this direction in order to e.g. highlight function names
-differently.
-
-Also, error reporting is not the scope of Pygments. It focuses on correctly
-highlighting syntactically valid documents, not finding and compensating errors.
-
-Who uses Pygments?
-------------------
-
-This is an (incomplete) list of projects and sites known to use the Pygments highlighter.
-
-* `Wikipedia <http://en.wikipedia.org>`_
-* `BitBucket <http://bitbucket.org/>`_, a Mercurial and Git hosting site
-* `The Sphinx documentation builder <http://sphinx.pocoo.org/>`_, for embedded source examples
-* `rst2pdf <http://code.google.com/p/rst2pdf/>`_, a reStructuredText to PDF converter
-* `Codecov <http://codecov.io/>`_, a code coverage CI service
-* `Trac <http://trac.edgewall.org/>`_, the universal project management tool
-* `AsciiDoc <http://www.methods.co.nz/asciidoc/>`_, a text-based documentation generator
-* `ActiveState Code <http://code.activestate.com/>`_, the Python Cookbook successor
-* `ViewVC <http://viewvc.org/>`_, a web-based version control repository browser
-* `BzrFruit <http://repo.or.cz/w/bzrfruit.git>`_, a Bazaar branch viewer
-* `QBzr <http://bazaar-vcs.org/QBzr>`_, a cross-platform Qt-based GUI front end for Bazaar
-* `Review Board <http://www.review-board.org/>`_, a collaborative code reviewing tool
-* `Diamanda <http://code.google.com/p/diamanda/>`_, a Django powered wiki system with support for Pygments
-* `Progopedia <http://progopedia.ru/>`_ (`English <http://progopedia.com/>`_),
-  an encyclopedia of programming languages
-* `Bruce <http://r1chardj0n3s.googlepages.com/bruce>`_, a reStructuredText presentation tool
-* `PIDA <http://pida.co.uk/>`_, a universal IDE written in Python
-* `BPython <http://www.noiseforfree.com/bpython/>`_, a curses-based intelligent Python shell
-* `PuDB <http://pypi.python.org/pypi/pudb>`_, a console Python debugger
-* `XWiki <http://www.xwiki.org/>`_, a wiki-based development framework in Java, using Jython
-* `roux <http://ananelson.com/software/roux/>`_, a script for running R scripts
-  and creating beautiful output including graphs
-* `hurl <http://hurl.it/>`_, a web service for making HTTP requests
-* `wxHTMLPygmentizer <http://colinbarnette.net/projects/wxHTMLPygmentizer>`_ is
-  a GUI utility, used to make code-colorization easier
-* `Postmarkup <http://code.google.com/p/postmarkup/>`_, a BBCode to XHTML generator
-* `WpPygments <http://blog.mirotin.net/?page_id=49>`_, and `WPygments
-  <https://github.com/capynet/WPygments>`_, highlighter plugins for WordPress
-* `Siafoo <http://siafoo.net>`_, a tool for sharing and storing useful code and programming experience
-* `D source <http://www.dsource.org/>`_, a community for the D programming language
-* `dpaste.com <http://dpaste.com/>`_, another Django pastebin
-* `Django snippets <http://www.djangosnippets.org/>`_, a pastebin for Django code
-* `Fayaa <http://www.fayaa.com/code/>`_, a Chinese pastebin
-* `Incollo.com <http://incollo.com>`_, a free collaborative debugging tool
-* `PasteBox <http://p.boxnet.eu/>`_, a pastebin focused on privacy
-* `hilite.me <http://www.hilite.me/>`_, a site to highlight code snippets
-* `patx.me <http://patx.me/paste>`_, a pastebin
-* `Fluidic <https://github.com/richsmith/fluidic>`_, an experiment in
-  integrating shells with a GUI
-* `pygments.rb <https://github.com/tmm1/pygments.rb>`_, a pygments wrapper for Ruby
-* `Clygments <https://github.com/bfontaine/clygments>`_, a pygments wrapper for
-  Clojure
-* `PHPygments <https://github.com/capynet/PHPygments>`_, a pygments wrapper for PHP
-
-
-If you have a project or web site using Pygments, drop me a line, and I'll add a
-link here.
-

data/vendor/pygments-main/doc/index.rst

@@ -1,54 +0,0 @@
-Welcome!
-========
-
-This is the home of Pygments.  It is a generic syntax highlighter suitable for
-use in code hosting, forums, wikis or other applications that need to prettify
-source code.  Highlights are:
-
-* a wide range of over 300 languages and other text formats is supported
-* special attention is paid to details that increase highlighting quality
-* support for new languages and formats are added easily; most languages use a
-  simple regex-based lexing mechanism
-* a number of output formats is available, among them HTML, RTF, LaTeX and ANSI
-  sequences
-* it is usable as a command-line tool and as a library
-* ... and it highlights even Perl 6!
-
-Read more in the :doc:`FAQ list <faq>` or the :doc:`documentation <docs/index>`,
-or `download the latest release <http://pypi.python.org/pypi/Pygments>`_.
-
-.. _contribute:
-
-Contribute
-----------
-
-Like every open-source project, we are always looking for volunteers to help us
-with programming. Python knowledge is required, but don't fear: Python is a very
-clear and easy to learn language.
-
-Development takes place on `Bitbucket
-<https://bitbucket.org/birkenfeld/pygments-main>`_, where the Mercurial
-repository, tickets and pull requests can be viewed.
-
-Our primary communication instrument is the IRC channel **#pocoo** on the
-Freenode network. To join it, let your IRC client connect to
-``irc.freenode.net`` and do ``/join #pocoo``.
-
-If you found a bug, just open a ticket in the Bitbucket tracker. Be sure to log
-in to be notified when the issue is fixed -- development is not fast-paced as
-the library is quite stable.  You can also send an e-mail to the developers, see
-below.
-
-The authors
------------
-
-Pygments is maintained by **Georg Brandl**, e-mail address *georg*\ *@*\ *python.org*.
-
-Many lexers and fixes have been contributed by **Armin Ronacher**, the rest of
-the `Pocoo <http://dev.pocoo.org/>`_ team and **Tim Hatch**.
-
-.. toctree::
-   :maxdepth: 1
-   :hidden:
-
-   docs/index