ctags.rb 1.0.19 → 1.1.3

data/ext/vendor/ctags/docs/internal.rst

@@ -0,0 +1,398 @@
+ctags Internal API
+---------------------------------------------------------------------
+
+Input text stream
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. figure:: input-text-stream.svg
+	    :scale: 80%
+
+Function prototypes for handling input text stream are declared in
+main/read.h. The file exists in exuberant ctags, too.  However, the
+names functions are changed when overhauling ``--line-directive``
+option. (In addition macros were converted to functions for making
+data structures for the input text stream opaque.)
+
+Ctags has 3 groups of functions for handling input: input, bypass, and
+raw. Parser developers should use input group. The rest of two
+are for ctags main part.
+
+
+.. _inputFile:
+
+`inputFile` type and the functions of input group
+......................................................................
+
+(The original version of this sub sub sub section was written
+ before `inputFile` type and `File` variable are made private. )
+
+`inputFile` is the type for representing the input file and stream for
+a parser. It was declared in main/read.h but now it is defined in
+main/read.c.
+
+Ctags uses a file static variable `File` having type `inputFile` for
+maintaining the input file and stream. `File` is also defined in
+main/read.c as `inputFile` is.
+
+`fp` and `line` are the essential fields of `File`. `fp` having type
+well known `MIO` declared in main/mio.h. By calling functions of input group
+(`getcFromInputFile` and `readLineFromInputFile`), a parser gets input
+text from `fp`.
+
+The functions of input group updates fields `input` and `source` of `File`
+These two fields has type `inputFileInfo`. These two fields are for mainly
+tracking the name of file and the current line number. Usually ctags uses
+only `input` field. `source` is used only when `#line` directive is found
+in the current input text stream.
+
+A case when a tool generates the input file from another file, a tool
+can record the original source file to the generated file with using
+the `#line` directive. `source` is used for tracking/recording the
+informations appeared on #line directives.
+
+Regex pattern matching are also done behind calling the functions of
+this group.
+
+
+The functions of bypass group
+......................................................................
+The functions of bypass group (`readLineFromBypass` and
+`readLineFromBypassSlow`) are used for reading text from `fp` field of
+`File` static variable without updating `input` and `source` fields of
+`File`.
+
+
+Parsers may not need the functions of this group.  The functions are
+used in ctags main part. The functions are used to make pattern
+fields of tags file, for example.
+
+
+The functions of raw group
+......................................................................
+The functions of this group(`readLineRaw` and `readLineRawWithNoSeek`)
+take a parameter having type `MIO`; and don't touch `File` static
+variable.
+
+Parsers may not need the functions of this group.  The functions are
+used in ctags main part. The functions are used to load option files,
+for example.
+
+
+promise API
+......................................................................
+(Currently the tagging via promise API is disabled by default.
+ Use `--extra=+s` optoin for enabling it.)
+
+Background and Idea
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+More than one programming languages can be used in one input text stream.
+promise API is introduced for running sub (guest) parser in the specified area
+of input text stream from the root (host) parser.
+
+e.g. Code written in c language (C code) is embedded
+in code written in Yacc language (Yacc code). Let's think about this
+input stream.
+
+.. code-block:: yacc
+
+   /* foo.y */
+    %token
+	    END_OF_FILE	0
+	    ERROR		255
+	    BELL		1
+
+    %{
+    /* C language */
+    int counter;
+    %}
+    %right	EQUALS
+    %left	PLUS MINUS
+    ...
+    %%
+    CfgFile		:	CfgEntryList
+			    { InterpretConfigs($1); }
+		    ;
+
+    ...
+    %%
+    int
+    yyerror(char *s)
+    {
+	(void)fprintf(stderr,"%s: line %d of %s\n",s,lineNum,
+					    (scanFile?scanFile:"(unknown)"));
+	if (scanStr)
+	    (void)fprintf(stderr,"last scanned symbol is: %s\n",scanStr);
+	return 1;
+    }
+
+In the input the area started from `%{` to `%}` and the area started from
+the second `%%` to the end of file are written in C. Yacc can be called
+host language, and C can be called guest language.
+
+Ctags may choose the Yacc parser for the input. However, the parser
+doesn't know about C syntax. Implementing C parser in the Yacc parser
+is one of approach. However, ctags has already C parser.  The Yacc
+parser should utilize the existing C parser. The promise API allows this.
+
+Usage
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+
+See a commit titled with "Yacc: run C parser in the areas where code
+is written in C".  I applied promise API to the Yacc parser.
+
+The parser for host language must track and record the `start` and the
+`end` of a guest language. Pairs of `line number` and `byte offset`
+represents the `start` and `end`. When the `start` and `end` are
+fixed, call `makePromise` with (1) the guest parser name, (2) start,
+and (3) end. (This description is a bit simplified the real usage.)
+
+
+Let's see the actual code from parsers/yacc.c.
+
+.. code-block:: c
+
+	struct cStart {
+		unsigned long input;
+		unsigned long source;
+	};
+
+The both two fields are for recording `start`. `input` field
+is for recording the value returned from `getInputLineNumber`.
+`source` is for `getSourceLineNumber`. See `inputFile`_ for the
+difference of the two.
+
+`enter_c_prologue` shown in the next is a function called when `%{` is
+found in the current input text stream. Remember, in yacc syntax, `%{`
+is a marker of C code area.
+
+.. code-block:: c
+
+    static void enter_c_prologue (const char *line CTAGS_ATTR_UNUSED,
+				 const regexMatch *matches CTAGS_ATTR_UNUSED,
+				 unsigned int count CTAGS_ATTR_UNUSED,
+				 void *data)
+    {
+	   struct cStart *cstart = data;
+
+
+	   readLineFromInputFile ();
+	   cstart->input  = getInputLineNumber ();
+	   cstart->source = getSourceLineNumber ();
+    }
+
+
+The function just records the start line.  It calls
+`readLineFromInputFile` because the C code may start the next line of
+the line where the marker is.
+
+`leave_c_prologue` shown in the next is a function called when `%}`,
+the end marker of C code area is found in the current input text stream.
+
+.. code-block:: c
+
+    static void leave_c_prologue (const char *line CTAGS_ATTR_UNUSED,
+				 const regexMatch *matches CTAGS_ATTR_UNUSED,
+				 unsigned int count CTAGS_ATTR_UNUSED,
+				 void *data)
+    {
+	   struct cStart *cstart = data;
+	   unsigned long c_end;
+
+	   c_end = getInputLineNumber ();
+	   makePromise ("C", cstart->input, 0, c_end, 0, cstart->source);
+    }
+
+After recording the line number of the end of the C code area,
+`leave_c_prologue` calls `makePromise`.
+
+Of course "C" stands for C language, the name of guest parser.
+Available parser names can be listed by running ctags with
+`--list-languages` option. In this example two `0` characters are provided as
+the 3rd and 5th argument. They are byte offsets of the start and the end of the
+C language area from the beginning of the line which is 0 in this case. In
+general, the guest language's section does not have to start at the beginning of
+the line in which case the two offsets have to be provided. Compilers reading
+the input character by character can obtain the current offset by calling
+`getInputLineOffset()`.
+
+Internal design
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+
+A host parser cannot run a guest parser directly. What the host parser
+can do is just asking the ctags main part scheduling of running the
+guest parser for specified area which defined with the `start` and
+`end`. These scheduling requests are called promises.
+
+After running the host parser, before closing the input stream, the
+ctags main part checks the existence of promise(s). If there is, the
+main part makes a sub input stream and run the guest parser specified
+in the promise. The sub input stream is made from the original input
+stream by narrowing as requested in the promise. The main part
+iterates the above process till there is no promise.
+
+Theoretically a guest parser can make more promises. It is just
+scheduled.  However, I have never tested such case.
+
+Why not running the guest parser directly from the context of the host
+parser? Remember many parsers have their own file static variables. If
+a parser is called from the parser, the variables may be crashed.
+
+
+Automatic parser guessing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Managing regular expression parsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Parser combination
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Parser written in C
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tag entfry info
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Output tag stream
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. figure:: output-tag-stream.svg
+	    :scale: 80%
+
+Ctags provides `makeTagEntry` to parsers as an entry point for writing
+tag informations to MIO. `makeTagEntry` calls `writeTagEntry` if the
+parser does not set `useCork` field. `writeTagEntry` calls `writerWriteTag`.
+`writerWriteTag` just calls `writeEntry` of writer backends.
+`writerTable` variable holds the four backends: ctagsWriter, etagsWriter,
+xrefWriter, and jsonWriter.
+One of them is chosen depending on the arguments passed to ctags.
+
+If `useCork` is set, the tag information goes to a queue on memory.
+The queue is flushed when `useCork` in unset. See `cork API` for more
+details.
+
+cork API
+......................................................................
+
+Background and Idea
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+cork API is introduced for recording scope information easier.
+
+Before introducing cork, a scope information must be recorded as
+strings. It is flexible but memory management is required.
+Following code is taken from clojure.c(with modifications).
+
+.. code-block:: c
+
+		if (vStringLength (parent) > 0)
+		{
+			current.extensionFields.scope[0] = ClojureKinds[K_NAMESPACE].name;
+			current.extensionFields.scope[1] = vStringValue (parent);
+		}
+
+		makeTagEntry (&current);
+
+`parent`, values stored to `scope [0]` and `scope [1]` are all
+kind of strings.
+
+cork API provides more solid way to hold scope information. cork API
+expects `parent`, which represents scope of a tag(`current`)
+currently parser dealing, is recorded to a *tags* file before recording
+the `current` tag via `makeTagEntry` function.
+
+For passing the information about `parent` to `makeTagEntry`,
+`tagEntryInfo` object was created. It was used just for recording; and
+freed after recording.  In cork API, it is not freed after recording;
+a parser can reused it as scope information.
+
+How to use
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+
+See a commit titled with "clojure: use cork". I applied cork
+API to the clojure parser.
+
+cork can be enabled and disabled per parser.
+cork is disabled by default. So there is no impact till you
+enables it in your parser.
+
+`useCork` field is introduced in `parserDefinition` type:
+
+.. code-block:: c
+
+		typedef struct {
+		...
+				boolean useCork;
+		...
+		} parserDefinition;
+
+Set `TRUE` to `useCork` like:
+
+.. code-block:: c
+
+    extern parserDefinition *ClojureParser (void)
+    {
+	    ...
+	    parserDefinition *def = parserNew ("Clojure");
+	    ...
+	    def->useCork = TRUE;
+	    return def;
+    }
+
+When ctags running a parser with `useCork` being `TRUE`, all output
+requested via `makeTagEntry` function calling is stored to an internal
+queue, not to `tags` file.  When parsing an input file is done, the
+tag information stored automatically to the queue are flushed to
+`tags` file in batch.
+
+When calling `makeTagEntry` with a `tagEntryInfo` object(`parent`),
+it returns an integer. The integer can be used as handle for referring
+the object after calling.
+
+
+.. code-block:: c
+
+		static int parent = CORK_NIL;
+		...
+		parent = makeTagEntry (&e);
+
+The handle can be used by setting to a `scopeIndex`
+field of `current` tag, which is in the scope of `parent`.
+
+.. code-block:: c
+
+		current.extensionFields.scopeIndex = parent;
+
+When passing `current` to `makeTagEntry`, the `scopeIndex` is
+refereed for emitting the scope information of `current`.
+
+`scopeIndex` must be set to `CORK_NIL` if a tag is not in any scope.
+When using `scopeIndex` of `current`, `NULL` must be assigned to both
+`current.extensionFields.scope[0]` and
+`current.extensionFields.scope[1]`.  `initTagEntry` function does this
+initialization internally, so you generally you don't have to write
+the initialization explicitly.
+
+Automatic full qualified tag generation
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+
+If a parser uses the cork for recording and emitting scope
+information, ctags can reuse it for generating full qualified(FQ)
+tags. Set `requestAutomaticFQTag` field of `parserDefinition` to
+`TRUE` then the main part of ctags emits FQ tags on behalf of the parser
+if `--extra=+q` is given.
+
+An example can be found in DTS parser:
+
+.. code-block:: c
+
+    extern parserDefinition* DTSParser (void)
+    {
+	    static const char *const extensions [] = { "dts", "dtsi", NULL };
+	    parserDefinition* const def = parserNew ("DTS");
+	    ...
+	    def->requestAutomaticFQTag = TRUE;
+	    return def;
+    }
+
+Setting `requestAutomaticFQTag` to `TRUE` implies setting
+`useCork` to `TRUE`.

data/ext/vendor/ctags/docs/news.rst

@@ -0,0 +1,1362 @@
+======================================================================
+Introduced changes
+======================================================================
+
+:Maintainer: Masatake YAMATO <yamato@redhat.com>
+
+.. contents:: `Table of contents`
+	:depth: 3
+	:local:
+
+----
+
+Many changes have been introduced in Universal-ctags. Use git-log to
+review changes not enumerated here, especially in language parsers.
+
+Importing changes from Exuberant-ctags
+---------------------------------------------------------------------
+See "Exuberant-ctags" in "Tracking other projects" for detailed
+information regarding imported changes.
+
+Some changes have also been imported from Fedora and Debian.
+
+Parser related changes
+---------------------------------------------------------------------
+
+Fully rewritten parsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* C (see :ref:`The new C/C++ parser <cxx>`)
+* C++ (see :ref:`The new C/C++ parser <cxx>`)
+* Python (see :ref:`The new Python parser <python>`)
+* HTML (see :ref:`The new HTML parser <html>`)
+
+New parsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The following parsers have been added:
+
+* Ada
+* AnsiblePlaybook *libyaml*
+* Autoconf
+* Automake
+* Clojure
+* CSS
+* D
+* DBusIntrospect *libxml*
+* Diff
+* DTS
+* Falcon
+* Glade *libxml*
+* Go
+* JavaProperties
+* JSON
+* man page *optlib*
+* Maven2 *libxml*
+* M4
+* ObjectiveC
+* Perl6
+* Pod *optlib*
+* PropertiyList(plist) *libxml*
+* Protobuf
+* PythonLoggingConfig
+* R
+* RelaxNG *libxml*
+* reStructuredText
+* RpmSpec
+* Rust
+* SystemdUnit
+* SystemVerilog
+* SVG *libxml*
+* TTCN
+* WindRes
+* XSLT v1.0 *libxml*
+* Yacc
+* Yaml *libyaml*
+* YumRepo
+* Zephir
+* CoffeeScript *xcmd*
+* ctags option library *optlib*
+* Myrddin
+
+See "Option library" for details on *optlib*.
+See "External parser command" for details on *xcmd*.
+Libxml2 is required to use the parser(s) marked with *libxml*.
+Libyaml is required to use the parser(s) marked with *libyaml*.
+
+TIPS: you can list newly introduced parsers if you also have
+Exuberant-ctags installed with following command line:
+
+.. code-block:: console
+
+		$ diff -ruN <(universal-ctags --list-languages) <(exuberant-ctags --list-languages)  | grep '^[-+]'
+
+
+Heavily improved parsers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Ant (rewritten with *libxml*)
+* PHP
+* Verilog
+
+
+New options
+---------------------------------------------------------------------
+
+Wildcard in options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For the purpose of gathering as much as information as possible from
+source code the "wildcard"(``*``) option value has been introduced.
+
+``--extra=*``
+
+	Enables all extra tags.
+
+``--fields=*``
+
+	Enables all available fields.
+
+``--<LANG>-kinds=*``
+
+	Enables all available kinds for ``LANG``.
+
+``--kinds-<LANG>=*``
+
+	Alternative representation of ``--<LANG>-kinds=*``.
+
+``--*-kinds=SPEC``
+
+	Applies SPEC as kinds to all available language parsers.
+
+``--*-kinds=*``
+
+	Enables all available kinds for all available language parsers.
+
+
+Long names in kinds, fields, and extra options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A letter is used for specifying a kind, a field, or an extra entry.
+In Universal-ctags a name can also be used.
+
+Surround the name with braces (`{` and `}`) in values assigned to the
+options, ``--kind-<LANG>=``, ``--fields=``, or ``--extra=``.
+
+.. code-block:: console
+
+	$ ./ctags --kinds-C=+L-d ...
+
+This command line uses the letters, `L` for enabling the label kind
+and `d` for disabling the macro kind of C. The command line can be
+rewritten with the associated names.
+
+.. code-block:: console
+
+	$ ./ctags --kinds-C='+{label}-{macro}' ...
+
+The quotes are needed because braces are interpreted as meta
+characters by the shell.
+
+The available names can be listed with ``--list-kinds-full``,
+``--list-fields``, or ``--list-extras``.
+
+
+
+Notice messages and ``--quiet``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+There were 3 classes of message in ctags:
+
+*fatal*
+
+	A critical error has occurred and ctags aborts the execution.
+
+*warning*
+
+	An error has occurred but ctags continues the execution.
+
+*verbose*
+
+	Mainly used for debugging purposes.
+
+
+*notice* is a new class of message. It is less important than
+*warning* but more important for users than *verbose*.
+
+Generally the user can ignore *notice* class messages and ``--quiet``
+can be used to disable them.
+
+``--input-encoding=ENCODING`` and ``--output-encoding=ENCODING``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. TODO: Review...
+
+Japanese programmers sometimes use the Japanese language in source
+code comments. Of course, it is not limited to Japanese. People may
+use their own native language and in such cases encoding becomes an
+issue.
+
+ctags doesn't consider the input encoding; it just reads input as a
+sequence of bytes and uses them as is when writing tags entries.
+
+On the other hand Vim does consider input encoding. When loading a
+file, Vim converts the file contents into an internal format with one
+of the encodings specified in its `fileencodings` option.
+
+As a result of this difference, Vim cannot always move the cursor to
+the definition of a tag as users expect when attempting to match the
+patterns in a tags file.
+
+The good news is that there is a way to notify Vim of the encoding
+used in a tags file with the ``TAG_FILE_ENCODING`` pseudo tag.
+
+Two new options have been introduced (``--input-encoding=IN`` and
+``--output-encoding=OUT``).
+
+Using the encoding specified with these options ctags converts input
+from ``IN`` to ``OUT``. ctags uses the converted strings when writing
+the pattern parts of each tag line. As a result the tags output is
+encoded in ``OUT`` encoding.
+
+In addition ``OUT`` is specified at the top the tags file as the
+value for the ``TAG_FILE_ENCODING`` pseudo tag. The default value of
+``OUT`` is UTF-8.
+
+NOTE: Converted input is NOT passed to language parsers.
+The parsers still deal with input as a byte sequence.
+
+With ``--input-encoding-<LANG>=IN``, you can specify a specific input
+encoding for ``LANG``. It overrides the global default value given
+with ``--input-encoding``.
+
+The example usage can be found in *Tmain/{input,output}-encoding-option.d*.
+
+Acceptable ``IN`` and ``OUT`` values can be listed with *iconv -l* or
+*iconv --list*. It is platform dependant.
+
+To enable the option, libiconv is needed on your platform. In addition
+``--enable-iconv`` must be given to configure before making ctags.
+On Windows mingw32, you must specify ``WITH_ICONV=yes`` like this::
+
+	C:\dev\ctags>mingw32-make -f mk_mingw.mak WITH_ICONV=yes
+
+Extra tag entries (``--extra``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+These extra tag entries are newly introduced.
+
+``F``
+
+	Equivalent to --file-scope.
+
+``p``
+
+	Include pseudo tags.
+
+
+Options for inspecting ctags internals
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Exuberant-ctags provides a way to inspect its internals via
+``--list-kinds``, ``--list-languages``, and ``--list-maps``.
+
+This idea has been expanded in Universal-ctags with
+``--list-kinds-full``, ``--list-extensions``,  ``--list-extra``,
+``--list-features``, ``--list-fields``, ``--list-patterns``, and
+``--list-pseudo-tags`` being added.
+
+The original three ``--list-`` options are not changed for
+compatibility reasons, however, the newly introduced options are
+recommended for all future use.
+
+By default, interactive use is assumed and ctags tries aligning the
+list output in columns for easier reading.
+
+When ``--machinable`` is given before a ``--list-`` option, ctags
+outputs the list in a format more suitable for processing by scripts.
+Tab characters are used as separators between columns. The alignment
+of columns is never considered when ``--machinable`` is given.
+
+Currently only ``--list-extra``, ``--list-fields`` and
+``--list-kinds-full`` support ``--machinable`` output.
+
+These new ``--list-`` options also print a column header, a line
+representing the name of each column. The header may help users and
+scripts to understand and recognize the columns. Ignoring the column
+header is easy because it starts with a `#` character.
+
+``--with-list-header=no`` suppresses output of the column header.
+
+Kinds synchronization
+----------------------------------------------------------------------
+
+In Universal-ctags, as in Exuberant-ctags, most kinds are parser
+local; enabling (or disabling) a kind in a parser has no effect on
+kinds in any other parsers even those with the same name and/or
+letter.
+
+However, there are exceptions, such as C and C++ for example. C++ can
+be considered a language extended from C. Therefore it is natural
+that all kinds defined in the C parser are also defined in the C++
+parser. Enabling a kind in the C parser also enables a kind having
+the same name in the C++ parser, and vice versa.
+
+A kind group is a group of kinds satisfying the following conditions:
+
+1. Having the same name and letter, and
+2. Being synchronized with each other
+
+A master parser manages the synchronization of a kind group. The
+`MASTER` column of ``--list-kinds-full`` shows the master parser of
+the kind.
+
+Internally, a state change (enabled or disabled with
+``--kind-<LANG>=[+|-]...``) of a kind in a kind group is reported to
+its master parser as an event. Then the master parser updates the
+state of all kinds in the kind group as specified with the option.
+
+.. code-block:: console
+
+    $ ./ctags --list-kinds-full=C++
+    #LETTER NAME            ENABLED  REFONLY NROLES MASTER     DESCRIPTION
+    d       macro           on       FALSE   1      C          macro definitions
+    ...
+    $ ./ctags --list-kinds-full=C
+    #LETTER NAME            ENABLED  REFONLY NROLES MASTER     DESCRIPTION
+    d       macro           on       FALSE   1      C          macro definitions
+    ...
+
+The example output indicates that the `d` kinds of both the C++ and C
+parsers are in the same group and that the `C` parser manages the
+group. 
+
+.. code-block:: console
+
+    $ ./ctags --kinds-C++=-d --list-kinds-full=C | head -2
+    #LETTER NAME            ENABLED  REFONLY NROLES MASTER     DESCRIPTION
+    d       macro           off      FALSE   1      C          macro definitions
+    $ ./ctags --kinds-C=-d --list-kinds-full=C | head -2
+    #LETTER NAME            ENABLED  REFONLY NROLES MASTER     DESCRIPTION
+    d       macro           off      FALSE   1      C          macro definitions
+    $ ./ctags --kinds-C++=-d --list-kinds-full=C++ | head -2
+    #LETTER NAME            ENABLED  REFONLY NROLES MASTER     DESCRIPTION
+    d       macro           off      FALSE   1      C          macro definitions
+    $ ./ctags --kinds-C=-d --list-kinds-full=C++ | head -2
+    #LETTER NAME            ENABLED  REFONLY NROLES MASTER     DESCRIPTION
+    d       macro           off      FALSE   1      C          macro definitions
+
+In the above example, the `d` kind is disabled via C or C++.
+Disabling a `d` kind via one language disables the `d` kind for the
+other parser, too.
+
+
+``--put-field-prefix`` options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some fields are newly introduced in Universal-ctags and more will be
+introduced in the future. Other tags generators may also introduce
+their own fields.
+
+In such a situation there is a concern about conflicting field names;
+mixing tags files generated by multiple tags generators including
+Universal-ctags is difficult.
+
+``--put-field-prefix`` provides a workaround for this use case. When
+``--put-field-prefix`` is given, ctags adds "UCTAGS" as a prefix to
+newly introduced fields.
+
+.. code-block:: console
+
+    $ cat /tmp/foo.h
+    #include <stdio.h>
+    $ ./ctags -o - --extra=+r --fields=+r /tmp/foo.h
+    stdio.h	/tmp/foo.h	/^#include <stdio.h>/;"	h	role:system
+    $ ./ctags --put-field-prefix -o - --extra=+r --fields=+r /tmp/foo.h
+    stdio.h	/tmp/foo.h	/^#include <stdio.h>/;"	h	UCTAGSrole:system
+
+In this example, ``role`` is prefixed.
+
+``--maxdepth`` option
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``--maxdepth`` limits the depth of directory recursion enabled with
+the ``-R`` option.
+
+``--map-<LANG>`` option
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``--map-<LANG>`` is newly introduced to control the file name
+to language mappings (langmap) with finer granularity than
+``--langmap`` allows.
+
+A langmap entry is defined as a pair; the name of the language and a
+file name extension (or pattern).
+
+Here we use "spec" as a generic term representing both file name
+extensions and patterns.
+
+``--langmap`` maps specs to languages exclusively::
+
+  $ ./ctags --langdef=FOO --langmap=FOO:+.ABC \
+	    --langdef=BAR --langmap=BAR:+.ABC  \
+	    --list-maps | grep '\*.ABC$'
+  BAR      *.ABC
+
+Though language `FOO` is added before `BAR`, only `BAR` is set as a
+handler for the spec `*.ABC`.
+
+Universal-ctags enables multiple parsers to be configured for a spec.
+The appropriate parser for a given input file can then be chosen by a
+variety of internal guessing strategies (see "Choosing a proper
+parser in ctags").
+
+Let's see how specs can be mapped non-exclusively with
+``--map-<LANG>``::
+
+    % ./ctags --langdef=FOO --map-FOO=+.ABC \
+	      --langdef=BAR --map-BAR=+.ABC \
+	      --list-maps | grep '\*.ABC$'
+    FOO      *.ABC
+    BAR      *.ABC
+
+Both `FOO` and `BAR` are registered as handlers for the spec `*.ABC`.
+
+``--map-<LANG>`` can also be used for removing a langmap entry.::
+
+    $ ./ctags --langdef=FOO --map-FOO=+.ABC \
+	      --langdef=BAR --map-BAR=+.ABC \
+	      --map-FOO=-.ABC --list-maps | grep '\*.ABC$'
+    BAR      *.ABC
+
+    $ ./ctags --langdef=FOO --map-FOO=+.ABC \
+	      --langdef=BAR --map-BAR=+.ABC \
+	      --map-BAR=-.ABC --list-maps | grep '\*.ABC$'
+    FOO      *.ABC
+
+    $./ctags --langdef=FOO --map-FOO=+.ABC \
+	     --langdef=BAR --map-BAR=+.ABC \
+	     --map-BAR=-.ABC --map-FOO=-.ABC  --list-maps | grep '\*.ABC$'
+    (NOTHING)
+
+``--langmap`` provides a way to manipulate the langmap in a
+spec-centric manner and ``--map-<LANG>`` provides a way to manipulate
+the langmap in a parser-centric manner.
+
+
+Guessing parser from file contents (``-G`` option)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See "Choosing a proper parser in ctags" section.
+
+
+Enabling/disabling pseudo tags (``--pseudo-tags`` option)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each pseudo tag can be enabled/disabled with ``--pseudo-tags``.
+::
+
+	--pseudo-tags=+ptag
+	--pseudo-tags=-ptag
+
+When prefixed with `+`, the pseudo tag specified as ``ptag`` is
+enabled.  When prefixed with `-`, the pseudo tag is disabled.
+``--list-pseudo-tags`` shows all recognized ptag names.
+
+All pseudo tags are enabled if `*` is given as the value of ptag
+like::
+
+	--pseudo-tags='*'
+
+All pseudo tags are disabled if no option value is given to
+``--pseudo-tags`` like::
+
+	--pseudo-tags=
+
+To specify only a single pseudo tag, omit the sign::
+
+	--pseudo-tags=ptag
+
+JSON output
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Experimental JSON output has been added. ``--output-format`` can be
+used to enable it.
+
+.. code-block:: console
+
+   $ ./ctags --output-format=json --fields=-s /tmp/foo.py
+   {"_type": "tag", "name": "Foo", "path": "/tmp/foo.py", "pattern": "/^class Foo:$/", "kind": "class"}
+   {"_type": "tag", "name": "doIt", "path": "/tmp/foo.py", "pattern": "/^    def doIt():$/", "kind": "member"}
+
+
+See :ref:`JSON output <output-json>` for more details.
+
+
+Defining a macro in CPreProcessor input
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Newly introduced ``-D`` option extends the function provided by
+``-I`` option.
+
+``-D`` emulates the behaviour of the corresponding gcc option:
+it defines a C preprocessor macro. All types of macros are supported,
+including the ones with parameters and variable arguments.
+Stringification, token pasting and recursive macro expansion are also supported.
+
+``-I`` is now simply a backward-compatible syntax to define a
+macro with no replacement.
+
+Some examples follow.
+
+.. code-block:: console
+
+	$ ctags ... -D IGNORE_THIS ...
+
+With this commandline the following C/C++ input
+
+.. code-block:: C
+
+	int IGNORE_THIS a;
+
+will be processed as if it was
+
+.. code-block:: C
+
+	int a;
+
+Defining a macro with parameters uses the following syntax:
+
+.. code-block:: console
+
+	$ ctags ... -D "foreach(arg)=for(arg;;)" ...
+
+This example defines `for(arg;;)` as the replacement `foreach(arg)`.
+So the following C/C++ input
+
+.. code-block:: C
+
+	foreach(char * p,pointers)
+	{
+
+	}
+
+is processed in new C/C++ parser as:
+
+.. code-block:: C
+
+	for(char * p;;)
+	{
+
+	}
+
+and the p local variable can be extracted.
+
+The previous commandline includes quotes since the macros generally contain
+characters that are treated specially by the shells. You may need some escaping.
+
+Token pasting is performed by the ## operator, just like in the normal
+C preprocessor.
+
+.. code-block:: console
+
+	$ ctags ... -D "DECLARE_FUNCTION(prefix)=int prefix ## Call();"
+
+So the following code
+
+.. code-block:: C
+
+	DECLARE_FUNCTION(a)
+	DECLARE_FUNCTION(b)
+
+will be processed as
+
+.. code-block:: C
+
+	int aCall();
+	int bCall();
+
+Macros with variable arguments use the gcc __VA_ARGS__ syntax.
+
+.. code-block:: console
+
+	$ ctags ... -D "DECLARE_FUNCTION(name,...)=int name(__VA_ARGS__);"
+
+So the following code
+
+.. code-block:: C
+
+	DECLARE_FUNCTION(x,int a,int b)
+
+will be processed as
+
+.. code-block:: C
+
+	int x(int a,int b);
+
+
+Changes to the tags file format
+---------------------------------------------------------------------
+
+
+Truncating the pattern for long input lines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To prevent generating overly large tags files, a pattern field is
+truncated, by default, when its size exceeds 96 bytes. A different
+limit can be specified with ``--pattern-length-limit=N``.
+
+An input source file with long lines and multiple tag matches per
+line can generate an excessively large tags file with an
+unconstrained pattern length. For example, running ctags on a
+minified JavaScript source file often exhibits this behaviour.
+
+Reference tags
+---------------------------------------------------------------------
+
+Traditionally ctags collects the information for locating where a
+language object is DEFINED.
+
+In addition Universal-ctags supports reference tags. If the extra-tag
+``r`` is enabled, Universal-ctags also collects the information for
+locating where a language object is REFERENCED. This feature was
+proposed by @shigio in #569 for GNU GLOBAL.
+
+Here are some examples. Here is the target input file named reftag.c.
+
+.. code-block:: c
+
+    #include <stdio.h>
+    #include "foo.h"
+    #define TYPE point
+    struct TYPE { int x, y; };
+    TYPE p;
+    #undef TYPE
+
+
+Traditional output:
+
+.. code-block:: console
+
+    $ ./ctags -o - reftag.c
+    TYPE	reftag.c	/^#define TYPE /;"	d	file:
+    TYPE	reftag.c	/^struct TYPE { int x, y; };$/;"	s	file:
+    p	reftag.c	/^TYPE p;$/;"	v	typeref:typename:TYPE
+    x	reftag.c	/^struct TYPE { int x, y; };$/;"	m	struct:TYPE	typeref:typename:int	file:
+    y	reftag.c	/^struct TYPE { int x, y; };$/;"	m	struct:TYPE	typeref:typename:int	file:
+
+Output with the extra-tag ``r`` enabled:
+
+.. code-block:: console
+
+    $ ./ctags --list-extra | grep ^r
+    r	Include reference tags	off
+    $ ./ctags -o - --extra=+r reftag.c
+    TYPE	reftag.c	/^#define TYPE /;"	d	file:
+    TYPE	reftag.c	/^#undef TYPE$/;"	d	file:
+    TYPE	reftag.c	/^struct TYPE { int x, y; };$/;"	s	file:
+    foo.h	reftag.c	/^#include "foo.h"/;"	h
+    p	reftag.c	/^TYPE p;$/;"	v	typeref:typename:TYPE
+    stdio.h	reftag.c	/^#include <stdio.h>/;"	h
+    x	reftag.c	/^struct TYPE { int x, y; };$/;"	m	struct:TYPE	typeref:typename:int	file:
+    y	reftag.c	/^struct TYPE { int x, y; };$/;"	m	struct:TYPE	typeref:typename:int	file:
+
+`#undef X` and two `#include` are newly collected.
+
+A reference tag may have "role" information representing how it is
+referenced.  Universal-ctags prints the role information when the `r`
+field is enabled with ``--fields=+r``. If a tag doesn't have a
+specialized role, `generic` is used as the name of role.
+
+.. code-block:: console
+
+    $  ./ctags -o - --extra=+r --fields=+r reftag.c
+    TYPE	reftag.c	/^#define TYPE /;"	d	file:
+    TYPE	reftag.c	/^#undef TYPE$/;"	d	file:	role:undef
+    TYPE	reftag.c	/^struct TYPE { int x, y; };$/;"	s	file:
+    foo.h	reftag.c	/^#include "foo.h"/;"	h	role:local
+    p	reftag.c	/^TYPE p;$/;"	v	typeref:typename:TYPE
+    stdio.h	reftag.c	/^#include <stdio.h>/;"	h	role:system
+    x	reftag.c	/^struct TYPE { int x, y; };$/;"	m	struct:TYPE	typeref:typename:int	file:
+    y	reftag.c	/^struct TYPE { int x, y; };$/;"	m	struct:TYPE	typeref:typename:int	file:
+
+The `Reference tag marker` field, ``R``, is a specialized GNU global
+requirement; D is used for the traditional definition tags, and R is
+used for the new reference tags. The field can be used only with
+``--_xformat``.
+
+.. code-block:: console
+
+    $ ./ctags -x --_xformat="%R %-16N %4n %-16F %C" --extra=+r reftag.c
+    D TYPE                3 reftag.c         #define TYPE point
+    D TYPE                4 reftag.c         struct TYPE { int x, y; };
+    D p                   5 reftag.c         TYPE p;
+    D x                   4 reftag.c         struct TYPE { int x, y; };
+    D y                   4 reftag.c         struct TYPE { int x, y; };
+    R TYPE                6 reftag.c         #undef TYPE
+    R foo.h               2 reftag.c         #include "foo.h"
+    R stdio.h             1 reftag.c         #include <stdio.h>
+
+See :ref:`Customizing xref output <xformat>` for more details about
+this option.
+
+Although the facility for collecting reference tags is implemented,
+only a few parsers currently utilize it. All available roles can be
+listed with ``--list-roles``:
+
+.. code-block:: console
+
+    $ ./ctags --_list-roles
+    C	d	undef	undefined	on
+    C	h	system	system header	on
+    C	h	local	local header	on
+    C++	d	undef	undefined	on
+    C++	h	system	system header	on
+    C++	h	local	local header	on
+    DTS	d	undef	undefined	on
+    DTS	h	system	system header	on
+    DTS	h	local	local header	on
+    Make	I	included	included	on
+    Make	I	optional	optionally included	on
+    Sh	s	loaded	loaded	on
+    Vera	d	undef	undefined	on
+    Vera	h	system	system header	on
+    Vera	h	local	local header	on
+
+The first column shows the name of the parser.
+The second column shows the name of the kind.
+The third column shows the name of the role.
+The fourth column shows the description of the role.
+The fifth column shows whether the role is enabled or not.
+
+Currently ctags doesn't provide the means for disabling a
+specified role.
+
+
+Automatic parser selection
+---------------------------------------------------------------------
+
+See "Choosing a proper parser in ctags" section.
+
+
+Incompatible changes to file name pattern and extension handling
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When guessing a proper parser for a given input file, Exuberant-ctags
+tests file name patterns AFTER file extensions (e-order).
+Universal-ctags does this differently; it tests file name patterns
+BEFORE file extensions (u-order).
+
+This incompatible change is introduced to deal with the following
+situation: "build.xml" is an input file. The Ant parser declares it
+handles a file name pattern "build.xml" and another parser, Foo,
+declares it handles a file extension "xml".
+
+Which parser should be used for parsing the input? The user may want
+to use the Ant parser because the pattern it declares is more
+specific than the extension Foo declares. However, in e-order, the
+other parser, Foo, is chosen.
+
+So Universal-ctags uses the u-order even though it introduces an
+incompatibility.
+
+
+Pseudo tags
+---------------------------------------------------------------------
+
+Pseudo tags are used to add meta data to a tags file. Universal-ctags
+will utilize pseudo tags aggressively.
+
+Universal-ctags is not mature yet; there is a possibility that
+incompatible changes will be introduced. As a result tools reading
+a tags file may not work as expected.
+
+To mitigate this issue pseudo tags are employed to make a tags file
+more self-descriptive. We hope some of the incompatibilities can be
+overcome in client tools by utilizing this approach.
+
+Example output:
+
+.. code-block:: console
+
+    $ ./ctags -o - --extra=p --pseudo-tags='TAG_KIND_DESCRIPTION' foo.c
+    !_TAG_KIND_DESCRIPTION!C	L,label	/goto label/
+    !_TAG_KIND_DESCRIPTION!C	c,class	/classes/
+    !_TAG_KIND_DESCRIPTION!C	d,macro	/macro definitions/
+    !_TAG_KIND_DESCRIPTION!C	e,enumerator	/enumerators (values inside an enumeration)/
+    !_TAG_KIND_DESCRIPTION!C	f,function	/function definitions/
+    !_TAG_KIND_DESCRIPTION!C	g,enum	/enumeration names/
+    !_TAG_KIND_DESCRIPTION!C	h,header	/included header files/
+    !_TAG_KIND_DESCRIPTION!C	l,local	/local variables/
+    !_TAG_KIND_DESCRIPTION!C	m,member	/class, struct, and union members/
+    !_TAG_KIND_DESCRIPTION!C	n,namespace	/namespaces/
+    !_TAG_KIND_DESCRIPTION!C	p,prototype	/function prototypes/
+    !_TAG_KIND_DESCRIPTION!C	s,struct	/structure names/
+    !_TAG_KIND_DESCRIPTION!C	t,typedef	/typedefs/
+    !_TAG_KIND_DESCRIPTION!C	u,union	/union names/
+    !_TAG_KIND_DESCRIPTION!C	v,variable	/variable definitions/
+    !_TAG_KIND_DESCRIPTION!C	x,externvar	/external and forward variable declarations/
+    foo	foo.c	/^foo (int i, int j)$/;"	f
+    main	foo.c	/^main (void)$/;"	f
+
+
+``TAG_KIND_DESCRIPTION``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a newly introduced pseudo tag. It is not emitted by default.
+It is emitted only when ``--pseudo-tags=+TAG_KIND_DESCRIPTION`` is
+given.
+
+This is for describing kinds; their letter, name, and description are
+enumerated in the tag.
+
+ctags emits ``TAG_KIND_DESCRIPTION`` with following format::
+
+	!_TAG_KIND_SEPARATOR!{parser}	{letter},{name}	/{description}/
+
+A backslash and a slash in {description} is escaped with a backslash.
+
+
+``TAG_KIND_SEPARATOR``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a newly introduced pseudo tag. It is not emitted by default.
+It is emitted only when ``--pseudo-tags=+TAG_KIND_SEPARATOR`` is
+given.
+
+This is for describing separators placed between two kinds in a
+language.
+
+Tag entries including the separators are emitted when ``--extra=+q``
+is given; fully qualified tags contain the separators. The separators
+are used in scope information, too.
+
+ctags emits ``TAG_KIND_SEPARATOR`` with following format::
+
+	!_TAG_KIND_SEPARATOR!{parser}	{sep}	/{upper}{lower}/
+
+or ::
+
+	!_TAG_KIND_SEPARATOR!{parser}	{sep}	/{lower}/
+
+Here {parser} is the name of language. e.g. PHP.
+{lower} is the letter representing the kind of the lower item.
+{upper} is the letter representing the kind of the upper item.
+{sep} is the separator placed between the upper item and the lower
+item.
+
+The format without {upper} is for representing a root separator. The
+root separator is used as prefix for an item which has no upper scope.
+
+`*` given as {upper} is a fallback wild card; if it is given, the
+{sep} is used in combination with any upper item and the item
+specified with {lower}.
+
+Each backslash character used in {sep} is escaped with an extra
+backslash character.
+
+Example output:
+
+.. code-block:: console
+
+    $ ./ctags -o - --extra=+p --pseudo-tags=  --pseudo-tags=+TAG_KIND_SEPARATOR input.php
+    !_TAG_KIND_SEPARATOR!PHP	::	/*c/
+    ...
+    !_TAG_KIND_SEPARATOR!PHP	\\	/c/
+    ...
+    !_TAG_KIND_SEPARATOR!PHP	\\	/nc/
+    ...
+
+The first line means `::` is used when combining something with an
+item of the class kind.
+
+The second line means `\\` is used when a class item is at the top
+level; no upper item is specified.
+
+The third line means `\\` is used when for combining a namespace item
+(upper) and a class item (lower).
+
+Of course, ctags uses the more specific line when choosing a
+separator; the third line has higher priority than the first.
+
+
+Parser own fields
+---------------------------------------------------------------------
+
+A tag has a `name`, an `input` file name, and a `pattern` as basic
+information. Some fields like `language:`, `signature:`, etc are
+attached to the tag as optional information.
+
+In Exuberant-ctags, fields are common to all languages.
+Universal-ctags extends the concept of fields; a parser can define
+its own field. This extension was proposed by @pragmaware in #857.
+
+For implementing the parser own fields, the options for listing and
+enabling/disabling fields are also extended.
+
+In the output of ``--list-fields``, the owner of the field is printed
+in the `LANGUAGE` column:
+
+.. code-block:: console
+
+	$ ./ctags --list-fields
+	#LETTER NAME            ENABLED LANGUAGE         XFMT  DESCRIPTION
+	...
+	-       end             off     C                TRUE   end lines of various constructs
+	-       properties      off     C                TRUE   properties (static, inline, mutable,...)
+	-       end             off     C++              TRUE   end lines of various constructs
+	-       template        off     C++              TRUE   template parameters
+	-       captures        off     C++              TRUE   lambda capture list
+	-       properties      off     C++              TRUE   properties (static, virtual, inline, mutable,...)
+	-       sectionMarker   off     reStructuredText TRUE   character used for declaring section
+	-       version         off     Maven2           TRUE   version of artifact
+
+e.g. reStructuredText is the owner of the sectionMarker field and
+both C and C++ own the end field.
+
+``--list-fields`` takes one optional argument, `LANGUAGE`. If it is
+given, ``--list-fields`` prints only the fields for that parser:
+
+.. code-block:: console
+
+	$ ./ctags --list-fields=Maven2
+	#LETTER NAME            ENABLED LANGUAGE        XFMT  DESCRIPTION
+	-       version         off     Maven2          TRUE  version of artifact
+
+A parser own field only has a long name, no letter. For
+enabling/disabling such fields, the name must be passed to
+``--fields-<LANG>``.
+
+e.g. for enabling the `sectionMarker` field owned by the
+`reStructuredText` parser, use the following command line:
+
+.. code-block:: console
+
+	$ ./ctags --fields-reStructuredText=+{sectionMarker} ...
+
+The wild card notation can be used for enabling/disabling parser own
+fields, too. The following example enables all fields owned by the
+`C++` parser.
+
+.. code-block:: console
+
+	$ ./ctags --fields-C++='*' ...
+
+`*` can also be used for specifying languages.
+
+The next example is for enabling `end` fields for all languages which
+have such a field.
+
+.. code-block:: console
+
+	$ ./ctags --fields-'*'=+'{end}' ...
+	...
+
+In this case, using wild card notation to specify the language, not
+only fields owned by parsers but also common fields having the name
+specified (`end` in this example) are enabled/disabled.
+
+Using the wild card notation to specify the language is helpful to
+avoid incompatibilities between versions of Universal-ctags itself
+(SELF INCOMPATIBLY).
+
+In Universal-ctags development, a parser developer may add a new
+parser own field for a certain language.  Sometimes other developers
+then recognize it is meaningful not only for the original language
+but also other languages. In this case the field may be promoted to a
+common field. Such a promotion will break the command line
+compatibility for ``--fields-<LANG>`` usage. The wild card for
+`<LANG>` will help in avoiding this unwanted effect of the promotion.
+
+With respect to the tags file format, nothing is changed when
+introducing parser own fields; `<fieldname>`:`<value>` is used as
+before and the name of field owner is never prefixed. The `language:`
+field of the tag identifies the owner.
+
+Parser own parameter
+---------------------------------------------------------------------
+
+.. NOT REVIEWED YET
+
+To control the detail of a parser, ``--param-<LANG>`` option.
+``--kinds-<LANG>``, ``--fields-<LANG>``, ``--extra-<LANG>`` (not implemented yet)
+can be used for customizing the behavior of a parser specified with ``<LANG>``.
+
+``--param-<LANG>`` should be used other aspects of the parser other
+than the options(kinds, fields, etras).
+
+A parser defines a set of parameters. Each parameter has name and
+takes an argument. A user can set a parameter with following notation
+::
+
+   --param-<LANG>:name=arg
+
+An example of specifying a parameter
+::
+
+   --param-CPreProcessor:if0=true
+
+Here `if0` is a name of parameter of CPreProcessor parser and
+`true` is the value of it.
+
+All available parameters can be listed with ``--list-params`` option.
+
+.. code-block:: console
+
+    $ ./ctags --list-params
+    #PARSER         NAME     DESCRIPTION
+    CPreProcessor   if0      examine code within "#if 0" branch (true or [false])
+    CPreProcessor   ignore   a token to be specially handled
+
+(At this time only CPreProcessor parser has parameters.)
+
+
+.. _xformat:
+
+Customizing xref output
+---------------------------------------------------------------------
+
+``--_xformat`` option allows a user to customize the cross reference
+(xref) output enabled with ``-x``.
+::
+
+   --_xformat=FORMAT
+
+
+The notation for FORMAT is similar to that employed by `printf(3)` in
+the C language; `%` represents a slot which is substituted with a
+field value when printing. You can specify multiple slots in FORMAT.
+Here field means an item listed with ``-list-fields`` option.
+
+The notation of a slot::
+
+   %[WIDTH-AND-ADJUSTMENT]FIELD-SPECIFIER
+
+``FIELD-SPECIFIER`` specifies a field whose value is printed.
+Short notation and long notation are available. They can be mixed
+in a FORMAT. Specifying a field with either notation, one or more
+fields are activated internally.
+
+The short notation is just a letter listed in the LETTER column of
+the ``--list-fields`` output.
+
+The long notation is a name string surrounded by braces(`{` and
+`}`). The name string is listed in the NAME column of the output of
+the same option. To specify a field owned by a parser, prepend
+the parser name to the name string with `.` as a separator.
+
+Wild card (`*`) can be used where a parser name is specified. In this
+case both common and parser own fields are activated and printed.
+If a common field and a parser own field have the same name,
+the common field has higher priority.
+
+`WIDTH-AND-ADJUSTMENT` is a positive or negative number.
+The absolute value of the number is used as the width of
+the column where a field is printed. The printing is
+right adjusted when a positive value is given, and left
+adjusted when negative.
+
+An example of specifying common fields:
+
+.. code-block:: console
+
+    $  ./ctags -x --_xformat="%-20N %4n %-16{input}|" main/main.c | head
+    CLOCKS_PER_SEC        360 main/main.c     |
+    CLOCKS_PER_SEC        364 main/main.c     |
+    CLOCK_AVAILABLE       358 main/main.c     |
+    CLOCK_AVAILABLE       363 main/main.c     |
+    Totals                 87 main/main.c     |
+    __anonae81ef0f0108     87 main/main.c     |
+    addTotals             100 main/main.c     |
+    batchMakeTags         436 main/main.c     |
+    bytes                  87 main/main.c     |
+    clock                 365 main/main.c     |
+
+Here `%-20N %4n %-16{input}|` is a format string. Let's look at the
+elements of the format.
+
+`%-20N`
+
+	The short notation is used here.
+	The element means filling the slot with the name of the tag.
+	The width of the column is 20 characters and left adjusted.
+
+`%4n`
+
+	The short notation is used here.
+	The element means filling the slot with the line number of
+	the tag. The width of the column is 4 characters and right
+        adjusted.
+
+`%-16{input}`
+
+	The long notation is used here.
+	The element means filling the slot with the input file name
+	where the tag is defined. The width of column is 16
+        characters and left adjusted.
+
+`|`
+
+	Printed as is.
+
+Another example of specifying parser own fields:
+
+.. code-block:: console
+
+	$  ./ctags -x --_xformat="%-20N [%10{C.properties}]" main/main.c
+	CLOCKS_PER_SEC       [          ]
+	CLOCK_AVAILABLE      [          ]
+	Totals               [          ]
+	__anonae81ef0f0108   [          ]
+	addTotals            [    extern]
+	batchMakeTags        [    static]
+	bytes                [          ]
+	clock                [          ]
+	clock                [    static]
+	...
+
+Here `"%-20N [%10{C.properties}]"` is a format string. Let's look at
+the elements of the format.
+
+`%-20N`
+
+	Already explained in the first example.
+
+`[` and `]`
+
+	Printed as is.
+
+`%10{C.properties}`
+
+	The long notation is used here.
+	The element means filling the slot with the value
+	of the properties field of the C parser.
+	The width of the column is 10 characters and right adjusted.
+
+
+.. TODO: An example of using WILDCARD
+
+
+Incompatible changes in command line
+---------------------------------------------------------------------
+
+.. NOT REVIEWED YET
+
+``-D`` option
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For ctags buinary that debugging output is enabled in build config
+stage, ``-D`` was used for specifying the level of debugging
+output. It is changed to ``-d``. This change is not critical because
+``-D`` option was not described in ctags.1 man page.
+
+Instead ``-D`` is used for defining a macro in CPreProcessor parser.
+
+
+Readtags
+---------------------------------------------------------------------
+
+Printing line numbers with ``-n``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+If both ``-e`` and ``-n`` are given, readtags prints the `line:`
+field.
+
+
+Filtering in readtags command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+readtags has ability to find tag entries by name.
+
+The concept of filtering is inspired by the display filter of
+Wireshark. You can specify more complex conditions for searching.
+Currently this feature is available only on platforms where
+`fmemopen` is available as part of libc. Filtering in readtags is an
+experimental feature.
+
+The syntax of filtering rules is based on the Scheme language, a
+variant of Lisp. The language has prefix notation and parentheses.
+
+Before printing an entry from the tags file, readtags evaluates an
+expression (S expression or sexp) given as an option argument to
+``-Q``. As the result of the evaluation, readtags gets a value. false
+represented as `#f`, indicates rejection: readtags doesn't print it.
+
+::
+
+   SEXP =
+	LIST
+	INTEGER
+	BOOLEAN
+	STRING
+	SYMBOL
+
+	LIST = ( SEXP... ) | ()
+	INTEGER = [0-9]+
+	BOOLEAN = #t | #f
+	STRING  = "..."
+	SYMBOL  = null?
+		    and
+		     or
+		    not
+		    eq?
+		      <
+		      >
+		     <=
+		     >=
+		prefix?
+		suffix?
+		substr?
+		 member
+		      $
+		  $name
+		 $input
+		$access
+		  $file
+	      $language
+	$implementation
+		  $line
+		  $kind
+		  $role
+	       $pattern
+	      $inherits
+	    $scope-kind
+	    $scope-name
+		   $end
+
+All symbols starting with `$` represent a field of a tag entry which
+is being tested against the S expression. Most will evaluate as a
+string or `#f`. It evaluates to `#f` when the field doesn't exist.
+`$inherits` is evaluated to a list of strings if the entry has an
+`inherits` field. The `scope` field holds structured data: the kind
+and name of the upper scope combined with `:`. The kind part is
+mapped to `$scope-kind`, and the name part to `$scope-name`.
+
+`$scope-kind` and `$scope-name` can only be used if the input tags
+file is generated by ctags with ``--fields=+Z``.
+
+All symbols not prefixed with `$` are operators. When using these,
+put them at the head(car) of list. The rest(cdr) of the list is
+passed to the operator as arguments. Many of them are also available
+in the Scheme language; see the other documents.
+
+prefix?, suffix?, and substr? may only be available in this
+implementation. All of them take two strings. The first one
+is called the target.
+
+The exception in the above naming convention is the `$` operator.
+`$` is a generic accessor for accessing extension fields.
+`$` takes one argument: the name of an extension field.
+It returns the value of the field as a string if a value
+is given, or `#f`.
+
+::
+
+	(prefix? "TARGET" "TA")
+	=> #t
+
+	(prefix? "TARGET" "RGET")
+	=> #f
+
+	(prefix? "TARGET" "RGE")
+	=> #f
+
+	(suffix? "TARGET" "TA")
+	=> #f
+
+	(suffix? "TARGET" "RGET")
+	=> #t
+
+	(suffix? "TARGET" "RGE")
+	=> #f
+
+	(substr? "TARGET" "TA")
+	=> #t
+
+	(suffix? "TARGET" "RGET")
+	=> #t
+
+	(suffix? "TARGET" "RGE")
+	=> #t
+
+	(and (suffix? "TARGET" "TARGET")
+	     (prefix? "TARGET" "TARGET")
+	     (substr? "TARGET" "TARGET")
+	=> #t
+
+
+Let's see examples.
+
+Examples of input
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Create the tags file (*foo.tags*) with following command line
+
+.. code-block:: console
+
+	$ ./ctags --fields='*' --extra='*' -o foo.tags foo.py
+
+for following input (*foo.py*)
+
+.. code-block:: python
+
+    class Foo:
+	def aq ():
+	    pass
+	def aw ():
+	    pass
+	def ae ():
+	    pass
+	class A:
+	    pass
+    class Bar (Foo):
+	def bq ():
+	    pass
+	def bw ():
+	    pass
+	class B:
+	    pass
+
+    class Baz (Foo):
+	def bq ():
+	    pass
+	def bw ():
+	    pass
+	class C:
+	    pass
+
+Examples of filter expressions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Print entries ending with "q"
+
+  .. code-block:: console
+
+	$ ./readtags -e -t foo.tags -Q '(suffix? $name "q")' -l
+	Bar.bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Bar	access:public	signature:()
+	Baz.bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+	Foo.aq	foo.py	/^    def aq ():$/;"	kind:member	language:Python	scope:class:Foo	access:public	signature:()
+	aq	foo.py	/^    def aq ():$/;"	kind:member	language:Python	scope:class:Foo	access:public	signature:()
+	bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Bar	access:public	signature:()
+	bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+
+* Print members of Baz
+
+  .. code-block:: console
+
+	$ ./readtags -e -t foo.tags -Q '(and (eq? $kind "member") (eq? "Baz" $scope-name))' -l
+	Baz.bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+	Baz.bw	foo.py	/^    def bw ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+	bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+	bw	foo.py	/^    def bw ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+
+* Print only fully qualified entries (assuming "." is used as the separator)
+
+  .. code-block:: console
+
+	$ ./readtags -e -t foo.tags -Q '(and (eq? $kind "member") (substr? $name "."))' -l
+	Bar.bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Bar	access:public	signature:()
+	Bar.bw	foo.py	/^    def bw ():$/;"	kind:member	language:Python	scope:class:Bar	access:public	signature:()
+	Baz.bq	foo.py	/^    def bq ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+	Baz.bw	foo.py	/^    def bw ():$/;"	kind:member	language:Python	scope:class:Baz	access:public	signature:()
+	Foo.ae	foo.py	/^    def ae ():$/;"	kind:member	language:Python	scope:class:Foo	access:public	signature:()
+	Foo.aq	foo.py	/^    def aq ():$/;"	kind:member	language:Python	scope:class:Foo	access:public	signature:()
+	Foo.aw	foo.py	/^    def aw ():$/;"	kind:member	language:Python	scope:class:Foo	access:public	signature:()
+
+* Print only classes inheriting Foo
+
+  .. code-block:: console
+
+	$ ./readtags  -e -t foo.tags -Q '(and (member "Foo" $inherits) (eq? $kind "class"))' -l
+	Bar	foo.py	/^class Bar (Foo):$/;"	kind:class	language:Python	inherits:Foo	access:public
+	Baz	foo.py	/^class Baz (Foo): $/;"	kind:class	language:Python	inherits:Foo	access:public