rroonga 5.0.0-x64-mingw32 → 5.0.1-x64-mingw32

data/vendor/local/share/doc/groonga/ja/html/_sources/reference/regular_expression.txt

@@ -0,0 +1,403 @@
+.. -*- rst -*-
+
+.. highlightlang:: none
+
+.. groonga-command
+.. database: regular_expression
+
+Regular expression
+==================
+
+Summary
+-------
+
+.. note::
+
+   Regular expression support is an experimental feature.
+
+.. versionadded:: 5.0.1
+
+Groonga supports pattern match by regular expression. Regular
+expression is widely used format to describe a pattern. Regular
+expression is useful to represent complex pattern.
+
+In most cases, pattern match by regular expression is evaluated as
+sequential search. It'll be slow for many records and many texts.
+
+In some cases, pattern match by regular expression can be evaluated
+by index. It's very fast rather than sequential search. Patterns
+that can be evaluated by index are described later.
+
+There are many regular expression syntaxes. Groonga uses the same
+syntax in Ruby. Because Groonga uses the same regular expression
+engine as Ruby. The regular expression engine is `Onigmo
+<https://github.com/k-takata/Onigmo/>`_. Characteristic difference
+with other regular expression syntax is ``^`` and ``$``. The regular
+expression syntax in Ruby, ``^`` means the beginning of line and ``$``
+means the end of line. ``^`` means the beginning of text and ``$``
+means the end of text in other most regular expression syntaxes. The regular
+expression syntax in Ruby uses ``\A`` for the beginning of text and
+``\z`` for the end of text.
+
+You can use regular expression in :ref:`select-query` and
+:ref:`select-filter` options of :doc:`/reference/commands/select`
+command.
+
+Usage
+-----
+
+Here are a schema definition and sample data to show usage. There is
+only one table, ``Logs``. ``Logs`` table has only ``message``
+column. Log messages are stored into the ``message`` column.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/usage_setup.log
+.. table_create Logs TABLE_NO_KEY
+.. column_create Logs message COLUMN_SCALAR Text
+..
+.. load --table Logs
+.. [
+.. {"message": "host1:[error]: No memory"},
+.. {"message": "host1:[warning]: Remained disk space is less than 30%"},
+.. {"message": "host1:[error]: Disk full"},
+.. {"message": "host2:[error]: No memory"},
+.. {"message": "host2:[info]: Shutdown"}
+.. ]
+
+Here is an example that uses regular expression in
+:ref:`select-query`. You need to use
+``${COLUMN}:~${REGULAR_EXPRESSION}`` syntax.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/usage_query.log
+.. select Logs --query 'message:~"[Dd]isk"'
+
+Here is an example that uses regular expression in
+:ref:`select-filter`. You need to use
+``${COLUMN} @~ ${REGULAR_EXPRESSION}`` syntax.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/usage_filter.log
+.. select Logs --filter 'message @~ "[Dd]isk"'
+
+Index
+-----
+
+Groonga can search records by regular expression with index. It's very
+fast rather than sequential search.
+
+But it doesn't support all regular expression patterns. It supports
+only the following regular expression patterns. The patterns will be
+increased in the future.
+
+  * Literal only pattern such as ``disk``
+  * The begging of text and literal only pattern such as ``\Adisk``
+  * The end of text and literal only pattern such as ``disk\z``
+
+You need to create an index for fast regular expression search. Here
+are requirements of index:
+
+  * Lexicon must be :ref:`table-pat-key` table.
+  * Lexicon must use :ref:`token-regexp` tokenizer.
+  * Index column must has ``WITH_POSITION`` flag.
+
+Other configurations such as lexicon's normalizer are optional. You
+can choose what you like. If you want to use case-insensitive search,
+use :ref:`normalizer-auto` normalizer.
+
+Here are recommended index definitions. In general, it's reasonable
+index definitions.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/index_definitions.log
+.. table_create RegexpLexicon TABLE_PAT_KEY ShortText \
+..   --default_tokenizer TokenRegexp \
+..   --normalizer NormalizerAuto
+.. column_create RegexpLexicon logs_message_index \
+..   COLUMN_INDEX|WITH_POSITION Logs message
+
+Now, you can use index for regular expression search. The following
+regular expression can be evaluated by index because it uses only "the
+beginning of text" and "literal".
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/search_by_index_query.log
+.. select Logs --query message:~\\\\Ahost1
+
+Here is an example that uses :ref:`select-filter` instead of
+:ref:`select-query`. It uses the same regular expression as the
+previous example.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/search_by_index_filter.log
+.. select Logs --filter 'message @~ "\\\\Ahost1:"'
+
+``\`` escape will confuse you because there are some steps that
+require escape between you and Groonga. Here are steps that require
+``\`` escape:
+
+  * Shell only when you pass Groonga command from command line the
+    following::
+
+      % groonga /tmp/db select Logs --filter '"message @~ \"\\\\Ahost1:"\"'
+
+    ``--filter '"message @~ \"\\\\Ahost1:\""'`` is evaluated as the
+    following two arguments by shell:
+
+      * ``--filter``
+      * ``"message @~ \"\\\\Ahost1:\""``
+
+  * Groonga command parser only when you pass Groonga command by
+    command line style (``COMMAND ARG1_VALUE ARG2_VALUE ...``) not
+    HTTP path style
+    (``/d/COMMAND?ARG1_NAME=ARG1_VALUE&ARG2_NAME=ARG3_VALUE``).
+
+    ``"message @~ \"\\\\Ahost1:\""`` is evaluated as the following
+    value by Groonga command parser:
+
+      * ``message @~ "\\Ahost1:"``
+
+  * :doc:`/reference/grn_expr` parser. ``\`` escape is required in both
+    :doc:`/reference/grn_expr/query_syntax` and
+    :doc:`/reference/grn_expr/script_syntax`.
+
+    ``"\\Ahost1:"`` string literal in script syntax is evaluated as
+    the following value:
+
+      * ``\Ahost1``
+
+    The value is evaluated as regular expression.
+
+Syntax
+------
+
+This section describes about only commonly used syntaxes. See `Onigmo
+syntax documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE>`_ for other
+syntaxes and details.
+
+.. _regular-expression-escape:
+
+Escape
+^^^^^^
+
+In regular expression, there are the following special characters:
+
+  * ``\``
+  * ``|``
+  * ``(``
+  * ``)``
+  * ``[``
+  * ``]``
+  * ``.``
+  * ``*``
+  * ``+``
+  * ``?``
+  * ``{``
+  * ``}``
+  * ``^``
+  * ``$``
+
+If you want to write pattern that matches these special character as
+is, you need to escape them.
+
+You can escape them by putting ``\`` before special character. Here
+are regular expressions that match special character itself:
+
+  * ``\\``
+  * ``\|``
+  * ``\(``
+  * ``\)``
+  * ``\[``
+  * ``\]``
+  * ``\.``
+  * ``\*``
+  * ``\+``
+  * ``\?``
+  * ``\{``
+  * ``\}``
+  * ``\^``
+  * ``\$``
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/choice.log
+.. select Logs --filter 'message @~ "\\\\[error\\\\]"'
+
+If your regular expression doesn't work as you expected, confirm that
+some special characters are used without escaping.
+
+.. _regular-expression-choice:
+
+Choice
+^^^^^^
+
+Choice syntax is ``A|B``. The regular expression matches when either
+``A`` pattern or ``B`` pattern is matched.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/choice.log
+.. select Logs --filter 'message @~ "warning|info"'
+
+.. caution::
+
+   Regular expression that uses this syntax can't be evaluated by
+   index.
+
+.. _regular-expression-group:
+
+Group
+^^^^^
+
+Group syntax is ``(...)``. Group provides the following features:
+
+  * Back reference
+  * Scope reducing
+
+You can refer matched groups by ``\n`` (``n`` is the group number)
+syntax. For example, ``e(r)\1o\1`` matches ``error``. Because ``\1``
+is replaced with match result (``r``) of the first group ``(r)``.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/group_back_reference.log
+.. select Logs --filter 'message @~ "e(r)\\\\1o\\\\1"'
+
+You can also use more powerful back reference features. See `"8. Back
+reference" section in Onigmo documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE#L302>`_ for
+details.
+
+Group syntax reduces scope. For example, ``\[(warning|info)\]``
+reduces choice syntax scope. The regular expression matches
+``[warning]`` and ``[info]``.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/group_scope_reducing.log
+.. select Logs --filter 'message @~ "\\\\[(warning|info)\\\\]"'
+
+You can also use more powerful group related features. See
+`"7. Extended groups" section in Onigmo documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE#L225>`_ for
+details.
+
+.. caution::
+
+   Regular expression that uses this syntax can't be evaluated by
+   index.
+
+.. _regular-expression-character-class:
+
+Character class
+^^^^^^^^^^^^^^^
+
+Character class syntax is ``[...]``. Character class is useful to
+specify multiple characters to be matched.
+
+For example, ``[Dd]`` matches ``D`` or ``d``.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/character_class_characters.log
+.. select Logs --filter 'message @~ "[Dd]isk"'
+
+You can specify characters by range. For example, ``[0-9]`` matches
+one digit.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/character_class_range.log
+.. select Logs --filter 'message @~ "[0-9][0-9]%"'
+
+You can also use more powerful character class related features. See
+`"6. Character class" section in Onigmo documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE#L164>`_ for
+details.
+
+.. caution::
+
+   Regular expression that uses this syntax can't be evaluated by
+   index.
+
+.. _regular-expression-anchor:
+
+Anchor
+^^^^^^
+
+There are the following commonly used anchor syntaxes. Some anchors
+can be evaluated by index.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Anchor
+     - Description
+     - Index ready
+   * - ``^``
+     - The beginning of line
+     - o
+   * - ``$``
+     - The end of line
+     - x
+   * - ``\A``
+     - The beginning of text
+     - o
+   * - ``\z``
+     - The end of text
+     - x
+
+Here is an example that uses ``\z``.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/anchor_z.log
+.. select Logs --filter 'message @~ "%\\\\z"'
+
+You can also use more anchors. See `"5. Anchors" section in Onigmo
+documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE#L152>`_ for
+details.
+
+.. caution::
+
+   Regular expression that uses this syntax except ``\A`` and ``\z``
+   can't be evaluated by index.
+
+.. _regular-expression-quantifier:
+
+Quantifier
+^^^^^^^^^^
+
+There are the following commonly used quantifier syntaxes.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Quantifier
+     - Description
+   * - ``?``
+     - 0 or 1 time
+   * - ``*``
+     - 0 or more times
+   * - ``+``
+     - 1 or more times
+
+For example, ``er+or`` matches ``error``, ``errror`` and so on.
+
+.. groonga-command
+.. include:: ../example/reference/regular_expression/quantifier_plus.log
+.. select Logs --filter 'message @~ "er+or"'
+
+You can also use more quantifiers. See `"4. Quantifier" section in Onigmo
+documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE#L119>`_ for
+details.
+
+.. caution::
+
+   Regular expression that uses this syntax can't be evaluated by
+   index.
+
+Others
+^^^^^^
+
+There are more syntaxes. If you're interested in them, see `Onigmo
+documentation
+<https://github.com/k-takata/Onigmo/blob/master/doc/RE>`_ for
+details. You may be interested in "character type" and "character"
+syntaxes.

data/vendor/local/share/doc/groonga/ja/html/_sources/reference/scorer.txt

@@ -0,0 +1,217 @@
+.. -*- rst -*-
+
+.. highlightlang:: none
+
+.. groonga-command
+.. database: scorer
+
+Scorer
+======
+
+Summary
+-------
+
+Groonga has scorer module that customizes score function. Score
+function computes score of matched record. The default scorer function
+uses the number of appeared terms. It is also known as TF (term
+frequency).
+
+TF is a fast score function but it's not suitable for the following
+cases:
+
+  * Search query contains one or more frequently-appearing words such
+    as "the" and "a".
+  * Document contains many same keywords such as "They are keyword,
+    keyword, keyword ... and keyword". Search engine spammer may use
+    the technique.
+
+Score function can solve these cases. For example, `TF-IDF
+<http://en.wikipedia.org/wiki/Tf%E2%80%93idf>`_ (term
+frequency-inverse document frequency) can solve the first case.
+`Okapi BM25 <http://en.wikipedia.org/wiki/Okapi_BM25>`_ can solve the
+second case. But their are slower than TF.
+
+Groonga provides TF-IDF based scorer as
+:doc:`/reference/scorers/scorer_tf_idf` but doesn't provide Okapi BM25
+based scorer yet.
+
+.. include:: scoring_note.rst
+
+Usage
+-----
+
+This section describes how to use scorer.
+
+Here are a schema definition and sample data to show usage.
+
+Sample schema:
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_setup_schema.log
+.. table_create Memos TABLE_HASH_KEY ShortText
+.. column_create Memos title COLUMN_SCALAR ShortText
+.. column_create Memos content COLUMN_SCALAR Text
+..
+.. table_create Terms TABLE_PAT_KEY ShortText \
+..   --default_tokenizer TokenBigram \
+..   --normalizer NormalizerAuto
+.. column_create Terms title_index COLUMN_INDEX|WITH_POSITION Memos title
+.. column_create Terms content_index COLUMN_INDEX|WITH_POSITION Memos content
+
+Sample data:
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_setup_data.log
+.. load --table Memos
+.. [
+.. {
+..   "_key": "memo1",
+..   "title": "Groonga is easy",
+..   "content": "Groonga is very easy full text search engine!"
+.. },
+.. {
+..   "_key": "memo2",
+..   "title": "Mroonga is easy",
+..   "content": "Mroonga is more easier full text search engine!"
+.. },
+.. {
+..   "_key": "memo3",
+..   "title": "Rroonga is easy",
+..   "content": "Ruby is very helpful."
+.. },
+.. {
+..   "_key": "memo4",
+..   "title": "Groonga is fast",
+..   "content": "Groonga! Groonga! Groonga! Groonga is very fast!"
+.. },
+.. {
+..   "_key": "memo5",
+..   "title": "PGroonga is fast",
+..   "content": "PGroonga is very fast!"
+.. },
+.. {
+..   "_key": "memo6",
+..   "title": "PGroonga is useful",
+..   "content": "SQL is easy because many client libraries exist."
+.. },
+.. {
+..   "_key": "memo7",
+..   "title": "Mroonga is also useful",
+..   "content": "MySQL has replication feature. Mroonga can use it."
+.. }
+.. ]
+
+You can specify custom score function in :ref:`select-match-columns`.
+There are some syntaxes.
+
+For score function that doesn't require any parameter such as
+:doc:`/reference/scorers/scorer_tf_idf`::
+
+  SCORE_FUNCTION(COLUMN)
+
+You can specify weight::
+
+  SCORE_FUNCTION(COLUMN) * WEIGHT
+
+For score function that requires one or more parameters such as
+:doc:`/reference/scorers/scorer_tf_at_most`::
+
+  SCORE_FUNCTION(COLUMN, ARGUMENT1, ARGUMENT2, ...)
+
+You can specify weight::
+
+  SCORE_FUNCTION(COLUMN, ARGUMENT1, ARGUMENT2, ...) * WEIGHT
+
+You can use different score function for each match column::
+
+  SCORE_FUNCTION1(COLUMN1) ||
+    SCORE_FUNCTION2(COLUMN2) * WEIGHT ||
+    SCORE_FUNCTION3(COLUMN3, ARGUMENT1) ||
+    ...
+
+Here is a simplest example:
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_one_no_argument_no_weight.log
+.. select Memos \
+..   --match_columns "scorer_tf_idf(content)" \
+..   --query "Groonga" \
+..   --output_columns "content, _score" \
+..   --sortby "-_score"
+
+``Groonga! Groonga! Groonga! Groonga is very fast!`` contains 4
+``Groonga``. If you use TF based scorer that is the default scorer,
+``_score`` is ``4``. But the actual ``_score`` is ``2``. Because the
+``select`` command uses TF-IDF based scorer ``scorer_tf_idf()``.
+
+Here is an example that uses weight:
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_one_no_argument_weight.log
+.. select Memos \
+..   --match_columns "scorer_tf_idf(content) * 10" \
+..   --query "Groonga" \
+..   --output_columns "content, _score" \
+..   --sortby "-_score"
+
+``Groonga! Groonga! Groonga! Groonga is very fast!`` has ``22`` as
+``_score``. It had ``2`` as ``_score`` in the previous example that
+doesn't specify weight.
+
+Here is an example that uses scorer that requires one
+argument. :doc:`/reference/scorers/scorer_tf_at_most` scorer requires
+one argument. You can limit TF score by the scorer.
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_one_one_argument_no_weight.log
+.. select Memos \
+..   --match_columns "scorer_tf_at_most(content, 2.0)" \
+..   --query "Groonga" \
+..   --output_columns "content, _score" \
+..   --sortby "-_score"
+
+``Groonga! Groonga! Groonga! Groonga is very fast!`` contains 4
+``Groonga``. If you use normal TF based scorer that is the default
+scorer, ``_score`` is ``4``. But the actual ``_score`` is ``2``.
+Because the scorer used in the ``select`` command limits the maximum
+score value to ``2``.
+
+Here is an example that uses multiple scorers::
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_multiple_scorers.log
+.. select Memos \
+..   --match_columns "scorer_tf_idf(title) || scorer_tf_at_most(content, 2.0)" \
+..   --query "Groonga" \
+..   --output_columns "title, content, _score" \
+..   --sortby "-_score"
+
+The ``--match_columns`` uses ``scorer_tf_idf(title)`` and
+``scorer_tf_at_most(content, 2.0)``. ``_score`` value is sum of them.
+
+You can use the default scorer and custom scorer in the same
+``--match_columns``. You can use the default scorer by just specifying
+a match column::
+
+.. groonga-command
+.. include:: ../example/reference/scorer/usage_default_and_custom_scorers.log
+.. select Memos \
+..   --match_columns "title || scorer_tf_at_most(content, 2.0)" \
+..   --query "Groonga" \
+..   --output_columns "title, content, _score" \
+..   --sortby "-_score"
+
+The ``--match_columns`` uses the default scorer (TF) for ``title`` and
+:doc:`/reference/scorers/scorer_tf_at_most` for
+``content``. ``_score`` value is sum of them.
+
+Built-in scorers
+----------------
+
+Here are built-in scores:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   scorers/*

data/vendor/local/share/doc/groonga/ja/html/_sources/reference/scorers/scorer_tf_at_most.txt

@@ -0,0 +1,22 @@
+.. -*- rst -*-
+
+.. highlightlang:: none
+
+.. groonga-command
+.. database: scorer_tf_at_most
+
+``scorer_tf_at_most``
+=====================
+
+.. note::
+
+   This scorer is an experimental feature.
+
+.. versionadded:: 5.0.1
+
+Summary
+-------
+
+``scorer_tf_at_most`` is a scorer based on TF (term frequency).
+
+TODO