nltk 3.2__zip → 3.3__zip

{nltk-3.2 → nltk-3.3}/nltk/test/compat.doctest

@@ -1,138 +1,134 @@
-
-=========================================
-NLTK Python 2.x - 3.x Compatibility Layer
-=========================================
-
-NLTK comes with a Python 2.x/3.x compatibility layer, nltk.compat
-(which is loosely based on `six <http://packages.python.org/six/>`_)::
-
-    >>> from nltk import compat
-    >>> compat.PY3
-    False
-    >>> compat.integer_types
-    (<type 'int'>, <type 'long'>)
-    >>> compat.string_types
-    (<type 'basestring'>,)
-    >>> # and so on
-
-@python_2_unicode_compatible
-----------------------------
-
-Under Python 2.x ``__str__`` and ``__repr__`` methods must
-return bytestrings.
-
-``@python_2_unicode_compatible`` decorator allows writing these methods
-in a way compatible with Python 3.x:
-
-1) wrap a class with this decorator,
-2) define ``__str__`` and ``__repr__`` methods returning unicode text
-   (that's what they must return under Python 3.x),
-
-and they would be fixed under Python 2.x to return byte strings::
-
-    >>> from nltk.compat import python_2_unicode_compatible
-
-    >>> @python_2_unicode_compatible
-    ... class Foo(object):
-    ...     def __str__(self):
-    ...         return u'__str__ is called'
-    ...     def __repr__(self):
-    ...         return u'__repr__ is called'
-
-    >>> foo = Foo()
-    >>> foo.__str__().__class__
-    <type 'str'>
-    >>> foo.__repr__().__class__
-    <type 'str'>
-    >>> print(foo)
-    __str__ is called
-    >>> foo
-    __repr__ is called
-
-Original versions of ``__str__`` and ``__repr__`` are available as
-``__unicode__`` and ``unicode_repr``::
-
-    >>> foo.__unicode__().__class__
-    <type 'unicode'>
-    >>> foo.unicode_repr().__class__
-    <type 'unicode'>
-    >>> unicode(foo)
-    u'__str__ is called'
-    >>> foo.unicode_repr()
-    u'__repr__ is called'
-
-There is no need to wrap a subclass with ``@python_2_unicode_compatible``
-if it doesn't override ``__str__`` and ``__repr__``::
-
-    >>> class Bar(Foo):
-    ...     pass
-    >>> bar = Bar()
-    >>> bar.__str__().__class__
-    <type 'str'>
-
-However, if a subclass overrides ``__str__`` or ``__repr__``,
-wrap it again::
-
-    >>> class BadBaz(Foo):
-    ...     def __str__(self):
-    ...         return u'Baz.__str__'
-    >>> baz = BadBaz()
-    >>> baz.__str__().__class__  # this is incorrect!
-    <type 'unicode'>
-
-    >>> @python_2_unicode_compatible
-    ... class GoodBaz(Foo):
-    ...     def __str__(self):
-    ...         return u'Baz.__str__'
-    >>> baz = GoodBaz()
-    >>> baz.__str__().__class__
-    <type 'str'>
-    >>> baz.__unicode__().__class__
-    <type 'unicode'>
-
-Applying ``@python_2_unicode_compatible`` to a subclass
-shouldn't break methods that was not overridden::
-
-    >>> baz.__repr__().__class__
-    <type 'str'>
-    >>> baz.unicode_repr().__class__
-    <type 'unicode'>
-
-unicode_repr
-------------
-
-Under Python 3.x ``repr(unicode_string)`` doesn't have a leading "u" letter.
-
-``nltk.compat.unicode_repr`` function may be used instead of ``repr`` and
-``"%r" % obj`` to make the output more consistent under Python 2.x and 3.x::
-
-    >>> from nltk.compat import unicode_repr
-    >>> print(repr(u"test"))
-    u'test'
-    >>> print(unicode_repr(u"test"))
-    'test'
-
-It may be also used to get an original unescaped repr (as unicode)
-of objects which class was fixed by ``@python_2_unicode_compatible``
-decorator::
-
-    >>> @python_2_unicode_compatible
-    ... class Foo(object):
-    ...     def __repr__(self):
-    ...         return u'<Foo: foo>'
-
-    >>> foo = Foo()
-    >>> repr(foo)
-    '<Foo: foo>'
-    >>> unicode_repr(foo)
-    u'<Foo: foo>'
-
-For other objects it returns the same value as ``repr``::
-
-    >>> unicode_repr(5)
-    '5'
-
-It may be a good idea to use ``unicode_repr`` instead of ``%r``
-string formatting specifier inside ``__repr__`` or ``__str__``
-methods of classes fixed by ``@python_2_unicode_compatible``
-to make the output consistent between Python 2.x and 3.x.
+
+=========================================
+NLTK Python 2.x - 3.x Compatibility Layer
+=========================================
+
+NLTK comes with a Python 2.x/3.x compatibility layer, nltk.compat
+(which is loosely based on `six <http://packages.python.org/six/>`_)::
+
+    >>> from nltk import compat
+    >>> compat.PY3
+    False
+    >>> # and so on
+
+@python_2_unicode_compatible
+----------------------------
+
+Under Python 2.x ``__str__`` and ``__repr__`` methods must
+return bytestrings.
+
+``@python_2_unicode_compatible`` decorator allows writing these methods
+in a way compatible with Python 3.x:
+
+1) wrap a class with this decorator,
+2) define ``__str__`` and ``__repr__`` methods returning unicode text
+   (that's what they must return under Python 3.x),
+
+and they would be fixed under Python 2.x to return byte strings::
+
+    >>> from nltk.compat import python_2_unicode_compatible
+
+    >>> @python_2_unicode_compatible
+    ... class Foo(object):
+    ...     def __str__(self):
+    ...         return u'__str__ is called'
+    ...     def __repr__(self):
+    ...         return u'__repr__ is called'
+
+    >>> foo = Foo()
+    >>> foo.__str__().__class__
+    <type 'str'>
+    >>> foo.__repr__().__class__
+    <type 'str'>
+    >>> print(foo)
+    __str__ is called
+    >>> foo
+    __repr__ is called
+
+Original versions of ``__str__`` and ``__repr__`` are available as
+``__unicode__`` and ``unicode_repr``::
+
+    >>> foo.__unicode__().__class__
+    <type 'unicode'>
+    >>> foo.unicode_repr().__class__
+    <type 'unicode'>
+    >>> unicode(foo)
+    u'__str__ is called'
+    >>> foo.unicode_repr()
+    u'__repr__ is called'
+
+There is no need to wrap a subclass with ``@python_2_unicode_compatible``
+if it doesn't override ``__str__`` and ``__repr__``::
+
+    >>> class Bar(Foo):
+    ...     pass
+    >>> bar = Bar()
+    >>> bar.__str__().__class__
+    <type 'str'>
+
+However, if a subclass overrides ``__str__`` or ``__repr__``,
+wrap it again::
+
+    >>> class BadBaz(Foo):
+    ...     def __str__(self):
+    ...         return u'Baz.__str__'
+    >>> baz = BadBaz()
+    >>> baz.__str__().__class__  # this is incorrect!
+    <type 'unicode'>
+
+    >>> @python_2_unicode_compatible
+    ... class GoodBaz(Foo):
+    ...     def __str__(self):
+    ...         return u'Baz.__str__'
+    >>> baz = GoodBaz()
+    >>> baz.__str__().__class__
+    <type 'str'>
+    >>> baz.__unicode__().__class__
+    <type 'unicode'>
+
+Applying ``@python_2_unicode_compatible`` to a subclass
+shouldn't break methods that was not overridden::
+
+    >>> baz.__repr__().__class__
+    <type 'str'>
+    >>> baz.unicode_repr().__class__
+    <type 'unicode'>
+
+unicode_repr
+------------
+
+Under Python 3.x ``repr(unicode_string)`` doesn't have a leading "u" letter.
+
+``nltk.compat.unicode_repr`` function may be used instead of ``repr`` and
+``"%r" % obj`` to make the output more consistent under Python 2.x and 3.x::
+
+    >>> from nltk.compat import unicode_repr
+    >>> print(repr(u"test"))
+    u'test'
+    >>> print(unicode_repr(u"test"))
+    'test'
+
+It may be also used to get an original unescaped repr (as unicode)
+of objects which class was fixed by ``@python_2_unicode_compatible``
+decorator::
+
+    >>> @python_2_unicode_compatible
+    ... class Foo(object):
+    ...     def __repr__(self):
+    ...         return u'<Foo: foo>'
+
+    >>> foo = Foo()
+    >>> repr(foo)
+    '<Foo: foo>'
+    >>> unicode_repr(foo)
+    u'<Foo: foo>'
+
+For other objects it returns the same value as ``repr``::
+
+    >>> unicode_repr(5)
+    '5'
+
+It may be a good idea to use ``unicode_repr`` instead of ``%r``
+string formatting specifier inside ``__repr__`` or ``__str__``
+methods of classes fixed by ``@python_2_unicode_compatible``
+to make the output consistent between Python 2.x and 3.x.

{nltk-3.2 → nltk-3.3}/nltk/test/compat_fixt.py

@@ -1,8 +1,9 @@
-# -*- coding: utf-8 -*-
-from __future__ import absolute_import
-from nltk.compat import PY3
-
-def setup_module(module):
-    from nose import SkipTest
-    if PY3:
-        raise SkipTest("compat.doctest is for Python 2.x")
+# -*- coding: utf-8 -*-
+from __future__ import absolute_import
+from nltk.compat import PY3
+
+
+def setup_module(module):
+    from nose import SkipTest
+    if PY3:
+        raise SkipTest("compat.doctest is for Python 2.x")

nltk-3.3/nltk/test/concordance.doctest

@@ -0,0 +1,55 @@
+.. Copyright (C) 2001-2016 NLTK Project
+.. For license information, see LICENSE.TXT
+
+==================================
+Concordance Example
+==================================
+
+A concordance view shows us every occurrence of a given
+word, together with some context. Here we look up the word monstrous
+in Moby Dick by entering text1 followed by a period, then the term
+concordance, and then placing "monstrous" in parentheses:
+
+>>> from nltk.corpus import gutenberg
+>>> from nltk.text import Text
+>>> corpus = gutenberg.words('melville-moby_dick.txt')
+>>> text = Text(corpus)
+
+>>> text.concordance("monstrous") # doctest:+NORMALIZE_WHITESPACE
+Displaying 11 of 11 matches:
+ong the former , one was of a most monstrous size . ... This came towards us ,
+ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r
+ll over with a heathenish array of monstrous clubs and spears . Some were thick
+d as you gazed , and wondered what monstrous cannibal and savage could ever hav
+that has survived the flood ; most monstrous and most mountainous ! That Himmal
+they might scout at Moby Dick as a monstrous fable , or still worse and more de
+th of Radney .'" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l
+ing Scenes . In connexion with the monstrous pictures of whales , I am strongly
+ere to enter upon those still more monstrous stories of them which are to be fo
+ght have been rummaged out of this monstrous cabinet there is no telling . But
+of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u
+
+>>> text.concordance("monstrous") # doctest:+ELLIPSIS, +NORMALIZE_WHITESPACE
+Displaying 11 of 11 matches:
+ong the former , one was of a most monstrous size . ... This came towards us ,
+ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r
+ll over with a heathenish array of monstrous clubs and spears . Some were thick
+...
+
+=================================
+Concordance List
+=================================
+
+Often we need to store the results of concordance for further usage.
+To do so, call the concordance function with the stdout argument set
+to false:
+
+>>> from nltk.corpus import gutenberg
+>>> from nltk.text import Text
+>>> corpus = gutenberg.words('melville-moby_dick.txt')
+>>> text = Text(corpus)
+>>> con_list = text.concordance_list("monstrous")
+>>> con_list[2].line
+'ll over with a heathenish array of monstrous clubs and spears . Some were thick'
+>>> len(con_list)
+11