@valueerror/ltjs-lang-en 0.0.0-stage → 0.1.0

package/licenses/resource_en_12dicts-readme.html

@@ -0,0 +1,738 @@
+<html>
+<head>
+<title>The 12dicts Word Lists</title>
+</head>
+<body>
+<h1>Introduction</h1>
+<p>
+12dicts is a collection of English word lists.  It differs in several important
+ways from most of the other free word lists you can download.
+<ul>
+<li> The 12dicts lists are oriented towards common words.  If you're looking for
+myriads of archaic, scientific or computer jargon words, you should look elsewhere.
+<li> The 12dicts lists have been rigorously checked for errors.  (This is not to
+say that they are error-free, merely that enough care has been taken that errors
+are rather infrequent.)
+<li> 12dicts contains a variety of lists, of different sizes and characteristics.
+One size does not fit all.  Because each list has different characteristics, I do
+not recommend combining them, except as noted below.
+</ul>
+<p>
+Originally, 12dicts was composed of lists derived from a specific set of 12 source
+dictionaries.  In addition to these "classic" lists, 12dicts now includes lists derived
+from other sources.  It would perhaps be appropriate to rename 12dicts to something
+more generic, such as BAWL (Beale's Assorted Word Lists), but I have not done so in
+order to preserve continuity.
+<p>
+A quick summary of the 12dicts lists and their characteristics is as follows:
+<p>
+<table border=1>
+<tr>
+<th></th><th>3esl</th><th>6of12</th><th>2of12</th><th>2of4brif</th><th>5desk</th><th>2of12inf</th>
+</tr><tr>
+<td>Size</td><td>21877</td><td>32153</td><td>41236</td><td>60387</td><td>61406</td><td>81520</td>
+</tr><tr>
+<td>Abbreviations</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>N</td><td>N</td>
+</tr><tr>
+<td>Acronyms</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>Y</td><td>N</td>
+</tr><tr>
+<td>British English</td><td>N</td><td>N</td><td>N</td><td>Y</td><td>N</td><td>N</td>
+</tr><tr>
+<td>Hyphenations</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>N</td>
+</tr><tr>
+<td>Inflections</td><td>N</td><td>N</td><td>N</td><td>Y</td><td>N</td><td>Y</td>
+</tr><tr>
+<td>Names</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>Y</td><td>N</td>
+</tr><tr>
+<td>Phrases</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>N</td><td>N</td>
+</tr>
+</table>
+<p>
+The remainder of this document is organized as follows:
+<ul>
+<li>
+<a href="#release">This release</a>
+<li>
+<a href="#classic">The classic 12dicts lists</a>
+<ul>
+<li>
+<a href="#nof12">The 6of12 and 2of12 lists</a>
+<li>
+<a href="#2of12inf">The 2of12inf list</a>
+</ul>
+<li>
+<a href="#3esl">The 3esl list</a>
+<li>
+<a href="#2of4brif">The 2of4brif list</a>
+<li>
+<a href="#5desk">The 5desk list</a>
+<li>
+<a href="#history">How 12dicts came to be</a>
+<li>
+<a href="#conclude">Conclusions</a>
+</ul>
+<h1><a name="release">This release</a></h1>
+<p>
+This is release 4.0 of 12dicts, released Jan. 18, 2003.
+It differs from previous versions by containing three additional lists
+which are not derived from the "classic" 12dicts sources.  Changes to
+the classic lists are limited to error corrections.
+<h1><a name="classic">The classic 12dicts lists</a></h1>
+<p>
+The 12dicts project began as the n-dicts projects, n being a variable whose
+value finally stabilized as 12.  The purpose of the project was to create a
+list of words approximating the common core of the vocabulary of American
+English.
+<p>
+The methodology of the project was to record and correlate the words
+listed in a number of small dictionaries.  The number of dictionaries
+so recorded is now 12, comprising 8 ESL (English as a Second Language)
+dictionaries and 4 "desk dictionaries".  The dictionaries chosen
+vary widely by publisher, by style, by completeness and by depth.
+In this version of 12dicts, all of them are dictionaries of American
+English (three from British publishers).  The smallest of them contains
+about 20,000 entries, and the largest 46,000.  (All totaled, there are
+about 75,000 entries, many of which appear in only a single dictionary.)
+All but two of them were published in the last seven years.
+<h2><a name="nof12">The 6of12 and 2of12 lists</a></h2>
+<p>
+I initially tried two different ways of winnowing the 12dicts data to
+produce lists of common words.  Both produced interesting results.
+One list, the 6of12 list, contains all words and phrases
+listed in 6 of the 12 dictionaries.  One way of describing this list
+is that it contains those words and phrases which a (seeming) majority
+of lexicographers believe are relevant to people learning English,
+and/or to everyday usage.  This list contains about 32,000 words and
+phrases. The other list, the 2of12 list, is more inclusive in that it
+includes words listed in as few as two of the source dictionaries, but
+less inclusive in that it excludes items of various sorts, including
+multiword phrases, proper names and abbreviations.  This list contains
+about 41,000 words.  It is perhaps more suitable for use in areas
+like spell checking or word games than the 6of12 list.  (Honesty
+compels me to admit that neither of these lists is, by itself, a good
+choice for spell checking, due to the absence of inflections, proper
+names, Roman numerals, etc.)
+<p>
+A third list, 2of12inf.txt, developed later, is of a rather different
+character, and is discussed in a later section.
+<p>
+A more precise description of the criteria by which the above lists
+were composed is as follows:
+<h3>6of12 list word selection</h3>
+<ul>
+<li>
+The 6of12 list contains all non-excluded words and phrases which
+appear in 6 or more of the source dictionaries.
+<li>
+Prefixes and suffixes are excluded.  Abbreviations are included;
+however, if they are entirely lower-case and alphabetic, they are
+terminated with a colon (":") so they can be easily distinguished
+from regular words.
+<li>
+Inflections of included words are not themselves included unless
+they are separately defined or irregular.
+<li>
+It sometimes occurs that a word is listed in several forms (e.g.,
+with and without hyphenation) in 6 or more dictionaries, even though
+no single form is so listed.  In this case, if one spelling is clearly
+more accepted, this spelling and this spelling only is listed.  If all
+spellings seem equally accepted, one spelling has been selected
+arbitrarily for inclusion.
+<li>The 6of12 list contains a significant number of words which do not
+meet either criterion 1 or 4 above.  These words, sometimes called
+"signature words", are discussed below.  All of these words are
+listed in at least one of the source dictionaries.
+<li>
+In addition to the ":" suffix discussed above, other special
+suffix characters are used to mark words with certain characteristics,
+as discussed below.
+</ul>
+<h3>2of12 list word selection</h3>
+<ul>
+<li>
+The 2of12 list contains all non-excluded words which appear in at
+least 2 of the source dictionaries.
+<li>
+This list excludes capitalized words, multiword phrases, and
+abbreviations, as well as prefixes and suffixes.  It does not
+exclude hyphenated words or contractions.  If a word occurs in
+both a hyphenated and an unhyphenated form, the unhyphenated
+form is listed, even if the hyphenated form is generally
+preferred.
+<li>
+The list excludes spellings which are considered (by a majority
+of the dictionaries listing it) to be non-American usage.  It
+also excludes secondary spellings which are mentioned by fewer
+than four of the source dictionaries.
+<li>
+Inflections of included words are not themselves included unless
+they are separately defined, or irregular.
+<li>
+Several of the source dictionaries include listings for obscure
+currencies, such as <b>ringgit, khoum</b> and <b>ngwee.</b>
+I was unable to regard such words as part of the English "core vocabulary",
+and so I required citation in over a third of the dictionaries for
+inclusion of monetary units.  A side-effect was the elimination
+of the word <b>lepton</b>, which, in addition to its use in particle
+physics, is also .01 Greek drachmas.
+<li>
+This list also includes a small number of signature words, as
+discussed below.
+</ul>
+<h3>Signature words</h3>
+As indicated, both lists have been augmented with words (and, in the
+case of the 6of12 list, phrases) which fail to meet the formal
+requirements for inclusion.  In the case of the 6of12 list, 1024
+words were added (about 3 % of the total).  These are all words which,
+in the judgment of the compiler, are as familiar as many of the words
+which met the criteria for inclusion.  Examples of some of the sorts
+of words which were added are:
+<ul>
+<li>
+Words of the same category as other included words.  An example is
+the astrological sign <b>Cancer</b>, which alone of all the
+astrological signs fails to appear in 6 or more of the dictionaries.
+Similarly added were the omitted holidays <b>Thanksgiving</b> and
+<b>Christmas Eve.</b>
+<li>
+Vulgarities, sexual terms and insults.  Some such words were
+already included, but most of the source dictionaries were quite
+squeamish about them.  These words are very widely known indeed;
+I hold that any list of "common" words which does not include the
+infamous f-word is simply discredited thereby.  Some may feel that
+it would have been better to leave some or all of these terms
+unmentioned.  Nevertheless, the expression of blasphemy,
+unwarranted contempt and perverse lust, whether in words or in
+deeds, is a very human trait.  Suppressing the evidence of these
+aspects of the human condition in our language makes no more sense
+than excluding <b>leprosy, gangrene</b> and <b>dementia</b>,
+no matter how unpleasant they may be to contemplate.
+<li>
+Conventional conversational phrases so common as to be practically
+invisible to native speakers.  Examples are <b>thank you, good
+night, uh-huh, of course</b> and <b>gesundheit.</b>
+<li>
+Sports terminology, especially for football and baseball.  (If I,
+who am practically sports-blind, noticed this deficiency, it must
+be of major proportions indeed.)
+</ul>
+Note that the signature words in the 6of12 list can be identified via
+the suffix character "+", and eliminated if desired.
+<p>
+A much smaller set of words (49) was added to the 2of12 list.  These
+were of two sorts:
+<ul>
+<li>
+Signature words from the 6of12 list which were not already present
+in the 2of12 list, and which are not excluded due to being
+abbreviations, phrases, etc.
+<li>
+Inflections of irregular verbs not explicitly mentioned in 2
+source dictionaries, such as <b>outfought</b> and <b>reheard.</b>
+</ul>
+<h3>Annotations</h3>
+Some of the 6of12 list entries are annotated with a suffix character,
+giving additional information about the associated word.  The
+annotations can be easily removed with an editor or script if
+they are unwanted.
+<p>
+These annotations are:
+<table>
+<tr>
+<td>:</td><td>The word is an otherwise unmarked abbreviation.  This suffix
+may appear in combination with another suffix.</td>
+</tr><tr>
+<td>&amp;</td><td>The word is primarily a non-American usage.</td>
+</tr><tr>
+<td>#</td><td>The word is generally held to be a variant or less preferred
+form of another word.</td>
+</tr><tr>
+<td>&lt;</td><td>This form of a word is held to be the primary form by fewer
+dictionaries than some other form of the word.</td>
+</tr><tr>
+<td>^</td><td>This form of the word was selected arbitrarily from a set of
+variants, none of which was clearly preferred.</td>
+</tr><tr>
+<td>=</td><td>Roughly, this indicates a "second class" word, as described
+below.</td>
+</tr><tr>
+<td>+</td><td>The word is a signature word.</td>
+</tr><tr>
+</table>
+The reasons a word might be marked with the = annotation are:
+<ul>
+<li>
+The word is an inflection which was defined in the same
+entry as the base word.
+<li>
+The word is a derived word (<b>-ly</b>, <b>-ness</b> or
+<b>-er/or</b>) which was not defined in a separate entry.
+<li>
+The word appeared in a list of undefined words with a
+common prefix, such as <b>un-</b> or <b>re-</b>.
+</ul>
+<p>
+The words in the 2of12 list are not annotated.
+<h2><a name="2of12inf">The 2of12inf list</a></h2>
+<p>
+The 2of12inf list is of a rather different character from the two
+original "classic" lists.  Conceptually,
+it is simple.  It consists of all the words in the 2of12 list, plus
+their inflections, amounting to about 81,000 words.  This list may
+be more useful than the other lists for applications like word games.
+It was created to help Kevin Atkinson in his Aspell and SCOWL projects
+(for which, follow <a href="http://aspell.sourceforge.net"> this link</a>).
+Unlike the 6of12 and
+2of12 lists, this list is not based exclusively on the contents of my
+12 source dictionaries, and for this reason it has, I feel, less
+authority than the other classic 12dicts lists.  It also probably has a
+significantly higher error rate than the other lists, for reasons
+explained below.
+<p>
+The criteria defining the 2of12inf list are as follows:
+<ul>
+<li>
+The 2of12inf list contains all non-excluded words which appear in
+at least 2 of the source dictionaries.
+<li>
+This list excludes capitalized words, multiword phrases,
+abbreviations, contractions, hyphenated words and single-letter
+words, as well as prefixes and suffixes.
+<li>
+The list does not exclude secondary spellings, non-American usages
+or monetary units.
+<li>
+The list includes inflections of all included words.  Any
+inflection mentioned or clearly implied by any of the source
+dictionaries is included (i.e., two citations are not required).
+Additionally, some inflections have been added from other sources.
+<li>
+Plurals of "uncountable" nouns were included, annotated with the
+"%" suffix character.  See below for an extended discussion of
+the inclusion of these words.
+<li>
+Signature words from the other lists, plus their inflections, were
+added.  No other signature words were added.
+</ul>
+<p>
+Though the 2of12inf list still consists mostly of very common words,
+criteria 3 through 5 above cause the 2of12inf list to contain a greater
+proportion of unfamiliar and unusual words than the other classic
+12dicts lists.
+<p>
+The 2of12inf list was not derived directly from the 12 source
+dictionaries.  The starting point was a subset of Kevin Atkinson's
+AGID list, a list of words, parts of speech and inflections derived
+from public-domain sources, notably Moby Words and WordNet.  (See the
+file agid.txt in the 12dicts archive, which is a copy of the AGID "readme",
+for more information on the antecedents of AGID.)  2of12inf was created
+by a process of editing the AGID subset to remove spurious entries and
+those which reflected a more esoteric English vocabulary than the other
+12dicts lists, and to add inflections which AGID failed to identify.
+This process required significantly less effort than would have been
+needed to derive the list directly from the source dictionaries.
+Unfortunately, a side effect of the process is that the result is
+likely to be somewhat less reliable than the other 12dicts lists.
+In particular, Moby Words is notoriously unreliable, and I find it
+unlikely that I have successfully identified all the spurious
+inflections its use has introduced.  It is my hope in the future to
+release another edition of 2of12inf which is not derived from AGID,
+and therefore not "infected" by Moby Words.
+<p>
+After the first version of the 2of12inf list was released, I replaced
+one of the source dictionaries, officially an international dictionary
+but in actuality rather British in its orientation, with a more
+American dictionary by  the same publisher.  It was not practical
+(nor necessarily desirable) for me to go through the list removing
+inflections endorsed only by the superseded dictionary.  For this
+reason, the 2of12inf list has a slightly more international character
+than the other 12dicts lists.  It is not altogether clear that this
+is a bad thing.
+<h3>Selection of inflections</h3>
+<p>
+Ideally, the 2of12inf list would contain only inflections listed in
+one of the 12dicts source dictionaries.  This proved not to be
+practical.  The reason for this has to do with the nature of these
+sources, which are mostly ESL dictionaries.  An ESL dictionary might
+well list the word <b>esophagus,</b> but, because an English learner is
+unlikely to need to talk about this organ in the plural, it will
+probably not bother to list the plural form <b>esophagi.</b>  For words of
+this sort, I therefore needed to obtain their inflections from other
+sources.  Obviously, the decisions on when to include additional
+inflections were judgment calls, as were the choices of which
+inflections to add.
+<p>
+Adjectival inflections (comparatives and superlatives) proved to be
+an especially annoying problem.  Only 2 of my 12 source dictionaries
+provided remotely reliable information of this sort.  In fact, such
+information is sparse and inconsistent in most dictionaries of any
+size.  I relied on a small set of additional dictionaries for this
+information, which was mostly disjoint from the sources for plurals
+and verb forms.  Several of these sources were Scrabble(r)-related,
+and therefore inclined to include forms of little plausibility such
+as <b>iller/illest</b> or <b>fertiler/fertilest.</b>
+Accordingly, I ended up rejecting some of the documented inflections on
+grounds of implausibility.  I have no doubt that, in the process, I made
+a number of errors of both inclusion and exclusion and, in any case, many
+of the forms listed have no connection with any of the 12dicts source
+dictionaries.
+<p>
+One additional problem in the creation of the 2of12inf list was that
+of "uncountable" nouns and their plurals.  Some English dictionaries,
+especially ESL dictionaries, as well as other linguistic sources
+attest to the existence of nouns which cannot be counted, or used in
+the plural.  Examples of such nouns include <b>mud, rayon, oregano,
+chess, fairness, wisdom, aluminum, training, materialism</b>
+and <b>chickenpox.</b>  This is an entirely commonsense notion, but a
+difficulty is the fact that the boundary between the countable and the
+uncountable is extremely vague and ill-defined.  For example, the word
+<b>coffee</b> is ordinarily uncountable, but not when ordering in a
+restaurant, as is the word <b>symmetry,</b> except in physics or math.
+In general, it is possible to contrive a context where use of the
+plural of any noun whatsoever is reasonable.
+<p>
+An alternate position, therefore, is that in fact no nouns are
+uncountable, and that any noun which is not already plural possesses
+a plural.  This position is especially useful in the context of word
+games, where words such as <b>zeals</b> and <b>anthraxes</b>
+may produce large scores.  For this reason, the official Scrabble
+dictionaries list words such as <b>thens, onces</b> and
+<b>mankinds</b>, which most people find
+rather implausible.  The fact that the 2of12inf list might well be
+useful in gaming contexts, together with the fact that the boundary
+between countable and uncountable nouns is so ill-defined, served as
+a powerful argument for inclusion of all plural forms, whether
+commonly used or not, while its derivation from ESL sources argued
+for including only the plurals of countable nouns, however
+distinguished.
+<p>
+In the end, I was unable to resolve this dilemma, and adopted a
+compromise.  The 2of12inf list includes all plurals, but with the
+plurals of uncountable nouns marked, making it easy to remove them
+if they are not wanted.  That left the issue of how to establish
+countability.  Six of my source dictionaries included information
+on countability, which was adequate to decide the status of most of
+the included nouns.  As for the rest, as usual, I used my best
+judgment.  I will confess to occasionally overriding the source
+dictionaries when I believed they were clearly incorrect.  (For
+instance, I chose not to mark the word <b>hatreds</b> as an
+uncountable plural, in defiance of the opinion of all my sources,
+on the grounds that it has been used in too many news stories from
+Bosnia to be considered unusual.)  It is interesting to note that
+most of the plurals I added from auxiliary sources were of words
+considered uncountable.
+<p>
+The difficulties listed above, and the fact that I was forced to
+exercise personal judgment frequently in creating it, emphasizes a
+fundamental difference between this list and the other classic 12dicts
+lists.  I have tried to make the 6of12 and 2of12 lists reflect only the
+source dictionaries, and to keep my own judgments and opinions out of
+the picture (except for my addition of signature words).  This has
+proved impossible to achieve for the 2of12inf list, which accordingly
+represents a less authoritative and more arbitrary collection.
+Additionally, the 2of12inf list has undergone less proofreading and
+validation than the other lists, and I suspect the error rate is
+considerably higher than the idealistic goal of 0.02 % I advocate
+elsewhere in this document.  Nevertheless, I hope it may prove to be
+of some use and interest.
+<p>
+I wish to offer my special thanks to Kevin Atkinson, for supplying me
+with the AGID list, and for encouraging me to add the inflections.  Of
+course, any errors that remain in the 2of12inf list are my own
+responsibility, and should not be blamed on Kevin, AGID, or even on
+Moby.
+<h1><a name="3esl">The 3esl list</a></h1>
+<p>
+The 3esl list represents another attempt to produce an English "core
+vocabulary" list.  It is about 2/3 of the size of the 6of12 list,
+which it resembles in terms of the sorts of words included.
+<p>
+The 3esl list is a far more subjective list than any of the classic
+12dicts lists.  It was compiled from 3 small ESL dictionaries, using
+the same criteria for eligibility as the 6of12 list.  I started with
+a list composed of all words from the smallest of the 3 sources, plus
+all words contained in both of the others.  This list was then edited
+in the following ways:
+<ol>
+<li>
+I removed alternate spellings for included words, such as <b>grey</b>
+and <b>off-stage</b>.  I also removed very similar synonyms for the
+same concept, for instance, removing <b>cable television</b> as a
+duplicate of <b>cable TV.</b>
+<li>
+I added one form of each word which would have been included if
+the sources had agreed on spelling, such as <b>shortchange</b> and
+<b>back seat</b>.
+<li>
+I removed some words which were present in the smallest of the
+sources but seemed too esoteric, such as the symbols of chemical
+elements.  I did this only for words which were not present in the
+other sources.
+<li>
+I added some words which were present in only one of the two
+larger sources, but which seemed appropriate to add.  These words
+were frequently of the sort added to the 6of12 list as signature
+words, as well as some inflections that often function as words
+with meanings of their own, such as <b>comforting</b> and
+<b>notes.</b>
+</ol>
+<p>
+All of these changes were quite subjective in nature, and quite
+numerous.  Probably more than 10 % of the candidate words were added
+or removed in this way.  For this reason, it is pointless to speak
+of signature words for this list; the composition of the list is too
+arbitrary for the term to make any sense.  (I will note that the list
+is still not entirely arbitrary, as I added only words found in
+some form in one of the sources, and removed no words present in two
+of the sources other than duplicates.  Thus, words like <b>front
+page</b> were not added, no matter how familiar, and words such
+as <b>lugubrious</b> were not removed, despite clearly not being
+part of any "core vocabulary".)
+<p>
+Like the 6of12 list, the 3esl list marks lower-case abbreviations
+with a ":" suffix, to prevent them from being mistaken for regular
+English words.
+<p>
+One final note on this list.  The 3esl list contains about 1500 words
+not present in the 6of12 list.  Because these two lists have the same
+rules for the kinds of words included, one could easily combine
+the two to produce a slightly larger list including a number of words
+whose omission from 6of12 is rather surprising.  Be warned that in a
+few cases, the spelling chosen for words with multiple spellings is
+different in the two lists, and I would recommend that the duplicates
+be removed.  (I'll be happy to provide a list of the duplicates if
+anyone wants one.)
+<h1><a name="2of4brif">The 2of4brif list</a></h1>
+<p>
+All of the classic 12dicts lists are unabashedly oriented towards
+American English.  I've received a few expressions of interest in a
+British English list.  The result is the 2of4brif list.  This list
+was compiled from 4 large "international" ESL dictionaries, published
+by British publishers.  To this American, they are more British than
+they are international; quite possibly, they seem more American than
+international to British readers.  It is interesting to note that,
+although there were only a third as many sources for this list as for
+the 12dicts lists, these dictionaries resembled each other far more
+closely than their American counterparts, which could mean that the
+2of4brif list is as good an approximation of a "core" British English
+vocabulary as the 6of12 list is for American English.  (Or, alternately,
+it may simply mean that my choice of sources was too narrow.)
+<p>
+This criteria for inclusion in this list were basically those of the
+2of12inf list.  In particular, inflections are included for all words,
+but hyphenated words, contractions, phrases, proper names and
+abbreviations are all excluded.  One important difference between
+the two is the way in which inflections were determined for inclusion.
+The 2of12inf list includes some inflections found in one (or even none)
+of its sources.  Further, as discussed in detail above,
+it includes plurals for words which are not normally
+considered to have plurals.  The 2of4brif list differs in both of
+these regards.  It includes only inflections endorsed by two or more
+of the sources, specifically excluding any plural forms for nouns
+listed as uncountable.
+<p>
+The 2of4brif list includes no signature words as such.  I made a small
+number of adjustments for consistency, such as making sure that
+<b>-ise</b> and <b>-ize</b> spellings were equally
+represented, and adding plurals for ordinal numbers.  (Why
+<b>fourteenth</b> would be defined as a fraction, but not
+<b>seventeenth</b>, I must simply regard as a mystery.)  These
+edits were so few, and so clearly harmless, that I have not marked them.
+<p>
+Prospective users of the 2of4brif list should realize that it was
+compiled by an American.  If my sources contained any glaring errors
+(and most dictionaries have a few), I might well not have noticed,
+and perpetuated them in the list.  The fact that two citations were
+required is some protection against such an event, but no guarantee.
+<p>
+As the 2of4brif list is very similar in makeup to the 2of12inf list,
+a user who wants a larger, more international list than either could
+reasonably merge the two.  If you do this, you should remove the
+unusual plurals (marked with a "%") from the 2of12inf list in the
+process, for consistency.
+<h1><a name="5desk">The 5desk list</a></h1>
+<p>
+I created the 5desk list in an attempt to do a better /usr/dict/words
+(about which I offer many harsh criticisms elsewhere in this document).
+The sorts of words admitted are the same sorts that /usr/dict/words
+contains.  Though somewhat larger in size than most versions of
+/usr/dict/words, this is still a short word list, striving for inclusion
+of words one is likely to encounter rather than the complete jargon of
+every possible scientific, artistic or occult endeavor.
+<p>
+5desk was assembled primarily from five "desk dictionaries".  It
+was augmented by words from five minor sources, including a "vocabulary
+builder" and a collection of proper names.  The list excludes
+prefixes, suffixes, phrases, hyphenated words, contractions and most
+abbreviations and acronyms.  There was no requirement for multiple
+listings; all qualifying words from each of the sources were included.
+Inflections of included words were not included themselves except when
+irregular, or separately defined.  Variant and non-American spellings
+were not excluded, and no signature words were added.
+<p>
+Words commonly considered to be abbreviations/acronyms were included
+if they contained at least one upper case character, and were defined
+with an explicit part of speech.  This excluded items like <b>Mr</b> and
+<b>Feb,</b> which are abbreviations in the classic sense, but allowed words
+like <b>DNA</b> and <b>ATM,</b> which are used far more frequently than that
+which they abbreviate.  While there is a trend in modern dictionaries
+to list such words as nouns (or occasionally verbs, adverbs, etc.),
+it is a trend in progress, and rather inconsistently applied.  For
+this reason, the set of such words in the 5desk list is somewhat
+incoherent, including <b>SPCA</b> but not <b>PETA</b>,
+<b>AIDS</b> but not <b>SIDS</b>, <b>KGB</b> but
+not <b>CIA</b>, and <b>PDQ</b> but not <b>ASAP</b>.
+<p>
+One class of commonly-used words is regrettably absent from the 5desk
+list, because I was unable to find a satisfactory source for them.
+This is the class of commercial names such as <b>Exxon, Tylenol,
+Pepsi</b> and <b>Chevy</b>.  This is probably forgivable,
+as this class of names is as ephemeral and transitory as teenage slang.
+The one-time household words <b>Kool, Ovaltine, Philco</b> and
+<b>Ipana</b> serve now only as answers to trivia questions,
+with modern wonders like <b>Starbucks, Google, Ritalin</b>
+and <b>TiVo</b> taking their place on the tongues of the trendy.
+<p>
+The 5desk list has clearly moved beyond any "core vocabulary" concept.
+It includes quite esoteric words (<b>ogee, pleonastic</b>), very
+uncommon spellings (<b>thiamine, yuppy</b>), and obscure geographical
+and historical names (<b>Paricutin, Nevelson</b>).  Like
+/usr/dict/words, it is frequently inconsistent and arbitrary, but I
+hope at the least I have avoided including spelling errors, and
+overlooking the stuff of everyday conversation.  Perhaps it will be
+useful as a compromise between basic lists such as 3esl, and truly
+massive lists like Mendel Cooper's ENABLE.
+<h1><a name="history">How 12dicts came to be</a></h1>
+<p>
+It may have occurred to some to wonder about how something like the
+n-dicts project came to be (though I assume that anyone who bothers
+to download this archive must already have some idea that such a
+project could be of interest).
+<p>
+Some years ago, there was a post to the sci.crypt Usenet newsgroup,
+on the subject of creating PGP passphrases using randomly selected
+entries from a supplied list of very short words.  (If this sounds
+interesting, follow <a href="http://world.std.com/~reinhold/diceware.html">
+this link</a> for an expanded version of the post.)  The word list,
+which was extracted from /usr/dict/words on some UNIX system, seemed
+to me ill-suited to its intended purpose.  It included arcane acronyms
+(<b>bstj, fmc</b>), misspellings (<b>diety, ouvre</b>) and
+words of amazing obscurity (<b>bhoy, kombu</b>).  I decided I
+could do better (and eventually did).
+
+This caused me to start downloading English word lists, of which there
+are many, from the Internet.  I was not impressed by the overall
+quality of these lists, and the few which were high-quality were
+all-inclusive, burying the everyday words under a mountain of archaisms
+and esoterica.
+
+The flaws of the vast majority of these lists are worth recounting:
+<ul>
+<li>
+Failure to proofread.  Many of these lists are littered with
+misspellings and typos, sometimes approaching gibberish.  (I
+presume, for instance, that the bizarre string <b>nondploe,</b>
+which was found in a purported Scrabble word list, is a typo
+for something more or less legitimate, but I have no idea what.)
+Working on my own lists has helped me understand that 100 %
+accuracy is a very demanding goal, seldom actually achieved, but
+I still feel it reasonable to expect no more than 1 or 2 errors
+per 10,000 words.
+<li>
+Acceptance of completely undocumented lazy spellings, such as
+<b>bullseye</b> and <b>courtmartial.</b>
+<li>
+Failure to respect capitalization.
+<li>
+Failure to distinguish abbreviations from other entries.
+<li>
+Treating esoteric computer jargon, and especially UNIX jargon,
+as everyday English.  (Beware any list which includes <b>bitblt,
+emacs, inode</b> or <b>lvalue</b>.)
+<li>
+Apparently random word selection.  For instance, the most common
+version of /usr/dicts/words contains a large set of apparently
+randomly chosen personal names (uncapitalized, of course, and
+missing <b>wanda, marge, polly</b> and <b>sid</b>).
+<li>
+Inconsistent inflection.  Some lists include all inflections of
+their vocabulary, while others include only singulars and
+infinitives.  Either policy is fine, and has its advantages.  I
+am personally very annoyed when inflected forms appear at random.
+I find this generally happens when a compiler merges several lists
+with different characteristics, with no attempt to reconcile their
+divergent styles.
+<li>
+Omission of everyday words.   I've seen a purported general-purpose list
+that includes <b>bremsstrahlung</b>, yet omits <b>log</b> and
+<b>beer</b>.  Or that includes <b>saxophone</b> but not
+<b>sax</b>, and <b>rhinoceros</b> but not <b>rhino</b>.
+Of course, due to my original purpose in seeking out common short
+words, I found this especially annoying.
+</ul>
+<p>
+One result of my frustration with this situation was my working with
+Mendel Cooper on ENABLE (for further information, check out
+<a href="http://personal.riverusers.com/~thegrendel/software.html">this
+link</a>), which was close to unique in having an active caretaker,
+one clearly concerned with quality, and in being oriented towards
+American rather than British English.  But ENABLE is an all-encompassing
+list and, even if it had been complete at the time I started my search
+for a list of common words, it would not have been what I wanted for
+that reason.
+<p>
+I finally decided that only starting from scratch with a systematic
+approach was likely to get me what I was looking for, and that
+dictionaries intended for non-native speakers of English were the
+best possible source for words that are in some cases so familiar
+that we never think of them.  This has led to the 12dicts lists,
+which I hope have managed to avoid the flaws recited above.
+</p>
+(I should acknowledge one form of inconsistency exhibited by the
+12dicts lists, which is that sometimes related words are spelled
+inconsistently.  For instance, the 2of12 list contains both
+<b>broadminded</b> and <b>broad-mindedness</b>.  This
+generally occurs as a result of the methodology used to build the lists.
+In the case of <b>broadminded</b>, only one source dictionary listed
+<b>broadmindedness</b>, which was therefore excluded.  I felt unequal
+to trying to correct these inconsistencies, some of which are real and not
+mere artifacts of 12dicts, such as the contrast between <b>self-conscious</b>
+and <b>unselfconscious</b>.)
+<h1><a name="conclude">Conclusions</a></h1>
+<p>
+When I released the first version of 12dicts in 1999, I assumed I was
+done with it.  It hasn't worked out that way.  Before I declare it finished
+for a second time, there are a few more things I'd like to accomplish.
+<ul>
+<li>
+As mentioned above, I would like to rework the 2of12inf list to remove
+the dependency on the Moby lists.
+<li>
+As may be seen by inspecting the table of file characteristics, the
+12dicts files now form a spectrum of word lists, with contents ranging
+from the extremely common to the mildly esoteric.  I would like to
+extend the spectrum further by applying the 12dicts methodology to
+dictionaries of larger size.  Whether I will ever get the time for a
+project this large remains to be seen.  If it ever comes to pass,
+it will probably be released separately from 12dicts itself, as
+anything larger than the 5desk list will be too large to even pretend
+to represent a "core English" vocabulary.  (Even the 5desk list itself
+is too large for that purpose.)
+<li>
+It is possible that in the future the "n" of n-dicts will increase
+again, but, in fact, consideration of an additional dictionary now
+generally ends with the discovery that its vocabulary matches 12dicts
+pretty closely.  At the very least, this phenomenon gives me hope that
+the 12dicts lists have now fulfilled their basic purpose.
+</ul>
+<p>
+The 12dicts lists were compiled by Alan Beale.  I explicitly release
+them to the public domain, but request acknowledgment of their use.
+(Actually, the dependency of the 2of12inf list on AGID prevents its
+release into the public domain.  However, I do not impose any additional
+requirements on its use beyond those imposed by AGID and its sources,
+as described in agid.txt.)  Feel free to send comments, suggestions,
+inquiries and/or large sums of money to me at <a href="mailto:biljir@pobox.com">
+biljir@pobox.com</a>.  If you find 12dicts useful, I'd love to hear about it.
+</body>
+</html>