treat 0.1.1

data/LICENSE

@@ -0,0 +1,28 @@
+Treat - Text Retrieval and Annotation Toolkit
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Author: Louis-Antoine Mullie (louis.mullie@gmail.com). Copyright 2012.
+
+Non-trivial amount of code has been incorporated and modified from
+other libraries, specifically for the following files:
+
+ - processors/tokenizers/macintyre.rb - Utiyama Masao (Ruby License)
+ - processors/tokenizers/tactful.rb - Matthew Bunday (GPL license)
+ - Inflectors/lemmatizers/e_lemma.rb - Utiyama Masao (GPL license)
+ - processors/tokenizers/perl.rb - Todd A. Fisher (MIT License)
+ - processors/tokenizers/punkt.rb - Steven Bird Edward Loper and Joel Nothman (Apache 2.0 license),
+ - extractors/topics/reuters.rb - Mark Watson (GPL license)
+ - inflectors/stemmers/porter.rb - Ray Pereda (No license information)
+ - tree.rb - Partyl based on work by 

data/TODO

@@ -0,0 +1,67 @@
+== Before first release
+
+- Linkers, inflectors
+- Check taggers for context
+- Stanford dependencies parse
+- Enju : make sure Enju parses constituents, remove_last, abbr
+- Ocropus => use better function
+- Optimize magic methods... bigger or smaller, is_token? type methods, phrase categories.
+- Move statistics?
+- Synset class move
+- general procedure for options, check that user doesn't want to change options...
+- Resources: dependencies vs. edges, PTB function tags
+- Check for # Fix everywhere
+- Check paths; parse bin paths
+- Ferret, Natural Inputs
+- Use consistently delegate
+- Text becomes section
+
+=== Eventually
+
+- RDF output
+- Apache OpenNLP
+- Ariel
+- Text2rdf
+- AI libraries
+- JAWS
+- NLTK
+- ghmm
+- Root
+- OpenCyc
+- Tapas Kanungo
+- Mark Watson Text Extractor & other Java tools
+- Citar 
+- PLDA
+- AI4R
+- Math
+- Abner
+- ACOPOST
+- Linkage integration
+- Lemmatizer
+- Claws tagger
+- YAWNI
+- Const missing instead of autoload
+- Probabilistic features: rchardet19, what_language
+- Enju multithreading ?
+- String type detector for other languages
+
+- Automatic benchmark
+
+=== Distant future
+
+- Spell Cheker Raspell
+- Multithreading
+
+=== Checklist before releasing
+
+- Remove code from main page
+- Remove lib path from tests and main page
+- Remove docs
+- gem19 build treat.gemspec
+
+=== Performance
+
+- Cache results that get computed often
+- Use .size == 0 instead of .empty?
+- Optimize method_missing using define_method, even dynamically
+- Array include is slow

data/bin/INFO

@@ -0,0 +1 @@
+This is where you can put your JAR files.

data/examples/benchmark.rb

@@ -0,0 +1,81 @@
+require 'benchmark'
+require 'treat'
+
+Benchmark.bmbm do |x|
+
+  Treat.edulcorate
+
+=begin
+  # Readers
+  x.report("Read:PDF") { doc = Document 'pages/hhmm_article.pdf'; doc.read }
+  x.report("Read:TXT") { doc = Document 'pages/kant_short.txt'; doc.read }
+  x.report("Read:YAML") { doc = Document 'pages/nanotechnology_article.yml'; doc.read }
+  # x.report("Read:XML") { doc = Document 'pages/test.xml'; doc.read }
+  x.report("Read:Image") { doc = Document 'pages/novel_page.jpg'; doc.read }
+  
+  # Read collection of texts.
+  coll = Collection 'pages'
+  coll.read
+=end
+
+  # Processors.
+  # x.report("Cluster:LDA") { coll.cluster(:lda) }
+  x.report("Chunk:txt ") { text.chunk(:txt) }
+  x.report("Segment:punkt ") { text.segment(:punkt) }
+  x.report("Segment:tactful ") { text.segment(:tactful) }
+  x.report("Segment:stanford ") { text.segment(:stanford) }
+  x.report("Tokenize:macintyre ") { text.tokenize(:macintyre) }
+  x.report("Tokenize:multilingual "){ text.tokenize(:multilingual) }
+  x.report("Tokenize:perl "){ text.tokenize(:perl) }
+  x.report("Tokenize:stanford ") { text.tokenize(:stanford) }
+  x.report("Parse:enju") { text = text.parse(:enju) }
+  # x.report("Parse:stanford") { text = text.parse(:stanford) }
+  # x.report("Parse:link") { text = text.parse(:link) }
+
+  doc = Document 'pages/kant_short.txt'
+  text = doc.read.text.chunk.segment.tokenize
+
+  # Formatters.
+  yaml = nil; xml = nil
+  x.report("Serialize:yaml") { yaml = text.serialize(:yaml) }
+  x.report("Serialize:xml") { xml = text.serialize(:xml) }
+  x.report("Visualize:tree") { text.visualize(:tree) }
+  x.report("Visualize:txt") { text.visualize(:txt) }
+  # x.report("Visualize:dot") { text.visualize(:dot) }
+  # x.report("Visualize:standoff") { text.visualize(:standoff) }
+  # x.report("Visualize:simple_html") { text.visualize(:html) }
+  # Clean: html
+
+  # Detectors
+  x.report("Langugage:what_language ") { text.language(:what_language) }
+  x.report("Encoding:r_chardet19 ") { text.encoding(:r_chardet19) }
+  x.report("Format:file ") { text.format(:file) }
+
+  # Extractors
+  x.report("Date:chronic") { '2007/02/12'.date(:chronic) }
+  x.report("Date:native") { '2007/02/12'.date(:native) }
+  x.report("Time:chronic") { '2007/02/12'.time(:chronic) }
+  x.report("Topic:reuters") { text.topic }
+  x.report("Statistics:frequency:") { text.each_token { |token| token.statistics(:frequency) } }
+  # x.report("Statistics:position:") { text.each_token { |token| token.statistics(:position) } }
+
+  # Inflectors
+  # x.report("Lemma:elemma") { text.each_word { |word| word.lemma(:elemma) } }
+  x.report("Stem:porter_r") { text.each_word { |word| word.stem(:porter) } }
+  x.report("Stem:porter_c") { text.each_word { |word| word.stem(:porter_c) } }
+  x.report("Stem:uea") { text.each_word { |word| word.stem(:uea) } }
+  x.report("Declense:granger") { text.each_word { |word| word.declense(:granger) } }
+  # x.report("Inflect:granger") { text.each_noun { |word| word.plural(:granger) } }
+
+  # Statistics
+  x.report("Entity:word_count") { text.word_count }
+
+  # puts text.words_with_cat(:noun).inspect
+
+  # Lexicalizers
+  x.report("Tag:stanford") { text.each_word { |word| word.tag(:stanford) } }
+  # x.report("Tag:brill") { text.each_word { |word| word.tag(:brill) } }
+  # x.report("Tag:lingua") { text.each_word { |word| word.tag(:lingua) } }
+  # x.report("Lex:wordnet") { text.each_word { |word| word.lex(:wordnet) } }
+
+end

data/examples/keywords.rb

@@ -0,0 +1,60 @@
+require 'benchmark'
+require 'treat'
+
+Treat.edulcorate
+
+c = Collection.from_serialized('texts/corpus.yml')
+
+=begin
+c.each_text do |t|
+  t.chunk.segment.parse(:stanford)
+  puts "Done text #{t.id}."
+end
+
+c.serialize(:yaml).save("economist/corpus.yml")
+
+=end
+
+topic_words = c.topic_words(
+  :lda,
+  :topics => 5,
+  :words_per_topic => 5,
+  :iterations => 20
+)
+  
+c.each_document do |d|
+  
+  sentences = d.key_sentences(
+    :topics_frequency,
+    :topic_words => topic_words,
+    :threshold => 4
+  )
+
+  tm = d.statistics(
+    :transition_matrix,
+    :features => [:tag],
+    :entity_type => :word,
+    :condition => lambda do |word|
+      word.has?(:is_keyword?) &&
+      word.is_keyword?
+    end
+  )
+
+  sentences.each do |sentence|
+    sentence.each_word do |word|
+      score = word.statistics(
+        :transition_probability,
+        :transition_matrix => tm,
+        :relationships => [:parent, :left, :right, :children]
+      )
+      if word.has?(:is_keyword?) && 
+         word.is_keyword?
+           score += 0.5 
+      end
+      if score > 1
+        puts word.to_s
+      end
+    end
+  end
+  
+end

data/examples/texts/bugged_out.txt

@@ -0,0 +1,26 @@
+Coup-plotting allegations in Turkey
+Bugged out
+
+HIS /O colleagues hailed him as a "soldier-intellectual." But yesterday Ilker Başbuğ, a former chief of Turkey's general staff, was jailed in Istanbul on charges of belonging to a terrorist gang bent on overthrowing Turkey's mildly Islamist government.
+
+Mr Başbuğ, who retired in 2010, is the highest-ranking officer to be arrested so far in the ongoing "Ergenekon" probe into assorted generals and their supposed allies in academia, the media and the business world who are accused of seeking to overthrow the government. Mr Başbuğ's alleged role in the conspiracy was to oversee a bevy of army-owned internet news portals that purported to prove that the government was seeking to introduce religious rule.
+
+Mr Başbuğ is the first former army chief to be prosecuted in a civilian court. His arrest brings to 139 the number of generals and admirals, both serving and retired, who have been jailed on suspicion of coup-plotting.
+
+Supporters of Recep Tayyip Erdoğan, the prime minister, say the move is a further step towards the government's consolidation of civilian control over an army that has toppled four governments since 1960. Until recently the generals were deemed untouchable, partly thanks to a swathe of laws they inserted into the constitution they wrote after their last coup, in 1980.
+
+Mr Erdoğan and his Justice and Development (AK) party have been steadily trimming their powers, most recently through a set of constitutional amendments that among other things allow the generals to be tried in civilian courts. These changes were approved by 58% of voters in a referendum in 2010, further cementing AK's power.
+
+Yet critics of the Ergenekon arrests say they have more to do with settling scores than with the rule of law. Defence lawyers have presented forensic reports suggesting that some of the evidence against their clients has been doctored.
+
+Eric Edelman, a former American ambassador to Turkey and number two at the Pentagon under George W. Bush, got to know Mr Başbuğ. He says: "The notion that he headed a terrorist organisation just strains credulity".
+
+It will also, he says, "underscore the serious questions about Turkey's continued commitment to press freedom and the rule of law." That commitment is waning. Nearly 500 students are in prison facing "terrorism" charges for such infractions as demanding free education. Some 97 journalists have also been imprisoned, many on similarly absurd grounds.
+
+Mr Erdoğan's increasingly authoritarian ways have prompted claims that his government has simply swapped places with the army. His supporters have a different worry. They say that having purged the old guard in the military, he is now cutting deals with the new set.
+
+That would help explain why the prime minister is supporting their renewed campaign against rebels of the separatist Kurdistan Workers' Party (PKK). This took a tragic turn last week when Turkish warplanes bombed a group of Kurdish civilians in northern Iraq, killing at least 34 of them. When conspiracists started to murmur Mr Erdogan rushed to the defence of the current chief of staff, Necdet Özel, saying it was out of the question that the army would deliberately bomb its own people.
+
+All this is a far cry from 2009 when Mr Erdoğan launched his "Kurdish opening", an attempt to solve Turkey's long-festering Kurdish problem through a flurry of democratic reforms rather than brutal military tactics. One element of this process was secret talks with the PKK, but these broke down last summer. In an ironic twist Mr Başbuğ, who was then chief of staff, firmly backed Mr Erdoğan.
+
+If found guilty, Mr Başbuğ will be convicted as a "terrorist", a label long reserved for the Kurdish separatists he sought to tame.

data/examples/texts/half_cocked_basel.txt

@@ -0,0 +1,16 @@
+Half-cocked Basel
+Stop-gap rules on banks’ trading books may add perilous complexity
+
+THE NEW-YEAR hangover throbbed agonisingly for investment bankers this year. Blame Basel 2.5, a new set of international rules which charges banks higher capital for the risks they run in their trading books (as opposed to their banking books, where they keep assets that they intend to hold to maturity). Those charges were too low before. And heaping higher costs on banks should please politicians and Joe Public. But they add another layer of complexity to banks’ risk management.
+
+Basel 2.5 came into force on December 31st in most European and major world financial jurisdictions. Switzerland applied the rules a year early, and the costs are substantial. Third-quarter figures for Credit Suisse show a 28% increase in risk-weighted assets, and hence capital charges, for its investment-banking activities purely because of Basel 2.5.
+
+The most notable laggard is America. US financial regulators do not oppose Basel 2.5, but it clashes with the Dodd-Frank act, America’s big wet blanket of a financial reform. Basel 2.5 uses credit ratings from recognised agencies such as Moody’s and Standard & Poor’s to calibrate capital charges. Dodd-Frank expressly forbids the use of such ratings agencies, whose poor judgments are held partly responsible for the crisis. Instead American regulators are working on their own cocktail of credit-risk calibrations for Basel 2.5, using market data and country-risk ratings from the OECD. Their solution is still months away from application (though not as distant as implementation by the Russians or Argentines).
+
+Basel 2.5 for the first time charges banks extra capital for the credit risk of what they hold in their trading portfolio (because the crisis showed that markets are not always liquid enough to be able to offload assets). That includes a charge for the risk that a counterparty goes bust. It also imposes heavy charges on securitised bundles of assets unless the credit risk of each piece of the bundle has an identifiable market price. Banks that have portfolios of trading positions which they reckon offset each other have to convince regulators that their risk models work or face being charged at a cruder, standardised rate.
+
+The problem with Basel 2.5, recognised by regulators and bankers alike, is its complexity. The risk of a trading portfolio must now be broken down into five “buckets”—value at risk (VaR), a measure of how much could be lost in an average trading day; stressed VaR (how much could be lost in extreme conditions); plus three types of credit risk ranging from the risk of single credits to those of securitised loans. Traders are understandably confused. For some banks, developing risk models and getting them approved is just too expensive: more complex businesses will be shut down. That will please those who want banks to be more boring.
+
+But unintended consequences will doubtless follow. Useful products may become less tradable. Trading of riskier products could migrate to unregulated entities. Banks may be tempted into new forms of regulatory arbitrage, by juggling assets between their trading book and their banking book. Worst of all, perhaps, is the increased risk of back-office bungling because of the extra complexity.
+
+Regulators recognise this risk. The Basel Committee on Banking Supervision, which drew up the rules and is also responsible for the full Basel 3 regime that will come into force in 2019, is still conducting what it calls a “fundamental review” of capital rules for banks’ trading books. Publication is not expected before March. Those sore heads will not soon clear.

data/examples/texts/hedge_funds.txt

@@ -0,0 +1,24 @@
+Rich managers, poor clients
+A devastating analysis of hedge-fund returns
+
+HEDGE-fund managers are the smartest investors around. With keen eyes and sharp brains, they spot and exploit inefficiencies in the markets. Or at least that is what the industry tells its clients.
+
+There is no doubt that hedge-fund managers have been good at making money for themselves. Many of America’s recently minted billionaires grew rich from hedge clippings. But as a new book* by Simon Lack, who spent many years studying hedge funds at JPMorgan, points out, it is hard to think of any clients that have become rich by investing in hedge funds (whereas Warren Buffett has made millionaires of many of his original investors). Indeed, since 1998, the effective return to hedge-fund clients has only been 2.1% a year, half the return they could have achieved by investing in boring old Treasury bills.
+
+How can that be, when traditional performance measures for the industry show average returns of 7% or so? The problem is a familiar one in fund management and is the equivalent of the “winner’s curse” that occurs with auctions (the successful bidder is doomed to overpay). Take a whole bunch of fund managers and give them an equal amount of money to invest. The managers that perform best initially will tend to attract more investors, and so will gradually become bigger than the moderate or poor performers (who will eventually go out of business).
+
+But the manager will not perform well indefinitely. By the time a bad year occurs, the manager will be running a much larger fund. In cash terms, the loss on the expanded fund may easily outweigh the gains made when the fund was smaller. The return of the average investor will be lower than the average return of the fund.
+
+What is true for individual funds also turns out to be true for the industry as a whole. Between 1998 and 2003 the average hedge fund earned positive returns every year, ranging from 5% in 2002 to 27% in 1999. Back then, however, the industry was quite small: overall assets only passed $200 billion in 2000.
+
+That strong performance attracted the attention of pension funds, charities and university endowments at a time when their portfolios had been clobbered by the bursting of the dotcom bubble. They duly piled into “alternative assets” like hedge funds and private equity. By early 2008 the hedge-fund industry had around $2 trillion under management.
+
+But that year turned out to be the annus horribilis for the hedge-fund sector. The average performance was a loss of 23%. In cash terms the loss for that single year was more than double the industry’s total assets under management in 2000, when it was still doing well. Mr Lack reckons that the industry may have lost enough money in 2008 to cancel out all the profits it made in the previous ten years.
+
+At this point, hedge-fund managers might cry foul. The losses suffered in 2008 make a huge impact on the way Mr Lack calculates his figures. If you use the same methodology on stockmarkets, hedge funds outperformed the S&P 500 between 2001 and the end of 2010.
+
+But private-equity managers are judged on a similar basis (the internal rate of return) to Mr Lack’s calculations. And his numbers probably flatter the hedge-fund industry. Indices of hedge-fund returns overstate the numbers because of factors such as “survivor bias” (poor performers stop reporting their numbers) and “backfill bias” (only successful newcomers start to report). These effects could add 3-5 percentage points a year to average returns. Many investors invest in the sector through funds of funds, which charge an additional layer of fees.
+
+Even if you allow for the rebound in markets (and hedge-fund returns) in 2009 and 2010, investors have still got the short end of the stick. They have yet to recover the losses suffered in 2008. But hedge-fund managers took home almost $100 billion in fees between 2008 and 2010 (and an aggregate haul of $379 billion between 1998 and 2010).
+
+Mr Lack’s book suggests the blind faith displayed by many institutional investors in hedge funds needs to be reconsidered. Individual managers may be brilliant but it is hard to spot them in advance. John Paulson was not particularly well-regarded before he made a fortune betting against subprime bonds—and his performance has slumped since. Investing in hedge funds will enable some lucky managers to enjoy an early retirement on their yachts. It will not enable pension funds to eliminate their deficits.

data/examples/texts/hose_and_dry.txt

@@ -0,0 +1,19 @@
+European banks
+Hose and dry
+The ECB fills banks with funds
+
+WHEN economists think of the financial system, it is usually as a frictionless conduit through which money flows to areas of the economy where it is most needed. A better analogy right now might be of a hosepipe with a knot tied in it. The European Central Bank (ECB) is pumping unprecedented amounts of money into one end of the pipe, but how much of that will find its way to the parched real economy is another question entirely.
+
+Start with the liquidity flowing into the euro-area banking system. On December 21st the ECB made available an eye-popping €489 billion ($628 billion) in three-year loans to more than 500 banks across Europe. The money was released in response to an almost total freeze since July in the bond markets that are an important source of long-term funding for banks. Demand for this ECB funding was much higher than expected, signalling just how much stress there is in the system.
+
+The ECB had previously tried to ease pressure by offering one-year loans to banks. Yet this had done little to encourage them to lend to companies or people, since the money banks owed the ECB would have to be repaid before they were due to be repaid by their customers. A shortage of longer-term funding also contributed to an increase in the riskiness of the whole financial system, since banks were being forced to rely ever more on short-term financing that needs to be rolled over.
+
+The flood of money being pumped into banks by the ECB goes a long way towards easing these funding pressures. Analysts at Morgan Stanley, an investment bank, reckon that the new facility adds about €235 billion in additional funding to the banks (since some of the money is being used to replace shorter-term loans) and takes the central bank’s total lending to the banking system to €979 billion. More important is the fact that it only has to be repaid in three years, which takes the average maturity on ECB loans to about 21 months, up from a mere ten weeks before the auction.
+
+This should help insulate most large banks from turmoil in the funding markets. But not all will be protected. Although the ECB has essentially made an offer of unlimited funds, banks are still constrained in how much they may borrow by the quality of the collateral that they are able to hand over. This is because their collateral is subject to an initial “haircut”, or reduction in value. A portfolio of relatively safe government bonds might attract a small haircut, whereas one consisting of loans to small businesses might be reduced in value by 40% or more. A bank focused on this area of business, as many of Europe’s smaller savings banks are, might thus only be able to borrow 60% of the value of its outstanding loans.
+
+A second constraint is that if the collateral the ECB holds falls in value, then banks need to post more. This also happens if there are credit downgrades on the bonds posted as collateral. The ECB’s haircut on sovereign bonds that are rated above “A-” is just 1.5%, but if they fall another notch this jumps up to 6.5%, Morgan Stanley notes. Weak banks in peripheral countries remain vulnerable to a downward spiral in which their holdings of government bonds are downgraded and fall in value, forcing them to come up with ever more collateral to keep borrowing from the central bank. It was this sort of collateral spiral that felled MF Global, a bust American broker, in its ill-fated bet on euro-area government bonds.
+
+Now that banks are getting funding, the big worry is whether they will do more than merely hoard it. The early signs are not encouraging. Over the Christmas weekend bank deposits at the ECB rose to a record €412 billion. That partly reflects a traditional year-end rush to tidy up banks’ balance-sheets but also nervousness about lending directly to others.
+
+Banks also have to present plans in January showing how they will raise an extra €115 billion in core capital to meet a new threshold imposed by the European Banking Authority. This capital shortfall has already prompted asset sales by banks, which would rather shrink their balance-sheets than tap shareholders. The ECB’s auction may make a credit crunch less severe, but it is not enough to avoid one.

data/examples/texts/hungarys_troubles.txt

@@ -0,0 +1,46 @@
+Hungary's troubles
+Not just a rap on the knuckles
+
+THE pressure is piling up on the beleaguered Hungarian government. Today the European Commission threatened it with legal action over several new "cardinal" laws that would require a two-thirds majority in parliament to overturn.
+
+The commission is still considering the laws, but today it highlighted concerns over three issues:
+
+ - The independence of the central bank. Late last year the Hungarian parliament passed a law which expands the monetary council and takes the power to nominate deputies away from the governor and hands it to the prime minister. A separate law opens the door to a merger between the bank and the financial regulator.
+
+ - The judiciary. More than 200 judges over the age of 62 have been forced into retirement and hundreds more face the sack. The new National Judicial Authority is headed by Tünde Handó, a friend of the family of Viktor Orban, the prime minister.
+
+ - The independence of the national data authority.
+
+That wasn't all the commission had to say today. Hungary also received a ticking-off from Olli Rehn (pictured), the economic-affairs commissioner, for not doing enough to tackle its budget deficit. It may now lose access to EU funds.
+
+Slammed in Brussels, the Hungarian government is also under pressure at home. Earlier this week Gordon Bajnai, who served as Socialist prime minister from 2009-10, fired off a broadside that sent shockwaves through the political and media establishments.
+
+After a year and a half of government by the right-wing Fidesz party, wrote Mr Bajnai in a lengthy article on the website of the Patriotism and Progress Public Policy Foundation, democracy has been destroyed in Hungary. The country, he warned, is scarred by division and is drifting towards bankruptcy and away from Europe.
+
+Mr Bajnai called for a radical change of government and a complete political re-orientation. “A new government must have a programme readily at hand that can be applied without delay: a programme that promotes the republic, reconciliation, and recovery.” 
+
+Fidesz is rattled by Mr Bajnai, who since leaving office has been teaching at Columbia University in New York. Understandably so. He headed a technocratic administration which stabilised the economy. Unlike his Socialist predecessor, Ferenc Gyurcsany, he was neither part of the old Communist elite nor connected to it by marriage, and so cannot be smeared as a "Komcsi". He is modern in outlook and well regarded internationally.
+
+Moreover, say those how know him, Mr Bajnai has little patience for the narcissistic exceptionalism that shapes Fidesz’s worldview. Exhibit A: the plaintive cry of Janos Martonyi, the foreign minister, who lamented recently: “The world will never understand our pains and spiritual wounds.” Such self-pity is unlikely to endear the Hungarian government to Brussels or Washington DC (to where it has sent an envoy this week to negotiate with the IMF).
+
+Fidesz won a two-thirds majority in 2010. But its support is evaporating, and analysts say there is a gap in the political market for a centrist pro-business party committed to democratic norms. Mr Bajnai, who has not ruled out a return to politics, would be an obvious candidate to lead it.
+
+Meanwhile, as Hungarians watch the value of their assets vaporise, in large part thanks to the government’s increasingly erratic policies, Mr Orban smirks his way through press conferences. Here he is dodging questions from a reporter from HVG, an economics weekly, about his responsibility for the crisis and trying to shift the blame to his old enemy Andras Simor, president of the central bank. The interview ran as follows:
+
+hvg.hu: Do you feel responsible for the falling/weakening forint?
+
+Mr Orban: You mean the president of the central bank? He did not comment on it.
+
+hvg.hu: No, you, Mr prime minister!
+
+Mr Orban: The personal responsibility of the president of the central bank was not discussed over the meeting.
+
+hvg.hu: You, your personal…!
+
+Mr Orban: That neither.
+
+Surrounded by yes-men and grinning flunkies, Mr Orban seems increasingly out of touch. His future will likely be decided not in the gilded corridors of the Hungarian parliament, but in Brussels and Washington DC.
+
+What happens next? If his hand is forced Mr Orban can probably endure policy reversals on the independence of the central bank and the data ombudsman. Sorry, he would say to his loyal followers: national crisis, what can you do.
+
+The dismantling of the judiciary would be another matter. If outsiders keep up the pressure and the judicial changes are judged to be in breach of the EU treaty, Mr Orban would be in a tricky spot. It’s hard to see how he could declare the 200-plus judges his government has forced into retirement ready for office after all, and still sit in his own.

data/examples/texts/indias_slowdown.txt

@@ -0,0 +1,15 @@
+India’s slowdown
+The case for the defence
+Why officials think investors are too bearish about India’s economy
+
+THE SMOG is so bad in Delhi right now that it seeps indoors. In one government building the far end of the corridor seems hazy. But the view of the mandarin working there is clear: India’s economic miracle is not over, regardless of the chatter among investors and howls about government paralysis from industrialists. He pokes fun at the latter. A year ago they were swanning around Davos proclaiming India could grow in its sleep, he says. Now, with growth dipping to 6.9% last quarter, from a peak of 10% (see chart), they are pleading for government action.
+
+Bears in Mumbai, India’s financial capital, worry that GDP growth might slip below 6% as confidence and investment slip. That partly reflects global woes, and partly too the gumming up of the bureaucracy due to a wave of graft allegations. But it is also because no big reforms have taken place for years; and such is the dire state of India’s politics that it is hard to imagine any being imminent. Things reached a nadir at the end of last year when the ruling coalition announced it would allow foreign supermarkets into the country, only to do a U-turn in the face of protests from the opposition and its own coalition partners. Shortly afterwards it failed to carry a key anti-corruption bill through parliament.
+
+The government is not blind to these concerns—in his new-year address the prime minister, Manmohan Singh, conceded that “it would be wrong to conclude that India is now unshakeably set on a process of rapid growth.” But officials in Delhi are more optimistic than the financial markets, for three reasons. First, they argue that growth is bottoming out. Inflation is showing signs of falling, which should allow the central bank to reverse its long series of interest-rate hikes. The recent drop in the rupee is a healthy adjustment, not cause for panic, they say. Meanwhile the euro zone is vaguely getting its act together and there are hints of a recovery in America. Growth, it is thought, will be about 7% for the fiscal year ending in March, respectable enough, and will pick up from there.
+
+Second, the long-term drivers of India’s boom are intact. There is “not much reason to change your mind,” says the mandarin. The rise in the savings rate, which allows more investment, will continue, partly thanks to a demographic bulge of people reaching working age. Even if there is a drop in capital expenditure, it should remain above 30% of GDP—a “handsome level”, says another official, that will boost the country’s potential. The government’s 12th five-year plan, which is due out soon, was originally expected to forecast growth of 9% between 2012 and 2017. That might fall to 8.5%, officials say, but no further.
+
+The final strut of the argument is that the politics are not as bad as they seem. Pessimists, the mandarin says, reckon “we’re just going to fiddle around and miss our opportunities.” But after important elections in February in Uttar Pradesh, the most populous state, the politicians may stop posturing and even co-operate to pass less contentious reforms such as a new national value-added tax. That would cut red tape and the fiscal deficit. And even if parliament stays gridlocked, there are lots of nuts-and-bolts reforms that do not require legislation. The government will try harder to tackle the bottlenecks that choke the power industry, for instance, and the paperwork that is snarling up big projects.
+
+The nub of the official argument is “calm down—and trust us to do just enough.” The trouble is the government has been saying this for a year, and business folk and investors seem to have lost heart. Firms have cut investment and the stockmarket was one of the world’s worst performers in dollar terms last year. Perhaps they are being too jumpy, but India does not have the luxury of dismissing what firms and investors think. The fiscal deficit, including the states and off-balance-sheet items, is running at 9-10% of GDP for the fourth year in a row. The current-account deficit is drifting towards 4% of GDP, officials admit, well above the country’s traditional comfort zone. India needs to command the confidence of domestic and foreign investors. Unless the reform process starts moving there is a risk that the financing of these deficits will become an acute problem—and that India’s economic miracle recedes further into the Delhi haze.

data/examples/texts/merkozy_rides_again.txt

@@ -0,0 +1,24 @@
+Saving the euro, part 473
+Merkozy rides again
+
+ANGELA MERKEL and Nicolas Sarkozy kicked off the 2012 season of the euro soap opera with a summit meeting in Berlin today. Neither said anything startling; certainly nothing that would betoken a swift and happy conclusion to the long-running saga.
+
+The German chancellor and the French president muted their differences over such issues as how quickly to introduce a tax on financial transactions and what the role of the European Central Bank (ECB) should be in supporting shaky members of the euro zone. “Our analysis is the same,” said Mr Sarkozy at the post-summit press conference.
+
+This did not calm markets’ nerves. The euro dropped to its lowest level against the dollar since September 2010 ($1.266) before the summit and recovered marginally as the two leaders met. Currency traders’ biggest worry is Greece’s failure to meet its fiscal targets, which means it may not get the fresh money it needs to avoid defaulting on its debt.
+
+At the opposite end of the confidence spectrum, investors are so eager to finance Germany that they accepted a negative interest rate on an auction of six-month paper, in effect paying Germany’s government for the privilege of lending to it. Germans will see this as vindication of their prudent policies, but it also serves to underline the dangerous economic divergences within the euro zone.
+
+The main significance of the Merkozy summit is that it seemed to signal a shift in emphasis. True, the austerity agenda—promoted by the Germans and grudgingly accepted by the French—is still there. Indeed, Mr Sarkozy boasted that France’s fiscal deficit was smaller than expected in 2011. Europe is making swift progress towards a “fiscal pact” to limit deficits, proclaimed Mrs Merkel, including German-style “debt brakes”. A new treaty should be signed by March.
+
+But fiscal self-denial will now be supplemented by what Mrs Merkel called a “second leg”, meaning economic growth and job creation. This is partly meant to help Mr Sarkozy, who faces a tough re-election fight this spring.
+
+All euro-zone countries, including Germany, are “prepared to do their homework” in this area, the chancellor promised, but it is not clear that much new is on offer. A big German stimulus package to boost growth in neighbouring countries is not in prospect (that would nobble the fiscal leg).
+
+Mrs Merkel spoke of spreading best practice in labour-market regulation across the euro zone (which is German practice, Mr Sarkozy admits) and spending existing European funds more quickly and effectively. Both ideas make sense; neither will prevent further financial turmoil, or a European recession. In the latest sign of fragility, German industrial production dropped 1% in November.
+
+The leaders tried to seem anything but complacent. Mr Sarkozy called the situation “very tense” and Mrs Merkel said they had “understood the needs of the hour.” The intention is to keep Greece from dropping out of the euro zone, but whatever happens Greece is an exceptional case, the leaders said (perhaps fearing that a Greek default or even an exit from the euro could not be avoided). As always, the chancellor dampened expectations of a quick “one-dimensional” solution to the crisis. The problem would be solved, she said, “step by step.”
+
+The next steps involve Italy, an indebted giant that poses a far greater threat to the euro than Greece. Mrs Merkel will meet Italy’s unelected prime minister, Mario Monti, in Berlin on Wednesday; she and Mr Sarkozy will hold a three-way summit with him in Rome on January 20th. European heads of government are to gather, probably on January 30th, to put the finishing touches to the fiscal pact.
+
+Also on the agenda, no doubt, will be a proposed financial-transactions tax. Britain is threatening a veto; Mr Sarkozy has said France will go it alone at first, if need be. Mrs Merkel wants the tax but her junior coalition partner, the Free Democrats, do not unless the British get on board. As the crisis sharpens, disagreements are likely to re-emerge over the role of the ECB and how to strengthen the euro zone's bail-out funds. The soap opera has a long way to run.

data/examples/texts/prada_is_not_walmart.txt

@@ -0,0 +1,9 @@
+Prada is not Walmart
+
+INDIA, if you believe the government, will be a land in which Starbucks and Prada thrive but where foreign firms will be prohibited from selling onions. It does not seem like much of a cause for celebration, but the announcement on January 11th that foreign “single brand” retailers could own 100% of their operations in India was meant to show the reform process was on track. It followed a debacle late last year when the government first announced that not only would single brand retailing be opened up, but foreign supermarkets would be allowed to operate in India too—and then was quickly forced into a U-turn on the latter promise after facing a rebellion within its own ranks and from the coalition parties it relies on in parliament.
+
+By emphasising that at least the single brand bit of retail reform is still on track, the government hopes to show the world that India is still open for business. But this is a meek change indeed. Single brand retailers, such as fashion chains, were already allowed to own 51% of their operations. And the political stink of last month is likely to scare those who are not already present because swathes of the political class have been shown to be populist and hostile for foreign firms. Individual states may still choose to override the central government’s rules. Lastly, the reform comes with a large catch: 30% of what is sold must be supplied from cottage industries in India. If you are selling a uniform product worldwide—a sofa or handbag made in China—that is a major hassle.
+
+The hope must be that India is on a journey to the right place, stumbling along the way. Perhaps the supplier rule will eventually be dropped, the argument goes. Maybe reluctant states will learn the error of their ways and open up too, after seeing the success of single brand retailers in other states. And maybe, after seeing an influx of investment from single brand retailers, the political climate will change and it will be easier to pass a reform that lets in supermarkets in too.
+
+Interviewed in Delhi earlier in January a government mandarin insisted that the supermarket reform was not dead. Yet all of this seems half hearted. India is a hard enough place as it is for foreign firms to make profits. Adding in a fickle polity just makes things worse. And it is a rather sorry day for progress when a rule tweak to allow Starbucks or Prada to own not 51%, but 100%, of their shops is presented as a meaningful economic reform.

data/examples/texts/republican_nomination.txt

@@ -0,0 +1,26 @@
+The Republican nomination
+Whose side are you on, Jon Huntsman?
+
+BEFORE it was clear whether Rick Perry would stay in the race or drop out, Erick Erickson, a conservative talking head for CNN and managing editor of Redstate.com, said, "If Rick Perry leaves the Republican race, there will not be a candidate in the field who authentically represents smaller government."
+
+But what about Ron Paul?
+
+For conventionally right-wing party stalwarts like Mr Erickson, Ron Paul doesn't count as real Republican, because of his principled anti-war stance. Of course, that makes him a more authentic representative of smaller government, war being the health of the state and all, but that's beside the point. The conservative tribe professes faith in smaller government, and it is membership in the tribe that determines the authenticity of one's devotion to the tribe's catechism. Ron Paul isn't really a member of the tribe, so he cannot "authentically represent smaller government". He may represent smaller government in fact, but not in the right way. As votes from the Iowa caucuses were being tallied, Mr Erickson saw fit to relate to his readers a rumour that "the Occupiers showed up for Ron Paul". You get the idea.
+
+And what about Jon Huntsman? According to the Cato Institute's "Fiscal Report Card on America's Governors" in 2008, the last year Messrs Perry and Huntsman were graded together, they received identical scores, tying for fifth place. Pretty good, huh? Since Cato's report card "grades the governors on their fiscal performance from a limited-government perspective", one wonders why Mr Perry but not Mr Huntsman is considered an authentic champion of smaller government. Indeed, Mr Huntsman's classically-conservative wariness of foreign entanglements in general, and his case for ending the conflict in Afghanistan in particular, contrast dramatically with Mr Perry's eagerness to revive the American occupation of Iraq, and suggest a rather more principled commitment to smaller government. Again, it all comes down to standing in the conservative tribe. 
+
+Long ago, Mr Erickson said he won't support Jon Huntsman ever because...well, here:
+
+While serving as the United States Ambassador to China, our greatest strategic adversary, Jon Huntsman began plotting to run against the President of the United States. This calls into question his loyalty not just to the President of the United States, but also his loyalty to his country over his own naked ambition.
+Whatever you think of this, it has nothing whatsoever to do with Mr Huntsman's commitment to smaller government, does it?
+
+In the comments to a follow-up post Mr Erickson explains himself further:
+
+Loyalty is a primary issue for me and there need be no further evidence that he is disloyal.It’s not a big issue in this day and age when we turn everything into partisan issues, but I guess I’m a bitter clinger to the way things used to be.
+According to Jonathan Haidt, a professor of psychology at the University of Virginia, in-group loyalty plays a much larger role in the moral judgments of conservatives than those of liberals. I don't mean to question Mr Erickson's judgment about Mr Huntsman's loyalty so much as point to it as a fairly representative expression of the conservative id operating precisely as Mr Haidt's studies describe.
+
+Now, I doubt many conservatives think Mr Huntsman was disloyal to America, as Mr Erickson charges. But I do think he is widely seen as a man of questionable loyalty. If you're like me, Mr Huntsman's willingness to set aside partisanship and serve in a Democratic administration, in spite of his high political aspirations, argues in favour of his loyalty to the country. But if you're like me you're not a conservative, and you don't really care that much about loyalty. Rock-ribbed conservatives I think see it like this: By agreeing to serve as ambassador to China under Barack Obama, Mr Huntsman picked a side, and it wasn't the side of the conservative tribe. And then he flaked on the Obama administration in order to run for president as a Republican. This is how I read Mr Erickson's denunciation of Mr Huntsman: "Are you crazy, Huntsman? You want back in? Now? No. Forget about it. You're dead to us."
+
+This leaves Mr Erickson, and millions of like-minded conservatives, in the odd position of preferring even Newt Gingrich, a man who has been disloyal to more than one wife. Indeed, a latter-day Dostoyevsky would be hard-pressed to imagine a nakeder embodiment of ambition than Newt Gingrich. Still, he's an honoured elder of the tribe. Meanwhile, an experienced fiscal-conservative governor with an outstanding grasp of foreign affairs who would stand an outstanding chance of defeating Barack Obama in the general election languishes in the polls. 
+
+In other words, good work Mr President.