biblicit 1.0 → 2.0.3

data/parscit/COPYING.LESSER

@@ -0,0 +1,165 @@
+		   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions. 
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version. 
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

data/parscit/INSTALL

@@ -0,0 +1,105 @@
+INSTALLATION:
+
+Please see the README.TXT first to familiarize yourself with the
+binaries (shell scripts) for this software package.  You should first
+attempt to get the basic command line clients working first.  The
+Troubleshooting section of the home page (in doc/index.html in this
+distribution; identical to the ParsCit home page at
+http://wing.comp.nus.edu.sg/parsCit/#t) also contains some notes of
+import.
+
+Please note: Currently the software is not supported by any auto
+install package -- we expect a UNIX-knowledgable person to be able to
+get the installation working within 30 minutes to an hour.
+
+------------------------------------------------------------
+**Prerequisites**
+
+You must have ruby and perl and a working C++ compiler on your local
+machine.
+
+ParsCit use the following Perl libraries from CPAN
+- Class::Struct
+- Getopt::Long
+- Getopt::Std
+- File::Basename
+- File::Spec
+- FindBin
+- HTML::Entities
+- IO::File
+- POSIX
+- XML::Parser
+- XML::Twig
+- XML::Writer
+- XML::Writer::String
+
+------------------------------------------------------------
+**Command-line client**
+
+These are the two scripts: citeExtract.pl and parseRefStrings.pl.  The
+second script is a subset of the first, and should be your first
+target to get running.
+
+1) You will first need to reinstall, recompile the crfpp package and
+place the binaries at crfpp/.  We are using CRF++ version 0.51.  To do
+this follow these instructions in the crfpp directory.  CRF++ is the
+core conditional random field learner, that is re-distributed in this
+ParsCit software.  It is due to Taku Kudo.
+
+  $ cd crfpp
+  # we're going to rebuild CRF 51
+  $ rm -Rf CRF++-0.51
+  $ tar -xvzf CRF++-0.51.tar.gz
+  $ cd CRF++-0.51
+
+  $ configure  
+  # important! the first "make" command fails for me; not sure why 
+  # Also see the Troubleshooting notes in the ParsCit home page
+  # (doc/index.html) for other possible problems with this step.
+  $ make
+  $ make clean
+  $ make
+  # optional, you may want to use sudo or root privileges to install CRF binaries so that other applications can use CRF.  Thanks to Priya Venkateshan for this.
+  $ sudo make install
+  # hopefully by here, you have a successful build of both crf_test and crf_learn
+
+  # move executables to where parscit expects to find them
+    $ cp crf_learn crf_test ..
+    # on Windows you may have to do this instead, as the executables are named with .exe
+    $ copy crf_learn.exe ../crf_learn
+    $ copy crf_test.exe ../crf_test
+  $ cd .libs
+  $ cp -Rf * ../../.libs
+
+2) Once the binaries are placed properly, you may need to edit the
+lib/ParsCit/Config.pm file to point to the proper directories on your
+machine.
+
+3) Edit the shebang lines (first line) of the scripts in the bin/
+directory to point to the proper versions of perl
+
+4) Try running parseRefStrings.pl and citeExtract.pl on the .txt data
+found in demodata.  It should produce the same result as found in the
+.out files. In the bin/ directory try:
+
+./citeExtract.pl -m extract_all ../demodata/sample2.txt
+
+./citeExtract.pl -i xml -m extract_all ../demodata/E06-1050.xml
+
+------------------------------------------------------------
+**Web Service**
+
+These are the remaining parts of the system.  There are two sets of
+client/server pairs (perl ones tested at IST and ruby ones tested at
+NUS).  You can use either and the installation does not require both.
+
+1) In order to use the ParsCit web service you will need the following
+modules in your perl library:
+
+Log::Log4perl
+Log::Dispatch
+
+2) Edit lib/ParsCit/Config.pm to provide values appropriate for your
+environment.  Also edit wsdl/ParsCit.wsdl to reflect any changes to
+your service URL.
+

data/parscit/README

@@ -0,0 +1,97 @@
+ParsCit README
+
+This software is copyrighted 2008, 2009, 2010, 2011 by Min-Yen Kan,
+Isaac G. Councill, C. Lee Giles, Minh-Thang Luong and Huy Nhat Hoang
+Do.  This program and all its source code is distributed are under the
+terms of the GNU General Public License (or the Lesser GPL).
+
+    ParsCit is free software: you can redistribute it and/or modify it
+    under the terms of the GNU Lesser General Public License as
+    published by the Free Software Foundation, either version 3 of the
+    License, or (at your option) any later version.
+
+    ParsCit is distributed in the hope that it will be useful, but
+    WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+    Lesser General Public License for more details.
+
+    You should have received a copy of the GNU Lesser General Public
+    License along with ParsCit.  If not, see
+    <http://www.gnu.org/licenses/>.
+
+See INSTALL.txt and USAGE.txt for more information.
+CHANGELOG.txt for updates on code and model changes.
+Also see doc/index.html for the web page describing this project.
+
+----------------------------------------------------------------------
+This software has been developed together at IST at Penn State
+University and at the National University of Singapore (NUS).
+
+Current development (from 2008 onwards) has been done at NUS only.
+The newer version of files are supported only through the newer
+scripts written at NUS.
+
+SectLabel has been developed at NUS (2010) to retrieve logical
+structure of scientific documents. For further information, please
+refer to the webpage description and TroubleShooting section.
+
+----------------------------------------------------------------------
+Files:
+
+bin/	- Binaries / scripts for running ParsCit
+   /archtest.pl		- Used by IST installation
+   /citeExtract.pl 	- Script for running the whole pipeline from a text/XML file 
+			  (logical structure parsing, body/references segmentation, reference string parsing)
+   /parsecit-client.pl	- Original perl web service client (tested at IST)
+   /parsecit-server.pl	- Original perl web service server (tested at IST)
+   /ParsCitClient.rb	- Ruby web service client for the parsCit/parsHed modules (tested at NUS)
+   /ParsCitClientWSDL.rb - Ruby web service client for parsCit/parsHed modules via WSDL (tested at NUS)
+   /ParsCitServer.rb	- Ruby web service server for the parsCit/parsHed modules (tested at NUS)
+   /redo.parsCit.pl	- Cross Validation training script and training notes 
+			  (at end of script) for ParsCit
+
+   /headExtract.pl	- Script for running header structure parsing.  
+   /parsHed/redo.parsHed.pl	- Cross Validation training script and 
+				  training notes (at end of script) for ParsHed
+
+   /sectExtract.pl	- Script for running sectLabeler module for logical structure parsing.
+   /sectLabel/README.txt	- Detailed descriptions of SectLabel scripts, including GenericSect ones
+   /sectLabel/redo.sectLabel.pl	- Cross Validation training script and 
+				  training notes (at end of script) for SectLabel 
+
+  /BiblioScript         - Include BiblioScript and BibUtils codes # Thang v100901
+CHANGELOG.TXT	- Changes between versions of the code.
+crfpp/		- The CRF++ machine learning package used within ParsCit
+     /traindata	- Sample training data for the CRF++ code of ParsCit,
+                  ParsHed, and SectLabel 
+demodata/	- Test data for sanity checking outputs
+doc/		- The CGI base installation and documentation (scientific 
+                  publications) about ParsCit	
+INSTALL.TXT	- installation and basic instructions
+
+lib
+   /ParsCit	- perl libraries for ParsCit 
+   /ParsHed	- perl libraries for ParsHed 
+   /SectLabel	- perl libraries for SectLabel
+   /cgiLog.txt	- CGI log for running web interface (nb, different than web 
+                  service interface)
+README.TXT	- this file
+resources/	- dictionary files and CRF++ trained models
+         /parsCitDict.txt - dictionary file shared by ParsCit and ParsHed
+         /parsCit.model - default model used by ParsCit
+
+         /parsHed/parsHed.model - default model used by ParsHed (trained at line level)
+         /parsHed/keywords - most frequent keywords
+         /parsHed/bigram - most frequent bigrams
+         /parsHed/archive/parsHed.model - old model (trained at token level)
+
+         /sectLabel/sectLabel.model - default model used by SectLabel for plain text input
+         /sectLabel/sectLabel.modelXml - default model used by SectLabel for XML input
+         /sectLabel/genericSect.model - default model used by GenericSect
+         /sectLabel/funcword - English function word list
+
+tmp/		- guess! :-)
+USAGE.TXT	- Usage files for the citeExtract and web service interface 
+wsdl/		- Web service description language file 
+    /parscit.wsdl - WSDL file from describing the ParsCit service from NUS (newer version)
+    /ParsCit.wsdl - WSDL file describing the ParsCit service from IST (older version)

data/{perl/ParsCit/README.TXT → parscit/USAGE}

@@ -1,22 +1,14 @@
 ParsCit README
-IGC
+(updated in 100401 distribution by MYK)
+Originally by IGC / MYK 
 
 ParsCit is a utility for extracting citations from research papers
 based on Conditional Random Fields and heuristic regularization.
+Please see the installation notes before attempting to use the
+package.  
 
-Installation:
-
-In order to use the ParsCit web service you will need the following
-modules in your perl library:
-
-Log::Log4perl
-Log::Dispatch
-
-Edit lib/ParsCit/Config.pm to provide values appropriate for your
-environment.  Also edit wsdl/ParsCitService.wsdl to reflect any
-changes to your service URL.
-
-Command Line Usage:
+------------------------------------------------------------
+COMMAND LINE USAGE:
 
 Command line utilities are provided for extracting citations from
 text documents.  By default, text files are expected to be encoded
@@ -24,11 +16,28 @@ in UTF-8, but the expected encoding can be adjusted using perl
 command line switches.  To run ParsCit on a single document, execute
 the following command:
 
-citeExtract.pl textfile [outfile]
+# Thang v100401: updated
+citeExtract.pl [-m mode] [-i <inputType>] <filename> [outfile]
 
 If "outfile" is specified, the XML output will be written to that
 file; otherwise, the XML will be printed to STDOUT.
 
+The mode switch (-m) has five settings: extract_citations,
+extract_header, extract_section, extract_meta, and extract_all.
+extract_citations is the default, which extracts and parses the
+citations in the reference/bibliography section of the input text
+file.  extract_header extracts and parses the header (first page) of
+the input text file.  extract_section extracts and parses the full
+document to retrieve logical structures.
+
+extract_meta performs both the extract_citations and extrac_header
+operations.  extract_all performs all three operations (_citations,
+_header, _section).
+
+The mode switch (-i) has two options: raw and xml. With "raw"
+(default), the system accepts raw text input; where as with "xml", the
+system expects XML output from Omnipage.
+
 There is also a web service interface available, using the
 SOAP::Lite perl module.  To start the service, just execute:
 
@@ -57,6 +66,7 @@ parscit-client.pl filePath repositoryID
 
 If the call is successful, the XML output will be printed to STDOUT.
 
+------------------------------------------------------------
 API:
 
 The ParsCit libraries may be used directly from external perl

data/parscit/bin/archtest.pl

@@ -0,0 +1,31 @@
+#!/usr/bin/perl -CSD
+#
+# Simple command script for executing ParsCit in an
+# offline mode (direct API call instead of going through
+# the web service).
+#
+# Isaac Councill, 08/23/07
+#
+use strict;
+use FindBin;
+use lib "$FindBin::Bin/../lib";
+use ParsCit::Controller;
+
+my $textFile = $ARGV[0];
+my $outFile = $ARGV[1];
+
+if (!defined $textFile) {
+    print "Usage: $0 textfile [outfile]\n";
+    exit;
+}
+
+my $rXML =
+    ParsCit::Controller::extractCitationsAlreadySegmented($textFile);
+
+if (defined $outFile) {
+    open (OUT, ">$outFile") or die "Could not open $outFile for writing: $!";
+    print OUT $$rXML;
+    close OUT;
+} else {
+    print "$$rXML";
+}