Text-based Maori Language Detection

The code and its necessary helper files and libraries, and this README, live at:

    http://trac.greenstone.org/browser/gs3-extensions/maori-lang-detection

You can checkout from svn with:

    svn co http://svn.greenstone.org/gs3-extensions/maori-lang-detection

This checkout contains:

• tarball of the OpenNLP version that was current at the time of the commit, but you can also get it from its original site

  http://opennlp.apache.org/download.html

• the folder "models-trainingdata-and-sampletxts", itself containing:
• langdetect-183.bin: The LanguageDetectionModel to be used by OpenNLP is also included (again, current at the time of commit), but you can get it from http://opennlp.apache.org/models.html (direct link: https://www.apache.org/dyn/closer.cgi/opennlp/models/langdetect/1.8.3/langdetect-183.bin)
• mri-sent_trained.bin: our own custom generated model for Maori Sentence Detection
• mri-sent.train: the training text file for generating the mri-sent_trained.bin Maori Sentence Detection model
• sample_mri_paragraphs.txt: contains some text to test the Maori Sentence Detection model on. Its content is from Niupepa collection page http://www.nzdl.org/cgi-bin/library?gg=text&e=p-00000-00---off-0niupepa--00-0----0-10-0---0---0direct-10---4-------0-1l--11-en-50---20-about---00-0-1-00-0-0-11-1-0utfZz-8-00&a=p&p=about&l=mi&nw=utf-8)
• sample_maori_shorttext.txt: to test the MaoriTextDetector.java with
• "src" folder for Java classes, currently just MaoriTextDetector.java and its classfile. MaoriTextDetector.java uses the aforementioned LanguageDetectionModel, langdetect-183.bin, to detect whether input text from a file or stdin is in Maori or not
• gen_SentenceDetection_model.sh, our custom script that generates both the mri-sent.train and model mri-sent_trained.bin files mentioned above
• the script works on opennlp's leipzig corpus of 100k Maori sentences from 2011 to get its sample sentences into the correct format in the mri-sent.train file
• from this file containing training sentences, it generates the Sentence Detector Model, mri-sent_trained.bin
• mri-opennlp-corpus.tar.gz: a tarball containing the 100k Maori sentences opennlp corpus checked out with svn in its original directory structure from https://svn.apache.org/repos/bigdata/opennlp/trunk/mri_web_2011_100K-sentences.txt

Once you've svn checked out the maori-lang-detection from gs3-extensions, create a folder called "models" and put the files langdetect-183.bin and mri-sent_trained.bin from the folder "models-trainingdata-and-sampletxts" into it. (These are just zip files, but have to remain with the .bin extension in order for OpenNLP to use them. If you ever wish to see the contents of such a .bin file, you can rename to .zip and use the Archive Manager or other Zip tool to inspect the contents.)

Next extract the apache-opennlp-1.9.1-bin.tar.gz. This will create a folder called apache-opennlp-1.9.1. Move the "models" folder you created in step 1 into this folder.

Before you can compile or run the MaoriTextDetector program, you always have to prepare a terminal by setting up the environment for OpenNLP as follows:

	   cd /type/here/path/to/your/extracted/apache-opennlp-1.9.1
	   export OPENNLP_HOME=`pwd`

If you want to recompile, go up into the checked out maori-lang-detection folder's "src" subfolder. To compile, make sure you have the JDK7+ bin folder on your PATH environment variable. Still in the SAME terminal as where you set up the OPENNLP_HOME environment in step 3, you can now run: maori-lang-detection/src$ javac -cp ".:$OPENNLP_HOME/lib/opennlp-tools-1.9.1.jar" org/greenstone/atea/MaoriTextDetector.java

To run the MaoriTextDetector program, you will need the JDK or JRE 7+ bin folder on your PATH and still in the SAME terminal as where you set up the OPENNLP_HOME environment in step 3, type one of the following:

	   maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" org.greenstone.atea.MaoriTextDetector --help
	   (prints the usage, including other options)                  
	 
	   maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" org.greenstone.atea.MaoriTextDetector --file <full/path/to/textfile>
	 
	   maori-lang-detection/src$ java -cp ".:$OPENNLP_HOME/lib/*" org.greenstone.atea.MaoriTextDetector -
	       which expects text to stream in from standard input.
	       If entering text manually, then remember to press Ctrl-D to indicate the end of StdIn as usual.