NLTK Installation with Python easy_install

Few weeks ago I wrote the NLTK on Ubuntu Quick Start Guide. Now with the release of NLTK (Natural Language Toolkit) 2.0b5 today the NLTK installation has been greatly simplified thanks to the nltk python egg (See Changelog).

To get started with NLTK install, you first need the python-setuptools package.

$ sudo apt-get install python-setuptools 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  python-setuptools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 195kB of archives.
After this operation, 909kB of additional disk space will be used.
Get:1 http://in.archive.ubuntu.com karmic/main python-setuptools 0.6c9-0ubuntu4 [195kB]
Fetched 195kB in 9s (20.2kB/s)                                                 
Selecting previously deselected package python-setuptools.
(Reading database ... 106971 files and directories currently installed.)
Unpacking python-setuptools (from .../python-setuptools_0.6c9-0ubuntu4_all.deb) ...
Setting up python-setuptools (0.6c9-0ubuntu4) ...



 Now lets install the NLTK with easy_install program.


$ sudo easy_install http://nltk.googlecode.com/files/nltk-2.0b5-py2.6.egg
 
Downloading http://nltk.googlecode.com/files/nltk-2.0b5-py2.6.egg
Processing nltk-2.0b5-py2.6.egg
creating /usr/local/lib/python2.6/dist-packages/nltk-2.0b5-py2.6.egg
Extracting nltk-2.0b5-py2.6.egg to /usr/local/lib/python2.6/dist-packages
Adding nltk 2.0b5 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/nltk-2.0b5-py2.6.egg
Processing dependencies for nltk==2.0b5
Searching for PyYAML==3.08
Reading http://pypi.python.org/simple/PyYAML/
Reading http://pyyaml.org/wiki/PyYAML
Best match: PyYAML 3.08
Downloading http://pyyaml.org/download/pyyaml/PyYAML-3.08.zip
Processing PyYAML-3.08.zip
Running PyYAML-3.08/setup.py -q bdist_egg --dist-dir /tmp/easy_install-T7Y0La/PyYAML-3.08/egg-dist-tmp-vRjvDM
build/temp.linux-i686-2.6/check_libyaml.c:2:18: error: yaml.h: No such file or directory
build/temp.linux-i686-2.6/check_libyaml.c: In function ‘main’:
build/temp.linux-i686-2.6/check_libyaml.c:5: error: ‘yaml_parser_t’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
build/temp.linux-i686-2.6/check_libyaml.c:5: error: for each function it appears in.)
build/temp.linux-i686-2.6/check_libyaml.c:5: error: expected ‘;’ before ‘parser’
build/temp.linux-i686-2.6/check_libyaml.c:6: error: ‘yaml_emitter_t’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:6: error: expected ‘;’ before ‘emitter’
build/temp.linux-i686-2.6/check_libyaml.c:8: warning: implicit declaration of function ‘yaml_parser_initialize’
build/temp.linux-i686-2.6/check_libyaml.c:8: error: ‘parser’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:9: warning: implicit declaration of function ‘yaml_parser_delete’
build/temp.linux-i686-2.6/check_libyaml.c:11: warning: implicit declaration of function ‘yaml_emitter_initialize’
build/temp.linux-i686-2.6/check_libyaml.c:11: error: ‘emitter’ undeclared (first use in this function)
build/temp.linux-i686-2.6/check_libyaml.c:12: warning: implicit declaration of function ‘yaml_emitter_delete’

libyaml is not found or a compiler error: forcing --without-libyaml
(if libyaml is installed correctly, you may need to
 specify the option --include-dirs or uncomment and
 modify the parameter include_dirs in setup.cfg)
zip_safe flag not set; analyzing archive contents...
Adding PyYAML 3.08 to easy-install.pth file

Installed /usr/local/lib/python2.6/dist-packages/PyYAML-3.08-py2.6-linux-i686.egg
Finished processing dependencies for nltk==2.0b5



 Now you done, import the NLTK and start downloading the NTLK data.


$ python
Python 2.6.2+ (release26-maint, Jun 19 2009, 15:14:35) 
[GCC 4.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.download()
NLTK Downloader
---------------------------------------------------------------------------
    d) Download      l) List      c) Config      h) Help      q) Quit
---------------------------------------------------------------------------
Downloader> l

Packages:
/usr/local/lib/python2.6/dist-packages/nltk-2.0b5-py2.6.egg/nltk/__init__.py:588: DeprecationWarning: object.__new__() takes no parameters
  [ ] maxent_ne_chunker... ACE Named Entity Chunker (Maximum entropy)
  [ ] abc................. Australian Broadcasting Commission 2006
  [ ] brown............... Brown Corpus
  [ ] alpino.............. Alpino Dutch Treebank
  [ ] cess_cat............ CESS-CAT Treebank
  [ ] brown_tei........... Brown Corpus (TEI XML Version)
  [ ] cmudict............. The Carnegie Mellon Pronouncing Dictionary (0.6)
  [ ] biocreative_ppi..... BioCreAtIvE (Critical Assessment of Information
                           Extraction Systems in Biology)
  [ ] cess_esp............ CESS-ESP Treebank
  [ ] chat80.............. Chat-80 Data Files
  [ ] city_database....... City Database
  [ ] conll2002........... CONLL 2002 Named Entity Recognition Corpus
  [ ] conll2000........... CONLL 2000 Chunking Corpus
  [ ] conll2007........... Dependency Treebanks from CoNLL 2007 (Catalan
                           and Basque Subset)
  [ ] dependency_treebank. Dependency Parsed Treebank
  [ ] floresta............ Portuguese Treebank
  [ ] genesis............. Genesis Corpus
  [ ] gazetteers.......... Gazeteer Lists
Hit Enter to continue: 

NLTK on Ubuntu Quick Start Guide

Update July 19, 2009 : You can now use nltk python egg instead, read the NLTK Installation with Python setuptools post.

While attending a short program in computational linguistics at Dravidian University, Dr. Arul introduced me to NLTK (Natural Language Toolkit). It was full two years before that I finally decided to have a close look at it. Like most linguists at the lab I used Perl programming language. With new version of NLTK 2.0 released last month, NLTK now works with python 2.6. Here a quick start guide for NLTK on Ubuntu Linux.

Installing NLTK on Ubuntu with Python 2.6

At the time of writing this post the Debian package on NLTK download page is built for Python 2.5. Ubuntu ships with Python 2.6 by default. So you need to download the source package from the NLTK download page.

NLTK needs some dependency modules, lets install them.
sudo apt-get install python-numpy python-matplotlib prover9

Uncompress the source package and run the NLTK setup.
$ unzip nltk-2.0b3.zip
$ cd nltk-2.0b3/
$ ls build LICENSE.txt nltk PKG-INFO README.txt setup.py yaml
$ sudo python setup.py install

After finishing the NLTK setup, you should download the NLTK data which contains various corpora, tagsets and treebank data etc.
$ python
Python 2.6.2+ (release26-maint, Jun 19 2009, 15:14:35)
[GCC 4.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.download()


Learning NLTK

The best place to start is the NLTK book Natural Language Processing with Python Analyzing Text with the Natural Language Toolkit. The book is released under public domain, so you can read it online on NLTK website itself. I would recommand you to buy a copy of this book as the procceds will go into the future development of NLTK.

There aren't many videos about NLTK. I recently stumbled upon this video lecture by the trinity of NLTK Steven Bird, Ewan Klein, and Edward Loper.

If you are new to computational linguistics and need good grounding in this field you should also consider reading these texts.

Speech and Language Processing (2nd Edition)

Natural Language Understanding (2nd Edition)

Foundations of Statistical Natural Language Processing

Lttoolbox Install Notes

Lttoolbox is a lexical and morphological analyser package from the Apertium machine translation system. Here is a brief install notes taken from my NLP summer school lab notes.

Build Lttoolbox

Download the lttoolbox source package from sourceforge.

tar zxvf lttoolbox-xxxxx.tgz cd lttoolbox-xxxxx/ make make install

You will need root level permission to install other dependency packages for lttoolbox . GNU Debian users however should use the lttoolbox debian package available in stable/testing/unstable and also read the Installing Apertium on Debian documentation page for more information.

Usage

If the tool is installed properly then you can look up the program install path with which lt-comp command.

Here I am compiling the Hindi dictionary with a left-to-right option lr, the output is stored in binary format in file 'mydata'. This binary file is used by the lt-proc processor to do the morph analysis.

lt-comp lr hi.dix.noun_f mydata lt-proc ./mydata

For more information do read the manual pages for the lt-comp and lt-proc tools or fire me a email, I would love to you help you.

Kompare to the rescue

If you have been reading this blog for the past week then you already know that I am in NLP Summer School in IIIT-Hyderabad. And while doing a project comparing the results of various dependency parsers we had to read through large text files to find the errors and its a real pain. Well, I strong believe in the motto of letting the computer do the dirty work. I would have preferred to use a editor like Vim or Emacs with multiple spilt windows to compare the files and their diff's output. However I found that the KDE's GUI based Kompare (frontend to diff, comp or cmp) to be better suited for this project since my team mates are linguists and quite new to gnu/Linux. Also almost all the lab machines have it installed KDE as default desktop envirnoment.

The above screenshot shows kompare in side-by-side mode and just by using arrow keys or the mouse we could manipulate both the windows at the same time and tally our results. It did help to finish the job quickly but its still a pain.

Stanford Parser Gotcha

I have aged 6 years since I last coded in Java. So I wasn't quite sure what to make of this Stanford Parser error in the NLP summer school lab. For a while I was quite sure that I didn't set the Java CLASSPATH properly and thus the parser is spewing blood and gore all over me. But after a hour of mucking around I realised that it wasn't so. The real problem seems to that the parser doesn't seem to run well with gij .

$ gij --version gij (GNU libgcj) version 3.4.6 20060404 (Red Hat 3.4.6-3)

$ ./lexparser.csh Warning: -server not understood. Ignoring. Warning: -mx150m not understood. Ignoring. Exception in thread "main" java.lang.ClassFormatError: edu.stanford.nlp.parser.lexparser.LexicalizedParser (erroneous method access flags) at _Jv_ClassReader.throw_class_format_error(byte) (/usr/lib/libgcj.so.5.0.0) at _Jv_ClassReader.handleMethod(int, int, int, int) (/usr/lib/libgcj.so.5.0.0) at _Jv_ClassReader.read_methods() (/usr/lib/libgcj.so.5.0.0) at _Jv_ClassReader.parse() (/usr/lib/libgcj.so.5.0.0) at _Jv_DefineClass(java.lang.Class, byte[], int, int) (/usr/lib/libgcj.so.5.0.0) at java.lang.VMClassLoader.defineClass(java.lang.ClassLoader, java.lang.String, byte[], int, int, java.security.ProtectionDomain) (/usr/lib/libgcj.so.5.0.0) at java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) (/usr/lib/libgcj.so.5.0.0) at java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) (/usr/lib/libgcj.so.5.0.0) at java.net.URLClassLoader.findClass(java.lang.String) (/usr/lib/libgcj.so.5.0.0) at gnu.gcj.runtime.VMClassLoader.findClass(java.lang.String) (/usr/lib/libgcj.so.5.0.0) at java.lang.ClassLoader.loadClass(java.lang.String, boolean) (/usr/lib/libgcj.so.5.0.0) at _Jv_FindClass(_Jv_Utf8Const, java.lang.ClassLoader) (/usr/lib/libgcj.so.5.0.0) at java.lang.Class.forName(java.lang.String, boolean, java.lang.ClassLoader) (/usr/lib/libgcj.so.5.0.0) at java.lang.Class.forName(java.lang.String) (/usr/lib/libgcj.so.5.0.0) at gnu.gcj.runtime.FirstThread.run() (/usr/lib/libgcj.so.5.0.0) at _Jv_ThreadRun(java.lang.Thread) (/usr/lib/libgcj.so.5.0.0) at _Jv_RunMain(java.lang.Class, byte const, int, byte const, boolean) (/usr/lib/libgcj.so.5.0.0) at __gcj_personality_v0 (/home/iasnlp2008/Desktop/stanford-parser-2007-08-19/java.version=1.4.2) at __libc_start_main (/lib/tls/libc-2.3.4.so) at _Jv_RegisterClasses (/home/iasnlp2008/Desktop/stanford-parser-2007-08-19/java.version=1.4.2)

To test my hunch I changed machines and tried with other version of Java (viz 1.5.x) and it worked without a breaking a sweat. Except for the minor problems like this mistake in script the exparser-gui.csh on line $scriptdir/stanford-parser.jar: everything seems to be alright so far.

$ java -version java version "1.5.0_05" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_05-b05) Java HotSpot(TM) Server VM (build 1.5.0_05-b05, mixed mode)