Sunday, November 28, 2010

NLTK and Mallet

I am quite impressed by the available open source NLP softwares. Recently, I have used two of them after playing with most the major softwares. I like NLTK because it is in python and hence it is easy to use and it has a lot of utilities. Mallet is the other one I like. It's in java and it seems to have good sequence modeling tools such as FSM, HMM and MRF etc. It's fantastic that there are also robust parser available in both python and in java. I remember that I wrote a HTML parser a long back after getting completely frustrated by mal formed htmls while working on a project. Today I found Jericho (java) and BeautifulSoup both work wonderfully for bad HTML data. However, the core theoretical NLP problem is still very hard because human languages are quite ambiguous and humen communication system is quite buggy (actually I should say it is high performance distributed and fault tolerant system and languages are inherently, intentionally vague) than a computer program. It is amazing that human can understand the meaning of a sentence in a language like English by seeing a few context words only.

No comments: