A study on part of speech tagging

Part of Speech (POS) tagging has high importance in the domain of Natural Language Processing (NLP). POS tagging determines grammatical category to any token, such as noun, verb, adjective, person, gender, etc. Some of the words are ambiguous in their categories and what tagging does is to clear of ambiguous word according to their context. Many taggers are designed with different approaches to reach high accuracy. In this paper we present a new tagging algorithm with a Hybrid algorithm. This algorithm combines the statistics and the rule based tagger to tag Persian unknown words. These algorithms use morphological and syntactical rules for tagging. These algorithms are applied in Gate package.This package has two parts in tagging; part of tokenization and part of tagging. Many problems depend on part of tokenization. Tokenization is detecting of tokens in a text. In this part, morphological analysis is very important and makes some problems in computational analysis. Persian morphological makes some problems in computational analysis. There is another case which causes some problems in tokenization and is called Persian script.In this paper, we elaborate some problems in Persian morphology in tokenization and Persian script.The purpose of this paper is to improve tagging and also to study problems in Gate package in tokenization part according to study of linguistics.After improving and studying of problems, this package was evaluated with two kinds of texts; standard and non standard texts. Accuracy of Gate package with the standard text and non standard text are 97 and 92%, respectively

کلیدواژه ها:

rule based ، statistical based ، tagging ، tokenization ، unknown word