چکیده:
Persian natural language processing (NLP) researchers have many limitations to
access linguistic tools which are suitable for text processing. Therefore, research
in Persian text processing is very limited. Since dataset is an important
requirement for experiments and their evaluation, we aimed to create appropriate
corpora for information retrieval and natural language processing in Persian. The
provided corpora in this article are based on HAMSHAHRI dataset which is
appropriate for simple information retrieval and simple natural language
processing because it has not been tagged. We converted this dataset to tagged
collection and increased its text quality. The new corpora minimize the text
preprocessing requirement. Here we have used STep-1 tools for text processing
and have proposed some ideas to remove the bugs of these tools in order to
increase their quality. At the end we used the new corpora for text retrieval and
results showed performance improvement.
خلاصه ماشینی:
Since dataset is an important requirement for experiments and their evaluation, we aimed to create appropriate corpora for information retrieval and natural language processing in Persian.
The provided corpora in this article are based on HAMSHAHRI dataset which is appropriate for simple information retrieval and simple natural language processing because it has not been tagged.
Since dataset is an important requirement for experiments and their evaluation, in this article we have tried to create appropriate corpora for information retrieval and natural language processing in Persian.
The provided corpora in this article are based on HAMSHAHRI collection which is appropriate for simple information retrieval and simple natural language processing because it has not been tagged.
, 2006) are the other Persian texts datasets which DOTIR is appropriate for information retrieval but has not been tagged and BIJAN KHAN is appropriate for only natural language processing but this dataset is not free.
Notice produced corpora in this article are useful for both of Persian text retrieval and natural language processing.
The used rules for comparing main word tag and stem tag Table 3 Some Samples for Incorrect Stems {مراجعه شود به فایل جدول الحاقی} The Structure of Produced Dataset The produced corpora are useful for information retrieval and natural language processing.
The results showed that the new dataset is effective in improving performance of many applications in Persian information retrieval and Persian natural language processing systems.