چکیده:
A bilingual corpus is considered as a very important knowledge source and an inevitable requirement for many natural language processing (NLP) applications in which two languages are involved. For some languages such as Persian, lack of such resources is much more significant. Several applications, including statistical and example-based machine translation needs bilingual corpora, in which large amounts of texts from two different languages have been aligned at the sentence or phrase levels. In order to meet this requirement, this paper aims to propose an accurate and hybrid sentence alignment method for construction of an English-Persian parallel corpus. As the first step, the proposed method uses statistical length based analysis for filtering of candidates. Punctuation marks are used as a directing feature to reduce the complexity and increase the accuracy. Finally, the proposed method makes use of some lexical knowledge in order to produce the final output. . In the phase of lexical analysis, a bilingual dictionary as well as a Persian semantic net (denoted as FarsNet) is used to calculate the extended semantic similarity. Experiments showed the positive effect of expansion on synonym words by extended semantic similarity on the accuracy of the sentence alignment process. In the proposed matching scheme, a semantic load based approach (which considers the verb as the pivot and the main part of a sentence) was also used in order for increasing the accuracy. The results obtained from the experiments were promising and the generated parallel corpus can be used as an effective knowledge source by researchers who work on Persian language.
خلاصه ماشینی:
A Hybrid Accurate Alignment method for large Persian-English corpus construction based on statistical analysis and Lexicon/Persian Word net hdastgheibHgmail.
Several applications, including statistical and example-based machine translation needs bilingual corpora, in which large amounts of texts from two different languages have been aligned at the sentence or phrase levels.
In order to meet this requirement, this paper aims to propose an accurate and hybrid sentence alignment method for construction of an English- Persian para11e1 corpus.
Experiments showed the positive effect of expansion on synonym words by extended semantic similarity on the accuracy of the sentence alignment process.
In the proposed matching scheme, a semantic load based approach (which considers the verb as the pivot and the main part of a sentence) was also used in order for increasing the accuracy.
Keywords: Parallel corpora, Hybrid sentence alignment, English-Persian corpus, Extended semantic similarity Introduction In recent years, due to the popularity of the Internet, the volume of online texts in distinct languages is increasing tremendously.
Thus, the goal of the present study is to meet this requirement by automatically generating a parallel English-Persian corpus, aligned at the sentence level.
This evaluation showed that the pure length based method cannot be accurate enough for pairs of languages that have different alphabet set like Persian- English.
In this work, it was intended to automatically produce a large parallel corpus for Persian- English pair, aligned at sentence level.