论文标题
英语泰卢固语代码中的单词级别语言标识混合数据
Word Level Language Identification in English Telugu Code Mixed Data
论文作者
论文摘要
如今，经常观察到多语言或社交配置内句内代码切换（ICS）或代码混合（CM）。在世界上，大多数人都知道一种以上的语言。在社交媒体平台中，CM使用尤其明显。此外，在技术，健康和法律的背景下，IC尤其重要，在技术，健康和法律的背景下，以母语的方式很难传达即将到来的发展。在对话系统，机器翻译，语义解析，浅解析等应用程序中。CM和代码转换构成了严重的挑战。为了在代码混合数据中进一步进步，必要的步骤是语言标识。在本文中，我们介绍了各种模型的研究 - 中殿贝叶斯分类器，随机森林分类器，有条件的随机字段（CRF）和隐藏的马尔可夫模型（HMM），用于英语 - 泰卢固语代码混合数据。考虑到代码混合语言中资源的匮乏，我们为单词级别的语言标识提出了CRF模型和HMM模型。我们最佳性能系统是基于CRF的F1得分为0.91。
In a multilingual or sociolingual configuration Intra-sentential Code Switching (ICS) or Code Mixing (CM) is frequently observed nowadays. In the world, most of the people know more than one language. CM usage is especially apparent in social media platforms. Moreover, ICS is particularly significant in the context of technology, health, and law where conveying the upcoming developments are difficult in one's native language. In applications like dialog systems, machine translation, semantic parsing, shallow parsing, etc. CM and Code Switching pose serious challenges. To do any further advancement in code-mixed data, the necessary step is Language Identification. In this paper, we present a study of various models - Nave Bayes Classifier, Random Forest Classifier, Conditional Random Field (CRF), and Hidden Markov Model (HMM) for Language Identification in English - Telugu Code Mixed Data. Considering the paucity of resources in code mixed languages, we proposed the CRF model and HMM model for word level language identification. Our best performing system is CRF-based with an f1-score of 0.91.
