论文标题
对BERT和GPT-2转移神经体系结构进行财务情感分析的敏感性分析
Sensitivity Analysis on Transferred Neural Architectures of BERT and GPT-2 for Financial Sentiment Analysis
论文作者
论文摘要
新型NLP单词嵌入和深度学习技术的爆炸引起了潜在应用的重大努力。这些方向之一是金融部门。尽管在GPT和BERT等最新模型中完成了很多工作，但是在预训练后通过微调进行的处理以及有关其参数的敏感性的信息，相对较少的作品。我们研究了预先训练的GPT-2和BERT模型转移神经体系结构的性能和灵敏度。我们基于冻结变压器层，批处理大小和学习率测试微调性能。我们发现BERT的参数对微调的随机性过敏，并且在这种实践中GPT-2更加稳定。同样很明显，GPT-2和BERT的较早层包含应该维护的基本单词模式信息。
The explosion in novel NLP word embedding and deep learning techniques has induced significant endeavors into potential applications. One of these directions is in the financial sector. Although there is a lot of work done in state-of-the-art models like GPT and BERT, there are relatively few works on how well these methods perform through fine-tuning after being pre-trained, as well as info on how sensitive their parameters are. We investigate the performance and sensitivity of transferred neural architectures from pre-trained GPT-2 and BERT models. We test the fine-tuning performance based on freezing transformer layers, batch size, and learning rate. We find the parameters of BERT are hypersensitive to stochasticity in fine-tuning and that GPT-2 is more stable in such practice. It is also clear that the earlier layers of GPT-2 and BERT contain essential word pattern information that should be maintained.
