论文标题
随机森林中的注意力和自我注意力
Attention and Self-Attention in Random Forests
论文作者
论文摘要
提出了使用注意力和自我发项机制共同解决回归问题的新模型。这些模型可以被视为基于注意力的随机森林的扩展，其思想源于将Nadaraya-Watson内核回归和Huber污染模型的结合施加到随机森林中。自我发作旨在捕获树木预测的依赖性，并消除随机森林中的噪声或异常预测。自我发场模块与用于计算权重的注意模块共同训练。结果表明，注意力重量的训练过程减少到解决单个二次或线性优化问题。提出并比较了一般方法的三个修改。还考虑了对随机森林的特定多头自我注意。自我注意事项的头部是通过更改其调谐参数（包括内核参数和模型的污染参数）来获得的。使用各种数据集的数值实验说明了所提出的模型，并表明自我发作的补充改善了许多数据集的模型性能。
New models of random forests jointly using the attention and self-attention mechanisms are proposed for solving the regression problem. The models can be regarded as extensions of the attention-based random forest whose idea stems from applying a combination of the Nadaraya-Watson kernel regression and the Huber's contamination model to random forests. The self-attention aims to capture dependencies of the tree predictions and to remove noise or anomalous predictions in the random forest. The self-attention module is trained jointly with the attention module for computing weights. It is shown that the training process of attention weights is reduced to solving a single quadratic or linear optimization problem. Three modifications of the general approach are proposed and compared. A specific multi-head self-attention for the random forest is also considered. Heads of the self-attention are obtained by changing its tuning parameters including the kernel parameters and the contamination parameter of models. Numerical experiments with various datasets illustrate the proposed models and show that the supplement of the self-attention improves the model performance for many datasets.
