论文标题
激活功能：潜入最佳激活函数
Activation Functions: Dive into an optimal activation function
论文作者
论文摘要
激活功能已成为神经网络的重要组成部分之一。适当激活函数的选择会影响这些方法的准确性。在这项研究中，我们通过将其定义为现有激活功能的加权总和，然后在训练网络时进一步优化这些权重，从而找到最佳的激活函数。该研究使用三个流行的图像数据集，MNIST，FashionMnist和KMNIST，使用了三个激活功能，即Relu，Tanh和Sin。我们观察到，Relu激活函数可以轻松忽略其他激活功能。同样，我们看到初始层更喜欢具有relu或levelelu类型的激活函数，但是更深的层倾向于更喜欢更收敛的激活函数。
Activation functions have come up as one of the essential components of neural networks. The choice of adequate activation function can impact the accuracy of these methods. In this study, we experiment for finding an optimal activation function by defining it as a weighted sum of existing activation functions and then further optimizing these weights while training the network. The study uses three activation functions, ReLU, tanh, and sin, over three popular image datasets, MNIST, FashionMNIST, and KMNIST. We observe that the ReLU activation function can easily overlook other activation functions. Also, we see that initial layers prefer to have ReLU or LeakyReLU type of activation functions, but deeper layers tend to prefer more convergent activation functions.
