论文标题
PocketNN：通过直接反馈对准和纯C ++激活神经网络的仅整数训练和推理
PocketNN: Integer-only Training and Inference of Neural Networks via Direct Feedback Alignment and Pocket Activations in Pure C++
论文作者
论文摘要
标准深度学习算法是使用浮点实数实现的。这为在可能没有专用浮点单元（FPU）的低端设备上实现它们提供了一个障碍。结果，Tinyml的研究人员考虑了机器学习算法，这些算法只能使用整数操作在低端设备上训练和运行深层神经网络（DNN）。在本文中，我们提出了Pocketnn，这是Pure C ++中的轻巧且独立的概念验证框架，用于仅使用整数训练和推断DNN。与其他方法不同，PocketNN直接在整数上运行，而无需任何明确的量化算法或自定义的固定点格式。这是通过口袋激活使这成为可能的，该激活是为仅整数DNN而设计的激活功能的家族，以及一种称为直接反馈比对（DFA）的新兴DNN训练算法。与标准反向传播（BP）不同，DFA独立训练每一层，从而避免整数溢出，这是将BP与仅整数操作一起使用时的关键问题。我们使用Pocketnn在两个著名的数据集（MNIST和时尚狂人）上培训了一些DNN。我们的实验表明，接受PocketNN培训的DNN分别在MNIST和时尚狂热数据集上获得了96.98％和87.7％的精度。这些精度非常接近使用BP训练的同等DNN，并具有浮点实际数字操作，因此准确性降解分别仅为1.02％P和2.09％P。最后，我们的PocketNN具有高端设备的高兼容性和可移植性，因为它是开源的，并在纯C ++中实现，而无需任何依赖关系。
Standard deep learning algorithms are implemented using floating-point real numbers. This presents an obstacle for implementing them on low-end devices which may not have dedicated floating-point units (FPUs). As a result, researchers in tinyML have considered machine learning algorithms that can train and run a deep neural network (DNN) on a low-end device using integer operations only. In this paper we propose PocketNN, a light and self-contained proof-of-concept framework in pure C++ for the training and inference of DNNs using only integers. Unlike other approaches, PocketNN directly operates on integers without requiring any explicit quantization algorithms or customized fixed-point formats. This was made possible by pocket activations, which are a family of activation functions devised for integer-only DNNs, and an emerging DNN training algorithm called direct feedback alignment (DFA). Unlike the standard backpropagation (BP), DFA trains each layer independently, thus avoiding integer overflow which is a key problem when using BP with integer-only operations. We used PocketNN to train some DNNs on two well-known datasets, MNIST and Fashion-MNIST. Our experiments show that the DNNs trained with our PocketNN achieved 96.98% and 87.7% accuracies on MNIST and Fashion-MNIST datasets, respectively. The accuracies are very close to the equivalent DNNs trained using BP with floating-point real number operations, such that accuracy degradations were just 1.02%p and 2.09%p, respectively. Finally, our PocketNN has high compatibility and portability for low-end devices as it is open source and implemented in pure C++ without any dependencies.
