论文标题
神经网是模块化的吗？通过可不同的重量面膜检查功能模块化
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
论文作者
论文摘要
其子网的神经网络（NNS）预计可重复使用的功能将提供许多优势，包括通过有效重组功能构件的有效重组，可解释性，防止灾难性干扰等。但是，当前的检查方法无法将模块与其功能联系起来。在本文中，我们提出了一种基于学习二进制重量面罩的新方法，以识别负责特定功能的单个权重和子网。使用此功能强大的工具，我们对NNS中新兴模块化的广泛研究涵盖了几个标准的体系结构和数据集。我们证明了常见的NN如何重复使用子模型，并就语言任务的系统概括相关问题提供了新的见解。
Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, including compositionality through efficient recombination of functional building blocks, interpretability, preventing catastrophic interference, etc. Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality. In this paper, we present a novel method based on learning binary weight masks to identify individual weights and subnets responsible for specific functions. Using this powerful tool, we contribute an extensive study of emerging modularity in NNs that covers several standard architectures and datasets. We demonstrate how common NNs fail to reuse submodules and offer new insights into the related issue of systematic generalization on language tasks.
