论文标题
在计算限制下的可用信息理论
A Theory of Usable Information Under Computational Constraints
论文作者
论文摘要
我们提出了一个新的框架，以推理复杂系统中的信息。我们的基础是基于香农信息理论的各种扩展，该理论考虑了观察者的建模能力和计算约束。所得\ emph {预测$ \ Mathcal {V} $ - 信息}涵盖了共同信息和其他信息性概念，例如确定系数。与香农的共同信息和违反数据处理不平等的行为不同，可以通过计算创建$ \ mathcal {v} $ - 信息。这与深度神经网络提取了表示学习中逐渐信息性特征的层次结构一致。此外，我们表明，通过合并计算约束，即使在具有PAC风格保证的高维度中，也可以从数据中可靠地从数据中可靠地估算信息。从经验上讲，我们证明了预测性$ \ MATHCAL {V} $ - 对于结构学习和公平表示学习而言，信息比共同信息更有效。
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive $\mathcal{V}$-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, $\mathcal{V}$-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, $\mathcal{V}$-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive $\mathcal{V}$-information is more effective than mutual information for structure learning and fair representation learning.
