论文标题
两次刻度参与者的全球融合用于解决线性二次调节器
Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator
论文作者
论文摘要
参与者批评（AC）的增强学习算法一直是许多具有挑战性的应用背后的强大力量。然而，它的收敛性一般都是脆弱的。为了研究其不稳定性，现有作品主要考虑具有有限状态和动作空间的罕见的双环变体或基本模型。我们研究了更实用的单样本两次尺度AC，用于解决规范线性二次调节器（LQR）问题，在每个迭代中，每个迭代中的一个样本在无界的连续状态和动作空间中仅更新一次。现有的分析无法得出这样一个挑战性案例的融合。我们开发了一个新的分析框架，该框架允许建立全局收敛到$ε$ - 最佳解决方案，最多最多是$ \ Mathcal {o}（ε^{ - 2.5}）$样本复杂性。据我们所知，这是单个样本两次尺度AC的第一个有限时间收敛分析，用于以全球最优性解决LQR。样本复杂性通过订单改善了其他变体的复杂性，从而阐明了单个样品算法的实际智慧。我们还通过全面的模拟比较进一步验证了理论发现。
The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an $ε$-optimal solution with at most an $\mathcal{O}(ε^{-2.5})$ sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.
