论文标题
将神经面部合成缩放到高FPS和通过神经缓存的低潜伏期
Scaling Neural Face Synthesis to High FPS and Low Latency by Neural Caching
论文作者
论文摘要
最近的神经渲染方法极大地提高了图像质量，接近光真相。但是，基础神经网络的运行时间很高，排除了需要在低延迟下高分辨率的虚拟现实应用程序。深网中层的顺序依赖性使它们的优化变得困难。我们通过将信息从上一个框架中缓存，以加快具有隐式扭曲的当前框架的处理来打破这种依赖性。使用浅网络的扭曲减小了延迟，可以将缓存操作进一步平行以提高帧速率。与现有的时间神经网络相反，我们的量身定制的是通过调节基础表面网格的变化来呈现面部的新型视野。我们在既定基准序列上都需要根据需要进行触觉的3D肖像头像的观点渲染方法。翘曲可将延迟减少70 $ \％$（从商品GPU上的49.4ms降低到14.9ms），并在多个GPU上相应地缩放了帧速率，同时仅将图像质量降低1 $ \％$，使其作为端到端视图依赖性3D Telectectrencencing应用程序的一部分。我们的项目页面可以在以下网址找到：https：//yu-frank.github.io/lowlatency/。
Recent neural rendering approaches greatly improve image quality, reaching near photorealism. However, the underlying neural networks have high runtime, precluding telepresence and virtual reality applications that require high resolution at low latency. The sequential dependency of layers in deep networks makes their optimization difficult. We break this dependency by caching information from the previous frame to speed up the processing of the current one with an implicit warp. The warping with a shallow network reduces latency and the caching operations can further be parallelized to improve the frame rate. In contrast to existing temporal neural networks, ours is tailored for the task of rendering novel views of faces by conditioning on the change of the underlying surface mesh. We test the approach on view-dependent rendering of 3D portrait avatars, as needed for telepresence, on established benchmark sequences. Warping reduces latency by 70$\%$ (from 49.4ms to 14.9ms on commodity GPUs) and scales frame rates accordingly over multiple GPUs while reducing image quality by only 1$\%$, making it suitable as part of end-to-end view-dependent 3D teleconferencing applications. Our project page can be found at: https://yu-frank.github.io/lowlatency/.
