论文标题
通过条件变形的自动编码器对未配对数据的新型视图综合
Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder
论文作者
论文摘要
新型视图合成通常需要来自源和目标视图的配对数据。本文提出了在CVAE-GAN框架下的视图翻译模型，而无需配对数据。我们设计了一个有条件的可变形模块（CDM），该模块使用视图向量作为过滤器来卷积VAE中主分支的特征图。它生成几对位移图以使特征变形，例如2D光流。将结果馈入基于变形的特征归一化模块（DFNM），该模块（DFNM）缩放并抵消了主要分支特征，鉴于其变形为侧分支的输入。凭借CDM和DFNM的优势，编码器输出了视图的后部，而解码器则将其从中汲取的代码综合了重建的和ViewTranslated的图像。为了进一步确保观点与其他因素之间的分离，我们在代码上添加了对抗性培训。对多重和3D椅子数据集的结果和消融研究验证了CVAE和设计模块中框架的有效性。
Novel view synthesis often needs the paired data from both the source and target views. This paper proposes a view translation model under cVAE-GAN framework without requiring the paired data. We design a conditional deformable module (CDM) which uses the view condition vectors as the filters to convolve the feature maps of the main branch in VAE. It generates several pairs of displacement maps to deform the features, like the 2D optical flows. The results are fed into the deformed feature based normalization module (DFNM), which scales and offsets the main branch feature, given its deformed one as the input from the side branch. Taking the advantage of the CDM and DFNM, the encoder outputs a view-irrelevant posterior, while the decoder takes the code drawn from it to synthesize the reconstructed and the viewtranslated images. To further ensure the disentanglement between the views and other factors, we add adversarial training on the code. The results and ablation studies on MultiPIE and 3D chair datasets validate the effectiveness of the framework in cVAE and the designed module.
