论文标题
部分可观测时空混沌系统的无模型预测
PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation
论文作者
论文摘要
尽管预先训练的语言模型最近在产生流利文本方面取得了进展，但现有的方法仍然遭受长期文本生成任务中的不连贯问题，这些任务需要适当的内容控制和计划以形成连贯的高级逻辑流。在这项工作中，我们提出了一个新的生成框架，它利用自回归自我发挥的机制来动态进行内容计划和表面实现。为了指导产量句子的产生，我们的框架丰富了具有潜在表示的变压器解码器，以维护以单词袋为基础的句子级别的语义计划。此外，我们引入了一个新的基于连贯的对比学习目标，以进一步提高产出的连贯性。对两个具有挑战性的长篇文本生成任务进行了广泛的实验，包括反驳生成和意见文章生成。自动评估和人类评估都表明，我们的方法显着胜过强大的基线，并生成更丰富的内容的更连贯的文本。
Despite recent progress of pre-trained language models on generating fluent text, existing methods still suffer from incoherence problems in long-form text generation tasks that require proper content control and planning to form a coherent high-level logical flow. In this work, we propose PLANET, a novel generation framework leveraging autoregressive self-attention mechanism to conduct content planning and surface realization dynamically. To guide the generation of output sentences, our framework enriches the Transformer decoder with latent representations to maintain sentence-level semantic plans grounded by bag-of-words. Moreover, we introduce a new coherence-based contrastive learning objective to further improve the coherence of output. Extensive experiments are conducted on two challenging long-form text generation tasks including counterargument generation and opinion article generation. Both automatic and human evaluations show that our method significantly outperforms strong baselines and generates more coherent texts with richer contents.
