论文标题
用RERM验证大型语言模型
Validating Large Language Models with ReLM
论文作者
论文摘要
尽管大型语言模型（LLMS）因其产生自然文本的能力而被吹捧，但人们对LLM的可能的负面影响（例如数据记忆，偏见和不适当的语言）越来越担心。不幸的是，LLMS的复杂性和发电能力使得（并纠正）此类担忧很困难。在这项工作中，我们介绍了RERM，这是一种使用标准正则表达式验证和查询LLM的系统。 RERM正式化并实现了广泛的语言模型评估，将复杂的评估规则降低到简单的正则表达查询。我们的结果探讨了围绕记忆，性别偏见，毒性和语言理解的查询，这表明，与最先进的临时查询相比，RELM可达到高达15倍的系统效率，2.5倍的数据效率以及提高统计和迅速调查的覆盖范围。 RERM为LLM验证的日益重要的问题提供了竞争性和一般的基线。
Although large language models (LLMs) have been touted for their ability to generate natural-sounding text, there are growing concerns around possible negative effects of LLMs such as data memorization, bias, and inappropriate language. Unfortunately, the complexity and generation capacities of LLMs make validating (and correcting) such concerns difficult. In this work, we introduce ReLM, a system for validating and querying LLMs using standard regular expressions. ReLM formalizes and enables a broad range of language model evaluations, reducing complex evaluation rules to simple regular expression queries. Our results exploring queries surrounding memorization, gender bias, toxicity, and language understanding show that ReLM achieves up to 15x higher system efficiency, 2.5x data efficiency, and increased statistical and prompt-tuning coverage compared to state-of-the-art ad-hoc queries. ReLM offers a competitive and general baseline for the increasingly important problem of LLM validation.
