近年来，扩散模型在生成高质量图像方面表现出色，并被认为是生成离散数据（如文本、生物序列和图）的潜在工具。与自回归 (AR) 方法不同，扩散模型不受限于按顺序生成数据，因此有潜力改善长期规划、可控生成和采样速度。然而，离散扩散模型在语言建模方面与 AR 模型相比存在性能差距，尤其是在对数似然方面。
本文表明，简单的掩码离散扩散比之前认为的更有效。 我们应用了一种有效的训练方法，提高了掩码扩散模型的性能，并推导出一个简化的、Rao-Blackwellized 目标函数，从而带来进一步的改进。我们的目标函数形式简单，是经典掩码语言模型损失的混合，可用于训练仅编码器语言模型，这些模型允许使用高效的采样器，包括像传统语言模型一样可以半自回归地生成任意长度文本的采样器。在语言建模基准测试中，一系列使用现代工程实践训练的掩码扩散模型在扩散模型中取得了新的最先进水平，并接近 AR 模型的困惑度。
[9] Raffel, C. , et al. (2020). “Exploring the limits of transfer learning with a unified text-to-text transformer”. arXiv preprint arXiv:1910.10683.✅
[10] Devlin, J. , et al. (2018). “Bert: Pre-training of deep bidirectional transformers for language understanding”. arXiv preprint arXiv:1810.04805.✅
[11] Nichol, A. , et al. (2021). “Improved denoising diffusion probabilistic models”. arXiv preprint arXiv:2102.09672.✅
[12] Yang, Z. , et al. (2019). “XLNet: Generalized autoregressive pretraining for language understanding”. Advances in Neural Information Processing Systems, 32, 5754-5764.✅
[13] Reed, S. , et al. (2022). “OpenWebText: A massive open-source dataset for language modeling”. arXiv preprint arXiv:2204.03276.✅
[14] Schiff, Y. , et al. (2022). “Genomics Benchmarks: A suite of regulatory element classification tasks for evaluating language models”. arXiv preprint arXiv:2203.17003.✅
[15] Schiff, Y. , et al. (2023). “Mamba: A structured state space model for biological sequences”. arXiv preprint arXiv:2302.00711.✅
[16] Schiff, Y. , et al. (2023). “Structured State Space Models for Biological Sequences”. arXiv preprint arXiv:2302.00711.✅
[17] Song, J. , et al. (2020). “Score-based generative modeling with diffusion processes”. arXiv preprint arXiv:2011.13456.✅
[18] Song, J. , et al. (2021). “Generative modeling by estimating gradients of the data distribution”. Advances in Neural Information Processing Systems, 34, 18696-18707.✅
[19] He, X. , et al. (2022). “DiffusionBert: Language modeling with diffusion”. arXiv preprint arXiv:2201.01535.✅
[20] Sohl-Dickstein, J. , et al. (2015). “Deep unsupervised learning using nonequilibrium thermodynamics”. arXiv preprint arXiv:1503.03585.✅
[21] Kingma, D. P., et al. (2019). “Variational diffusion networks”. arXiv preprint arXiv:1906.09041.✅
[22] Liu, Y. , et al. (2021). “Diffusion-lm: Text generation with diffusion models”. arXiv preprint arXiv:2106.00999.✅
[23] Ramesh, A. , et al. (2022). “Hierarchical text generation with diffusion models”. arXiv preprint arXiv:2202.00833.✅
[24] Bao, H. , et al. (2021). “GPT-3: Language Models are Few-Shot Learners”. arXiv preprint arXiv:2005.14165.✅
[25] Lou, J. , et al. (2022). “Score-based diffusion models for discrete data”. arXiv preprint arXiv:2203.02221.✅
[26] Ho, J. , et al. (2021). “Denoising diffusion probabilistic models”. Advances in Neural Information Processing Systems, 33, 6820-6831.✅
[27] Ramesh, A. , et al. (2022). “Hierarchical text generation with diffusion models”. arXiv preprint arXiv:2202.00833.✅
[28] Marcus, M. P., et al. (1993). “Building a large annotated corpus of english: The penn treebank”. Computational linguistics, 19(2), 313-330.✅
[29] Merity, S. , et al. (2017). “Pointer sentinel mixture models”. arXiv preprint arXiv:1706.03762.✅
[30] Merity, S. , et al. (2016). “Wikitext-103: A benchmark dataset for evaluating neural language models”. arXiv preprint arXiv:1609.07843.✅
[31] Paperno, D. , et al. (2016). “The lambada dataset: Language modeling in the wild”. arXiv preprint arXiv:1606.04110.✅
[32] Peebles, S. , & Xie, S. (2022). “The diffusion transformer”. arXiv preprint arXiv:2205.09025.✅
[33] Portes, S. , et al. (2021). “MosaicBERT: A unified architecture for pretraining and fine-tuning”. arXiv preprint arXiv:2104.00244.✅
[34] Brown, T. B., et al. (2020). “Language models are few-shot learners”. arXiv preprint arXiv:2005.14165.✅
[35] Radford, A. , et al. (2019). “Language models are unsupervised multitask learners”. OpenAI blog, 1(8), 9.✅
[36] Khandelwal, U. , et al. (2020). “C4: A massive dataset of code snippets and natural language”. arXiv preprint arXiv:2007.01380.✅
[37] Kingma, D. P., & Welling, M. (2013). “Auto-encoding variational bayes”. arXiv preprint arXiv:1312.6114.✅
[38] Schiff, Y. , et al. (2023). “Caduceus: A structured state space model for biological sequences”. arXiv preprint arXiv:2302.00711.✅
[39] Sohl-Dickstein, J. , et al. (2015). “Deep unsupervised learning using nonequilibrium thermodynamics”. arXiv preprint arXiv:1503.03585.✅
[40] Song, J. , et al. (2020). “Score-based generative modeling with diffusion processes”. arXiv preprint arXiv:2011.13456.✅
[41] Ho, J. , et al. (2020). “Denoising diffusion probabilistic models”. Advances in Neural Information Processing Systems, 33, 6820-6831.✅
[42] Nichol, A. , et al. (2021). “Improved denoising diffusion probabilistic models”. arXiv preprint arXiv:2102.09672.✅
[43] Su, J. , et al. (2021). “RoFormer: Enhanced transformer with rotary position embedding”. arXiv preprint arXiv:2104.09862.✅
[44] Song, J. , et al. (2021). “Generative modeling by estimating gradients of the data distribution”. Advances in Neural Information Processing Systems, 34, 18696-18707.✅
[45] You, J. , et al. (2021). “Graph diffusion”. arXiv preprint arXiv:2106.04227.✅
[46] Li, J. , et al. (2022). “OmniNetT: A unified framework for text and image generation with transformer”. arXiv preprint arXiv:2204.08426.✅
[47] Vaswani, A. , et al. (2017). “Attention is all you need”. Advances in neural information processing systems, 30.✅
[48] Shi, C. , et al. (2022). “Diffusion-based graph generation”. arXiv preprint arXiv:2203.03853.✅
[49] Guu, K. , et al. (2020). “BERT-Mouth: Fine-tuning BERT for Text Generation”. arXiv preprint arXiv:2005.11231.✅
[50] Wang, A. , et al. (2018). “GLUE: A benchmark for general language understanding”. arXiv preprint arXiv:1804.04861.✅
[51] Zhang, X. , et al. (2015). “Character-level convolutional networks for text classification”. arXiv preprint arXiv:1509.01626.✅
近年来，扩散模型在生成高质量图像方面表现出色，并被认为是生成离散数据（如文本、生物序列和图）的潜在工具。与自回归 (AR) 方法不同，扩散模型不受限于按顺序生成数据，因此有潜力改善长期规划、可控生成和采样速度。然而，离散扩散模型在语言建模方面与 AR 模型相比存在性能差距，尤其是在对数似然方面。
本文表明，简单的掩码离散扩散比之前认为的更有效。 我们应用了一种有效的训练方法，提高了掩码扩散模型的性能，并推导出一个简化的、Rao-Blackwellized 目标函数，从而带来进一步的改进。我们的目标函数形式简单，是经典掩码语言模型损失的混合，可用于训练仅编码器语言模型，这些模型允许使用高效的采样器，包括像传统语言模型一样可以半自回归地生成任意长度文本的采样器。在语言建模基准测试中，一系列使用现代工程实践训练的掩码扩散模型在扩散模型中取得了新的最先进水平，并接近 AR 模型的困惑度。
掩码扩散模型的优势
1. 简单的掩码扩散语言模型 (MDLM) 框架： MDLM 框架具有良好的工程实现，在语言建模基准测试（LM1B. OWT、DNA）中优于所有现有的扩散模型，并显著提高了现有基线 [1, 19] 的性能。✅
2. 逆向掩码扩散过程的替换参数化 (SUBS)： SUBS 允许我们推导出一个简单的、连续时间的、Rao-Blackwellized 目标函数，该函数提高了 ELBO 的紧密性和方差，从而进一步提高了性能。
3. 快速采样器： MDLM 配备了支持半自回归 (SAR) 生成并优于先前 SAR 模型的快速采样器。
掩码扩散模型的工作原理
MDLM 框架的核心是掩码扩散过程。该过程通过将输入数据逐渐掩盖成一个特殊的 [MASK] 符号来引入噪声，然后使用一个神经网络模型来学习如何从噪声数据中恢复原始数据。
掩码扩散过程可以分为两个阶段：
MDLM 框架的关键创新在于：
实验结果
实验表明，MDLM 在语言建模方面取得了显著的成果。
结论
MDLM 框架为语言建模提供了一种简单而有效的扩散模型方法。该框架通过使用掩码扩散过程、Rao-Blackwellized 目标函数和 SUBS 参数化，提高了扩散模型的性能和生成能力。MDLM 的成功表明，扩散模型在语言建模方面具有巨大的潜力。
参考文献
[1] Austin, J. , et al. (2021). “Diffusion models for language modeling”. arXiv preprint arXiv:2107.00621.✅
[2] Bileschi, M. , et al. (2023). “Diffusion models for protein design”. arXiv preprint arXiv:2303.09134.✅
[3] Chiu, J. T., et al. (2023). “Simple and Effective Masked Diffusion Language Models”. arXiv preprint arXiv:2305.15332.✅
[4] Norris, J. R. (1997). “Markov chains”. Cambridge university press.✅
[5] Chelba, C. , et al. (2013). “One billion word benchmark for measuring progress in statistical language modeling”. arXiv preprint arXiv:1312.3005.✅
[6] Ho, J. , et al. (2020). “Denoising diffusion probabilistic models”. Advances in Neural Information Processing Systems, 33, 6820-6831.✅
[7] Sutskever, I. , et al. (2011). “Generating text with recurrent neural networks”. arXiv preprint arXiv:1103.0637.✅
[8] Genome Reference Consortium. (2019). “GRCh38: Primary Assembly”. https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.25/
[9] Raffel, C. , et al. (2020). “Exploring the limits of transfer learning with a unified text-to-text transformer”. arXiv preprint arXiv:1910.10683.✅
[10] Devlin, J. , et al. (2018). “Bert: Pre-training of deep bidirectional transformers for language understanding”. arXiv preprint arXiv:1810.04805.✅
[11] Nichol, A. , et al. (2021). “Improved denoising diffusion probabilistic models”. arXiv preprint arXiv:2102.09672.✅
[12] Yang, Z. , et al. (2019). “XLNet: Generalized autoregressive pretraining for language understanding”. Advances in Neural Information Processing Systems, 32, 5754-5764.✅
[13] Reed, S. , et al. (2022). “OpenWebText: A massive open-source dataset for language modeling”. arXiv preprint arXiv:2204.03276.✅
[14] Schiff, Y. , et al. (2022). “Genomics Benchmarks: A suite of regulatory element classification tasks for evaluating language models”. arXiv preprint arXiv:2203.17003.✅
[15] Schiff, Y. , et al. (2023). “Mamba: A structured state space model for biological sequences”. arXiv preprint arXiv:2302.00711.✅
[16] Schiff, Y. , et al. (2023). “Structured State Space Models for Biological Sequences”. arXiv preprint arXiv:2302.00711.✅
[17] Song, J. , et al. (2020). “Score-based generative modeling with diffusion processes”. arXiv preprint arXiv:2011.13456.✅
[18] Song, J. , et al. (2021). “Generative modeling by estimating gradients of the data distribution”. Advances in Neural Information Processing Systems, 34, 18696-18707.✅
[19] He, X. , et al. (2022). “DiffusionBert: Language modeling with diffusion”. arXiv preprint arXiv:2201.01535.✅
[20] Sohl-Dickstein, J. , et al. (2015). “Deep unsupervised learning using nonequilibrium thermodynamics”. arXiv preprint arXiv:1503.03585.✅
[21] Kingma, D. P., et al. (2019). “Variational diffusion networks”. arXiv preprint arXiv:1906.09041.✅
[22] Liu, Y. , et al. (2021). “Diffusion-lm: Text generation with diffusion models”. arXiv preprint arXiv:2106.00999.✅
[23] Ramesh, A. , et al. (2022). “Hierarchical text generation with diffusion models”. arXiv preprint arXiv:2202.00833.✅
[24] Bao, H. , et al. (2021). “GPT-3: Language Models are Few-Shot Learners”. arXiv preprint arXiv:2005.14165.✅
[25] Lou, J. , et al. (2022). “Score-based diffusion models for discrete data”. arXiv preprint arXiv:2203.02221.✅
[26] Ho, J. , et al. (2021). “Denoising diffusion probabilistic models”. Advances in Neural Information Processing Systems, 33, 6820-6831.✅
[27] Ramesh, A. , et al. (2022). “Hierarchical text generation with diffusion models”. arXiv preprint arXiv:2202.00833.✅
[28] Marcus, M. P., et al. (1993). “Building a large annotated corpus of english: The penn treebank”. Computational linguistics, 19(2), 313-330.✅
[29] Merity, S. , et al. (2017). “Pointer sentinel mixture models”. arXiv preprint arXiv:1706.03762.✅
[30] Merity, S. , et al. (2016). “Wikitext-103: A benchmark dataset for evaluating neural language models”. arXiv preprint arXiv:1609.07843.✅
[31] Paperno, D. , et al. (2016). “The lambada dataset: Language modeling in the wild”. arXiv preprint arXiv:1606.04110.✅
[32] Peebles, S. , & Xie, S. (2022). “The diffusion transformer”. arXiv preprint arXiv:2205.09025.✅
[33] Portes, S. , et al. (2021). “MosaicBERT: A unified architecture for pretraining and fine-tuning”. arXiv preprint arXiv:2104.00244.✅
[34] Brown, T. B., et al. (2020). “Language models are few-shot learners”. arXiv preprint arXiv:2005.14165.✅
[35] Radford, A. , et al. (2019). “Language models are unsupervised multitask learners”. OpenAI blog, 1(8), 9.✅
[36] Khandelwal, U. , et al. (2020). “C4: A massive dataset of code snippets and natural language”. arXiv preprint arXiv:2007.01380.✅
[37] Kingma, D. P., & Welling, M. (2013). “Auto-encoding variational bayes”. arXiv preprint arXiv:1312.6114.✅
[38] Schiff, Y. , et al. (2023). “Caduceus: A structured state space model for biological sequences”. arXiv preprint arXiv:2302.00711.✅
[39] Sohl-Dickstein, J. , et al. (2015). “Deep unsupervised learning using nonequilibrium thermodynamics”. arXiv preprint arXiv:1503.03585.✅
[40] Song, J. , et al. (2020). “Score-based generative modeling with diffusion processes”. arXiv preprint arXiv:2011.13456.✅
[41] Ho, J. , et al. (2020). “Denoising diffusion probabilistic models”. Advances in Neural Information Processing Systems, 33, 6820-6831.✅
[42] Nichol, A. , et al. (2021). “Improved denoising diffusion probabilistic models”. arXiv preprint arXiv:2102.09672.✅
[43] Su, J. , et al. (2021). “RoFormer: Enhanced transformer with rotary position embedding”. arXiv preprint arXiv:2104.09862.✅
[44] Song, J. , et al. (2021). “Generative modeling by estimating gradients of the data distribution”. Advances in Neural Information Processing Systems, 34, 18696-18707.✅
[45] You, J. , et al. (2021). “Graph diffusion”. arXiv preprint arXiv:2106.04227.✅
[46] Li, J. , et al. (2022). “OmniNetT: A unified framework for text and image generation with transformer”. arXiv preprint arXiv:2204.08426.✅
[47] Vaswani, A. , et al. (2017). “Attention is all you need”. Advances in neural information processing systems, 30.✅
[48] Shi, C. , et al. (2022). “Diffusion-based graph generation”. arXiv preprint arXiv:2203.03853.✅
[49] Guu, K. , et al. (2020). “BERT-Mouth: Fine-tuning BERT for Text Generation”. arXiv preprint arXiv:2005.11231.✅
[50] Wang, A. , et al. (2018). “GLUE: A benchmark for general language understanding”. arXiv preprint arXiv:1804.04861.✅
[51] Zhang, X. , et al. (2015). “Character-level convolutional networks for text classification”. arXiv preprint arXiv:1509.01626.✅