Preprints & Publications

2024

1. Plug-and-Play Controllable Generation for Discrete Masked Models

   Wei Guo, Yuchen Zhu, Molei Tao, and Yongxin Chen

   arXiv preprint arXiv:2410.02143, 2024

   Abs HTML

   This article makes discrete masked models for the generative modeling of discrete data controllable. The goal is to generate samples of a discrete random variable that adheres to a posterior distribution, satisfies specific constraints, or optimizes a reward function. This methodological development enables broad applications across downstream tasks such as class-specific image generation and protein design. Existing approaches for controllable generation of masked models typically rely on task-specific fine-tuning or additional modifications, which can be inefficient and resource-intensive. To overcome these limitations, we propose a novel plug-and-play framework based on importance sampling that bypasses the need for training a conditional score. Our framework is agnostic to the choice of control criteria, requires no gradient information, and is well-suited for tasks such as posterior sampling, Bayesian inverse problems, and constrained generation. We demonstrate the effectiveness of our approach through extensive experiments, showcasing its versatility across multiple domains, including protein design.

2. Provable Benefit of Annealed Langevin Monte Carlo for Non-log-concave Sampling

   Wei Guo, Molei Tao, and Yongxin Chen

   arXiv preprint arXiv:2407.16936, 2024

   Abs HTML

   We consider the outstanding problem of sampling from an unnormalized density that may be non-log-concave and multimodal. To enhance the performance of simple Markov chain Monte Carlo (MCMC) methods, techniques of annealing type have been widely used. However, quantitative theoretical guarantees of these techniques are under-explored. This study takes a first step toward providing a non-asymptotic analysis of annealed MCMC. Specifically, we establish, for the first time, an oracle complexity of \widetildeO\left(\fracdβ^2\cal A^2\varepsilon^6\right) for the simple annealed Langevin Monte Carlo algorithm to achieve \varepsilon^2 accuracy in Kullback-Leibler divergence to the target distribution \pi∝\rm e^-V on \mathbbR^d with β-smooth potential V. Here, \cal A represents the action of a curve of probability measures interpolating the target distribution \pi and a readily sampleable distribution.

2023

1. [Bachelor’s Thesis] Theoretical Analysis of the Approximation Properties of Score-Based Generative Models

   Wei Guo

   2023

   Abs PDF

   Score-based generative models leverage a neural network to approximate the score function of the data distribution and employ stochastic or ordinary differential equations to sample from the learned model, which have achieved state-of-the-art performance in tasks such as text-to-image generation and audio synthesis. To elucidate and demystify their empirical success, we investigate their approximation properties in this paper, and focus on two main algorithms: the inexact Langevin Monte Carlo (LMC), and the diffusion models. We establish convergence guarantees of inexact LMC in various metrics (e.g., total-variational distance, Wasserstein-2 distance, and Rényi divergence) under different assumptions of accuracy in score estimation (namely, accuracy in the sense of L^∞, L^2, and moment generating function). We also provide a comprehensive review of different approaches to analyze the approximation properties of diffusion models, including the variational approach, the Fokker-Planck approach, the Girsanov approach, the L^∞\to L^2 approach, the restoration-degradation approach, and the KL divergence decomposition approach. Our analysis reveals that under mild assumptions of the target distribution and the discretization scheme, score-based generative models can arbitrarily approximate the target distribution provided that the score estimate is sufficiently precise, which partially sheds light on the theoretical foundations of score-based generative models.