Academic Report of School of Mathematical Sciences [2026] No. 059

(Series Report for High-Level University Construction No. 1318)

Title：Accelerating RLHF Training with Reward Variance Increase

Speaker：Yancheng Yuan, Assistant Professor (The Hong Kong Polytechnic University)

Time：16:30-17:30, June 16, 2026

Location：Room 304, Alumni Plaza, Yuehai Campus

Abstract: Reinforcement learning from human feedback (RLHF) is an essential technique for ensuring that large language models (LLMs) are aligned with human values and preferences during the post-training phase. As an effective RLHF approach, group relative policy optimization (GRPO) has demonstrated success in many LLM-based applications. However, efficient GRPO-based RLHF training remains a challenge. Recent studies reveal that a higher reward variance of the initial policy model leads to faster RLHF training. Inspired by this finding, we propose a practical reward adjustment model to accelerate RLHF training by provably increasing the reward variance and preserving the relative preferences and reward expectation. Our reward adjustment method inherently poses a nonconvex optimization problem, which is NP-hard to solve in general. To overcome the computational challenges, we design a novel $O(n \log n)$ algorithm to find a global solution of the nonconvex reward adjustment model by explicitly characterizing the extreme points of the feasible set. As an important application, we naturally integrate this reward adjustment model into the GRPO algorithm, leading to a more efficient GRPO with reward variance increase (GRPOVI) algorithm for RLHF training. As an interesting byproduct, we provide an indirect explanation for the empirical effectiveness of GRPO with rule-based reward for RLHF training, as demonstrated in DeepSeek-R1. Experiment results demonstrate that the GRPOVI algorithm can significantly improve the RLHF training efficiency compared to the original GRPO algorithm. This is a joint work with Zonglin Yang, Zhexuan Gu, and Houduo Qi.

Speaker Profile：Dr. Yancheng Yuan is currently an Assistant Professor in the Department of Applied Mathematics at The Hong Kong Polytechnic University (PolyU), Deputy Director of the PolyU-CITIC Joint Laboratory for Artificial Intelligence and Digital Innovation, and Assistant Director of the Center for Intelligent Operations Research at PolyU. His primary research interests include continuous optimization, the mathematical foundations of artificial intelligence, and its applications in large-scale models, recommendation systems, and healthcare. His research has been published in prestigious academic journals such as SIAM Journal on Optimization, Mathematical Programming: Computation, Journal of the American Statistical Association, Journal of Machine Learning Research, and IEEE Transactions on Pattern Analysis and Machine Intelligence, as well as presented at major AI conferences including NeurIPS, ICML, ICLR, ACM WWW, and ACM SIGIR. His research has been selected as a Best Paper Award Finalist at major AI conferences (ACM WWW 2021, ACM SIGIR 2024).

Faculty and students are welcome to attend!

Invited by: Yuqia Wu

School of Mathematical Sciences

June 15, 2026