Actor-Critic with Synthesis Loss for Solving Approximation Biases

Bo Wen Guo, Fei Chao*, Xiang Chang, Changjing Shang, Qiang Shen

*Awdur cyfatebol y gwaith hwn

Allbwn ymchwil: Cyfraniad at gyfnodolynErthygladolygiad gan gymheiriaid

1 Dyfyniad (Scopus)

Crynodeb

Approximation biases of value functions are considered a key problem in reinforcement learning (RL). In particular, existing RL algorithms are hindered by overestimation and underestimation biases, i.e., value mismatching between RL’s actual returns and action–value approximations limits the performance of RL algorithms. In this article, we first develop a new synthesis loss function for RL’s action–value estimation integrating a regularization term and a modified “clipped double Q -learning” structure for solving overestimation and underestimation biases. To minimize the differences between action–value estimations and actual returns in RL, we develop a new discrepancy function to determine the type and magnitude of approximation biases. Then, two coefficients embedded in the synthesis loss are automatically tuned by minimizing the discrepancy function during training to minimize approximation biases. We further design a new actor–critic (AC) algorithm, named AC with synthesis loss (ACSL), by integrating the synthesis loss function and an error-controlled mechanism. Experimental results on continuous control tasks illustrate that the proposed ACSL algorithm outperforms other cutting-edge RL methods in many tasks and that the proposed synthesis loss function is easily implemented into other algorithms and significantly reduces approximation biases while improving performance. The proposed method can successfully handle many complex continuous control tasks and can greatly outperform other state-of-the-art algorithms on most tasks.
Iaith wreiddiolSaesneg
Tudalennau (o-i)5323-5336
Nifer y tudalennau14
CyfnodolynIEEE Transactions on Cybernetics
Cyfrol54
Rhif cyhoeddi9
Dyddiad ar-lein cynnar03 Mai 2024
StatwsCyhoeddwyd - 01 Medi 2024

Ôl bys

Gweld gwybodaeth am bynciau ymchwil 'Actor-Critic with Synthesis Loss for Solving Approximation Biases'. Gyda’i gilydd, maen nhw’n ffurfio ôl bys unigryw.

Dyfynnu hyn