Sampled-data Control through Model-Free Reinforcement Learning with Effective Experience Replay

Xiao, Bo and Lam, Hak-Keung and Su, Xiaojie and Wang, Ziwei and P.W.Lo, Frank and Chen, Shihong and Yeatman, Eric (2023) Sampled-data Control through Model-Free Reinforcement Learning with Effective Experience Replay. Journal of Automation and Intelligence, 2 (1). ISSN 2949-8554

Text (jai100018)
jai100018.pdf - Draft Version
Available under License Creative Commons Attribution-NonCommercial-NoDerivs.
Download (1MB)

Abstract

Reinforcement Learning (RL) based control algorithms can learn the control strategies for nonlinear and uncertain environment during interacting with it. Guided by the rewards generated by environment, a RL agent can learn the control strategy directly in a model-free way instead of investigating the dynamic model of the environment. In the paper, we propose the sampled-data RL control strategy to reduce the computational demand. In the sampled-data control strategy, the whole control system is of a hybrid structure, in which the plant is of continuous structure while the controller (RL agent) adopts a discrete structure. Given that the continuous states of the plant will be the input of the agent, the state–action value function is approximated by the fully connected feed-forward neural networks (FCFFNN). Instead of learning the controller at every step during the interaction with the environment, the learning and acting stages are decoupled to learn the control strategy more effectively through experience replay. In the acting stage, the most effective experience obtained during the interaction with the environment will be stored and during the learning stage, the stored experience will be replayed to customized times, which helps enhance the experience replay process. The effectiveness of proposed approach will be verified by simulation examples.