A Novel Implementation of Q-Learning for the Whittle Index

Gibson, Lachlan J. and Jacko, Peter and Nazarathy, Yoni (2021) A Novel Implementation of Q-Learning for the Whittle Index. In: Performance Evaluation Methodologies and Tools - 14th EAI International Conference, VALUETOOLS 2021, Proceedings :. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST . Springer, Cham, pp. 154-170. ISBN 9783030925109

Full text not available from this repository.


We develop a method for learning index rules for multi-armed bandits, restless bandits, and dynamic resource allocation where the underlying transition probabilities and reward structure of the system is not known. Our approach builds on an understanding of both stochastic optimisation (specifically, the Whittle index) and reinforcement learning (specifically, Q-learning). We propose a novel implementation of Q-learning, which exploits the structure of the problem considered, in which the algorithm maintains two sets of Q-values for each project: one for reward and one for resource consumption. Based on these ideas we design a learning algorithm and illustrate its performance by comparing it to the state-of-the-art Q-learning algorithm for the Whittle index by Avrachenkov and Borkar. Both algorithms rely on Q-learning to estimate the Whittle index policy, however the nature in which Q-learning is used in each algorithm is dramatically different. Our approach seems to be able to deliver similar or better performance and is potentially applicable to a much broader and more general set of problems.

Item Type:
Contribution in Book/Report/Proceedings
Uncontrolled Keywords:
?? multi-armed banditsrestless banditsreinforcement learningq-learningmarkov decision processescomputer networks and communications ??
ID Code:
Deposited By:
Deposited On:
22 Jun 2022 08:10
Last Modified:
16 Jul 2024 05:08