Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications

Page view(s)
111
Checked on Jun 21, 2024
Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
Title:
Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
Journal Title:
Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Keywords:
Publication Date:
04 August 2023
Citation:
Nambiar, M., Ghosh, S., Ong, P., Chan, Y. E., Bee, Y. M., & Krishnaswamy, P. (2023). Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. https://doi.org/10.1145/3580305.3599800
Abstract:
There is increasing interest in data-driven approaches for recommending optimal treatment strategies in many chronic disease management and critical care applications. Reinforcement learning methods are well-suited to this sequential decision-making problem, but must be trained and evaluated exclusively on retrospective medical record datasets as direct online exploration is unsafe and infeasible. Despite this requirement, the vast majority of treatment optimization studies use off-policy RL methods (e.g., Double Deep Q Networks (DDQN) or its variants) that are known to perform poorly in purely offline settings. Recent advances in offline RL, such as Conservative Q-Learning (CQL), offer a suitable alternative. But there remain challenges in adapting these approaches to real-world applications where suboptimal examples dominate the retrospective dataset and strict safety constraints need to be satisfied. In this work, we introduce a practical and theoretically grounded transition sampling approach to address action imbalance during offline RL training. We perform extensive experiments on two real-world tasks for diabetes and sepsis treatment optimization to compare performance of the proposed approach against prominent off-policy and offline RL baselines (DDQN and CQL). Across a range of principled and clinically relevant metrics, we show that our proposed approach enables substantial improvements in expected health outcomes and in consistency with relevant practice and safety guidelines.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This research is supported by core funding from: A*STAR
Grant Reference no. : Grant No. H19/01/a0/023 – Diabetes Clinic of the Future
Description:
ISBN:
979-8-4007-0103-0
Files uploaded:

File Size Format Action
35803053599800.pdf 1.15 MB PDF Open