Type 1 diabetes is a common metabolic disease that is harmful to people's health. Individuals with this disease need to inject insulin dosages to maintain the blood glucose in a regular range. Over the years, several methods have been proposed to utilize continuous glucose monitoring (CGM) to optimize insulin dosages in an artificial pancreas. However, most of these methods only optimize the insulin dosages without considering the insulin injection time, which remains a gap for deployment in realistic scenarios. In this paper, we formulate the problem of optimizing the insulin injection time and dosages in the reinforcement learning framework. To solve the vast action space challenge in the insulin injection problem, a heuristic search schema is proposed to narrow down the action space and achieve efficient exploration. A double-score strategy is further introduced for robust action initialization. Extensive experiments are conducted on an FDA-approved simulator for 30 virtual patients over 10 days and based on different meal schedules. The results demonstrate that the proposed method outperforms other basal-bolus and PID controller methods.
