Abstract
Behavioral weight loss (WL) trials show that, on average, participants regain lost weight unless provided long-term, intensive—and thus costly—intervention. Optimization solutions have shown mixed success. The artificial intelligence principle of “reinforcement learning” (RL) offers a new and more sophisticated form of optimization in which the intensity of each individual’s intervention is continuously adjusted depending on patterns of response. In this pilot, we evaluated the feasibility and acceptability of a RL-based WL intervention, and whether optimization would achieve equivalent benefit at a reduced cost compared to a non-optimized intensive intervention. Participants (n = 52) completed a 1-month, group-based in-person behavioral WL intervention and then (in Phase II) were randomly assigned to receive 3 months of twice-weekly remote interventions that were non-optimized (NO; 10-min phone calls) or optimized (a combination of phone calls, text exchanges, and automated messages selected by an algorithm). The Individually-Optimized (IO) and Group-Optimized (GO) algorithms selected interventions based on past performance of each intervention for each participant, and for each group member that fit into a fixed amount of time (e.g., 1 h), respectively. Results indicated that the system was feasible to deploy and acceptable to participants and coaches. As hypothesized, we were able to achieve equivalent Phase II weight losses (NO = 4.42%, IO = 4.56%, GO = 4.39%) at roughly one-third the cost (1.73 and 1.77 coaching hours/participant for IO and GO, versus 4.38 for NO), indicating strong promise for a RL system approach to weight loss and maintenance.
Similar content being viewed by others
References
Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Paper presented at the Proceedings of the twenty-first international conference on Machine learning.
Anderson, J. W., Vichitbandra, S., Qian, W., & Kryscio, R. J. (1999). Long-term weight maintenance after an intensive weight-loss program. Journal of the American College of Nutrition, 18, 620–627.
Association of Behavioral and Cognitive Therapies. (2018). Find a CBT Therapist. Retrieved May 23, 2018, from http://www.findcbt.org.
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.
Aydin, M. E., & Öztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33, 169–178.
Brindal, E., Freyne, J., Saunders, I., Berkovsky, S., Smith, G., & Noakes, M. (2012). Features predicting weight loss in overweight or obese participants in a web-based intervention: Randomized trial. Journal of medical Internet research, 14, e173.
Butryn, M. L., Forman, E. M., Lowe, M. R., Gorin, A. A., Zhang, F., & Schaumberg, K. (2017). Efficacy of environmental and acceptance-based enhancements to behavioral weight loss treatment: The ENACT trial. Obesity, 25, 866–872.
Butryn, M. L., Zhang, F., Remmert, J. E., Roberts, S. R., & Forman, E. M. (2018). Baseline Executive Functioning Predicts Weight Loss and Physical Activity Outcomes in A Lifestyle Modification Program. Paper presented at the 39th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine, New Orleans, LA.
Carels, R. A., Darby, L., Cacciapaglia, H. M., Douglass, O. M., Harper, J., Kaplar, M. E., et al. (2005). Applying a stepped-care approach to the treatment of obesity. Journal of Psychosomatic Research, 59, 375–383.
Carels, R. A., Darby, L., Cacciapaglia, H. M., Konrad, K., Coit, C., Harper, J., et al. (2007). Using motivational interviewing as a supplement to obesity treatment: A stepped-care approach. Health Psychology, 26, 369.
Carels, R. A., Hoffmann, D. A., Hinman, N., Burmeister, J. M., Koball, A., Ashrafioun, L., et al. (2013). Step-down approach to behavioural weight loss treatment: A pilot of a randomised clinical trial. Psychology & Health, 28, 1121–1134.
Carels, R. A., Selensky, J. C., Rossi, J., Solar, C., & Hlavka, R. (2017). A novel stepped-care approach to weight loss: The role of self-monitoring and health literacy in treatment outcomes. Eating Behaviors, 26, 76–82.
Centers for Disease Control and Prevention (CDC). (2014). National Health and Nutrition Examination Survey Data. Hyattsville, MD. Retrieved May 13, 2018, from https://www.cdc.gov/nchs/fastats/obesity-overweight.htm.
Diabetes Prevention Program Research Group. (2003). Costs associated with the primary prevention of type 2 diabetes mellitus in the diabetes prevention program. Diabetes Care, 26, 36–47.
Forman, E. M., Butryn, M. L., Manasse, S. M., Crosby, R. D., Goldstein, S. P., Wyckoff, E. P., et al. (2016). Acceptance-based versus standard behavioral treatment for obesity: Results from the mind your health randomized controlled trial. Obesity, 24, 2050–2056.
Jakicic, J. M., Tate, D. F., Lang, W., Davis, K. K., Polzien, K., Rickman, A. D., et al. (2012). Effect of a stepped-care intervention approach on weight loss in adults: A randomized clinical trial. JAMA, 307, 2617–2626.
Jeffery, R. W., Epstein, L. H., Wilson, G. T., Drewnowski, A., Stunkard, A. J., & Wing, R. R. (2000). Long-term maintenance of weight loss: Current status. Health Psychology, 19, 5.
Joo, N.-S., & Kim, B.-T. (2007). Mobile phone short message service messaging for behaviour modification in a community-based weight control programme in Korea. Journal of Telemedicine and Telecare, 13, 416–420.
Korinek, E. V., Phatak, S. S., Martin, C. A., Freigoun, M. T., Rivera, D. E., Adams, M. A., et al. (2018). Adaptive step goals and rewards: A longitudinal growth model of daily steps for a smartphone-based walking intervention. Journal of Behavioral Medicine, 41, 74–86.
Kramer, F. M., Jeffery, R. W., Forster, J. L., & Snell, M. K. (1989). Long-term follow-up of behavioral treatment for obesity: Patterns of weight regain among men and women. International Journal of Obesity, 13, 123–136.
Krukowski, R. A., Tilford, J. M., Harvey-Berino, J., & West, D. S. (2011). Comparing behavioral weight loss modalities: Incremental cost-effectiveness of an internet-based versus an in-person condition. Obesity, 19, 1629–1635.
Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. Paper presented at the 2015 IEEE International Conference on Robotics and Automation (ICRA).
McGraa, K. L. K. (2010). The effects of persuasive motivational text messaging on adherence to diet and exercise programs across different personality traits. Santa Barbara: Fielding Graduate University.
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., et al. (2006). Autonomous inverted helicopter flight via reinforcement learning. In M. H. Ang & O. Khatib (Eds.), Experimental Robotics IX (pp. 363–372). Berlin: Springer.
Ontanón, S. (2013). The combinatorial multi-armed bandit problem and its application to real-time strategy games. Paper presented at the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.
Ontanón, S. (2017). Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58, 665–702.
Paredes, P., Gilad-Bachrach, R., Czerwinski, M., Roseway, A., Rowan, K., & Hernandez, J. (2014). PopTherapy: Coping with stress through pop-culture. Paper presented at the Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare.
Patrick, K., Raab, F., Adams, M. A., Dillon, L., Zabinski, M., Rock, C. L., et al. (2009). A text message–based intervention for weight loss: Randomized controlled trial. Journal of medical Internet research, 11, e1.
Phatak, S. S., Freigoun, M. T., Martín, C. A., Rivera, D. E., Korinek, E. V., Adams, M. A., et al. (2018). Modeling individual differences: A case study of the application of system identification for personalizing a physical activity intervention. Journal of Biomedical Informatics, 79, 82–97.
Piette, J. D., Farris, K. B., Newman, S., An, L., Sussman, J., & Singh, S. (2014). The potential impact of intelligent systems for mobile health self-management support: Monte Carlo simulations of text message support for medication adherence. Annals of Behavioral Medicine, 49, 84–94.
Piette, J. D., Krein, S. L., Striplin, D., Marinec, N., Kerns, R. D., Farris, K. B., et al. (2016). Patient-centered pain care using artificial intelligence and mobile health tools: Protocol for a randomized study funded by the US Department of Veterans Affairs Health Services Research and Development Program. JMIR Research Protocols, 5, e53.
Poirier, J., Bennett, W. L., Jerome, G. J., Shah, N. G., Lazo, M., Yeh, H.-C., et al. (2016). Effectiveness of an activity tracker-and internet-based adaptive walking program for adults: A randomized controlled trial. Journal of Medical Internet Research, 18, e34.
Rabbi, M., Aung, M. H., Zhang, M., & Choudhury, T. (2015). MyBehavior: Automatic personalized health feedback from user behaviors and preferences using smartphones. Paper presented at the Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.
Snook, K. R., Hansen, A. R., Duke, C. H., Finch, K. C., Hackney, A. A., & Zhang, J. (2017). Change in percentages of adults with overweight or obesity trying to lose weight, 1988–2014. JAMA, 317, 971–973.
Society of Clinical Psychology. (2018). Therapists Near Me. Retrieved May 23, 2018, from https://www.div12.org/therapist-search/.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT press Cambridge.
Wadden, T. A., Butryn, M. L., & Wilson, C. (2007). Lifestyle modification for the management of obesity. Gastroenterology, 132, 2226–2238.
Wadden, T. A., Sternberg, J., Letizia, K., Stunkard, A., & Foster, G. (1989). Treatment of obesity by very low calorie diet, behavior therapy, and their combination: A five-year perspective. International Journal of Obesity, 13, 39–46.
Webber, K. H., Tate, D. F., & Bowling, J. M. (2008). A randomized comparison of two motivationally enhanced Internet behavioral weight loss programs. Behaviour Research and Therapy, 46, 1090–1095.
Wilson, G. T. (1994). Behavioral treatment of obesity: Thirty years and counting. Advances in Behaviour Research and Therapy, 16, 31–75.
Wilson, G. T., & Brownell, K. D. (2002). Behavioral treatment for obesity. In C. G. Fairburn & K. D. Brownell (Eds.), Eating disorders and obesity. New York: Guilford Press.
Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., & Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research, 19, e338.
Zhou, M., Fukuoka, Y., Mintz, Y., Goldberg, K., Kaminsky, P., Flowers, E., et al. (2018). Evaluating machine learning-based automated personalized daily step goals delivered through a Mobile Phone App: Randomized controlled trial. JMIR mHealth and uHealth, 6, e28.
Acknowledgements
This project was funded by a Drexel University DARE (Drexel Areas of Research Excellence) Award to Evan Forman. We would also like to acknowledge the assistance of Pedrag Krasnja whom we consulted during the planning of this study.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Evan M. Forman, Stephanie G. Kerrigan, Meghan L. Butryn, Adrienne S. Juarascio, Stephanie M. Manasse, Santiago Ontañón, Diane H. Dallal, Rebecca J. Crochiere and Danielle Moskow declare that they have no conflict of interest.
Human and animal rights and Informed consent
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.
Rights and permissions
About this article
Cite this article
Forman, E.M., Kerrigan, S.G., Butryn, M.L. et al. Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?. J Behav Med 42, 276–290 (2019). https://doi.org/10.1007/s10865-018-9964-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10865-018-9964-1