Korean Journal of Culture and Social Issue (한국심리학회지 : 문화 및 사회문제)

Korean Psychological Association of Culture and Social Issues (한국문화및사회문제심리학회)
권호 표지
DOI QR코드

DOI QR Code

Comparison between Machine Learning and Traditional Tecnique for Suicide Prediction based on Meta-analysis

메타분석에 기반한 자살 예측 연구에서 전통적 통계 기법과 머신러닝 기반 접근법의 예측력 비교

  • Received : 2024.02.16
  • Accepted : 2024.04.29
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

The purpose of this study was to compare the predictive accuracy of traditional prediction models (methods) and machine learning algorithms in predicting suicidal behaviors. The research aimed to go beyond a systematic review level and scientifically examine the predictive capabilities of these two techniques through meta-analysis, analyzing variables identified through domestic research, particularly at the regional level. In order to achieve this, a total of 124 studies, including 50 studies utilizing machine learning and 74 studies employing traditional methods, were included in the meta-analysis. The results of the study revealed that the integrated area under the curve (AUC) for studies using traditional methods was .770, which was lower than the integrated AUC value of .853 for studies using machine learning. Particularly, studies conducted in Asia (AUC = .944) demonstrated higher accuracy compared to studies in Western countries (AUC = .820) and Korea (AUC = .864). Additional analysis of the moderating effects in domestic research indicated that a higher proportion of males and the prediction of suicide attempts were associated with higher prediction accuracy. On the other hand, prediction accuracy was lower when the prediction target was suicide deaths and when studies utilized neural network analysis. This study synthesized various research findings on the prediction of suicidal behaviors, verified the effectiveness of prediction using machine learning, and holds significance in exploring variables applicable in the context of South Korea.

본 연구는 자살 관련 행동에 대해 전통적인 예측 모형(기법)과 머신러닝 알고리즘을 활용한 연구의 예측력을 비교하기 위한 목적에서 수행되었다. 따라서 체계적 리뷰 수준에서 벗어나 메타분석을 통해 과학적으로 두 가지 기법의 예측력에 대해 살펴보고, 지역적인 수준에서 특히 국내 연구를 통해 알 수 있는 변인들을 분석하여 추후 자살 관련 행동 예측 연구에 도움을 주고자 하였다. 이를 위해 머신러닝을 사용한 연구 50개와 전통적 기법을 활용한 연구 74개로 총 124개의 문헌이 메타분석에 포함되었다. 연구 결과 전통적 기법을 활용한 연구들의 통합 AUC는 .770으로 머신러닝을 활용한 연구들의 통합 AUC값인 .853보다 낮은 것으로 나타났다. 특히 아시아권의 연구(AUC = .944)가 서양(AUC = .820)과 한국(AUC = .864)의 연구에 비해 높은 정확도를 나타내었다. 국내 연구에서의 조절효과를 추가적으로 분석한 결과 남성의 비율이 많을수록, 예측 대상이 자살 시도일수록 예측 정확도가 높았으며, 예측 대상이 자살 사망일수록, 그리고 신경망분석(Neural Network)을 활용한 연구일수록 예측 정확도가 낮았다. 본 연구는 자살 관련 행동의 예측에 대한 다양한 연구결과를 종합하고, 머신러닝을 활용한 예측의 효과성을 검증하는 한편, 국내에서 활용가능한 변인을 탐색하는 데 그 의의가 있다.

Keywords

References

  1. 권혁준, & 서종한 (2022). 머신러닝 기반 한국 경찰 자살 생각 위험요인 탐색: 랜덤포레스트를 중심으로. 치안정책연구, 36(4), 7-48.
  2. 김효창 (2010). 자살: 문화심리학적 관점에서의 조망. 한국심리학회지: 문화 및 사회문제, 16(2), 165-178.
  3. 이택호, 김선영, 한윤선 (2022). 랜덤포레스트 머신러닝 기법을 활용한 전통적 비행이론 기반 청소년 온, 오프라인 비행 예측요인 연구. 한국심리학회지: 문화 및 사회문제, 28(4), 661-690. https://doi.org/10.20406/KJCS.2022.11.28.4.661
  4. 정기성 (2018). 인공신경망 분석을 활용한 인천시 주거취약계층의 행복주택 입주의향에 미치는 요인 분석. 주택연구, 26(3), 55-78.
  5. 홍기혜 (2020). 랜덤포레스트 머신러닝 알고리즘 기반 남․여 청소년의 자살생각 예측 및 분석. 한국사회복지학, 72(3), 157-180.
  6. Adamou, M., Antoniou, G., Greasidou, E., Lagani, V., Charonyktakis, P., Tsamardinos, I., & Doyle, M. (2018). Toward automatic risk assessment to support suicide prevention. Crisis, 40(4). https://doi.org/10.1027/0227-5910/a000561
  7. Bae, S. M., Lee, S. A., & Lee, S. H. (2015). Prediction by data mining, of suicide attempts in Korean adolescents: a national study. Neuropsychiatric disease and treatment, 2367-2375.
  8. Bayramli, I., Castro, V., Barak-Corren, Y., Madsen, E. M., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2022a). Predictive structured-unstructured interactions in EHR models: A case study of suicide prediction. NPJ Digital Medicine, 5(1), 15. https://doi.org/10.1038/s41746-022-00558-0
  9. Bayramli, I., Castro, V., Barak-Corren, Y., Madsen, E. M., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2022b). Temporally informed random forests for suicide risk prediction. Journal of the American Medical Informatics Association, 29(1), 62-71. https://doi.org/10.1093/jamia/ocab225
  10. Beck, A. T., Schuyler, D., & Herman, I. (1974). Development of suicidal intent scales. Charles Press Publishers.
  11. Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., ... & Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019). Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry, 76(6), 642-651. 10.1001/jamapsychiatry.2019.0174
  12. Berkelmans, G., van der Mei, R., Bhulai, S., & Gilissen, R. (2021). Identifying socio-demographic risk factors for suicide using data on an individual level. BMC Public Health, 21(1), 1-8. https://doi.org/10.1186/s12889-021-11743-3
  13. Bhak, Y., Jeong, H. O., Cho, Y. S., Jeon, S., Cho, J., Gim, J. A., Jeon, Y., Blazyte, A., Park, S. G., Kim, H. M., Shin, E. S., Paik, J. W., Lee, H. W., Kang, W., Kim, A., Kim, Y., Kim, B. C., Ham, B. J., Bhak, J., & Lee, S. (2019). Depression and suicide risk prediction models using blood-derived multi-omics data. Translational Psychiatry, 9(1), 262. https://doi.org/10.1038/s41398-019-0595-2
  14. Bhat, H. S., & Goldman-Mellor, S. J. (2017). Predicting adolescent suicide attempts with neural networks. arXiv preprint arXiv: 1711.10057. https://doi.org/10.48550/arXiv.1711.10057
  15. Boudreaux, E. D., Rundensteiner, E., Liu, F., Wang, B., Larkin, C., Agu, E., Ghosh, S., Semeter, J., Simon, G., & Davis-Martin, R. E. (2021). Applying machine learning approaches to suicide prediction using healthcare data: overview and future directions. Frontiers in Psychiatry, 12, 707916. https://doi.org/10.3389/fpsyt.2021.707916
  16. Bryan, C. J., & Rudd, M. D. (2006). Advances in the assessment of suicide risk. Journal of Clinical Psychology, 62(2), 185-200. https://doi.org/10.1002/jclp.20222
  17. Burke, T. A., Ammerman, B. A., & Jacobucci, R. (2019). The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: A systematic review. Journal of Offective Disorders, 245, 869-884. https://doi.org/10.1016/j.jad.2018.11.073
  18. Byeon, H. (2022). Prediction of adolescent suicidal ideation after the COVID-19 pandemic: A nationwide survey of a representative sample of Korea. Frontiers in Pediatrics, 10, 951439. https://doi.org/10.3389/fped.2022.951439
  19. Chan, L. F., Maniam, T., & Shamsul, A. S. (2011). Suicide attempts among depressed inpatients with depressive disorder in a Malaysian sample: psychosocial and clinical risk factors. Crisis, 32(5), 283-287. https://doi.org/10.1027/0227-5910/a000088
  20. Cheng, Q., Li, T. M., Kwok, C. L., Zhu, T., & Yip, P. S. (2017). Assessing suicide risk and emotional distress in Chinese social media: a text mining and machine learning study. Journal of Medical Internet Research, 19(7), e243. 10.2196/jmir.7276
  21. Choi, S. B., Lee, W., Yoon, J. H., Won, J. U., & Kim, D. W. (2018). Ten-year prediction of suicide death using Cox regression and machine learning in a nationwide retrospective cohort study in South Korea. Journal of Affective Disorders, 231, 8-14. https://doi.org/10.1016/j.jad.2018.01.019
  22. Chou, P. H., Wang, S. C., Wu, C. S., Horikoshi, M., & Ito, M. (2022). A machine-learning model to predict suicide risk in Japan based on national survey data. Frontiers in Psychiatry, 13, 918667. https://doi.org/10.3389/fpsyt.2022.918667
  23. Cohen, J., Wright-Berryman, J., Rohlfs, L., Trocinski, D., Daniel, L., & Klatt, T. W. (2022). Integration and validation of a natural language processing machine learning suicide risk prediction model based on open-ended interview language in the emergency department. Frontiers in Digital Health, 4, 818705. https://doi.org/10.3389/fdgth.2022.818705
  24. Cohen, J., Wright-Berryman, J., Rohlfs, L., Wright, D., Campbell, M., Gingrich, D., Santel, D., & Pestian, J. (2020). A feasibility study using a machine learning suicide risk prediction model based on open-ended interview language in adolescent therapy sessions. International Journal of Environmental Research and Public Health, 17(21), 8187. https://doi.org/10.3390/ijerph17218187
  25. Cohen, S. (1986). Contrasting the Hassles Scale and the Perceived Stress Scale: Who's really measuring appraised stress?. American Psychologist, 41(6), 716-718. https://doi.org/10.1037/0003-066X.41.6.716
  26. Coley, R. Y., Walker, R. L., Cruz, M., Simon, G. E., & Shortreed, S. M. (2021). Clinical risk prediction models and informative cluster size: Assessing the performance of a suicide risk prediction algorithm. Biometrical Journal, 63(7), 1375-1388. https://doi.org/10.1002/bimj.202000199
  27. Corke, M., Mullin, K., Angel-Scott, H., Xia, S., & Large, M. (2021). Meta-analysis of the strength of exploratory suicide prediction models; from clinicians to computers. BJPsych open, 7(1), e26. https://doi.org/10.1192/bjo.2020.162
  28. Czyz, E. K., Koo, H. J., Al-Dajani, N., King, C. A., & Nahum-Shani, I. (2023). Predicting short-term suicidal thoughts in adolescents using machine learning: Developing decision tools to identify daily level risk after hospitalization. Psychological Medicine, 53(7), 2982-2991. https://doi.org/10.1017/S0033291721005006
  29. Delgado-Gomez, D., Blasco-Fontecilla, H., Sukno, F., Ramos-Plasencia, M. S., & Baca-Garcia, E. (2012). Suicide attempters classification: Toward predictive models of suicidal behavior. Neurocomputing, 92, 3-8. https://doi.org/10.1016/j.neucom.2011.08.033
  30. Demi̇rbas, H., Celik, S., Ilhan, I. O., & Dogan, Y. B. (2003). An examination of suicide probability in alcoholic in-patients. Alcohol and Alcoholism, 38(1), 67-70. https://doi.org/10.1093/alcalc/agg019
  31. Edgcomb, J. B., Shaddox, T., Hellemann, G., & Brooks III, J. O. (2021). Predicting suicidal behavior and self-harm after general hospitalization of adults with serious mental illness. Journal of Psychiatric Research, 136, 515-521. https://doi.org/10.1016/j.jpsychires.2020.10.024
  32. Franklin, J. C., Huang, X., & Bastidas, D. (2019). Virtual reality suicide: Development of a translational approach for studying suicide causes. Behaviour Research and Therapy, 120, 103360. https://doi.org/10.1016/j.brat.2018.12.013
  33. Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017). Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychological Bulletin, 143(2), 187-232. https://doi.org/10.1037/bul0000084
  34. Fulginiti, A., Segal, A., Wilson, J., Hill, C., Tambe, M., Castro, C., & Rice, E. (2022). Getting to the root of the problem: A decision-tree analysis for suicide risk among young people experiencing homelessness. Journal of the Society for Social Work and Research, 13(2), 327-352. https://doi.org/10.1086/715211
  35. Gladstone, G. L., Mitchell, P. B., Parker, G., Wilhelm, K., Austin, M. P., & Eyers, K. (2001). Indicators of suicide over 10 years in a specialist mood disorders unit sample. Journal of Clinical Psychiatry, 62(12), 945-951. https://doi.org/10.4088/JCP.v62n1205
  36. Goldston, D. B., Daniel, S. S., Reboussin, D. M., Reboussin, B. A., Frazier, P. H., & Kelley, A. E. (1999). Suicide attempts among formerly hospitalized adolescents: A prospective naturalistic study of risk during the first 5 years after discharge. Journal of the American Academy of Child & Adolescent Psychiatry, 38(6), 660-671. https://doi.org/10.1097/00004583-199906000-00012
  37. Gomez, J. M. (2014). Language technologies for suicide prevention in social media. In Proceedings of the Workshop on Natural Language Processing in the 5th Information Systems Research Working Days (JISIC) (pp. 21-29).
  38. Gradus, J. L., Rosellini, A. J., Horvath-Puho, E., Jiang, T., Street, A. E., Galatzer-Levy, I., Lash, T. L., & Sorensen, H. T. (2021). Predicting sex-specific nonfatal suicide attempt risk using machine learning and data from Danish national registries. American Journal of Epidemiology, 190(12), 2517-2527. https://doi.org/10.1093/aje/kwab112
  39. Han, K., Ji, L., Chen, C., Hou, B., Ren, D., Yuan, F., Liu, L., Bi, Y., Guo, Z., Wu, N., Feng, M., Su, K., Wang, C., Yang, F., Wu, X., Li, X., Liu, C., Zuo, Z., Zhang, R., ... He, G. (2022). College students' screening early warning factors in identification of suicide risk. Frontiers in Genetics, 13, 977007. https://doi.org/10.3389/fgene.2022.977007
  40. Haroz, E. E., Walsh, C. G., Goklish, N., Cwik, M. F., O'Keefe, V., & Barlow, A. (2020). Reaching those at highest risk for suicide: development of a model using machine learning methods for use with Native American communities. Suicide and Life-Threatening Behavior, 50(2), 422-436. https://doi.org/10.1111/sltb.12598
  41. Hawkins, D. M. (2004). The problem of overfitting. Journal of Chemical Information and Computer Sciences, 44(1), 1-12. https://doi.org/10.1021/ci0342472
  42. Hopkins, D., Rickwood, D. J., Hallford, D. J., & Wats ford, C. (2022). Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: A systematic review and meta-analysis. Frontiers in Digital Health, 4, 945006. https://doi.org/10.3389/fdgth.2022.945006
  43. Horvath, A., Dras, M., Lai, C. C., & Boag, S. (2021). Predicting suicidal behavior without asking about suicidal ideation: machine learning and the role of borderline personality disorder criteria. Suicide and Life-Threatening Behavior, 51(3), 455-466. https://doi.org/10.1111/sltb.12719
  44. Jiang, T., Rosellini, A. J., Horvath-Puho, E., Shiner, B., Street, A. E., Lash, T. L., Sorensen, H. T., & Gradus, J. L. (2021). Using machine learning to predict suicide in the 30 days after discharge from psychiatric hospital in Denmark. The British Journal of Psychiatry, 219(2), 440-447. doi:10.1192/bjp.2021.19
  45. Joiner, T., Rudd, M. D. (2002). The hopelessness theory of suicidality. In Abramson, L. Y., Alloy, L. B., Hogan, M. E., Whitehouse, W. G., Gibb, B. E., Hankin, B. L., & Cornette, M. M. (Ed.), Suicide Science: Expanding the Boundaries (pp. 17-32).
  46. Jung, J. S., Park, S. J., Kim, E. Y., Na, K. S., Kim, Y. J., & Kim, K. G. (2019). Prediction models for high risk of suicide in Korean adolescents using machine learning techniques. PLoS one, 14(6), e0217639. https://doi.org/10.1371/journal.pone.0217639
  47. Kessler, R. C. (2019). Clinical epidemiological research on suicide-related behaviors-where we are and where we need to go. JAMA Psychiatry, 76(8), 777-778. 10.1001/jamapsychiatry.2019.1238
  48. Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R. (2020). Suicide prediction models: a critical review of recent research with recommendations for the way forward. Molecular Psychiatry, 25(1), 168-179. https://doi.org/10.1038/s41380-019-0531-0
  49. Kim, K. W., Lim, J. S., Yang, C. M., Jang, S. H., & Lee, S. Y. (2021). Classification of adolescent psychiatric patients at high risk of suicide using the personality assessment inventory by machine learning. Psychiatry Investigation, 18(11), 1137. doi: 10.30773/pi.2021.0191
  50. Kim, S., & Lee, K. (2022). The effectiveness of predicting suicidal ideation through depressive symptoms and social isolation using machine learning techniques. Journal of Personalized Medicine, 12(4), 516. https://doi.org/10.3390/jpm12040516
  51. Kim, S., Lee, H. K., & Lee, K. (2021). Detecting suicidal risk using MMPI-2 based on machine learning algorithm. Scientific Reports, 11(1), 15310. https://doi.org/10.1038/s41598-021-94839-5
  52. Kleespies, P. M., & Dettmer, E. L. (2000). An evidence-based approach to evaluating and managing suicidal emergencies. Journal of Clinical Psychology, 56(9), 1109-1130. https://doi.org/10.1002/1097-4679(200009)56:9<1109::AID-JCLP2>3.0.CO;2-C
  53. Large, M. M., Ryan, C. J., Carter, G., & Kapur, N. (2017). Can we usefully stratify patients according to suicide risk?. Bmj, 359.
  54. Levis, M., Westgate, C. L., Gui, J., Watts, B. V., & Shiner, B. (2021). Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models. Psychological Medicine, 51(8), 1382-1391. https://doi.org/10.1017/S0033291720000173
  55. Lim, J. S., Yang, C. M., Baek, J. W., Lee, S. Y., & Kim, B. N. (2022). Prediction models for suicide attempts among adolescents using machine learning techniques. Clinical Psychopharmacology and Neuroscience, 20(4), 609. doi: 10.9758/cpn.2022.20.4.609
  56. Lin, G. M., Nagamine, M., Yang, S. N., Tai, Y. M., Lin, C., & Sato, H. (2020). Machine learning based suicide ideation prediction for military personnel. IEEE Journal of Biomedical and Health Informatics, 24(7), 1907-1916. doi: 10.1109/JBHI.2020.2988393
  57. Linthicum, K. P., Schafer, K. M., & Ribeiro, J. D. (2019). Machine learning in suicide science: Applications and ethics. Behavioral Sciences & the Law, 37(3), 214-222. https://doi.org/10.1002/bsl.2392
  58. Lu, H., Tang, J., Huang, Z., Zou, Q., Guo, M., Huang, X., & Hu, F. (2020). Development and Value Evaluation of a Simple Prediction Model of Suicidal Behavior in Depressive Disorder. Chinese General Practice, 23(26), 3247. DOI: 10.12114/j.issn.1007-9572.2020.00.430
  59. Macalli, M., Navarro, M., Orri, M., Tournier, M., Thiebaut, R., Cote, S. M., & Tzourio, C. (2021). A machine learning approach for predicting suicidal thoughts and behaviours among college students. Scientific Reports, 11(1), 11363. https://doi.org/10.1038/s41598-021-90728-z
  60. Mann, J. J., Currier, D., Stanley, B., Oquendo, M. A., Amsel, L. V., & Ellis, S. P. (2006). Can biological tests assist prediction of suicide in mood disorders?. International Journal of Neuropsychopharmacology, 9(4), 465-474. https://doi.org/10.1017/S1461145705005687 
  61. Mann, J. J., Waternaux, C., Haas, G. L., & Malone, K. M. (1999). Toward a clinical model of suicidal behavior in psychiatric patients. American Journal of Psychiatry, 156(2), 181-189. https://doi.org/10.1176/ajp.156.2.181
  62. Matykiewicz, P., Duch, W., & Pestian, J. (2009, June). Clustering Semantic Spaces of Suicide Notes and Newsgroups Articles. In Proceedings of the BioNLP 2009 Workshop (pp. 179-184).
  63. McHugh, C. M., Ho, N., Iorfino, F., Crouse, J. J., Nichles, A., Zmicerevska, N., ... & Hickie, I. B. (2023). Predictive modelling of deliberate self-harm and suicide attempts in young people accessing primary care: a machine learning analysis of a longitudinal study. Social Psychiatry and Psychiatric Epidemiology, 58(6), 893-905. https://doi.org/10.1007/s00127-022-02415-7
  64. Min, S., Shin, D., Rhee, S. J., Park, C. H. K., Yang, J. H., Song, Y., Kim, M. J., Kim, K., Cho, W. I., Kwon, O. C., Ahn, Y. M., & Lee, H. (2023). Acoustic analysis of speech for screening for suicide risk: machine learning classifiers for between-and within-person evaluation of suicidality. Journal of Medical Internet Research, 25, e45456. doi: 10.2196/45456
  65. Nordin, N., Zainol, Z., Mohd Noor, M. H., & Lai Fong, C. (2021). A comparative study of machine learning techniques for suicide attempts predictive model. Health Informatics Journal, 27(1), 1460458221989395. https://doi.org/10.1177/14604582219893
  66. Obeid, J. S., Dahne, J., Christensen, S., Howard, S., Crawford, T., Frey, L. J., Stecker, T., & Bunnell, B. E. (2020). Identifying and predicting intentional self-harm in electronic health record clinical notes: deep learning approach. JMIR Medical Informatics, 8(7), e17784. doi: 10.2196/17784
  67. Oh, B., Yun, J. Y., Yeo, E. C., Kim, D. H., Kim, J., & Cho, B. J. (2020). Prediction of suicidal ideation among Korean adults using machine learning: a cross-sectional study. Psychiatry Investigation, 17(4), 331. doi: 10.30773/pi.2019.0270
  68. Oh, J., Yun, K., Hwang, J. H., & Chae, J. H. (2017). Classification of suicide attempts through a machine learning algorithm based on multiple systemic psychiatric scales. Frontiers in Psychiatry, 8, 192. https://doi.org/10.3389/fpsyt.2017.00192
  69. Park, H., & Lee, K. (2022a). A Machine Learning Approach for Predicting Wage Workers' Suicidal Ideation. Journal of Personalized Medicine, 12(6), 945. https://doi.org/10.3390/jpm12060945
  70. Park, H., & Lee, K. (2022b). Using Boosted Machine Learning to Predict Suicidal Ideation by Socioeconomic Status among Adolescents. Journal of Personalized Medicine, 12(9), 1357. https://doi.org/10.3390/jpm12091357
  71. Rabani, S. T., Khan, Q. R., & Khanday, A. M. U. D. (2020). Detection of suicidal ideation on Twitter using machine learning & ensemble approaches. Baghdad Science Journal, 17(4), 1328-1328. https://doi.org/10.21123/bsj.2020.17.4.1328
  72. Ryu, S., Lee, H., Lee, D. K., Kim, S. W., & Kim, C. E. (2019). Detection of suicide attempters among suicide ideators using machine learning. Psychiatry Investigation, 16(8), 588. doi: 10.30773/pi.2019.06.19.
  73. Ryu, S., Lee, H., Lee, D. K., & Park, K. (2018). Use of a machine learning algorithm to predict individuals with suicide ideation in the general population. Psychiatry Investigation, 15(11), 1030. doi: 10.30773/pi.2018.08.27
  74. Sanderson, M., Bulloch, A. G., Wang, J., Williams, K. G., Williamson, T., & Patten, S. B. (2020). Predicting death by suicide following an emergency department visit for parasuicide with administrative health care system data and machine learning. EClinicalMedicine, 20. https://doi.org/10.1016/j.eclinm.2020.100281
  75. Schafer, K. M., Kennedy, G., Gallyer, A., & Resnik, P. (2021). A direct comparison of theory-driven and machine learning prediction of suicide: A meta-analysis. PloS one, 16(4), e0249833. https://doi.org/10.1371/journal.pone.0249833
  76. Shen, Y., Zhang, W., Chan, B. S. M., Zhang, Y., Meng, F., Kennon, E. A., Wu, H. E., Luo, X., & Zhang, X. (2020). Detecting risk of suicide attempts among Chinese medical college students using a machine learning algorithm. Journal of Affective Disorders, 273, 18-23. https://doi.org/10.1016/j.jad.2020.04.057
  77. Shin, D., Kim, K., Lee, S. B., Lee, C., Bae, Y. S., Cho, W. I., ... & Ahn, Y. M. (2022). Detection of depression and suicide risk based on text from clinical interviews using machine learning: possibility of a new objective diagnostic marker. Frontiers in Psychiatry, 13, 801301. https://doi.org/10.3389/fpsyt.2022.801301
  78. Simundic, A. M. (2009). Measures of diagnostic accuracy: basic definitions. ejifcc, 19(4), 203.
  79. Skogman, K., Alsen, M., & Ojehagen, A. (2004). Sex differences in risk factors for suicide after attempted suicide: a follow-up study of 1052 suicide attempters. Social Psychiatry and Psychiatric Epidemiology, 39, 113-120. https://doi.org/10.1007/s00127-004-0709-9
  80. Song, S. I., Hong, H. T., Lee, C., & Lee, S. B. (2022). A machine learning approach for predicting suicidal ideation in post stroke patients. Scientific Reports, 12(1), 15906. https://doi.org/10.1038/s41598-022-19828-8
  81. Su, C., Aseltine, R., Doshi, R., Chen, K., Rogers, S. C., & Wang, F. (2020). Machine learning for suicide risk prediction in children and adolescents with electronic health records. Translational Psychiatry, 10(1), 413. https://doi.org/10.1038/s41398-020-01100-0
  82. Su, R., John, J. R., & Lin, P. I. (2023). Machine learning-based prediction for self-harm and suicide attempts in adolescents. Psychiatry Research, 328, 115446. https://doi.org/10.1016/j.psychres.2023.115446
  83. Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016). An empirical comparison of model validation techniques for defect prediction models. IEEE Transactions on Software Engineering, 43(1), 1-18. 10.1109/TSE.2016.2584050
  84. Tsui, F. R., Shi, L., Ruiz, V., Ryan, N. D., Biernesser, C., Iyengar, S., Walsh, C. G., & Brent, D. A. (2021). Natural language processing and machine learning of electronic health records for prediction of first-time suicide attempts. JAMIA open, 4(1), ooab011. https://doi.org/10.1093/jamiaopen/ooab011
  85. Tu, J. V. (1996). Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. Journal of Clinical Epidemiology, 49(11), 1225-1231. https://doi.org/10.1016/S0895-4356(96)00002-9
  86. Van Vuuren, C. L., Van Mens, K., De Beurs, D., Lokkerbol, J., Van der Wal, M. F., Cuijpers, P., & Chinapaw, M. J. M. (2021). Comparing machine learning to a rule-based approach for predicting suicidal behavior among adolescents: Results from a longitudinal population-based survey. Journal of Affective Disorders, 295, 1415-1420. https://doi.org/10.1016/j.jad.2021.09.018
  87. Velupillai, S., Hadlaczky, G., Baca-Garcia, E., Gorrell, G. M., Werbeloff, N., Nguyen, D., Patel, R., Leightley, D., Downs, J., Hotopf, M., & Dutta, R. (2019). Risk assessment tools and data-driven approaches for predicting and preventing suicidal behavior. Frontiers in Psychiatry, 10, 36. https://doi.org/10.3389/fpsyt.2019.00036
  88. Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017). Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science, 5(3), 457-469. https://doi.org/10.1177/2167702617691560
  89. Zheng, S., Zeng, W., Xin, Q., Ye, Y., Xue, X., Li, E., Liu, T., Yan, N., Chen, W., & Yin, H. (2022). Can cognition help predict suicide risk in patients with major depressive disorder? A machine learning study. BMC psychiatry, 22(1), 1-13. https://doi.org/10.1186/s12888-022-04223-4
  90. Zuromski, K. L., Bernecker, S. L., Gutierrez, P. M., Joiner, T. E., King, A. J., Liu, H., Naifeh, J. A., Nock, M. K., Sampson, N. A., Zaslavsky, A. M., Stein, M. B., Ursano, R. J., & Kessler, R. C. (2019). Assessment of a risk index for suicide attempts among US army soldiers with suicide ideation: analysis of data from the army study to assess risk and resilience in servicemembers (Army STARRS). JAMA network open, 2(3), e190766-e190766. 10.1001/jamanetworkopen.2019.0766