This paper is in the following e-collection/theme issue:

Articles (1) Anxiety and Stress Disorders (999) Machine Learning (1370) e-Mental Health and Cyberpsychology (1386) Student/Learners Perceptions and Experiences with Educational Technology (63) Depression and Mood Disorders; Suicide Prevention (1271) Psychological, -behavioural, -social, and/or -cultural Experiments and Interventions (35) mHealth for Wellness, Behavior Change and Prevention (2728)

Published on in Vol 13 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/50982, first published .
Predicting the Effectiveness of a Mindfulness Virtual Community Intervention for University Students: Machine Learning Model

Predicting the Effectiveness of a Mindfulness Virtual Community Intervention for University Students: Machine Learning Model

Predicting the Effectiveness of a Mindfulness Virtual Community Intervention for University Students: Machine Learning Model

Authors of this article:

Christo El Morr1 Author Orcid Image ;   Farideh Tavangar2 Author Orcid Image ;   Farah Ahmad1 Author Orcid Image ;   Paul Ritvo3 Author Orcid Image ;   MVC Team4
Article Authors Cited by Tweetations (3) Metrics

    Original Paper

    1School of Health Policy and Management, York University, Toronto, ON, Canada

    2Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, ON, Canada

    3Kinesiology & Health Science, York University, Toronto, ON, Canada

    4See Acknowledgments

    Corresponding Author:

    Christo El Morr, PhD

    School of Health Policy and Management

    York University

    4700 Keele Street

    Toronto, ON, M3J 1P3

    Canada

    Phone: 1 426 736 2100 ext 22053

    Email: elmorr@yorku.ca


    Background: Students’ mental health crisis was recognized before the COVID-19 pandemic. Mindfulness virtual community (MVC), an 8-week web-based mindfulness and cognitive behavioral therapy program, has proven to be an effective web-based program to reduce symptoms of depression, anxiety, and stress. Predicting the success of MVC before a student enrolls in the program is essential to advise students accordingly.

    Objective: The objectives of this study were to investigate (1) whether we can predict MVC’s effectiveness using sociodemographic and self-reported features and (2) whether exposure to mindfulness videos is highly predictive of the intervention’s success.

    Methods: Machine learning models were developed to predict MVC’s effectiveness, defined as success in reducing symptoms of depression, anxiety, and stress as measured using the Patient Health Questionnaire-9 (PHQ-9), the Beck Anxiety Inventory (BAI), and the Perceived Stress Scale (PSS), to at least the minimal clinically important difference. A data set representing a sample of undergraduate students (N=209) who took the MVC intervention between fall 2017 and fall 2018 was used for this secondary analysis. Random forest was used to measure the features’ importance.

    Results: Gradient boosting achieved the best performance both in terms of area under the curve (AUC) and accuracy for predicting PHQ-9 (AUC=0.85 and accuracy=0.83) and PSS (AUC=1 and accuracy=1), and random forest had the best performance for predicting BAI (AUC=0.93 and accuracy=0.93). Exposure to online mindfulness videos was the most important predictor for the intervention’s effectiveness for PHQ-9, BAI, and PSS, followed by the number of working hours per week.

    Conclusions: The performance of the models to predict MVC intervention effectiveness for depression, anxiety, and stress is high. These models might be helpful for professionals to advise students early enough on taking the intervention or choosing other alternatives. The students’ exposure to online mindfulness videos is the most important predictor for the effectiveness of the MVC intervention.

    Trial Registration: ISRCTN Registry ISRCTN12249616; https://www.isrctn.com/ISRCTN12249616

    Interact J Med Res 2024;13:e50982

    doi:10.2196/50982

    Keywords

    machine learning; virtual community; virtual care; mindfulness; depression; anxiety; stress; students; online; randomized controlled trial; Canada; virtual; artificial intelligence; symptoms; behavioral therapy; sociodemographic; mindfulness video; online video 


    Students’ mental health crises were recognized before the COVID-19 pandemic and deepened during the pandemic. University students are experiencing an increase in psychological distress on North American campuses. A student survey of 32 Canadian postsecondary institutions reported high anxiety (56.5%), hopelessness (54%), seriously depressed mood (37.5%), and overwhelming anger (42%) [1]. A similar survey in 2016 revealed higher distress levels [2]. In 2013, a study of 997 students at York University (site of this study) indicated that 57% reported depression scores sufficient for diagnosable clinical depression, while 33% reported anxiety scores in ranges typically indicative of panic disorder and generalized anxiety disorder [3]. The situation appears similar at universities in the United States [4,5] and worldwide; in 2018, the World Health Organization reported increasing mental disorders in college and university students worldwide [6]. Mental health challenges among university students demand attention. This is a vulnerable period, as 70% of mental health problems emerge before the age of 25 years. Without intervention, these problems can worsen and hinder students’ personal and academic success [7]. COVID-19 has negatively impacted university students’ mental health [8-10].

    University student distress is both an individual and societal challenge. Losses in productivity during the study and at work due to distress and mental disorders are associated with indirect but significant economic burdens [11]. Canadian estimates show that mental disorders cost nearly US $37 billion yearly, with 9.8% due to direct medical costs, 16.6% and 18.2% due to long-term loss and short-term work loss, respectively, and 55.4% due to the loss of healthy function (ie, loss of the utilities of vision, hearing, speech, mobility, dexterity, emotion, cognition, and pain as assessed in the Health Utilities Index Mark 3 system) [12].

    While mental distress and disorder are becoming more prevalent in students, the counseling offered in colleges and universities needs to catch up with demand. For example, from 2007 to 2012, full-time enrollment in the Ontario college system increased from 167,000 to 210,600 (a 26% increase), while the number of counselors employed in the college system increased from 146 to 152.7 (a 4.6% increase) [13]. This discrepancy leaves students underserved and counselors overwhelmed amid the increasing distress [14].

    Mindfulness-based interventions have been demonstrated to positively impact psychological and physical health [15-17], with multiple meta-analyses demonstrating positive impacts in clinical and nonclinical populations [18-22]. However, with large numbers of students (50,000 to 60,000 on some campuses), there may not be enough trained personnel to convey helpful mindfulness-based practices directly. Instead, in the eHealth domain, virtual communities (VCs) [23], that is, online communities, have been used in health care to provide e-education tools and online support to empower active participants in health enhancement [24-26]. VCs can scale up mindfulness interventions at lower costs to a broader range of students, especially those restricted from attending clinics due to time-place discontinuities. VCs preserve anonymity (with reduced stigmatization) while promoting voluntary, supportive, interpersonal connections.

    We developed a web-delivered mindfulness program (mindfulness virtual community [MVC]) to reduce symptoms of depression, anxiety, and stress in university students and conducted a randomized controlled trial (RCT) targeting university students at a Canadian university to examine its effectiveness. Following a successful RCT [26-29], we wanted in this secondary analysis (1) to develop a machine learning (ML) model to predict the effectiveness of the online mindfulness intervention on mental health outcomes using sociodemographic and self-reported features and (2) to investigate if exposure to mindfulness videos was highly predictive of the intervention’s success.


    Prediction Problem

    This study aims to predict the effectiveness (ie, success vs nonsuccess) of the online mindfulness intervention on mental health outcomes; as such, this is a retrospective prognostic analysis of a classification problem per individual (ie, participants in the MVC mindfulness intervention).

    Data Set Source

    This is a retrospective analysis, where we analyzed an anonymized data set. The data were deidentified, and consent was obtained during the RCT; no further consent was sought for this secondary data analysis since nonidentifiable data were used. The data set was collected via an RCT described in detail elsewhere [28]. The parent study design consisted of a 2-arm parallel-design RCT, comparing a group assigned to the web-based MVC program to a waitlist control group. Participants in the study were students who were at least 18 years of age, reported English language fluency, self-reported high confidence in completing the study, and actively enrolled in an undergraduate program. This paper is based on the MVC intervention sample recruited in fall 2017, winter 2017, and fall 2018. The MVC intervention was an 8-week program and was comprised of three components: (1) 12 online videos for mental health education; (2) 3 anonymous discussion boards on depression, anxiety, and stress; and (3) anonymous, 20-minute group-based live videoconferences led by a mental health professional with training in mindfulness during which students could raise questions related to mindfulness (Figure 1).

    Each of the 12 mental health modules consisted of 1 educational content video and 1 mindfulness practice video recorded in both male and female voices and offered in both high and low resolution (a total of 8 videos per module); participants could choose the type of video they wanted to watch for each module. The videos were available for participants 24 hours a day to watch or listen to on computers, phones, or tablets at their convenience. The module scripts and audio recordings were created by one of the investigators with extensive experience as a clinical psychologist and researcher in mindfulness. They were based on mindfulness and cognitive behavioral therapy principles and informed by the prior student-based focus group study [30,31]—the choice of moving and still images used in creating the videos involved collaborative work. The topics of the 12 modules included the following: overcoming stress, anxiety, and depression; mindfulness and being a student; mindfulness for better sleep; thriving in a fast-changing world; healthy intimacy; destigmatization; no more procrastination; pain reduction and mindfulness; healthy body image; healthier eating; overcoming trauma; and relationships with family and friends.

    The primary RCT outcomes were depression, anxiety, and perceived stress, following hypotheses that symptom scores for depression, anxiety, and stress at T2 (after 8 weeks) would be significantly better in the MVC group when compared with the waitlist control group. The outcomes were measured with the following validated scales: Patient Health Questionnaire-9 (PHQ-9) [32], Beck Anxiety Inventory (BAI) [33], and Perceived Stress Scale (PSS) [34]. The secondary aim was to assess the impact of 3 elements of the MVC intervention on the outcomes. Participants also completed a sociodemographic questionnaire section at the T1 (baseline) survey.

    ‎
    Figure 1. The mindfulness virtual community design.

    Ethical Considerations

    The previous study received ethics approval from the Human Participant Research Committee (certificate e2016-345) of the York University. This ML study received ethics approval from the same committee (certificate e2023-012); the approval covers secondary analysis without additional consent. Participants in the original study had the option to receive an honorarium of CAD $50 (US $37.5) or 2% in course grade (for professors who gave this permission) or 3 credits (equivalent to 2% course grade) in the Undergraduate Research Participation Pool of the Department of Psychology. The participants’ data were anonymized.

    Participants

    We aimed to build a model to predict who will likely benefit from the intervention, unlike the RCT study, where overall intervention effectiveness was determined (and supported by analysis) by comparing intervention and control groups. That is why we have analyzed intervention group data only to understand individual differences in response to the intervention.

    Data Preparation

    The data set consisted of 209 students who took the MVC intervention during fall 2017, winter 2018, and fall 2018. The effectiveness of the intervention was determined using the minimal clinically important difference (MCID), that is, the level of reduction in symptoms that psychologists consider clinically meaningful, for each of the mental health outcomes. We adopted evidence from psychology that determines the MCID to be a 5-point reduction in PHQ-9 for depression [35,36], an 8.8-point reduction in BAI for anxiety [30,31], and an 11-point reduction in PSS for stress [37,38]. Any reduction equal to or above the MCID was labeled an effective intervention (label=1); otherwise, it was deemed ineffective (label=0).

    To build a good prediction model from the training set, the data must be balanced. The class labels of the target variables, PHQ-9, BAI, and PSS, used in this study were not balanced. In our case, the percentage of instances with label=1 was extremely low: 50 (23%) for PHQ-9, 48 (24%) for BAI, and 8 (3.8%) for PSS, leading to a substantial imbalance. To alleviate the imbalanced data, we applied an oversampling method using the sklearn.resample function available in Python (version 3; Python Software Foundation).

    Missing Data

    Missing data in the outcomes were 12 (5.7%) for BAI and PHQ-9 and 13 (6.2%) for PSS of the 209 records. Missing data for the outcomes were dropped from the data set. There were no missing values in the predictors.

    Labels and Features

    The outcome variables were the 3 MCIDs associated with PHQ-9, BAI, and PSS being met or not for each instance. To investigate whether we can predict MVC’s effectiveness using sociodemographic and self-reported features, the following features were used: sex (male and female), country of birth (Canada and other), first language (English and other), education level (bachelor degree and other), ethnicity (White and non-White), marital status (married and other), age, number of weekly working hours, and self-rated health (poor, fair, good, very good, and excellent). To investigate the importance of exposure to mindfulness videos, in comparison with these features, in the prediction of intervention success, we added the total number of mindfulness videos watched to the previous data set.

    Algorithms

    Seven different classification algorithms, representing different learning paradigms, were used in this study: logistic regression (LR), support vector machine (SVM), random forest (RF), decision tree (DT), k-nearest neighbor (KNN), adaptive boosting (AdaBoost), and gradient boosting that showed good performance in previous studies that targeted depression, anxiety, and stress [35,39,40]. The implementations of the classification algorithms provided in the scikit-learn ML library [41] were used. The data set was split into 80% for training and 20% for testing. Hyperparameter tuning for each algorithm was performed using a grid search over a 10-fold cross-validation on the training data set. The optimal hyperparameters for the classification algorithms and their values for the data set without exposure to videos and the data set with exposure to videos are presented in Tables 1 and 2, respectively.

    Each classifier’s performance was compared with the best overall performance, leading to the selection of the best prediction model for the psychological outcomes. The classifiers’ performances were assessed based on several evaluation metrics, including the percentage of correctly classified instances or the accuracy, sensitivity, specificity, and area under the curve (AUC) of the receiver operating characteristic curve. The best performance, as measured by the AUC score, was chosen for each algorithm.

    To evaluate the features’ importance in predicting intervention success, the data set with the total exposure to mindfulness videos was used to build predictive models. The RF algorithm was used to measure the features’ importance. The hyperparameters used for the classification algorithms and their values that provided the optimal model are presented in Table 2.

    Table 1. Algorithms and their corresponding optimal hyperparameters found by grid search (data set without videos).
    AlgorithmParameters
    Logistic regression

    		PHQ-9aC=1, penalty=l1, solver=liblinear

    		BAIbC=1, penalty=l1, solver=liblinear

    		PSScC=1, penalty=l1, solver=liblinear
    Support vector machine

    		PHQ-9C=10, γ=0.1, kernel=rbf

    		BAIC=10, γ=0.1, kernel=rbf

    		PSSC=10, γ=0.1, kernel=rbf
    Random forest

    		PHQ-9Max_features=auto, n_estimators=500, max_depth=8, criterion=entropy

    		BAIMax_features=auto, n_estimators=500, max_depth=8, criterion=gini

    		PSSMax_features=auto, n_estimators=500, max_depth=8, criterion=gini
    Decision tree

    		PHQ-9Max_leaf_nodes=59, random_state=42, min_samples_split=2, criterion=entropy

    		BAIMax_leaf_nodes=56, random_state=42, min_samples_split=2, criterion=entropy

    		PSSMax_leaf_nodes=16, random_state=42, min_samples_split=2, criterion=entropy
    K-nearest neighbor

    		PHQ-9N_neighbors=2, weight=distance, leaf size=27, P=1

    		BAIN_neighbors=2, weight=distance, leaf size=1, P=1

    		PSSN_neighbors=1, weight=dniform, leaf size=1, P=1
    Adaptive boosting

    		PHQ-9n-estimators=5000, max_depth=3, learning rate=0.5

    		BAIn-estimators=5000, max_depth=3, learning rate=0.9

    		PSSn-estimators=500, max_depth=3, learning rate=0.9
    Gradient boosting

    		PHQ-9Learning rate=0.05, max depth=6, n-estimators=100, subsample=0.9, max_features=none, min_samples_split=2

    		BAILearning rate=0.02, max depth=10, n-estimators=1000, subsample=1.0, max_features=none, min_samples_split=2

    		PSSLearning rate=0.01, max depth=6, n-estimators=1000, subsample=0.9, max_features=sqrt, min_samples_split=2

    aPHQ-9: Patient Health Questionnaire-9.

    bBAI: Beck Anxiety Inventory.

    cPSS: Perceived Stress Scale.

    Table 2. Algorithms and their corresponding optimal hyperparameters found by grid search (data set with exposure to videos).
    AlgorithmParameters
    Logistic regression

    		PHQ-9aC=1, penalty=l1, solver=liblinear

    		BAIbC=0.1, penalty=l2, solver=newton-cg

    		PSScC=100, penalty=l2, solver=lbfgs
    Support vector machine

    		PHQ-9C=10, γ=0.01, kernel=rbf

    		BAIC=1, γ=1, kernel=rbf

    		PSSC=1, γ=1, kernel=rbf
    Random forest

    		PHQ-9Max_features=auto, n_estimators=500, max_depth=7, criterion=entropy

    		BAIMax_features=auto, n_estimators=200, max_depth=8, criterion=gini

    		PSSMax_features=auto, n_estimators=500, max_depth=8, criterion=gini
    Decision tree

    		PHQ-9Max_leaf_nodes=39, random_state=42, min_samples_split=2, criterion=entropy

    		BAIMax_leaf_nodes=53, random_state=42, min_samples_split=3, criterion=gini

    		PSSMax_leaf_nodes=16, random_state=42, min_samples_split=2, criterion=gini
    K-nearest neighbor

    		PHQ-9N_neighbors=1, weight=uniform, leaf size=14, P=1

    		BAIN_neighbors=2, weight=distance, leaf size=1, P=1

    		PSSN_neighbors=2, weight=uniform, leaf size=1, P=1
    Adaptive boosting

    		PHQ-9n-estimators=500, max_depth=3, learning rate=0.5

    		BAIn-estimators=500, max_depth=3, learning rate=0.7

    		PSSn-estimators=2000, max_depth=3, learning rate=0.7
    Gradient boosting

    		PHQ-9Learning rate=0.5, max depth=50, n-estimators=50, subsample=0.9, max_features=sqrt, min_samples_split=2

    		BAILearning rate=0.04, max depth=10, n-estimators=1000, subsample=0.5, max_features=none, min_samples_split=2

    		PSSLearning rate=0.03, max depth=8, n-estimators=1000, subsample=0.5, max_features=none, min_samples_split=2

    aPHQ-9: Patient Health Questionnaire-9.

    bBAI: Beck Anxiety Inventory.

    cPSS: Perceived Stress Scale.


    Demographics

    Table 3 presents the demographic characteristics of participants at baseline. Of 209 students, 73.2% (n=153) were female, 8.1% (n=17) were married, and 21.1% (n=44) were White. Most participants were born in Canada, and English was their first language. The median (IQR) of age, work hours per week, and the total number of mindfulness videos watched were 21 (19-23) years, 10 (0-18), and 16 (9-30), respectively.

    Table 3. Characteristics of participants at baseline (N=209).
    CharacteristicsValues
    Sex, n (%)

    		Male56 (26.8)

    		Female153 (73.2)
    Marital status, n (%)

    		Married17 (8.1)

    		Other192 (91.9)
    Ethnicity, n (%)

    		White44 (21.1)

    		Non-White165 (78.9)
    Language, n (%)

    		English136 (65.1)

    		Other73 (34.9)
    Country of birth, n (%)

    		Canada119 (56.9)

    		Other90 (43.1)
    Education, n (%)

    		High school diploma or General Education Development or college degree or certificate program182 (87.1)

    		Bachelor degree27 (12.9)
    Self-reported general health, n (%)

    		Poor or fair43 (20.6)

    		Good or very good or excellent166 (79.4)
    Age (years), median (IQR)21 (19-23)
    Average number of hours at work per week, median (IQR)10 (0-18)
    Total number of mindfulness videos watched, median (IQR)16 (9-30)

    Objective 1: Predicting MVC’s Effectiveness Using Sociodemographic and Self-Reported Features

    Table 4 summarizes the evaluated models’ performances: sensitivity, specificity, accuracy, and AUC, using 10-fold cross-validation.

    The results showed that both gradient boosting (AUC=0.85 and accuracy=0.83) and DT (AUC=0.84 and accuracy=0.81) are slightly better compared to AdaBoost and KNN (AUC=0.82 and accuracy=0.80) as well as SVM (AUC=0.81 and accuracy=0.80) and outperformed the remaining classification algorithms for predicting a clinically significant reduction in PHQ-9. The best classifiers for predicting a clinically significant reduction in BAI were RF (AUC=0.93 and accuracy=0.93), followed by AdaBoost (AUC=0.92 and accuracy=0.92) and gradient boosting (AUC=0.87 and accuracy=0.87), which outperformed the remaining classifiers. Two classifiers, gradient boosting and DT, gained the perfect accuracy and AUC (AUC=1 and accuracy=1) for predicting a clinically significant reduction in PSS, followed by the near-perfect scores for SVM and AdaBoost (AUC=0.99 and accuracy=0.99). Meanwhile, LR had the lowest performance for PHQ-9, BAI, and PSS in terms of AUC (0.64, 0.75, and 0.73, respectively) and accuracy (0.66, 0.75, and 0.73, respectively).

    The results were close to those found in the models built without video exposure. Gradient boosting (AUC=0.89 and accuracy=0.88) was the best predictor for a significant reduction in PHQ-9, followed closely by AdaBoost and DT (AUC=0.84 and accuracy=0.81), which outperformed the remaining classification algorithms. The best classifiers for predicting a clinically significant reduction in BAI were AdaBoost and SVM (AUC=0.93 and accuracy=0.93), followed closely by gradient boosting (AUC=0.91 and accuracy=0.92) and RF (AUC=0.90 and accuracy=0.90), which outperformed the remaining classifiers. Four classifiers, gradient boosting, AdaBoost, RF, and SVM, gained the perfect AUC and accuracy (AUC=1 and accuracy=1) for predicting a clinically significant reduction in PSS, followed by the near-perfect score for KNN (AUC=0.99 and accuracy=0.99) and DT (AUC=0.97 and accuracy=0.97). Meanwhile, LR had the lowest performance for PHQ-9, BAI, and PSS in terms of AUC (0.62, 0.60, and 0.79, respectively) and accuracy (0.63, 0.60, and 0.80, respectively).

    Using the second data set (ie, enriched with the exposure to videos), RF was used to detect features’ importance in relation to the 3 outcomes. The most predictive feature for the PHQ-9, BAI, and PSS was the total exposure to the mindfulness videos, followed by the average number of working hours per week and age for PHQ-9 and BAI. In contrast, age and the average number of working hours per week were the second and third most important predictors for PSS, respectively.

    Table 4. Classification report of the machine learning algorithms for outcomes.
    AlgorithmAUCaAccuracySensitivitySpecificity
    Logistic regression

    		PHQ-9b0.640.660.570.72

    		BAIc0.750.750.750.75

    		PSSd0.730.730.730.74
    Support vector machine

    		PHQ-90.810.800.900.75

    		BAI0.770.770.790.75

    		PSS0.960.961.00.91
    Random forest

    		PHQ-90.780.760.870.69

    		BAI0.930.930.861.0

    		PSS0.990.991.00.97
    Decision tree

    		PHQ-90.840.810.960.72

    		BAI0.840.830.930.75

    		PSS1.01.01.01.0
    K-nearest neighbor

    		PHQ-90.820.800.910.72

    		BAI0.780.780.680.88

    		PSS0.960.961.00.91
    Adaptive boosting

    		PHQ-90.820.800.910.72

    		BAI0.920.920.890.94

    		PSS0.990.991.00.97
    Gradient boosting

    		PHQ-90.850.830.910.78

    		BAI0.870.870.860.88

    		PSS1.01.01.01.0

    aAUC: area under the curve.

    bPHQ-9: Patient Health Questionnaire-9.

    cBAI: Beck Anxiety Inventory.

    dPSS: Perceived Stress Scale.

    Objective 2: Importance of Exposure to Mindfulness Videos in Comparison With Sociodemographics and Self-Reported Features in Predicting Intervention Success

    After the introduction of the total exposure to the mindfulness videos to the data set, new predictive models were built (Table 5).

    Table 5. Classification report of the machine learning algorithms for outcomes (data set with exposure to videos).
    AlgorithmAUCaAccuracySensitivitySpecificity
    Logistic regression

    		PHQ-9b0.620.630.570.67

    		BAIc0.600.600.610.59

    		PSSd0.790.800.850.74
    Support vector machine

    		PHQ-90.780.750.960.61

    		BAI0.930.930.861.00

    		PSS1.001.001.001.00
    Random forest

    		PHQ-90.840.830.870.81

    		BAI0.900.900.860.94

    		PSS1.001.001.001.00
    Decision tree

    		PHQ-90.840.810.960.72

    		BAI0.800.800.860.75

    		PSS0.970.971.000.94
    K-nearest neighbor

    		PHQ-90.810.780.960.67

    		BAI0.840.830.890.78

    		PSS0.990.991.000.97
    Adaptive boosting

    		PHQ-90.840.810.960.72

    		BAI0.930.930.890.97

    		PSS1.001.001.001.00
    Gradient boosting

    		PHQ-90.890.880.960.83

    		BAI0.910.920.860.97

    		PSS1.001.001.001.00

    aAUC: area under the curve.

    bPHQ-9: Patient Health Questionnaire-9.

    cBAI: Beck Anxiety Inventory.

    dPSS: Perceived Stress Scale.


    Principal Results

    The study investigated the predictability of the effectiveness of an MVC designed for undergraduate students to reduce symptoms of depression, anxiety, and stress as measured by PHQ-9, BAI, and PSS. The effectiveness was measured by the MCID for PHQ-9, BAI, and SPSS. Several algorithms were used to predict the MCID.

    Predicting Intervention Success With Sociodemographic and Self-Reported Measures

    We successfully built ML-based models that predicted the effectiveness of the MVC intervention. The highest AUC was achieved for gradient boosting to predict the intervention effectiveness for PHQ-9 and PSS (AUC=0.85 and AUC=1, respectively), followed closely by DT (AUC=0.84 and AUC=1, respectively) and AdaBoost (AUC=0.82 and AUC=0.99, respectively). The RF model had the highest AUC to predict BAI (AUC=0.93), followed closely by AdaBoost (AUC=0.92). AdaBoost might be the algorithm of choice for the 3 outcomes, as it is fairing a close second best for BAI and a close third best for PHQ-9 and PSS. Gradient boosting and AdaBoost are both good choices to predict the intervention success for the 3 outcomes. It might be argued that AdaBoost might be preferable, given that it is usually less prone to overfitting than gradient boosting; however, there is no need to use the same algorithm to build the 3 predictors for the 3 outcomes.

    We could not make a direct comparison with other studies that measured the 3 outcomes among university students using the same validated scales (PHQ-9, BAI, and SPSS). However, for PHQ-9, the performance of our model is higher than the one found in a previous study among adults in Korea using the Center for Epidemiologic Studies—Depression Scale 11 (AUC=0.87 and accuracy=0.86) [40] as well as the one found in a study in the United States that defined the success of the intervention as a 5-point reduction in PHQ-9 or a 4-point reduction in the General Anxiety Disorder screener-7 values (AUC=0.60 and accuracy=0.71) [35]. Regarding anxiety, the predictive model developed in this study had a higher performance (accuracy=0.92) than another study that used the Self-Rating Anxiety Scale, which did not report AUC but reported an accuracy of 0.84.

    Feature Importance

    Exposure to mindfulness videos was the most important factor in predicting the intervention’s success. This study has demonstrated a link between the MVC intervention’s success and exposure to mindfulness videos. It also confirms the results of the previous MVC pilot study that proved that exposure to mindfulness videos alone, without interaction between participants via an online discussion forum and without weekly videoconferencing with a coach, effectively reduced symptoms of depression, anxiety, and stress [26]. In other words, it indicates the ability of MVC to be deployed at a large scale without an increase in human resources. Scalability is a critical factor for eHealth intervention deployment in large populations. This finding suggests that scaling up an effective e-mental health MVC is possible in a cost-effective manner; scalability is one of the recognized failures in eHealth implementations [42].

    Practical and Policy Implications

    The MVC intervention does not provide clinical support; it is a platform that offers self-management of mental health symptoms (depression, anxiety, and stress). The MVC intervention proved to be effective [26-28] in reducing symptoms of depression, anxiety, and stress in university students. This study builds a predictive model that predicts intervention success using sociodemographic and self-reported measures; this will allow counseling services on university campuses to assess the usefulness of MVC for a particular student before taking the intervention and advise them accordingly to use MVC or to opt for another type of intervention. This will enable counseling services to personalize the advice to students’ profiles and allow students to manage their symptoms with the most appropriate intervention.

    The other finding related to videos being the most important factor in predicting intervention success confirms the ability of MVC to be deployed at a large scale without an increase in human resources. The number of working hours is another important predictor of the success of the intervention. Although the provincial governments in Canada support university education, students must pay for their education and bear the cost of living. Not surprisingly, they work long hours, especially if they belong to a marginalized community. Our findings align with other studies that suggest that longer working hours outside the university and difficulty paying bills were recognized as predictors of poor mental health among students [43]. In Ontario, where the sample was taken, Statistics Canada recently reported an increased reliance of academic institutions on students’ fees in higher education to the extent that 54% of all college revenues in 2019/2020 were downloaded on students, which translates into an overall decline in public funding [44]. This situation pushed students to longer working hours; one can argue that since student debt has been recognized as negatively associated with mental well-being and academic outcomes [45,46], providing access to free higher education, supported by taxes such as in most of Europe, could enhance students’ mental well-being as it would relieve them from the need for long working hours.

    Strengths and Limitations

    One of the strengths of this study is the ability to predict the intervention’s success based on a few demographics and one question about self-rated health. Hence, the predictive model can be used in real life to indicate the suitability of online mindfulness intervention for specific individuals and possibly suggest alternatives if the model predicts noneffectiveness. The excellent AUC and accuracy measures make the models suitable for implementation and evaluation in real-life scenarios. However, the ML models must be monitored continuously if implemented for daily use (eg, a counseling service) [47,48].

    A limitation of this study is that it relied on research done on 1 site; future research with larger samples with participants from multiple universities and colleges would better test the generalizability of results as it allows us to test the effectiveness of the models on external data.

    Conclusions

    Our results suggest that we can build high-performing models to predict MVC intervention effectiveness for depression, anxiety, and stress based on simple sociodemographics and self-reported features and that exposure to mindfulness videos is the most important predictor for the effectiveness of the intervention. Our findings provide evidence that scaling MVC can be done without additional cost for support and that the predictive models might be useful for professionals to advise students early enough on taking the intervention or choosing other alternatives.

    Acknowledgments

    The authors acknowledge the contribution of all students who spent valuable time participating in the study. The MVC Team members are Sahir Abbas, BSc; Yvonne Bohr, PhD; Manuela Ferrari, PhD; Wai Lun Alan Fung, MD, ScD, FRCPC; Louise Hartley, PhD; Amin Mawani, PhD; Kwame McKenzie, MD, FRCPC; and Jan E Odai, BA. They made contributions to several aspects of the project and the results’ development. They approve the final version and agree to be accountable for all aspects of the submitted paper. The work reported in this paper was funded by the Canadian Institutes for Health Research and eHealth Innovations Partnership Program grant (EH1-143553). The project’s principal investigators are CE, FA, and PR.

    Data Availability

    The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.

    Authors' Contributions

    CE, FA, and PR designed the original mindfulness virtual community study questionnaire, received the funds, and contributed equally. CE supervised FT, who performed and reported the analysis. CE verified the analysis and prepared the first draft. All authors provided critical feedback and revised it.

    Conflicts of Interest

    It is the understanding of the university and researchers that the Project Intellectual Property belongs to the CE, FA, and PR. The industry partner ForaHealthyMe.com owns all rights and titles to the copyrights of any computer source code software developed from this research project.

    1. American College Health Association-National College Health Assessment II: Canadian reference group executive summary spring 2013. American College Health Association. 2013. URL: https://tinyurl.com/vxn2swse [accessed 2024-04-18]
    2. American College Health Association-National College Health Assessment II: Canadian reference group executive summary spring 2016. American College Health Association. 2016. URL: https://tinyurl.com/bdfpv8b4 [accessed 2024-04-18]
    3. Pirbaglou M, Cribbie R, Irvine J, Radhu N, Vora K, Ritvo P. Perfectionism, anxiety, and depressive distress: evidence for the mediating role of negative automatic thoughts and anxiety sensitivity. J Am Coll Health. 2013;61(8):477-483. [FREE Full text] [CrossRef] [Medline]
    4. Eisenberg D, Hunt J, Speer N, Zivin K. Mental health service utilization among college students in the United States. J Nerv Ment Dis. 2011;199(5):301-308. [CrossRef] [Medline]
    5. Han B, Compton WM, Eisenberg D, Milazzo-Sayre L, McKeon R, Hughes A. Prevalence and mental health treatment of suicidal ideation and behavior among college students aged 18-25 years and their non-college-attending peers in the United States. J Clin Psychiatry. 2016;77(6):815-824. [FREE Full text] [CrossRef] [Medline]
    6. Auerbach RP, Mortier P, Bruffaerts R, Alonso J, Benjet C, Cuijpers P, et al. WHO World Mental Health Surveys International College Student project: prevalence and distribution of mental disorders. J Abnorm Psychol. 2018;127(7):623-638. [FREE Full text] [CrossRef] [Medline]
    7. The human face of mental health and mental illness in Canada 2006. Government of Canada. 2006. URL: https://www.phac-aspc.gc.ca/publicat/human-humain06/pdf/human_face_e.pdf [accessed 2024-04-18]
    8. Allen R, Kannangara C, Vyas M, Carson J. European university students' mental health during Covid-19: exploring attitudes towards Covid-19 and governmental response. Curr Psychol. 2022.:1-14. [FREE Full text] [CrossRef] [Medline]
    9. Koelen JA, Mansueto AC, Finnemann A, de Koning L, van der Heijde CM, Vonk P, et al. COVID-19 and mental health among at-risk university students: a prospective study into risk and protective factors. Int J Methods Psychiatr Res. 2022;31(1):e1901. [FREE Full text] [CrossRef] [Medline]
    10. Zapata-Ospina JP, Patiño-Lugo DF, Vélez CM, Campos-Ortiz S, Madrid-Martínez P, Pemberthy-Quintero S, et al. Mental health interventions for college and university students during the COVID-19 pandemic: a critical synthesis of the literature. Rev Colomb Psiquiatr (Engl Ed). 2021;50(3):199-213. [FREE Full text] [CrossRef] [Medline]
    11. Ungar T. The health care payment game is rigged. National Post. Apr 28, 2015. URL: https://nationalpost.com/opinion/thomas-ungar-the-health-care-payment-game-is-rigged [accessed 2024-05-03]
    12. Lim KL, Jacobs P, Ohinmaa A, Schopflocher D, Dewa CS. A new population-based measure of the economic burden of mental illness in Canada. Chronic Dis Can. 2008;28(3):92-98. [FREE Full text] [Medline]
    13. Bayram N, Bilgel N. The prevalence and socio-demographic correlations of depression, anxiety and stress among a group of university students. Soc Psychiatry Psychiatr Epidemiol. 2008;43(8):667-672. [FREE Full text] [CrossRef] [Medline]
    14. Pfeffer A. Ontario campus counsellors say they're drowning in mental health needs. CBC. 2019. URL: https://www.cbc.ca/news/canada/ottawa/mental-health-ontario-campus-crisis-1.3771682 [accessed 2019-07-07]
    15. Brown KW, Ryan RM. The benefits of being present: mindfulness and its role in psychological well-being. J Pers Soc Psychol. 2003;84(4):822-848. [CrossRef] [Medline]
    16. Keng SL, Smoski MJ, Robins CJ. Effects of mindfulness on psychological health: a review of empirical studies. Clin Psychol Rev. 2011;31(6):1041-1056. [FREE Full text] [CrossRef] [Medline]
    17. Grossman P, Niemann L, Schmidt S, Walach H. Mindfulness-based stress reduction and health benefits. A meta-analysis. J Psychosom Res. 2004;57(1):35-43. [FREE Full text] [CrossRef] [Medline]
    18. Chiesa A, Serretti A. Mindfulness-based stress reduction for stress management in healthy people: a review and meta-analysis. J Altern Complement Med. 2009;15(5):593-600. [FREE Full text] [CrossRef] [Medline]
    19. Hofmann SG, Sawyer AT, Witt AA, Oh D. The effect of mindfulness-based therapy on anxiety and depression: a meta-analytic review. J Consult Clin Psychol. 2010;78(2):169-183. [FREE Full text] [CrossRef] [Medline]
    20. Vøllestad J, Nielsen MB, Nielsen GH. Mindfulness- and acceptance-based interventions for anxiety disorders: a systematic review and meta-analysis. Br J Clin Psychol. 2012;51(3):239-260. [FREE Full text] [CrossRef] [Medline]
    21. Eberth J, Sedlmeier P. The effects of mindfulness meditation: a meta-analysis. Mindfulness. 2012;3(3):174-189. [FREE Full text] [CrossRef]
    22. Sedlmeier P, Eberth J, Schwarz M, Zimmermann D, Haarig F, Jaeger S, et al. The psychological effects of meditation: a meta-analysis. Psychol Bull. 2012;138(6):1139-1171. [CrossRef] [Medline]
    23. Bender JL, Jimenez-Marroquin MC, Ferris LE, Katz J, Jadad AR. Online communities for breast cancer survivors: a review and analysis of their characteristics and levels of use. Support Care Cancer. 2013;21(5):1253-1263. [FREE Full text] [CrossRef] [Medline]
    24. El Morr C. Mobile virtual communities in healthcare: managed self care on the move. 2007. Presented at: Telehealth '07: The Third IASTED International Conference on Telehealth; May 31, 2007-June 1, 2007; Montreal, Canada. [CrossRef]
    25. Morr CE, Kawash J. Mobile virtual communities research: a synthesis of current trends and a look at future perspectives. Int J Web Based Communities. 2007;3(4):386-403. [FREE Full text] [CrossRef]
    26. Ahmad F, El Morr C, Ritvo P, Othman N, Moineddin R, MVC Team. An eight-week, web-based mindfulness virtual community intervention for students' mental health: randomized controlled trial. JMIR Ment Health. 2020;7(2):e15520. [FREE Full text] [CrossRef] [Medline]
    27. Ritvo P, Ahmad F, El Morr C, Pirbaglou M, Moineddin R, MVC Team. A mindfulness-based intervention for student depression, anxiety, and stress: randomized controlled trial. JMIR Ment Health. 2021;8(1):e23491. [FREE Full text] [CrossRef] [Medline]
    28. El Morr C, Ritvo P, Ahmad F, Moineddin R, MVC Team. Effectiveness of an 8-week web-based mindfulness virtual community intervention for university students on symptoms of stress, anxiety, and depression: randomized controlled trial. JMIR Ment Health. 2020;7(7):e18595. [FREE Full text] [CrossRef] [Medline]
    29. El Morr C. Virtual communities, machine learning and IoT: opportunities and challenges in mental health research. Int J Extreme Autom Connectivity Healthc. 2019;1(1):4-11. [FREE Full text] [CrossRef]
    30. Carter S, Greenberg J, Funes CJ, Macklin EA, Vranceanu AM. Effects of a mind-body program on symptoms of depression and perceived stress among adults with neurofibromatosis type 2 who are deaf: a live-video randomized controlled trial. Complement Ther Med. 2021;56:102581. [FREE Full text] [CrossRef] [Medline]
    31. Eskildsen A, Dalgaard VL, Nielsen KJ, Andersen JH, Zachariae R, Olsen LR, et al. Cross-cultural adaptation and validation of the Danish consensus version of the 10-item Perceived Stress Scale. Scand J Work Environ Health. 2015;41(5):486-490. [FREE Full text] [CrossRef] [Medline]
    32. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary care evaluation of mental disorders. Patient Health Questionnaire. JAMA. 1999;282(18):1737-1744. [FREE Full text] [CrossRef] [Medline]
    33. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol. 1988;56(6):893-897. [CrossRef] [Medline]
    34. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983;24(4):385-396. [Medline]
    35. Hornstein S, Forman-Hoffman V, Nazander A, Ranta K, Hilbert K. Predicting therapy outcome in a digital mental health intervention for depression and anxiety: a machine learning approach. Digit Health. 2021;7:20552076211060659. [FREE Full text] [CrossRef] [Medline]
    36. Löwe B, Unützer J, Callahan CM, Perkins AJ, Kroenke K. Monitoring depression treatment outcomes with the Patient Health Questionnaire-9. Med Care. 2004;42(12):1194-1201. [FREE Full text] [CrossRef] [Medline]
    37. Goldsworthy S, Palmer S, Latour JM, McNair H, Cramp M. A systematic review of effectiveness of interventions applicable to radiotherapy that are administered to improve patient comfort, increase patient compliance, and reduce patient distress or anxiety. Radiography (Lond). 2020;26(4):314-324. [FREE Full text] [CrossRef] [Medline]
    38. Leentjens AFG, Dujardin K, Marsh L, Richard IH, Starkstein SE, Martinez-Martin P. Anxiety rating scales in Parkinson's disease: a validation study of the Hamilton anxiety rating scale, the Beck anxiety inventory, and the hospital anxiety and depression scale. Mov Disord. 2011;26(3):407-415. [FREE Full text] [CrossRef] [Medline]
    39. Wang C, Zhao H, Zhang H. Chinese college students have higher anxiety in new semester of online learning during COVID-19: a machine learning approach. Front Psychol. 2020;11:587413. [FREE Full text] [CrossRef] [Medline]
    40. Na KS, Cho SE, Geem ZW, Kim YK. Predicting future onset of depression among community dwelling adults in the Republic of Korea using a machine learning algorithm. Neurosci Lett. 2020;721:134804. [FREE Full text] [CrossRef] [Medline]
    41. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825-2830. [FREE Full text]
    42. Moroz N, Moroz I, D'Angelo MS. Mental health services in Canada: barriers and cost-effective solutions to increase access. Healthc Manage Forum. 2020;33(6):282-287. [FREE Full text] [CrossRef] [Medline]
    43. Roberts R, Golding J, Towell T, Reid S, Woodford S, Vetere A, et al. Mental and physical health in students: the role of economic circumstances. Br J Health Psychol. 2010;5(3):289-297. [FREE Full text] [CrossRef]
    44. Trends in private and public funding in Canadian colleges, 2019/2020. Statistics Canada. 2023. URL: https://www150.statcan.gc.ca/n1/daily-quotidien/220120/dq220120c-eng.htm [accessed 2023-03-26]
    45. Pisaniello MS, Asahina AT, Bacchi S, Wagner M, Perry SW, Wong ML, et al. Effect of medical student debt on mental health, academic performance and specialty choice: a systematic review. BMJ Open. 2019;9(7):e029980. [FREE Full text] [CrossRef] [Medline]
    46. Boyles JD, Ahmed B. Does student debt affect dental students' and dentists' stress levels? Br Dent J. 2017;223(8):601-606. [FREE Full text] [CrossRef] [Medline]
    47. Gurevich E, El Hassan B, El Morr C. Equity within AI systems: what can health leaders expect? Healthc Manage Forum. 2023;36(2):119-124. [FREE Full text] [CrossRef] [Medline]
    48. Kundi B, El Morr C, Gorman R, Dua E. Artificial intelligence and bias: a scoping review. In: Yampolskiy RV, editor. AI and Society: Tensions and Opportunities. New York. CRC Press Taylor & Francis; 2023;199-213.

    ‎
    AdaBoost: adaptive boosting
    AUC: area under the curve
    BAI: Beck Anxiety Inventory
    DT: decision tree
    KNN: k-nearest neighbor
    LR: logistic regression
    MCID: minimal clinically important difference
    ML: machine learning
    MVC: mindfulness virtual community
    PHQ-9: Patient Health Questionnaire-9
    PSS: Perceived Stress Scale
    RCT: randomized controlled trial
    RF: random forest
    SVM: support vector machine
    VC: virtual community

    Edited by T de Azevedo Cardoso; submitted 18.07.23; peer-reviewed by M Musker, D Carvalho; comments to author 24.11.23; revised version received 05.12.23; accepted 05.04.24; published 13.05.24.

    Copyright

    ©Christo El Morr, Farideh Tavangar, Farah Ahmad, Paul Ritvo, MVC Team. Originally published in the Interactive Journal of Medical Research (https://www.i-jmr.org/), 13.05.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.i-jmr.org/, as well as this copyright and license information must be included.