Original Paper
Abstract
Background: Personality disorders (PDs) are typically associated with higher mental health service use; however, individual patterns of engagement among patients with complex needs are poorly understood.
Objective: The study aimed to identify subgroups of individuals based on patterns of service receipt in secondary mental health services and examine how routinely collected information is associated with these subgroups.
Methods: A sample of 3941 patients diagnosed with a personality disorder and receiving care from secondary services in South London was identified using health care records covering an 11-year period from 2007 to 2018. Basic demographic information, service use, and treatment data were included in the analysis. Service use measures included the number of contacts with clinical teams and instances of did-not-attend.
Results: Using a large sample of 3941 patients with a diagnosis of PD, latent profile analysis identified 2 subgroups characterized by low and high service receipt, denoted as profile 1 (n=2879, 73.05%) and profile 2 (n=1062, 26.95%), respectively. A 2-profile solution (P<.01) was preferred over a 3-profile solution, which was nonsignificant. In unconditional (t3941,3939=19.53; P<.001; B=7.27; 95% CI 6.54-8) and conditional (t3941,3937=−3.31; P<.001; B=−74.94; 95% CI −119.34 to −30.56) models, cluster membership was significantly related to receipt of nursing contacts, over and above other team contacts.
Conclusions: These results suggest that routinely collected data may be used to classify likely engagement subtypes among patients with complex needs. The algorithm identified factors associated with service use and has the potential to inform clinical decision-making to improve treatment for individuals with complex needs.
doi:10.2196/55348
Keywords
Introduction
Personality disorders (PDs) are typically associated with poorer health outcomes [] and high mental health service use [], yet PD treatments remain poorly understood and are often not specific to the individual []. Understanding patterns of service use across the diagnostic spectra of PD comprises an important and novel area of research in the applied domain. There are also considerations of optimizing UK National Health Service (NHS) service capacity and improving relationships between services and service users, which are often challenging given the relational sequelae component within a PD diagnosis [].
The longstanding needs of patients with PDs require effective management and treatment approaches. Being able to understand likely patterns of service use could help allocate resources in secondary care, given limited resources. Predictive modeling for more personalized treatments is vanishingly rare in conventional health care models of service delivery [], yet there are growing calls in the methodology and policy literature for improvements in both treatments and relationships [] for patients with complex needs.
Effective prediction of treatment outcomes is an area of continued interest in general mental health care []. To our knowledge, this body of work has yet to be applied to PD treatments; however, there are promising opportunities offered by large-scale applied health data research, greater availability of relevant datasets [], and increasing sophistication of relevant computer software and analytical approaches. This paper expounds formative work in this sphere by conducting formal validation in understanding diagnostic patterns of service receipt, as a key step toward formal prediction and more tailored treatments.
Given the engagement difficulties that come with a PD diagnosis [], there is clinical interest in better understanding patterns of service use across the swathe of International Classification of Diseases, Tenth Revision (ICD-10) PD diagnoses [,]. Enabling services to predict likely patterns of service contact can help personalize the treatment pathway, given likely service demand, and optimize resource allocation. In turn, if treatment pathways can be tailored to the needs of service users at presentation, this may reduce the propensity for harmful treatments [], directly improve therapeutic relationships, indirectly reduce long-term service use [], and aid capacity management and resource allocation [].
Individuals with a diagnosis of PD are reported as misunderstood, at least in mental health care []. In addressing this impasse, this study provides mental health services with the necessary functionality to move away from uniform PD treatments and toward more personalized approaches []. Decision support studies in health care are gaining momentum [], and although the adoption of such systems has been slower in mental health care, the past decade has seen a rise in pilot implementations [,]. There is also a need for more advanced methodologies beyond “first-generation” regression-based approaches, which face increasing challenges and scrutiny in the age of data scientific [], such as mediation and structural equation models, both in PD management [] and in clinical psychology [,].
There has been growing interest in person-centered approaches [] driven by calls for methodological and analytical advancements [], efforts to improve access [,], initiatives to democratize treatments [], and a rising focus on the personalization of mental health care []. These, in turn, have resulted in developments in theory [] and modeling [,]. For example, latent class analysis (LCA) comprises an increasingly researched, novel approach, especially relevant in unpicking potential sources of observed heterogeneity [,]. This paper integrates these research perspectives using latent profile analysis (LPA). LPA is an extension of LCA, statistically described as finite Gaussian mixture models [].
Person-centered approaches, such as clustering methods, LCA, and LPA, provide a useful first step in dimension reduction [], argued as a key focus in reducing heterogeneity encountered in PD treatments, research, and theory []. In the context of psychiatric comorbidity, the number of potential co-occurring factors that make up disorders can challenge traditional variable-centered approaches, which may struggle to model all potential interactions []. In contrast, clustering approaches comprise an important first step in the aim to personalize treatments and treatment pathways, such as with physical medicine in cancer treatments []. In addition to these clinical and methodological motivations, the search for subgroups in complex mental health treatment is further motivated by relevant policy calls in the treatment literature [].
The aim of this study was to characterize the patterns of service receipt in a large sample of patients with a diagnosis of PD in an urban, secondary care setting. The research question considered how to best characterize service receipt, as part of the specialist PD treatment pathway, using routinely available treatment, diagnostic, and demographic data.
Methods
Overview
LPA was conducted to identify statistically distinct patterns of service contact within a sample of individuals with a diagnosis of PD receiving treatment in secondary care in South London. In addition to identifying the optimal class solution in this setting, formal validation aimed to inform generalization estimation, reporting both average-level and individual-level classification errors. This paper further extends LPA research through consideration and evaluation of the structural model and extends such evaluation through protocolized resampling.
Validation-focused analyses assessed the structural model [], with parameters tested for relevance when considering cluster membership. Out-of-sample approaches to prediction are well researched in wider structural equation modeling perspectives, for example, partial least squares structural equation modeling []. Importantly, while findings of mixture models may prove interesting, classes may not necessarily reside in the wider population [], further motivating explicit consideration of generalization in study design.
LPA guidelines suggest using 3 sampling frames as good practice to validate findings [,]. To begin, the initial whole sample was “sample frame 1,” comprising the final analytical set. Validation samples randomly split the whole sample into two equal groups of 3941, resulting in “sample frame 2” [,] and 2 validation samples (1970/3941, 49.98%). Finally, cross-validation requires a holdout procedure [], which partitions the dataset into test and training samples, to assess “out-of-sample” performance. This procedure split the whole sample into 5 parts, with 80% or k−4 parts designated as the training sample (3153/3941, 80%) and the remaining 20% or k−1 part designated as the test sample (788/3941, 20%). As shown in 
Figure 1, there were 3 sampling frames as a result.
In this study, posterior probabilities of cluster membership were identified using sample frames 1 and 2. The comparison between sampling frames (Tables S1 and S3 in ) describes the formal relevance of the model regarding validation. Posterior probabilities of cluster membership were calculated from the e-step of the expectation maximization algorithm [] and describe how the person i belongs to the class k [,], also described as their modal class assignment, according to the selected model [].
Setting
Patient data for the study were extracted from the Clinical Record Interactive Search platform at the South London and Maudsley NHS Foundation Trust. South London and Maudsley (SLaM) provides comprehensive, near-monopoly specialist mental health services to a geographic catchment area of 4 boroughs of southeast London (Croydon, Lambeth, Lewisham, and Southwark) with a population of approximately 1.3 million residents.
Ethical Considerations
The Clinical Records Interactive Search (CRIS) system was set up in 2007-2008 through codevelopment of a data security and governance model with service users []. The CRIS system and related linked data have received successive research ethics approvals since 2008 as an anonymized data resource for secondary analyses (current approval: Oxford Research Ethics Committee C, approval number 23/SC/0257). The institutionally approved governance model has been described in detail elsewhere [], but in summary, it includes automated deidentification of the record, standard researcher approval, a locally publicized opt-out facility, and an oversight committee with patient leadership and membership reviewing all data use.
Sample
The inclusion criteria comprised having an ICD-10 PD diagnosis [] at any time, being aged between 20 and 101 years at initial presentation, and being in contact with SLaM due to referral, although 10.38% (409/3941) of the sample had no recorded team contacts in the analysis set. Given the population of interest and generalization intention, those aged <20 years (318/3941, 8.07%) and >101 years (8/3941, 2.02%) were excluded to reduce possible inclusion bias and to encourage greater sample representativeness.
The inclusion criteria were applied over an exposure period of 11 years, with care episodes sampled between 2007 and 2018. This study sought to identify a matched, complete set of service use data at service users’ first and final points of contact with services (time 1 and time 2, respectively). Time points were denoted by episode start and end dates, such that time 1 represented the start and end of the first episodes of care and time 2 represented the same at the final episodes of care. Dataset assembly resulted in a complete, matched set of data points, ascribed by time 1 and time 2 and organized by episode of care. An episode of care was calculated as the elapsed time between patients being referred to and in contact with services and subsequent discharge from secondary services. In the final matched pairs set, the average length of an episode of care was 131.86 (SD 309.69) days. Moreover, “discharge” is defined by SLaM Trust policy as “discharging a patient from an in-patient setting to a community team or from the Trust completely to primary care or another provider” []. Importantly, this follows when a patient is deemed to no longer benefit from either reduced risk or improved well-being [] and suggests a shift in presentation and overall needs.
Working with the CRIS team, a custom data extract resulted in 82,265 episodes of care from 8510 service users. The inclusion of contacts from clinical teams data resulted in a reduced set of 6068 individuals, which was further processed to include complete “matched pairs” data for 3941 service users, at their first and final episodes of care (T1 and T2, respectively; Figure S1 in ). A complete dataset was preferred as there were no missing data, which strengthened the design using a large sample. T1 and T2 contained episode start and end dates, respectively, as discussed in the Statistical Analysis section. The authors acknowledge that the nature of this experimental design necessitated service users engaging with services on more than one occasion. Therefore, individuals with only a single episode of care (2127/3941, 53.97%) were omitted from the analysis. Notwithstanding, PD is typically a chronic condition [,], and in general, service users are likely to have ongoing needs and therefore present to services on more than one occasion.
Measurements
Quantitative Variables
Mixture modeling research typically uses cross-sectional data []. In the search for the best model, LPA compared model solutions using both repeated measures (T1 and T2) and T1 data only. Analytically, the episodic shaping of the data did not account for the timing or point of diagnosis, resulting in some patients being diagnosed after the initial T1 episode of care. Once again, however, given the longstanding nature of PD [] and difficulties in accessing effective treatments [], the precise timing of when a patient receives a diagnosis can be considered less important than it may initially appear.
Preliminary modeling [] identified negligible “loading” or lambda values when including more granular detail regarding ethnicity (Table S1 in ). For example, including ethnicity categories such as Asian (λ=0.25) and mixed ethnicity (λ=−0.23) in the demographic construct failed to meet statistical significance, and the strongest performing model was the one that only included the presence or absence of White ethnicity.
Demographic data were modeled to include ethnicity (White or non-White), neighborhood-level deprivation, age, and gender (proportion of female and other). Deprivation was measured using the Multiple Indices of Deprivation []. This measure comprises nationally recorded data and measures multiple deprivation at the Lower Super Output Area level (a standard administrative geography encompassing approximately 1500 residents), including a nominal variable of 10 categories, with 1 describing most deprived and 10 describing least deprived [].
Service receipt was modeled to include the total number of face-to-face contacts, as well as the number of unattended appointments, for each episode of care. Clinical input data were counted as the number of contacts from each treating team during each episode of care, that is, the cumulative total number of contacts within the specified time frame of a care episode. The number of contacts from care teams included medical, psychology, nursing, therapy, social work, and other. The CRIS system uses the General Architecture for Text Engineering [], a natural language processing engine that captures recorded diagnostic statements. This output was used to ascertain PD diagnoses (Table S2 in ) in addition to prestructured primary diagnosis fields, which provide all options from the ICD-10 [].
ICD-10 lists a range of PD diagnoses, denoted by category membership, in contrast to ICD-11, which characterizes PD by severity. Accumulating evidence indicates limited support for the foregoing polythetic approaches [,]. To aid meaningful comparisons, diagnoses were collated and assigned to relevant clusters used routinely in psychiatric care (Table S2 in Multimedia Appendix 4). Diagnostic and Statistical Manual cluster A diagnoses are characterized by “odd and eccentric” presentations, including paranoid, schizoid, and schizotypal PD diagnoses. These were distinguished from cluster B diagnoses, described as “overly dramatic, overly emotional, or unpredictable” presentations, as well as cluster C diagnoses, namely, “fearful or anxious” presentations.
As with ethnicity data, preliminary statistical modeling [] highlighted the negligible impact on model performance of including more specific diagnostic information (Table S1 in ), beyond the presence or absence of cluster A diagnoses alone. Specifically, this was exemplified by low lambda loading values (λ<0.40) when including cluster B (λ=0.25), cluster C (λ=−0.20), and cluster other (λ=−0.07) onto the overall construct of diagnosis (refer to Tables S1 and S2 in for data cleaning descriptions of ethnicity and diagnostic data, respectively). Following these inferences, the proportion of DSM A diagnoses were taken as the proportion of internalizing diagnoses [].
Statistical Methods
As formulated in the study by Nylund et al [] and discussed in the study by Masyn [], the null hypothesis in mixture modeling is the one-class model. Comparative likelihood ratio approaches, using the likelihood ratio test (LRT), describe and compare the likelihood of the two models under comparison, given the data []. In the first stage, different model solutions (namely, 2-, 3-, or 4-cluster solutions) were tested against this null sample. In the second stage, the number of groups, or “G,” was varied using mclust [], and relevant likelihood ratio statistics were then entered into tidyLPA []. Second-stage testing specified the null (ie, the 2-class solution) versus the alternative (ie, the 3-class solution) hypothesis. The LRT statistic then compares the likelihood of either model, given the data [].
Model evaluation then proceeds from the measurement model, in terms of number of classes, to the structural model, in terms of the relation of parameters to classes. Across the range of “stepwise” configurations, 1- and 3-step procedures are typical of the literature [,], with different approaches varying in their inclusivity and efficiency [,]. In practice, comparing the relationship between parameters and clusters can assist in model evaluation. These relationships describe the structural model of the mixture model, and structural model evaluation compares these relationships.
Discriminant analyses are popular in machine learning perspectives on mixture models [,]. Discriminant validity assesses the classification accuracy of a mixture model by comparing the performance of a testing subsample on a training subsample (Figure 1). In this study, discriminant analysis was developed using the eigenvalue decomposition discriminant analysis [,]. The eigenvalue decomposition discriminant analysis constrains the discriminant analysis model to have a single component for each class, with the same covariance structure among classes. More recent discriminant analysis models are available []; however, these models impose stricter requirements, such as a different number of mixture components and covariance structures for each class, which were deemed unsuitable for this analysis.
Discriminant analysis was conducted over 10 folds. For each “fold,” a separate dataset was assembled, comparing actual values versus those predicted by the model. As is typical in prediction studies, prediction error tests the predictive performance of the model on a holdout sample, with results summarized using common prediction metrics []. Typically, these include the root mean squared error, mean absolute error (MAE), and mean absolute percentage error (MAPE) [,]. In predictive applications, the MAE was selected as the metric of choice, given its greater resistance to distributional skewness []. To assist interpretation, both the MAE and MAPE are reported, with the MAPE calculated using the Metrics package []. The MAPE is calculated using the absolute error in each fold divided by the observed values that are evident for that fold. Averaging these fixed percentages yields the MAPE, which indicates how much error arises from the prediction model compared with the true value [].
LPA was chosen for its flexibility in data requirements and its suitability for modeling a heterogeneous population that cannot be effectively represented by a single-component distribution []. Indicator variables included demographic variables, number of face-to-face contacts, number of did-not-attends, and number of contacts with clinical teams, per episode of care. Data preparation was assisted using the tidyverse package []. Analyses were conducted using the packages mclust [] and tidyLPA [] using R statistical software (R Core Team) [,].
Statistical Analysis
Fitting the Measurement Model
The purpose of this analysis was to identify the patterns of service receipt in a large sample of patients with a diagnosis of PD in a secondary care setting, with the goal of enhancing contemporary models of care []. LPA scholarship was further extended through formal validation, as a precursor to prediction, which used protocolized resampling to ascertain the discriminatory relevance and possible scale potential of the proposed model solution.
To establish the best-fitting model, a range of metrics were considered. These included the Vuong-Lo-Mendall-Rubin LRT [] and the bootstrapped LRT [,]. Alongside likelihood ratio tests, fit statistics included the Akaike information criterion, the Bayesian information criterion (BIC), and values of entropy. Both the LRTs, the Vuong-Lo-Mendall-Rubin LRT and bootstrapped LRT compare the K model (the current model with a number of k profiles) to K−1 model (a model with one less profile). Significant P values from these tests assert that the K model fits the data better than a comparable model with one less profile []. Conversely, a nonsignificant P value (P≤.05) indicates that the model with one less profile provides a better fit for the data, with more parsimonious models preferred. Smaller Akaike information criterion and BIC values indicate better model fit, while higher values of entropy suggest higher accuracy in classification of the model.
As no prior hypotheses for the number of profile groups were suggested, model selection compared solutions derived from either a repeated measures design, consisting of T1 and T2 data, or a contemporaneous design, consisting of T1 data only. Episodes of care were defined across two levels: (1) at the patient level, including demographic and diagnostic information, and (2) at the episode level, including “start” and “end” date of episodes, whereby service receipt measures were collected as properties, or descriptors, of that episode.
Episode descriptors were used as indicators for the LPA, including sociodemographic measures (age, ethnicity, and gender); diagnostic factors (percentage of internalizing diagnoses); general measures of contact (the number of face-to-face contacts and the number of did-not-attends); and the number of contacts from specific clinical teams (medical, nurse, psychology, therapy, and other).
Fitting the Structural Model
Assessment of the structural model considered the explanatory relevance of parameter values on cluster membership. Exogenous, pretreatment variables (ie, demographics) were limited to 1-step evaluation procedures. For service contact parameters, inferential tests were developed for both 1- and 3-step procedures []. Three-step procedures assert the importance of conditional probability of class membership in post hoc testing. Corrected by assignment probability [], the 3-step procedure helps control for the individual validity of profiles. In contrast, the 1- and 2-step approaches can often omit the nonzero classification accuracy of derived clusters []. However, in the case of models with high classification accuracy, 1-step procedures may be justified [,]. As such, this paper reports both 1- and 3-step procedures, comparing unconditional versus conditional structural mixture models.
For the adapted 3-step procedure, the first step identified and enumerated cluster solutions, which were compared using model fit statistics. In the second step, conditional probabilities were extracted from the e-step of the EM algorithm using mclust [] and vectorized “row wise” at the level of each individual. For the third step, individual conditional probabilities were included as covariates (ie, general linear models mapped the interaction between parameters with conditional probabilities). General linear models compared cluster assignment as a function of each parameter. Results of these analyses are summarized in 
Figure 2 and presented in 
Table 1.
| t test (df) | Direct effect, 95% CI | t test (df) | Controlled effect (95% CI) | ||
| Engagement | |||||
| Did-not-attend | 19.05 (3941,3939) | 4.30 (3.85 to 4.74) | 0.05 (3941,3937) | 2 (4 to 10) | |
| Face to face | 28.70 (3941,3939) | 31.33 (29.19 to 33.47) | −1.04 (3941,3937) | −68.86 (−199.13 to 61.41) | |
| Clinical contacts | |||||
| Medical | 23 (3941,3939) | 4.91 (4.49 to 5.33) | −1.30 (3941,3937) | −16.89 (−42.38 to 8.60) | |
| Nurse | 19.53 (3941,3939) | 7.27 (6.54 to 8) | −3.31a (3941,3937) | −74.94 (−119.34 to −30.56) | |
| Psychology | 23.83 (3941,3939) | 5.71 (5.24 to 6.18) | 1.72 (3941,3937) | 25.01 (−3.57 to 53.59) | |
| Therapy | 15.37 (3941,3939) | 3.21 (2.80 to 3.61) | 1.07 (3941,3937) | 13.57 (−11.35 to 38.49) | |
| Other | 12.64 (3941,3939) | 4.84 (4.09 to 5.59) | 0.45 (3941,3937) | 10.60 (−35.14 to 56.34) | |
aP<.001.
Latent Profile Cross-Validation Strategy
Cross-validation uses the data to test “the predictive power” of the model []. For the procedure, cross-validation splits the dataset into k subsamples of equal size, repeating k times, each time using a different test set (Figure 1). The best-fitting parameters of the model are then fit using k−1 subsamples, with the model’s error then computed on the unused subsample or test set [].
In line with the discriminant analysis procedure, cross-validation was conducted using sample frames 1 and 2 (in sample and out of sample, respectively), as shown in Figure 1. The Brier score [,] indicates the predictive usefulness of the model for individuals who, theoretically, may be unseen. Applied research in mental health has shown increased experimentation with the Brier score as a forecasting tool, such as for predicting suicidal ideation in individuals at near-term suicide risk [] or identifying patients likely to drop out of psychotherapy []. In related psychotherapy research, the Brier score ranges from 0 (best prediction) to 1 (worst prediction), and it measures the accuracy of probabilistic predictions [].
A detailed sampling strategy enabled both explanatory and predictive appraisal of the derived model solution, as shown in Figure 1. As well as the whole sample (frame 1), validation samples (frame 2) divided the dataset randomly, 2 × 50%. LPA was conducted on both frames 1 and 2 (total sample and validation samples, respectively), as shown in Figure 1, informing sample comparisons and confirming the profile structure []. Subsequently, LPA was rerun on the whole sample to aid interpretation of the findings.
Sample frame 3 for discriminant analyses split the total sample into 5 parts, with 80% (3153/3941) or k−4 parts designated as the training sample and the remaining 20% (788/3941) or k−1 part designated as the test sample. The holdout procedure was repeated over 10 folds, in line with best practices []. Discriminant analyses tests were then applied over sampling frames 1 and 3 (Figure 1), with cross-validation describing classification accuracy for both total and holdout sampling frames using the classification error and Brier scores. As discussed earlier, the Brier score is a proper score function that measures the accuracy of probabilistic predictions []. More technically, the Brier score comprises the mean square difference between the true classes and predicted probabilities [,].
Latent Profile Resampling Testing Strategy
Discussions on prediction-focused structural equation modeling consider discriminant validity in terms of in-sample versus out-of-sample perspectives []. Bootstrapping has gained prominence in sample validation; however, this remains a decidedly in-sample approach. In this study, larger sample sizes permitted a combination of k-fold assessments of discriminant validity, combined with cross-validation modeling, which ascertain both in-sample and out-of-sample classification accuracy.
LPA was then conducted on the “T1 and T2” dataset, while a holdout procedure (Figure 1) tested the predictive relevance of the profile solution across 10 partitions, or “folds,” of this procedure. Predictive performance was summarized by prediction metrics, namely, the MAE and MAPE [,]. The MAPE was calculated using the absolute error in each fold divided by the observed values that are evident for that fold. Averaging these fixed percentages yields the MAPE, which indicates how much error arises from the prediction model compared with the true value [].
Results
This study broadly follows the reporting guidelines from the Consolidated Standards of Reporting Trials (CONSORT) [], although it specifically adheres to the Strengthening the Reporting of Observational Studies in Epidemiology [] and Reporting of Studies Conducted Using Observational Routinely-Collected Data [] guidelines, which were more suited for improving the reporting of observational epidemiological research.
Participants
The flow of participants throughout the study is shown in Figure S1 in . In brief, the initial analytical set comprised 82,265 episodes and 8510 patients. Data were prepared to include team contacts and ensure completeness, resulting in 170,365 episodes and 6421 patients. Further processing resulted in the final analytical set of 7882 episodes from 3941 patients to optimize the analysis, with this complete dataset having no missing values.
Descriptive Statistics
Tables 2 and 3 display the distribution of patient characteristics from the LPA and across sampling frames. Descriptive statistics for holdout samples were limited to the initial fold (of 10). Comparing latent profiles to the whole sample means and distribution provides an understanding of the characteristics of each group of patients across identified latent profiles (LPs)—LP1 and LP2—in the 2-profile solution. Service receipt, measured as the number of contacts from clinical teams, was markedly higher for LP2 than for LP1. The LPA best characterized service receipt as being either high or low, captured through LP2 and LP1, respectively. While LP1 counted no psychology contacts, LP2 on average had almost 6 per episode of care. This pattern was consistent for medical teams and, in particular, nursing teams.
In general, LP2 received treatment at a rate 10 times higher than LP1. The derived profiles were not differentiated by sociodemographic characteristics; both latent profiles had similar proportions of female and White individuals as well as comparable levels of neighborhood deprivation. In total, 11.3% (120/1062) of LP2 had internalizing diagnoses, with no such diagnoses present in LP1.
| Sample frame 1 (N=3941) | LP1a (n=2879; 73.05%) | LP2 (n=1062; 26.95%) | Sample frame 2 | Sample frame 3 (holdout) | |||||||
| Validation sample a (n=1970) | Validation sample b (n=1970) | Train (n=3153) | Test (n=788) | ||||||||
| Demographics | |||||||||||
| Age (y), mean (SD) | 44.33 (14.30) | 44.10 (14.1) | 44.95 (14.90) | 43.90 (14.23) | 44.38 (14.29) | 44.49 (14.26) | 43.68 (14.41) | ||||
| Gender(n=2482, 63%) | |||||||||||
| Female, mean (SD) | 2482.8 (1891.7) | 1805.1 (1381.9) | 660.6 (1062) | 1221.4 (965.3) | 1241.1 (945.6) | 1986.4 (1513.4) | 472.8 (386.1) | ||||
| Deprivation | |||||||||||
| IMDb, mean (SD) | 3.68 (1.73) | 3.66 (1.75) | 3.70 (1.67) | 3.74 (1.76) | 3.64 (1.65) | 3.66 (1.69) | 3.74 (1.87) | ||||
| Ethnicity (n=1261, 32%) | |||||||||||
| White, mean (SD) | 1261.1 (1852.3) | 878.1 (1324.3) | 386.6 (509.8) | 650.1 (925.9) | 650.1 (925.9) | 1008.9 (1481.9) | 252.16 (370.4) | ||||
| Diagnostic data | |||||||||||
| Internalizing (y), mean (SD) | 118.23 (669.9) | 0 (0) | 120 (339.8) | 59.1 (315.2) | 59.1 (354.6) | 94.59 (536) | 23.64 (126.08) | ||||
aLP: latent profile.
bIMD: indices of multiple deprivation.
| Sample frame 1 (N=3941) | LP1 (n=2879; 73.05%) | LP2 (n=1062; 26.95%) | Sample frame 2 | Sample frame 3 (holdout) | ||||
| Validation sample a (n=1970) | Validation sample b (n=1970) | Train (n=3153) | Test (n=788) | |||||
| Engagement, mean (SD) | ||||||||
| F2Fa | 12.88 (33.44) | 4.43 (5.17) | 35.78 (58) | 12.06 (30.25) | 13.03 (33.93) | 12.93 (33.49) | 12.65 (33.24) | |
| Did-not-attend | 1.59 (6.57) | 0.43 (1.03) | 4.73 (12) | 1.57 (6.51) | 1.59 (6.79) | 1.60 (6.62) | 1.55 (6.35) | |
| Clinical team input, mean (SD) | ||||||||
| Medical | 2.32 (6.33) | 1 (1.50) | 5.90 (11.2) | 2.20 (6.13) | 2.33 (6.78) | 2.32 (6.17) | 2.34 (6.94) | |
| Nurse | 4.11 (10.86) | 2.15 (3.36) | 9.43 (19.20) | 3.98 (10.23) | 4.14 (10.13) | 4.13 (11.41) | 4.05 (8.29) | |
| Psychology | 1.54 (7.13) | 0 (0) | 5.70 (12.90) | 1.49 (7.01) | 1.46 (6.76) | 1.64 (7.57) | 1.15 (4.98) | |
| Therapy | 0.91 (5.98) | .05 (0.21) | 3.25 (11.20) | 0.74 (3.79) | 0.94 (6.52) | 0.89 (5.36) | 0.99 (8.01) | |
| Other | 1.56 (10.88) | 0.25 (0.72) | 5.10 (20.5) | 1.23 (9.27) | 3.64 (1.65) | 1.55 (11.09) | 1.60 (9.99) | |
aF2F: face to face.
Evaluation of the Measurement Model
Analyses identified 2 profiles emerging from the contemporaneous (T1) dataset and 4 profiles from the repeated measures dataset (T1 and T2; Table 4). Profiles were enumerated according to the BIC using mclust [] for T1 versus T1/T2 dataset, selecting 2 and 4 profiles, respectively. The comparison of classification and likelihood indices shown in Table 4 suggested that the 2-profile solution from T1 data was preferred, showing superior model fit. Stage-1 LRT tests compared 2 and 3 class solutions against the null hypothesis, that is, the 1-class model. Both solutions demonstrated superior likelihood than the null hypothesis; however, the 2-profile solution was preferred. Stage-2 LRT tests compared 3-profile (the alternative) versus 2-profile (the null) solutions and failed to demonstrate any advantage of a 3-profile solution over a 2-profile solution.
| Fit values | 1-cluster | 2-cluster | 3-cluster | 4-cluster | |
| Log-likelihood criteria | |||||
| Log likelihood | −119,761 | −72,497.47 | −81,918.54 | −118,721.9 | |
| BICb | −240,267.2 | −146,402.4 | −164,267.6 | −245,317.3 | |
| Classification criteria | |||||
| Entropy | —c | 13.14 | 90 | 29 | |
| Normalized entropy difference | — | 0.034 | 0.025 | 0.0052 | |
| LRTsd | 1 versus 2 | 2 versus 4 | 3 versus 1 | 3 versus 2 | |
| VLMR-LRTe | 90,868.55f | −90,624.52 | 92,661.71f | nsg | |
| B-LRTh | 94,527.1i | — | — | — | |
aMultivariate indicators: multivariate skewness=1425.64 (statistic 936,405.6); multivariate kurtosis=2840.37 (statistic 4576.15). For a formal mathematical definition, see Figure S2 in .
bBIC: Bayesian information criterion.
cNot applicable.
dLRT: likelihood ratio test.
eVLMR-LRT: Ven-Lo-Mendall-Rubin likelihood ratio test.
fP<.001.
gns: nonsignificant.
hB-LRT: bootstrapped likelihood ratio test.
iP<.01.
In model selection, comparative LRTs described how a 3-cluster solution significantly outperformed the 1-class model but not the 2-class model (Table 4); therefore, a 2-cluster solution was preferred.
Posterior probabilities were calculated as the proportion of correctly assigned patients to the respective clusters. These probabilities are “posterior,” in the sense that they are calculated after individuals have been assigned a cluster membership []. Table S2 in displays classification indicators and posterior probabilities, and the classification error of the model was also negligible, being near 0. The Brier scores further demonstrate near-perfect classification accuracy of the model in forecasting individuals’ likely engagement subtype, given the values of routinely collected indicators. High posterior probabilities are also considered clear indicators of classification accuracy, with values ideally exceeding 0.80, in line with established guidelines [].
Evaluation of the Structural Model
Findings from evaluation of the measurement model, taken with the near-perfect degree of classification accuracy (Table S3 in ), suggest the usefulness of combined parameters in predicting cluster membership.
Relating parameter values to clusters is expressed in the language of direct and controlled direct effects []. The direct effect describes the (unconditional) relationship between parameters and clusters, while the controlled direct effect describes this relationship based on the individual probabilities of cluster assignment. In the unconditional model, all parameters determined class assignment, apart from gender (Table 5). Figure 2 displays conditional model evaluation, where face-to-face contacts (t3940,3939=5.73; P<.001; B=0.006; 95% CI 0.005-0.006) and the number of nursing contacts (t3940,3939=5.52; P<.001; B=0.0005; 95% CI −0.003 to 0.0006) significantly determined class assignment.
Likelihood-based approaches tend to dominate the mixture modeling landscape; however, there can be inferential challenges when adhering to the orthodoxy [,]. A more Bayesian approach may consider the likely parameter value, given the cluster membership. Structural model evaluation using this approach followed the same 3-step procedure as advised by Nylund-Gibson et al [] and described in the section Fitting the Structural Model. Results from these analyses are presented in Table 1 and further detailed in 
Figure 3.
In the unconditional model and of all care contacts, psychology demonstrated the strongest relationship to clusters (t3940,3937=23.83; P<.001; B=0.02; 95% CI 0.02-0.02); however, this was not upheld in the conditional model (t3940,3937=1.72; P=.09). Overall, key prognostic targets of service receipt given the knowledge of cluster membership were nurse contacts. In both unconditional (t3940=19.53; P<.001; B=7.27; 95% CI 6.54-8) and conditional (t3939=5.52; P<.001; B=−74.94; 95% CI −119.34 to −30.56) models, cluster membership was significantly related to the receipt of nursing contacts.
Further testing modeled an interaction term between nursing contacts and cluster membership in determining nonattendance, as shown in Figure 3. The interaction term was significant both for face-to-face contacts (t3940,3937=5.81; P<.001; B=0.74; 95% CI 0.49-0.99) and did-not-attends (t3940,3937=3.62; P<.001; B=0.12; 95% CI 0.06-0.20), with confirmatory testing indicating that the rate of did-not-attends did not significantly differ from a Poisson distribution (N=3941, χ28=10.3; P=.20).
| t test (df) | Direct effect, 95% CI | t test (df=3939) | Controlled effect, 95% CI | |||
| Engagement | ||||||
| Did-not-attend | 2.93 (3940) | 0.004 (0.001 to 0.006) | 1.53 | 0.0004 (−0.0001 to 9.86) | ||
| Face to face | 20.76 (3940) | 0.005 (0.004 to 0.006) | 5.73a | 0.006 (0.005 to 0.006) | ||
| Clinical contacts | ||||||
| Medical | 23 (3940) | 0.024 (0.022 to 0.026) | 1.38 | 0.0003 (−0.0001 to 0.0007) | ||
| Nurse | 19.53 (3940) | 0.01 (0.01 to 0.01) | 5.52a | 0.0005 (−0.003 to 0.0006) | ||
| Psychology | 23.83 (3940) | 0.02 (0.02 to 0.02) | 1.72 | 0.0003 (−0.001 to 0.003) | ||
| Therapy | 15.37 (3940) | 0.02 (0.02 to 0.02) | 0.15 | 0.0002 (−0.002 to 0.003) | ||
| Other: sociodemographics | ||||||
| Internalizing diagnoses | 19.07 (3940) | 0.75 (0.68 to 0.81) | —b | — | ||
| Ethnicity (White) | 2.97 (3940) | 0.04 (0.01 to 0.06) | — | — | ||
| Gender (female) | 1.16 (3940) | 0.001 (−0.01 to 0.04) | — | — | ||
aP<.001.
bNot applicable.
Figure 3 shows this relationship visually and indicates how did-not-attend rates vary between clusters as a function of nursing contacts. The assignment of profiles is shown probabilistically between either latent profile 1 and latent profile 2, which allows for assignment likelihood or uncertainty in clinical practice. For LP2, the number of nursing contacts made less of a difference in fitted rates of did-not-attends, compared to LP1. As it represented the majority of the sample, LP1 was more sensitive to the effect of nursing contacts regarding fitted rates of attended appointments.
Resampling Performance
Discriminant analysis modeling indicated high classification accuracy of the selected model (Table S3 in ). Both the MAE and MAPE were taken as summative indicators regarding the overall prediction error of the model. Over 10 folds of the holdout procedure, or train × test runs, the “grand” MAE was 0.004 (Table S3 in ), meaning that the model predicted correct class assignment with near-perfect accuracy. Conversely, out-of-sample tests on the replicative utility of engagement measures were less favorable. For face-to-face contacts, the MAE was 11.27, whereas for the number of did-not-attends, the MAE was 1.97. The MAPE may aid interpretation of these results, specifying that the number of face-to-face contacts erroneously predicts cluster membership in 11% (86/788) of cases (of 10 “folds”; Table S2 in ).
In the explanation-focused in-sample approach, the Brier score was 0.005. From an out-of-sample perspective, the average Brier score was 0.19 (Table S1 in ). The average classification error over 10 folds was 0.005, indicating substantially high classification accuracy. In addition, the average (out-of-sample) Brier score was 0.19 (SE 0.005), suggesting a nearly 80% rate of forecasted classification accuracy from an out-of-sample perspective.
Discussion
Summary of Findings
This study aimed to characterize the patterns of service receipt in a large sample of patients with a diagnosis of PD in an urban, secondary care setting. The main findings of this study were identification of 2 LPs that best characterize service receipt in specialist PD treatments. LP2, at 26% (1062/3941) of the sample, received almost 5-fold as many nursing contacts as LP1 (9.43 vs 3.98) and 6-fold as many medical contacts (5.90 vs 1). In addition, general service use was almost 10 times greater in LP2 than LP1. Sociodemographic characteristics were not associated with LPs, but diagnostic factors were associated. Specifically, LP1 had no internalizing diagnoses. These findings suggest important distinctions of service receipt as a function of routinely collected data and suggest the discriminatory relevance of such descriptors in determining likely profile membership.
LPA identified statistically reliable categories of service use using routine observational data for those with a diagnosis of PD. Profiles distinguish service receipt for those with a diagnosis of PD and were replicated across a range of sampling frames. Discriminant analyses and cross-validation modeling suggest substantial discriminatory performance in secondary settings using these types of data. This study described 2 LPs, primarily distinguished by diagnostic information. Of the approximately 3000 patients in LP1 (n=2879, 73.05%), none had internalizing PD diagnoses. In inferential testing, patients in LP1 were twice as likely to have less service contact.
Interpretation
The literature discussing PD and service use describes greater service use in the general presence of PD [,]; however, this has yet to be robustly applied between different PD diagnoses. One study delineates the interplay between personality (disorder) and health [], and there is rigorous work, such as comparisons of service use based on ethnicity in PD treatments []. Despite economies of scale offered by population-based interventions, these programs should be customized for subgroups on the basis of personality []. This study details foundational work in this area using a large sample in secondary care and attempts to export this comparative focus to diagnostic factors within PD treatments.
In a large study using administrative data in Quebec, emergency department visits were compared between “externalizing” or Diagnostic and Statistical Manual cluster B PD and schizophrenia, those diagnosed with both conditions, and the general population []. General practitioner visits were highest in the cluster B group, while the use of psychiatric (vs nonpsychiatric specialist or generalist) services differed in the presence of a PD diagnosis, with 1 mean annual visit versus 4, for schizophrenia.
Similar findings were captured in this study, which demonstrates lower service use for noninternalizing diagnostic groups. Greater service use was shown for the smaller latent profile, LP2. In practice, greater patterns of service use from the minority group arguably have predictive relevance in profile determination and tailoring treatment pathways accordingly. “Hotspotting,” as it is known in general practice [], is a data-driven practice that aims to identify patterns of “extreme” use []. Complex needs and PD populations are similarly characterized by high use []. This responsive concept is well known in general practice but has yet to be specifically applied to mental health care. In the case of PD and complex needs, general practice is often a key contact []. As demonstrated in this study, understanding individual (stratified) patterns of service use offers an example for expanding hotspotting practices, commonly used by general practitioners, to address historical challenges in generalist service provision for complex needs. By applying advanced techniques such as LPA, this approach can help create more tailored treatment pathways.
Therefore, this study argues that more data-enabled, proactive service provision, engendered by hotspotting practices, may assist modernization of mental health provision and encourage more individually intelligible and tailored treatment pathways. The assertion is that the determination of cluster membership, given the routinely measured parameters, may help inform similar practices in secondary care with complex needs. Related work in improving access to psychological therapies highlights 8 LPs [], which contrasts with the solution presented in this study.
Interestingly, the rates of nonattendance were stable across profiles, averaging around 1 in 10, which is broadly in line with the literature in general practice []. Service use was poorly distinguished as a function of the treating team. Structural evaluation revealed nursing contacts and did-not-attends of key interest; modeling the interaction term showed that the fitted rate of did-not-attends differs between clusters as a function of nursing contacts. The implication is that more intensive treatment pathways with more nursing contacts result in greater fitted nonattendance for LP1. However, for LP2, greater nursing contacts had much less of an effect on did-not-attends.
The use of a large sample, the multiplicity of analyses, and the development of a protocolized resampling scheme for validation comprise key strengths of this study. Discriminant validity and classification accuracy tests showed near-perfect classification for individuals with a diagnosis of PD into LPs (Table S2 in ). Generalizability was explicitly considered pursuant to resampling and validation, which may have implications regarding wider (national) policy implementation. From an out-of-sample perspective, for example, the model accurately predicted cluster assignment in nearly 80% of cases, across 10 “folds” of the holdout procedure, or train × test runs, compared with a Brier score of 0.005 for in-sample model performance (Table S1 in ). At least, for explanatory purposes, the model was able to classify all cases correctly.
Limitations
A limitation of this study is that both exposure and outcome were measured over the same periods. This contemporaneous design was indicated in model selection; however, more frequent measurement may have resulted in more precise model solutions. Comparative modeling between both T1 and T1/T2 data suggested that a 2-cluster solution using T1 measurement showed superior model performance. In addition, the analytical design only included those who had more than 1 episode of care, which may exclude more sporadic or crisis presentations. Covariates were also drawn over time, rather than at presentation, rendering this study more suitable for validation than for strict prediction.
In addition, clinical outcomes were not assessed in this study, although there is scope to include these in future research. Finally, the direction of causality remains unclear—specifically, whether service receipt determines latent profile, or vice versa—along with the relationship between service receipt, profile, and clinical outcomes.
Conclusions
Truly “predictive” models of care are currently uncommon in mental health []. However, this study plays an important role in moving toward more predictive models of care in the medium term, following the formal validation of LPs and deployment of a rigorous resampling procedure in testing classification uncertainty at both average and individual levels. Through the identification and description of LPs, services are envisioned to be equipped to include such information in more personalized treatment approaches. More reflexively, such approaches can be seen as facilitating preventative models of care rather than reactive ones [,], representing a radical and, if not timely, necessary shift in care delivery.
Acknowledgments
The first author would like to thank Dr William Marsh for their assistance in the development of the required code to sample episodes of care as well as their supervisory assistance for work related to this study. In addition, JF would like to thank MF for their indispensable assistance and supervision throughout this study and for highlighting the funding opportunity. Finally, R Saunders helped inspire the conceptualization of this research, offering invaluable methodological and theoretical assistance at key periods, for which the authors are profoundly grateful. The authors would also like to thank the dedicated editors and peer reviewers from JMIR Interactive Journal of Medical Research for their insightful comments and attention to detail, which helped this study fulfill its potential. This report is an independent research funded by the National Institute for Health and Care Research Applied Research Collaboration North Thames. This paper represents an independent research partly funded by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley National Health Service (NHS) Foundation Trust and King’s College London. The views expressed are those of the authors and not necessarily those of the NHS, NIHR, or Department of Health and Social Care. This project was made possible through additional NIHR funding through the NIHR Short Placement for Applied Research Collaboration award.
R Stewart is partly funded by (1) the NIHR Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London; (2) the NIHR Applied Research Collaboration South London at King’s College Hospital NHS Foundation Trust; (3) the Datamind Health Data Research United Kingdom Mental Health Data Hub (Medical Research Council grant MR/W014386); and (4) the United Kingdom Prevention Research Partnership (Violence, Health and Society; MR-VO49879/1), an initiative funded by United Kingdom Research and Innovation Councils, the Department of Health and Social Care (England) and the United Kingdom–devolved administrations, and leading health research charities.
Data Availability
The datasets generated or analyzed during this study are not publicly available.
Authors' Contributions
The project was developed by JS with assistance from R Stewart and supervisory support from MF. Data were curated by R Stewart. JS analyzed the data and drafted the paper as first author. Formal analyses were conducted by JS. R Saunders assisted in conceptualization at key points of designing the methods and analysis. Resources were provided by R Stewart. All authors reviewed the manuscript, which was drafted by the first author, JS.
Conflicts of Interest
R Stewart received research support from GSK and Takeda in the last 3 years prior to manuscript submission. All other authors declare no conflicts of interest.
References
- Moran P, Romaniuk H, Coffey C, Chanen A, Degenhardt L, Borschmann R, et al. The influence of personality disorder on the future mental health and social adjustment of young adults: a population-based, longitudinal cohort study. Lancet Psychiatry. Jul 2016;3(7):636-645. [FREE Full text] [CrossRef] [Medline]
- Coid J, Yang M, Tyrer P, Roberts A, Ullrich S. Prevalence and correlates of personality disorder in Great Britain. Br J Psychiatry. May 2006;188:423-431. [CrossRef] [Medline]
- Bach B, Simonsen S. How does level of personality functioning inform clinical management and treatment? Implications for ICD-11 classification of personality disorder severity. Curr Opin Psychiatry. Jan 2021;34(1):54-63. [CrossRef] [Medline]
- Barnicot K, Redknap C, Coath F, Hommel J, Couldrey L, Crawford M. Patient experiences of therapy for borderline personality disorder: commonalities and differences between dialectical behaviour therapy and mentalization-based therapy and relation to outcomes. Psychol Psychother. Mar 2022;95(1):212-233. [CrossRef] [Medline]
- Deisenhofer AK, Barkham M, Beierl ET, Schwartz B, Aafjes-van Doorn K, Beevers CG, et al. Implementing precision methods in personalizing psychological therapies: barriers and possible ways forward. Behav Res Ther. Jan 2024;172:104443. [FREE Full text] [CrossRef] [Medline]
- Delamain H, Buckman JE, O'Driscoll C, Suh JW, Stott J, Singh S, et al. Predicting post-treatment symptom severity for adults receiving psychological therapy in routine care for generalised anxiety disorder: a machine learning approach. Psychiatry Res. Jun 2024;336:115910. [FREE Full text] [CrossRef] [Medline]
- Stewart R. The big case register. Acta Psychiatr Scand. Aug 2014;130(2):83-86. [CrossRef] [Medline]
- Crawford MJ, Price K, Gordon F, Josson M, Taylor B, Bateman A, et al. Engagement and retention in specialist services for people with personality disorder. Acta Psychiatr Scand. Apr 2009;119(4):304-311. [CrossRef] [Medline]
- Hansen SJ, Christensen S, Kongerslev MT, First MB, Widiger TA, Simonsen E, et al. Mental health professionals' perceived clinical utility of the ICD-10 vs. ICD-11 classification of personality disorders. Personal Ment Health. May 2019;13(2):84-95. [CrossRef] [Medline]
- Siefert CJ, Porcerelli J, Meehan KB, Dauphin B. Clinical utility of proposed dimensional personality disorder models: considerations and opportunities. In: Huprich SK, editor. Personality Disorders and Pathology: Integrating Clinical Assessment and Practice in the DSM-5 and ICD-11 Era. New York, NY. American Psychological Association; 2022:53-73.
- Bach B, Kramer U, Doering S, di Giacomo E, Hutsebaut J, Kaera A, et al. The ICD-11 classification of personality disorders: a European perspective on challenges and opportunities. Borderline Personal Disord Emot Dysregul. Apr 01, 2022;9(1):12-63. [FREE Full text] [CrossRef] [Medline]
- Campbell K, Lakeman R. Borderline personality disorder: a case for the right treatment, at the right dose, at the right time. Issues Ment Health Nurs. Jun 2021;42(6):608-613. [CrossRef] [Medline]
- Lau P, Amestoy ME, Roth M, Monson C. Patient-related factors associated with patient retention and non-completion in psychosocial treatment of borderline personality disorder: a systematic review. Personal Ment Health. Nov 2024;18(4):300-322. [CrossRef] [Medline]
- Sampson M, Berry C, Jerry A, Gray C. Models of SCM implementation— United Kingdom. In: Mitchell S, Sampson M, Bateman A, editors. Structured Clinical Management (SCM) for Personality Disorder: An Implementation Guide. Oxford, UK. Oxford University Press; 2022:87-112.
- Lamph G, Baker J, Dickinson T, Lovell K. Personality disorder co-morbidity in primary care 'Improving Access to Psychological Therapy' (IAPT) services: a qualitative study exploring patient perspectives on treatment experience. Behav Cogn Psychother. Mar 2021;49(2):144-158. [FREE Full text] [CrossRef] [Medline]
- Fok ML, Stewart R, Hayes RD, Moran P. The impact of co-morbid personality disorder on use of psychiatric services and involuntary hospitalization in people with severe mental illness. Soc Psychiatry Psychiatr Epidemiol. Oct 2014;49(10):1631-1640. [FREE Full text] [CrossRef] [Medline]
- Lutz W, Deisenhofer AK, Rubel J, Bennemann B, Giesemann J, Poster K, et al. Prospective evaluation of a clinical decision support system in psychological therapy. J Consult Clin Psychol. Jan 2022;90(1):90-106. [CrossRef] [Medline]
- Bonfils KA, Dreison KC, Luther L, Fukui S, Dempsey AE, Rapp CA, et al. Implementing CommonGround in a community mental health center: lessons in a computerized decision support system. Psychiatr Rehabil J. Sep 2018;41(3):216-223. [FREE Full text] [CrossRef] [Medline]
- Dawoodbhoy FM, Delaney J, Cecula P, Yu J, Peacock I, Tan J, et al. AI in patient flow: applications of artificial intelligence to improve patient flow in NHS acute mental health inpatient units. Heliyon. May 2021;7(5):e06993. [FREE Full text] [CrossRef] [Medline]
- Rose S, McGuire TG. Limitations of P-values and R-squared for stepwise regression building: a fairness demonstration in health policy risk adjustment. Am Stat. 2019;73(Suppl 1):152-156. [FREE Full text] [CrossRef] [Medline]
- Mathlin G, Freestone M, Taylor C, Shaw J. Offenders with personality disorder who fail to progress: a case-control study using partial least squares structural equation modeling path analysis. JMIRx Med. Oct 29, 2021;2(4):e27907. [FREE Full text] [CrossRef] [Medline]
- Ebrahimi A, Poursharifi H, Dolatshahi B, Rezaee O, Hassanabadi HR, Naeem F. The cognitive model of negative symptoms in schizophrenia: a hierarchical component model with PLS-SEM. Front Psychiatry. 2021;12:707291. [FREE Full text] [CrossRef] [Medline]
- Ostovar S, Bagheri R, Griffiths MD, Mohd Hashima IH. Internet addiction and maladaptive schemas: the potential role of disconnection/rejection and impaired autonomy/performance. Clin Psychol Psychother. Nov 23, 2021;28(6):1509-1524. [CrossRef] [Medline]
- Saunders R, Buckman JE, Pilling S. Latent variable mixture modelling and individual treatment prediction. Behav Res Ther. Jan 2020;124:103505. [FREE Full text] [CrossRef] [Medline]
- Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p < 0.05”. Am Stat. Mar 20, 2019;73(sup1):1-19. [CrossRef]
- Amiot O, Sauvaget A, Alamome I, Bulteau S, Charpeaud T, Clair A, et al. Prospective acceptability of digital therapy for major depressive disorder in France: multicentric real-life study. JMIR Form Res. May 20, 2024;8:e53204. [FREE Full text] [CrossRef] [Medline]
- Friesen L, Gaine G, Klaver E, Klingle K, Parmar D, Hrabok M, et al. Bridging the gap in community care for patients with borderline personality disorder: protocol for qualitative inquiry into patient, caregiver, and clinician perspectives on service gaps and potential solutions for severe emotion dysregulation. JMIR Res Protoc. Aug 20, 2020;9(8):e14885. [FREE Full text] [CrossRef] [Medline]
- Stringer B, van Meijel B, Karman P, Koekkoek B, Hoogendoorn AW, Kerkhof AJ, et al. Collaborative care for patients with severe personality disorders: preliminary results and active ingredients from a pilot study (Part I). Perspect Psychiatr Care. Jul 2015;51(3):180-189. [CrossRef] [Medline]
- Singham T, Saunders R, Brooker H, Creese B, Aarsland D, Hampshire A, et al. Are subtypes of affective symptoms differentially associated with change in cognition over time: a latent class analysis. J Affect Disord. Jul 15, 2022;309:437-445. [FREE Full text] [CrossRef] [Medline]
- Cole VT, Bauer DJ. A note on the use of mixture models for individual prediction. Struct Equ Modeling. 2016;23(4):615-631. [FREE Full text] [CrossRef] [Medline]
- Tueller S, Lubke G. Evaluation of structural equation mixture models parameter estimates and correct class assignment. Struct Equ Modeling. Apr 01, 2010;17(2):165-192. [FREE Full text] [CrossRef] [Medline]
- Berlin KS, Williams NA, Parra GR. An introduction to latent variable mixture modeling (part 1): overview and cross-sectional latent class and latent profile analyses. J Pediatr Psychol. Mar 2014;39(2):174-187. [CrossRef] [Medline]
- Vermunt JK, Magidson J. Latent class cluster analysis. In: Hagenaars JA, McCutcheon AL, editors. Applied Latent Class Analysis. Cambridge, MA. Cambridge University Press; 2002:89-106.
- McLachlan GJ, Lee SX, Rathnayake SI. Finite mixture models. Annu Rev Stat Appl. Mar 07, 2019;6(1):355-378. [CrossRef]
- Scrucca L, Fop M, Murphy TB, Raftery AE. mclust 5: clustering, classification and density estimation using gaussian finite mixture models. R J. Aug 2016;8(1):289-317. [FREE Full text] [Medline]
- Newton-Howes G, Cunningham R, Atkinson J. Personality disorder prevalence and correlates in a whole of nation dataset. Soc Psychiatry Psychiatr Epidemiol. Apr 2021;56(4):679-685. [CrossRef] [Medline]
- Bell D. Delivering personalised cancer care to enhance patients’ quality of life. Cancer Nurs Pract. 2022;21(5):52. [CrossRef]
- Fonagy P, Allison E. The role of mentalizing and epistemic trust in the therapeutic relationship. Psychotherapy (Chic). Sep 2014;51(3):372-380. [CrossRef] [Medline]
- Nylund-Gibson K, Grimm RP, Masyn KE. Prediction from latent classes: a demonstration of different approaches to include distal outcomes in mixture models. Struct Equ Modeling. Apr 24, 2019;26(6):967-985. [CrossRef]
- Hair JF, Risher JJ, Sarstedt M, Ringle CM. When to use and how to report the results of PLS-SEM. Eur Bus Rev. Jan 14, 2019;31(1):2-24. [CrossRef]
- Bensmail H, Celeux G. Regularized gaussian discriminant analysis through eigenvalue decomposition. J Am Stat Assoc. Dec 1996;91(436):1743-1748. [CrossRef]
- Masyn KE. Latent class analysis and finite mixture modeling. In: Little TD, editor. The Oxford Handbook of Quantitative Methods in Psychology: Vol. 2: Statistical Analysis. Oxford, UK. Oxford Academic Press; 2013:551-611.
- Saunders R, Cape J, Fearon P, Pilling S. Predicting treatment outcome in psychological treatment services by identifying latent profiles of patients. J Affect Disord. Jun 2016;197:107-115. [FREE Full text] [CrossRef] [Medline]
- Colling C, Khondoker M, Patel R, Fok M, Harland R, Broadbent M, et al. Predicting high-cost care in a mental health setting. BJPsych Open. Jan 17, 2020;6(1):e10. [FREE Full text] [CrossRef] [Medline]
- Shmueli G, Ray S, Velasquez Estrada JM, Chatla SB. The elephant in the room: predictive performance of PLS models. J Bus Res. Oct 2016;69(10):4552-4564. [CrossRef]
- Takano K, Hino H, Akaho S, Murata N. Nonparametric e-mixture estimation. Neural Comput. Dec 2016;28(12):2687-2725. [CrossRef] [Medline]
- Grimm KJ, Houpt R, Rodgers D. Model fit and comparison in finite mixture models: a review and a novel approach. Front Educ. Mar 4, 2021;6:25. [CrossRef]
- Melnykov V. On the distribution of posterior probabilities in finite mixture models with application in clustering. J Multivar Anal. Nov 2013;122:175-189. [CrossRef]
- Perera G, Broadbent M, Callard F, Chang C, Downs J, Dutta R, et al. Cohort profile of the South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLaM BRC) case register: current status and recent enhancement of an electronic mental health record-derived data resource. BMJ Open. Mar 01, 2016;6(3):e008721. [FREE Full text] [CrossRef] [Medline]
- Fernandes AC, Cloete D, Broadbent MT, Hayes RD, Chang C, Jackson RG, et al. Development and evaluation of a de-identification procedure for a case register sourced from mental health electronic records. BMC Med Inform Decis Mak. Jul 11, 2013;13:71. [FREE Full text] [CrossRef] [Medline]
- Ranga R. South London and Maudsley discharge & transfer policy. National Health Service, England. URL: https://slam.nhs.uk/download.cfm?doc=docm93jijm4n1241.pdf&ver=1721 [accessed 2021-11-26]
- Tyrer P. Response: Personality disorder is here to stay across the lifespan - a response to Commentaries on the May 2022 Debate. Child Adolesc Ment Health. Sep 2022;27(3):253-255. [CrossRef] [Medline]
- Pan B, Wang W. Practical implications of ICD-11 personality disorder classifications. BMC Psychiatry. Mar 07, 2024;24(1):191-202. [FREE Full text] [CrossRef] [Medline]
- Coulacoglou C, Saklofske DH. Advances in latent variable measurement modeling. In: Coulacoglou C, Saklofske DH, editors. Psychometrics and Psychological Assessment: Principles and Applications. New York, NY. Academic Press; 2017:67-88.
- Katakis P, Schlief M, Barnett P, Rains LS, Rowe S, Pilling S, et al. Effectiveness of outpatient and community treatments for people with a diagnosis of 'personality disorder': systematic review and meta-analysis. BMC Psychiatry. Jan 21, 2023;23(1):57-86. [FREE Full text] [CrossRef] [Medline]
- Steadman J. Mapping the complex care pathways for personality disorder in South London (in press). Dissertation. Queen Mary, University of London. Aug 12, 2025. URL: https://qmro.qmul.ac.uk/xmlui/handle/123456789/98877 [accessed 2025-03-12]
- McLennan D, Noble S, Plunkett E, Wright G, Gutacker N. The English indices of deprivation. Ministry of Housing CLG. 2019. URL: https://www.gov.uk/government/collections/english-indices-of-deprivation [accessed 2024-04-29]
- Cunningham H. GATE: a framework and graphical development environment for robust NLP tools and applications. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002. Presented at: ACL '02; July 6-12, 2002:33-86; Philadelphia, PA. [CrossRef]
- The ICD-10 classification of mental and behavioural disorders. World Health Organization. 1993. URL: https://icd.who.int/browse10/2019/en [accessed 2024-04-29]
- Tyrer P, Mulder R, Kim YR, Crawford MJ. The development of the ICD-11 classification of personality disorders: an amalgam of science, pragmatism, and politics. Annu Rev Clin Psychol. May 07, 2019;15:481-502. [CrossRef] [Medline]
- Kotov R, Krueger RF, Watson D, Achenbach TM, Althoff RR, Bagby RM, et al. The Hierarchical Taxonomy of Psychopathology (HiTOP): a dimensional alternative to traditional nosologies. J Abnorm Psychol. May 2017;126(4):454-477. [CrossRef] [Medline]
- Nylund KL, Asparouhov T, Muthén BO. Deciding on the number of classes in latent class analysis and growth mixture modeling: a Monte Carlo simulation study. Struct Equ Modeling. Dec 05, 2007;14(4):535-569. [CrossRef]
- Almodóvar-Rivera IA, Pericchi-Guerra LR. An objective and robust bayes factor for the hypothesis test one sample and two population means. Entropy (Basel). Jan 20, 2024;26(1):e26010088. [FREE Full text] [CrossRef] [Medline]
- Rosenberg JM, Beymer PN, Anderson DJ, van Lissa CJ, Schmidt JA. tidyLPA: an R package to easily carry out latent profile analysis (LPA) using open-source or commercial software. J Open Source Softw. Oct 2018;3(30):978. [CrossRef]
- Bray BC, Lanza ST, Tan X. Eliminating bias in classify-analyze approaches for latent class analysis. Struct Equ Modeling. Jan 2015;22(1):1-11. [FREE Full text] [CrossRef] [Medline]
- Feingold A, Tiberio SS, Capaldi DM. New approaches for examining associations with latent categorical variables: applications to substance abuse and aggression. Psychol Addict Behav. Mar 2014;28(1):257-267. [FREE Full text] [CrossRef] [Medline]
- Celeux G, Soromenho G. An entropy criterion for assessing the number of clusters in a mixture model. J Classif. Sep 1996;13(2):195-212. [CrossRef] [Medline]
- McLachlan GJ. Discriminant analysis. Wiley Interdiscip Rev Comput Stat. Jul 16, 2012;4(5):421-431. [CrossRef]
- Baudry JP, Raftery AE, Celeux G, Lo K, Gottardo R. Combining mixture components for clustering. J Comput Graph Stat. Jun 01, 2010;9(2):332-353. [FREE Full text] [CrossRef] [Medline]
- Hair Jr JF, Hult GT, Ringle CM, Sarstedt M, Danks NP, Ray S. Partial Least Squares Structural Equation Modeling (PLS-SEM) Using R: A Workbook. Cham, Switzerland. Springer; 2021.
- Hamner B, Frasco M, LeDell E. Metrics. GitHub. URL: https://github.com/mfrasco/Metrics [accessed 2024-04-29]
- Khair U, Fahmi H, Al Hakim S, Rahim R. Forecasting error calculation with mean absolute deviation and mean absolute percentage error. J Phys Conf Ser. Dec 14, 2017;930:012002. [CrossRef]
- Wickham H, Averick M, Bryan J, Chang W, McGowan L, François R, et al. Welcome to the Tidyverse. J Open Source Softw. Nov 2019;4(43):1686. [CrossRef]
- R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. URL: https://www.R-project.org/ [accessed 2024-04-29]
- Integrated development for R. RStudio. URL: http://www.rstudio.com/ [accessed 2024-04-29]
- DeLeo K, Maconick L, McCabe R, Broeckelmann E, Sheridan Rains L, Rowe S, et al. Experiences of crisis care among service users with complex emotional needs or a diagnosis of 'personality disorder', and other stakeholders: systematic review and meta-synthesis of the qualitative literature. BJPsych Open. Feb 24, 2022;8(2):e53. [FREE Full text] [CrossRef] [Medline]
- Lo Y, Mendell NR, Rubin DB. Testing the number of components in a normal mixture. Biometrika. Oct 01, 2001;88(3):767-778. [CrossRef]
- McLachlan GJ. On bootstrapping the likelihood ratio test stastistic for the number of components in a normal mixture. R Stat Soc C Appl Stat. 1987;36(3):318-324. [CrossRef]
- Bolck A, Croon M, Hagenaars J. Estimating latent structure models with categorical variables: one-step versus three-step estimators. Polit Anal. Jan 04, 2017;12(1):3-27. [CrossRef]
- Nielsen JD, Rosenthal JS, Sun Y, Day DM, Bevc I, Duchesne T. Group-based criminal trajectory analysis using cross-validation criteria. Commun Stat Theory Methods. Sep 30, 2014;43(20):4337-4356. [CrossRef]
- Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence. 1995. Presented at: IJCAI '95; August 20-25, 1995:1137-1143; Montreal, QC. URL: https://dl.acm.org/doi/10.5555/1643031.1643047 [CrossRef]
- Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. Jan 1950;78(1):1-3. [CrossRef]
- Merkle EC, Flores RD. Forecaster evaluation as model comparison. Decision. May 16, 2024:43-62. [CrossRef]
- McMullen L, Parghi N, Rogers ML, Yao H, Bloch-Elkouby S, Galynker I. The role of suicide ideation in assessing near-term suicide risk: a machine learning approach. Psychiatry Res. Oct 2021;304:114118. [CrossRef] [Medline]
- Bennemann B, Schwartz B, Giesemann J, Lutz W. Predicting patients who will drop out of out-patient psychotherapy using machine learning algorithms. Br J Psychiatry. Feb 18, 2022;220(4):1-10. [CrossRef] [Medline]
- Shmueli G, Sarstedt M, Hair JF, Cheah J, Ting H, Vaithilingam S, et al. Predictive model assessment in PLS-SEM: guidelines for using PLSpredict. Eur J Mark. Nov 11, 2019;53(11):2322-2347. [CrossRef]
- Hair JF, Astrachan CB, Moisescu OI, Radomir L, Sarstedt M, Vaithilingam S, et al. Executing and interpreting applications of PLS-SEM: updates for family business researchers. J Fam Bus Strateg. Sep 2021;12(3):100392. [CrossRef]
- Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. Dec 31, 2011;13(4):e126. [FREE Full text] [CrossRef] [Medline]
- von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. STROBE Initiative. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. Oct 20, 2007;335(7624):806-808. [FREE Full text] [CrossRef] [Medline]
- Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. RECORD Working Committee. The REporting of studies conducted using observational routinely-collected health data (RECORD) statement: methods for arriving at consensus and developing reporting guidelines. PLoS Med. Oct 2015;12(10):e1001885. [FREE Full text] [CrossRef] [Medline]
- van der Nest G, Lima Passos V, Candel MJ, van Breukelen GJ. An overview of mixture modelling for latent evolutions in longitudinal data: modelling approaches, fit statistics and software. Adv Life Course Res. Mar 2020;43:100323. [CrossRef] [Medline]
- Nylund-Gibson K, Masyn KE. Covariates and mixture modeling: results of a simulation study exploring the impact of misspecified effects on class enumeration. Struct Equ Modeling. Sep 22, 2016;23(6):782-797. [CrossRef]
- Simola U, Cisewski-Kehe J, Wolpert RL. Approximate Bayesian computation for finite mixture models. J Stat Comput Simul. Nov 06, 2020;91(6):1155-1174. [CrossRef]
- Bourke J, Murphy A, Flynn D, Kells M, Joyce M, Hurley J. Borderline personality disorder: resource utilisation costs in Ireland. Ir J Psychol Med. Sep 2021;38(3):169-176. [CrossRef] [Medline]
- Sveen CA, Pedersen G, Ulvestad DA, Zahl KE, Wilberg T, Kvarstein EH. Societal costs of personality disorders among treatment-seeking patients in Norway: the relative contribution of specific DSM-5 categories. Eur Arch Psychiatry Clin Neurosci. Feb 2024;274(1):139-149. [FREE Full text] [CrossRef] [Medline]
- Durvasula SR, Halkitis PN. Delineating the interplay of personality disorders and health. Behav Med. 2017;43(3):151-155. [FREE Full text] [CrossRef] [Medline]
- Bender DS, Skodol AE, Dyck IR, Markowitz JC, Shea MT, Yen S, et al. Ethnicity and mental health treatment utilization by patients with personality disorders. J Consult Clin Psychol. Dec 2007;75(6):992-999. [CrossRef] [Medline]
- Cailhol L, Pelletier É, Rochette L, Renaud S, Koch M, David P, et al. Utilization of health care services by patients with cluster B personality disorders or schizophrenia. Psychiatr Serv. Dec 01, 2021;72(12):1392-1399. [CrossRef] [Medline]
- Finkelstein A, Zhou A, Taubman S, Doyle J. Health care hotspotting - a randomized, controlled trial. N Engl J Med. Jan 09, 2020;382(2):152-162. [FREE Full text] [CrossRef] [Medline]
- Capp R, Misky GJ, Lindrooth RC, Honigman B, Logan H, Hardy R, et al. Coordination program reduced acute care use and increased primary care visits among frequent emergency care users. Health Aff (Millwood). Oct 01, 2017;36(10):1705-1711. [CrossRef] [Medline]
- Botham J, Simpson A, McCrone P. Mental health service use and costs associated with complex emotional needs and a diagnosis of personality disorder: analysis of routine data. BJPsych Bull. Apr 2024;48(2):85-92. [FREE Full text] [CrossRef] [Medline]
- Catalao R, Broadbent M, Ashworth M, Das-Munshi J, L Hatch S, Hotopf M, et al. Access to psychological therapies amongst patients with a mental health diagnosis in primary care: a data linkage study. Soc Psychiatry Psychiatr Epidemiol (Forthcoming). Nov 06, 2024. [CrossRef] [Medline]
- Parsons J, Bryce C, Atherton H. Which patients miss appointments with general practice and the reasons why: a systematic review. Br J Gen Pract. Jun 2021;71(707):e406-e412. [FREE Full text] [CrossRef] [Medline]
- Talukder AK, Sanz JB, Samajpati J. ‘Precision health’: balancing reactive care and proactive care through the evidence based knowledge graph constructed from real-world electronic health records, disease trajectories, diseasome, and patholome. In: Proceedings of the 8th International Conference on Big Data Analytics. 2020. Presented at: BDA '20; December 15-18, 2020:113-133; Sonepat, India. URL: https://link.springer.com/chapter/10.1007/978-3-030-66665-1_9 [CrossRef]
- Population health management in primary health care: a proactive approach to improve health and well-being: primary heath care policy paper series. World Health Organization. 2023. URL: https://apps.who.int/iris/bitstream/handle/10665/368805/WHO-EURO-2023-7497-47264-69316-eng.pdf [accessed 2024-04-29]
Abbreviations
| BIC: Bayesian information criterion |
| CONSORT: Consolidated Standards of Reporting Trials |
| CRIS: Clinical Records Interactive Search |
| ICD-10: International Classification of Diseases, Tenth Revision |
| ICD-11: International Classification of Diseases, Eleventh Revision |
| LCA: latent class analysis |
| LP: latent profile |
| LPA: latent profile analysis |
| LRT: likelihood ratio test |
| MAE: mean absolute error |
| MAPE: mean absolute percentage error |
| NHS: National Health Service |
| NIHR: National Institute for Health Research |
| PD: personality disorder |
| SLaM: South London and Maudsley |
Edited by T de Azevedo Cardoso; submitted 11.12.23; peer-reviewed by P Moran, J Ballard, P Roma; comments to author 01.03.24; revised version received 06.07.24; accepted 10.12.24; published 15.04.25.
Copyright©Jack Steadman, Rob Saunders, Mark Freestone, Robert Stewart. Originally published in the Interactive Journal of Medical Research (https://www.i-jmr.org/), 15.04.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.i-jmr.org/, as well as this copyright and license information must be included.