Beyond verbal fluency in the verbal fluency task: semantic clustering as a predictor of remission in individuals at clinical high risk for psychosis
====================================================================================================================================================

* Eugenie Choe
* Minji Ha
* Sunah Choi
* Sunghyun Park
* Moonyoung Jang
* Minah Kim
* Jun Soo Kwon

## Abstract

**Background:** There have been conflicting reports on whether conventional verbal fluency measures can predict the prognosis of individuals at clinical high risk (CHR) for psychosis. We aimed to investigate whether verbal fluency task measures that represent semantic processing more directly than conventional measures could be more reliable predictors of later remission in CHR individuals.

**Methods:** We recruited CHR individuals and healthy controls to participate in a baseline verbal fluency assessment. We identified semantic clusters within the verbal fluency task responses based on cosine similarity between consecutive words, calculated from the word embedding model. Binomial logistic regression was performed to test whether average semantic cluster size and number of words produced could be predictors of remission in CHR individuals.

**Results:** Our study sample included 96 CHR individuals and 178 healthy controls. According to clinical assessment at the last follow-up, 23 CHR individuals were classified as remitters and 73 as nonremitters, including 29 individuals who converted to psychosis. The CHR remitters showed larger average and maximum semantic cluster sizes than CHR nonremitters and healthy controls. Average semantic cluster size, but not the number of words, was a significant predictor of later remission in CHR individuals.

**Limitations:** Our sample included only native Korean speakers.

**Conclusion:** A verbal fluency task measure that more specifically represents semantic processing may be a better neurocognitive predictive marker for remission in CHR individuals than conventional verbal fluency measures. Our results provide an explanation for heterogeneous reports on whether verbal fluency can predict prognosis in CHR individuals and suggest that semantic processing is a putative cognitive predictor of their prognosis.

## Introduction

The clinical high risk state (CHR) for psychosis is a concept that was established for early intervention to prevent or delay the onset of psychotic disorders.1 However, it has been consistently reported that a substantial portion of individuals who meet the criteria for CHR do not convert to psychotic disorders and that those who do not convert to psychosis show heterogeneous clinical and functional outcomes. While there are some who remit from the high-risk state, some continue to show poor functional outcomes.2,3 Predicting remission, therefore, has special clinical implications in that it may reduce unnecessary intervention for those who would remit and enable more intensive treatment for those whose prognosis is expected to be poor.4–6

It is widely accepted that the prognostic outcome of psychotic disorders is heavily affected by cognition.7 Nevertheless, in the CHR state, there have been inconsistent reports on whether specific neurocognitive domains can be used to predict remission or functional recovery.8,9 In those reports, verbal fluency has frequently been mentioned as a potential predictor,8,10,11 or at least a correlate, of functional recovery12 or remission.13 On the other hand, there have also been studies in which verbal fluency was not found to be a predictor of the functional outcome of CHR individuals.9,14

A putative reason for such conflicting results is the heterogeneity of cognitive functions involved in the verbal fluency task. Although performing the task involves semantic processing, several other neurocognitive functions, including psychomotor speed, set shifting and inhibition, are concurrently represented in task performance.15 This highlights the need to investigate task performance measures that directly reflect neurocognitive functions of interest. In the verbal fluency task, measures that represent semantic clustering and switching in the task response are potential candidates.16 During the task, participants tend to produce a cluster of words that are related and then shift to another group of semantically related words to produce another cluster. Clustering is regarded as a phenomenon that reflects the activation of semantically related representations, while shifting is deemed to be more related to executive function, such as cognitive flexibility.17

Since deficient semantic processing is one of the hallmark features of schizophrenia,18,19 we hypothesized that individuals who would remit from CHR would be less likely to share this feature. We therefore aimed to explore whether semantic clustering, a phenomenon that reflects semantic processing during the verbal fluency task, could be a reliable predictor of remission in CHR individuals. We sought to investigate whether CHR individuals show differential patterns of semantic clustering during the category verbal fluency task according to their clinical course and whether category verbal fluency performance measures related to semantic clustering can be used to predict the prognosis of individuals at CHR, especially remission.

## Methods

### Participants

We recruited individuals at CHR for psychosis and healthy controls to participate in the baseline verbal fluency assessment between November 2004 and November 2019. All participants were native Korean speakers. The CHR individuals were recruited from Seoul Youth Clinic ([www.youthclinic.org](http://www.youthclinic.org)) in the Department of Neuropsychiatry at Seoul National University Hospital (SNUH). The CHR participants underwent the Structured Interview for Prodromal Symptoms (SIPS), and their prodromal symptoms were evaluated using the validated Korean version of the Scale of Prodromal Symptoms (SOPS).20 The Global Assessment of Functioning (GAF) was used to define general functional status. Healthy controls were recruited via Internet advertisements and were screened using the Structured Clinical Interview for DSM Disorders I (SCID-I) Nonpatient Edition (SCID-NP). Potential healthy controls were excluded if they had a history of any psychiatric disorder or had first- to third-degree relatives with psychotic disorders. The exclusion criteria for CHR and healthy controls included a lifetime diagnosis of psychotic disorder, a history of antipsychotic use, substance abuse or dependence, neurologic disease or significant head trauma, medical illness with cognitive sequelae, sensory impairments and intellectual disability (IQ < 70).

The CHR participants were followed up regularly for 1–6 years with the provision of regular treatment such as supportive therapy, cognitive behavioural therapy and medication. Remission in CHR individuals was defined as achieving a score of 2 or lower on the SOPS positive symptom subscale and a score of 60 or higher on the GAF at the last follow-up point.6,13 We classified CHR participants as remitters or nonremitters according to these remission criteria. Medication use during the follow-up period was documented, and the mean olanzapine equivalent dose was calculated for antipsychotic use.21

All participants provided written informed consent after receiving a thorough explanation of the study procedure (IRB no. H-1110–009–380). For participants younger than the age of majority, informed consent was obtained from both the participants and their parents. This study was conducted in accordance with the Declaration of Helsinki and was approved by the Institutional Review Board of SNUH (IRB no. 2211–162–1383).

### Verbal fluency task performance measures

Participants underwent the category verbal fluency task22 as a part of a larger battery of neuropsychological tests. During the task, participants were asked to name as many animals as they could in 1 minute. Lists of words produced during the “animal” category task were subject to analysis.

Semantic clusters within a participant’s verbal fluency task response were defined as a series of consecutive words in which semantic similarities exceeded a cutoff similarity value. To quantify semantic similarity between a pair of consecutive words, we used a publicly released Korean word embedding model trained with the FastText algorithm.23 The word embedding model was trained with a corpus that includes Korean Wikipedia, KorQuAD and Naver movie reviews ([https://github.com/ratsgo/embedding](https://github.com/ratsgo/embedding)). Cosine similarity between vector representations of each word in the pair of words was regarded as semantic similarity between the words.

The cutoff similarity value to distinguish between clusters was calculated for each individual task response. An average of cosine similarity values of word pairs in each task response was calculated after randomly shuffling the words, and then the grand average of the average similarity values was taken by repeating the shuffling 50 times. The grand average was used in the analysis as the cutoff similarity value.

The average cluster size and maximum cluster size of each individual response were taken as measures that represent clustering processes during the task. These measures, along with the number of words in a response, were investigated as possible predictors of the clinical course of individuals at CHR for psychosis.

### Statistical analysis

All statistical analyses were conducted using R version 4.2.3. Statistical significance was set at *p* < 0.05. We compared demographic variables, clinical variables and verbal fluency task performance measures across groups using analysis of variance (ANOVA), independent *t* test, Welch *t* test or χ2 test as appropriate. Tukey tests were performed as post hoc ANOVA. To explore associations between verbal fluency task performance measures, we calculated Pearson correlation coefficients between each measure for each group. Multiple comparison corrections were performed using the Bonferroni correction.

Binomial logistic regression analyses with the backward selection method were performed to elucidate whether baseline verbal fluency task performance could predict remission in CHR individuals. Three separate logistic regression models were established for analysis: 1) a model with only demographic and clinical variables as independent variables; 2) a model with the number of words added as another independent variable, along with variables included in model 1; and 3) a model with average cluster size included as another independent variable, along with variables included in model 2. Common demographic and clinical variables included in the models were age, sex, years of education, IQ, baseline GAF score, baseline SOPS subscale scores (positive symptoms, negative symptoms, disorganization symptoms, general symptoms) and antipsychotic use during the follow-up period. The maximum cluster size was excluded due to collinearity with the average cluster size.

#### Sensitivity analyses

To determine whether the definition of clusters affected the main results, sensitivity analyses of the cutoff cosine similarity values used to define the clusters were conducted. Details are described in Appendix 1, available at [www.jpn.ca/lookup/doi/10.1503/jpn.230074/tab-related-content](http://www.jpn.ca/lookup/doi/10.1503/jpn.230074/tab-related-content)

## Results

### Participants

A total of 96 individuals at CHR for psychosis and 178 healthy controls participated in the baseline verbal fluency assessment. Of the CHR individuals, 23 were classified as remitters, 73 were classified as nonremitters, and 29 of the 73 nonremitters had converted to a psychotic disorder according to SIPS criteria by the end of the follow-up period. Table 1 summarizes the demographic and clinical characteristics of the participants at baseline. Healthy controls were significantly older and more educated than both CHR remitters (age, Tukey honestly significant difference (HSD)-adjusted *p* < 0.001; education, Tukey HSD-adjusted *p* < 0.001) and CHR nonremitters (age, Tukey HSD-adjusted *p* < 0.001; education, Tukey HSD-adjusted *p* < 0.001). The CHR nonremitters had lower IQ than healthy controls (Tukey HSD-adjusted *p* = 0.005). Among CHR participants, SOPS positive symptom and disorganization subscale scores at baseline were higher in nonremitters than remitters (SOPS positive, *t* = 2.68, *p* = 0.011; SOPS disorganization, *t* = 3.03, *p* = 0.003). During follow-up, CHR nonremitters were exposed to higher olanzapine-equivalent doses of antipsychotics than remitters (*t* = 2.85, *p* = 0.006).

View this table:
[Table 1](http://jpn.ca/content/48/6/E414/T1)

Table 1 
Demographic, clinical and verbal fluency task performance characteristics of study participants at baseline

### Group difference in the verbal fluency task performance measure

Group comparison results for the category verbal fluency task performance measures are presented in Table 1 and Figure 1. The CHR nonremitters produced fewer words than healthy controls (Tukey HSD-adjusted *p* = 0.006). The CHR remitters showed larger average and maximum semantic cluster sizes than both CHR nonremitters (average cluster size, Tukey HSD-adjusted *p* = 0.013; maximum cluster size, Tukey HSD-adjusted *p* = 0.035) and healthy controls (average cluster size, Tukey HSD-adjusted *p* = 0.006; maximum cluster size, *p* = 0.014).

![Figure 1](http://jpn.ca/https://www.jpn.ca/content/jpn/48/6/E414/F1.medium.gif)

[Figure 1](http://jpn.ca/content/48/6/E414/F1)

Figure 1 
Group comparison of (A) the number of words produced during the verbal fluency task, (B) average semantic cluster size in the task response and (C) maximum semantic cluster size in the task response. CHR-NR = participants at clinical high risk for psychosis who were nonremitters; CHR-R = participants at clinical high risk for psychosis who remitted; HC = healthy controls. *Post hoc Tukey honestly significant difference (HSD)-adjusted *p* < 0.05; **post hoc Tukey HSD-adjusted *p* < 0.01.

### Correlation between task performance measures

While we observed a statistically significant positive correlation between average cluster size and number of words produced in CHR nonremitters (*r* = 0.57, Bonferroni-adjusted *p* = 0.006) and healthy controls (*r* = 0.23, Bonferroni-adjusted *p* = 0.010), remitters did not show a significant correlation between the 2 measures (Figure 2A). Such discrepancy was also observed in the correlation between maximum cluster size and number of words produced (Figure 2B).

![Figure 2](http://jpn.ca/https://www.jpn.ca/content/jpn/48/6/E414/F2.medium.gif)

[Figure 2](http://jpn.ca/content/48/6/E414/F2)

Figure 2 
Correlation between (A) average semantic cluster size and number of words produced during the verbal fluency task and (B) maximum semantic cluster size and number of words produced during the verbal fluency task. Pearson correlation coefficient, Bonferroni-corrected *p* values and linear regression line are shown. The confidence bands in each plot represent 95% confidence intervals of each linear regression model. CHR-NR = participants at clinical high risk for psychosis who were nonremitters; CHR-R = participants at clinical high risk for psychosis who remitted; HC = healthy controls.

### Predicting remission of CHR individuals

The results of binary logistic regression analyses performed on each of the 3 logistic regression models are summarized in Table 2. The SOPS positive symptom subscore (β = 0.821, 95% confidence interval [CI] 0.695–0.945, *p* = 0.009) was the only statistically significant predictor of remission in the model with demographic and clinical variables only. In the model with the number of words in the verbal fluency task added, the SOPS positive symptom subscore (β = 0.831, 95% CI 0.709–0.958, *p* = 0.015) was also the only significant predictor. When the average cluster size was added, the average cluster size (β = 4.510, 95% CI 1.804–13.704, *p* = 0.002), SOPS positive symptom subscore (β = 0.801, 95% CI 0.672–0.937, *p* = 0.008) and sex (β = 0.243, 95% CI 0.066–0.802, *p* = 0.024) were significant predictors.

View this table:
[Table 2](http://jpn.ca/content/48/6/E414/T2)

Table 2 
Selected predictors of remission from clinical high risk for psychosis (CHR) in binary logistic regression analyses with the backward selection method

### Sensitivity analyses

The results of the sensitivity analyses for the definition of clusters are described in Appendix 1. Average semantic cluster size, but not the number of words, was a significant predictor of later remission in CHR individuals at all thresholds.

## Discussion

In the present study, we aimed to investigate whether the verbal fluency task performance measure that is more directly reflective of semantic processing could better predict remission in CHR individuals. As task performance measures that represent semantic processing, semantic clustering in the task response was quantified as the average cluster size and the maximum cluster size. The CHR remitters showed significantly larger semantic cluster size during the task, and the average cluster size was revealed to be a predictor of remission in CHR individuals, while the number of words produced, which is a conventional verbal fluency task measure, did not predict remission. To our knowledge, this is the first study to address semantic clustering during the verbal fluency task in CHR individuals and to show that the task measure related to semantic clustering better predicts remission in CHR individuals.

Semantic processing is a neurocognitive function that has been investigated extensively in schizophrenia, and the relationship between aberrant semantic processing and symptom severity in patients with schizophrenia has long been reported.24,25 In CHR individuals, it has recently been suggested that altered semantic processing may be a marker of prognosis.26–28 Although most of these studies focused on conversion to psychosis as a prognostic outcome, an electrophysiological study showed that the baseline N400 semantic priming effect, which is regarded as an electrophysiological marker of semantic processing, could predict social functioning after 1 year in CHR participants.26 The results of the present study add to the literature, confirming that altered semantic processing at baseline and its relationship with clinical outcome is evident not only at the electrophysiological level, but also at the neurocognitive performance level.

Our results have shown that the number of words produced during the verbal fluency task could not predict later remission, while semantic clustering measures could. A possible reason for this result is that the difference in semantic processing at baseline between CHR remitters and nonremitters may not be so overt as to be differentiated by the number of words produced during the verbal fluency task but may be detected by more focused measures of semantic processing, such as cluster size. This may provide an explanation for the aforementioned heterogeneous reports on whether verbal fluency can predict remission or functional recovery in CHR individuals8,9 since the results are likely to differ across samples if the effect is not sufficiently large.

Another finding of the present study is that CHR remitters showed significantly larger semantic cluster sizes than healthy controls. Notably, while a strong correlation between the number of words produced during the task and the average/maximum size of semantic clusters was observed in CHR nonremitters and healthy contrrols, in CHR remitters there was no statistically significant correlation between those measures. These results imply that CHR remitters have a distinctive pattern of neurocognitive functions during the task. One speculative explanation for this phenomenon is that, while performing the task, CHR remitters might tend to rely on relatively intact semantic processing capacity to compensate for other deficient cognitive processes.29 Further studies to clarify such characteristics are warranted and are expected to provide insight into the nature of the CHR state.

### Limitations

There are several limitations to this study. First, all participants in this study were native Korean speakers. Because most previous studies on semantic clustering in the verbal fluency task involved English speakers, to overcome linguistic and cultural differences, we used an unsupervised, agnostic word embedding model trained with a large Korean corpus, which was expected to capture latent meanings of Korean words30 while not relying on arbitrary ratings by a human rater. Nevertheless, caution is required when generalizing the results to speakers from other linguistic backgrounds. Second, the criteria for remission used in this study, which are determined by clinical assessment at the last follow-up point, may not suffice to reflect the clinical outcome of individuals with a fluctuating clinical course. Third, while other clinical variables, such as negative, disorganization and general symptom scores, may also have implications in the individual’s clinical state, remission was defined based on scores on the SOPS positive subscale and GAF scale. Fourth, CHR individuals and healthy controls were not matched for several demographic and cognitive variables. However, because significant differences in participant characteristics, such as lower IQ and younger age in CHR participants, reflect characteristics of the CHR state, matching for every demographic variable may have led to selection bias. Therefore, the effects of unmatched variables were controlled by including the variables as covariates in logistic regression models instead of by group matching. Finally, our sample size was relatively small.

## Conclusion

Our results have shown that task performance measures related to semantic clustering at baseline could predict later remission in CHR individuals, while conventional verbal fluency measures could not. By showing that task performance measures that are more specific to semantic processing had better predictive capacity, our results potentially explain previous conflicting reports on whether verbal fluency can predict remission in CHR individuals. They also suggest a need to consider which neurocognitive functions are represented by the neurocognitive task of interest, especially when interpreting results from tasks, such as the verbal fluency task, that involve multiple neurocognitive functions. Among those, semantic processing is suggested as a putative cognitive predictor of remission in CHR individuals.

## Acknowledgements

This research was supported by the Basic Science Research Program and the Brain Science Convergence Research Program through the National Research Foundation of Korea (NRF) and the Basic Research Program of the Korea Brain Research Institute (KBRI), funded by the Ministry of Science & ICT (grant nos. 2019R1C1C1002457, 2020M3E5D9079910, RS-2023-00266120, and 21-BR-03-01).

## Footnotes

*   **Competing interests:** None declared.

*   **Contributors:** E. Choe, M. Kim and J.S. Kwon designed the study. M. Ha and S. Park acquired the data, which S. Choi and M. Jang analyzed. E. Choe wrote the article. All of the authors revised it critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.

*   Received May 14, 2023.
*   Revision received July 12, 2023.
*   Revision received August 4, 2023.
*   Accepted August 4, 2023.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: [https://creativecommons.org/licenses/by-nc-nd/4.0/](https://creativecommons.org/licenses/by-nc-nd/4.0/)

## References

1.  Fusar-Poli P, Borgwardt S, Bechdolf A, et al. The psychosis high-risk state: a comprehensive state-of-the-art review. JAMA Psychiatry 2013;70:107–20.
    
    

2.  Allswede DM, Addington J, Bearden CE, et al. Characterizing covariant trajectories of individuals at clinical high risk for psychosis across symptomatic and functional domains. Am J Psychiatry 2020;177:164–71.
    
    

3.  Polari A, Yuen HP, Amminger P, et al. Prediction of clinical outcomes beyond psychosis in the ultra-high risk for psychosis population. Early Interv Psychiatry 2021;15:642–51.
    
    

4.  Beck K, Andreou C, Studerus E, et al. Clinical and functional long-term outcome of patients at clinical high risk (CHR) for psychosis without transition to psychosis: a systematic review. Schizophr Res 2019;210:39–47.
    
    

5.  Kim M, Kim T, Hwang WJ, et al. Forecasting prognostic trajectories with mismatch negativity in early psychosis. Psychol Med 2023;53:1489–99.
    
    

6.  Kim M, Lee TH, Yoon YB, et al. Predicting remission in subjects at clinical high risk for psychosis using mismatch negativity. Schizophr Bull 2018;44:575–83.
    
    

7.  Kahn RS, Keefe RS. Schizophrenia is a cognitive illness: time for a change in focus. JAMA Psychiatry 2013;70:1107–12.
    
    

8.  Bolt LK, Amminger GP, Farhall J, et al. Neurocognition as a predictor of transition to psychotic disorder and functional outcomes in ultra-high risk participants: findings from the NEURAPRO randomized clinical trial. Schizophr Res 2019;206:67–74.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1016/j.schres.2018.12.013&link_type=DOI) 

9.  Glenthøj LB, Kristensen TD, Wenneberg C, et al. Predictors of remission from the ultra-high risk state for psychosis. Early Interv Psychiatry 2021;15:104–12.
    
    

10. Lin A, Wood SJ, Nelson B, et al. Neurocognitive predictors of functional outcome two to 13 years after identification as ultra-high risk for psychosis. Schizophr Res 2011;132:1–7.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1016/j.schres.2011.06.014&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=21763109&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

11. Niendam TA, Bearden CE, Zinberg J, et al. The course of neurocognition and social functioning in individuals at ultra high risk for psychosis. Schizophr Bull 2007;33:772–81.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1093/schbul/sbm020&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=17420177&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 
    
    [Web of Science](http://jpn.ca/lookup/external-ref?access_num=000247241700023&link_type=ISI) 

12. Shin YS, Kim SY, Lee TY, et al. Longitudinal change in neurocognition and its relation to symptomatic and functional changes over 2years in individuals at clinical high-risk for psychosis. Schizophr Res 2016;174:50–7.
    
    

13. Lee TY, Kim SN, Correll CU, et al. Symptomatic and functional remission of subjects at clinical high risk for psychosis: a 2-year naturalistic observational study. Schizophr Res 2014;156:266–71.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1016/j.schres.2014.04.002&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=24815568&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

14. Hedges EP, Dickson H, Tognin S, et al. Verbal memory performance predicts remission and functional outcome in people at clinical high-risk for psychosis. Schizophr Res Cogn 2022;28:100222.
    
    

15. Shao Z, Janse E, Visser K, et al. What do verbal fluency tasks measure? Predictors of verbal fluency performance in older adults. Front Psychol 2014;5:772.
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

16. Troyer AK. Normative data for clustering and switching on verbal fluency tasks. J Clin Exp Neuropsychol 2000;22:370–8.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1076/1380-3395(200006)22:3;1-V;FT370&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=10855044&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 
    
    [Web of Science](http://jpn.ca/lookup/external-ref?access_num=000087811000006&link_type=ISI) 

17. Carpenter E, Peñaloza C, Rao L, et al. Clustering and switching in verbal fluency across varying degrees of cognitive control demands: evidence from healthy bilinguals and bilingual patients with aphasia. Neurobiol Lang (Camb) 2021;2:532–57.
    
    

18. Liu Y, Wang G, Jin H, et al. Cognitive deficits in subjects at risk for psychosis, first-episode and chronic schizophrenia patients. Psychiatry Res 2019;274:235–42.
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

19. Wang K, Cheung EF, Gong QY, et al. Semantic processing disturbance in patients with schizophrenia: a meta-analysis of the N400 component. PLoS One 2011;6:e25435.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1371/journal.pone.0025435&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=22022395&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

20. Jung MH, Jang JH, Kang DH, et al. The reliability and validity of the korean version of the structured interview for prodromal syndrome. Psychiatry Investig 2010;7:257–63.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.4306/pi.2010.7.4.257&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=21253409&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 
    
    [Web of Science](http://jpn.ca/lookup/external-ref?access_num=000285956500005&link_type=ISI) 

21. Gardner DM, Murphy AL, O’Donnell H, et al. International consensus study of antipsychotic dosing. Am J Psychiatry 2010; 167:686–93.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1176/appi.ajp.2009.09060802&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=20360319&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 
    
    [Web of Science](http://jpn.ca/lookup/external-ref?access_num=000278269500013&link_type=ISI) 

22. Robert PH, Lafont V, Medecin I, et al. Clustering and switching strategies in verbal fluency tasks: comparison between schizophrenics and healthy adults. J Int Neuropsychol Soc 1998;4:539–46.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1017/S1355617798466025&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=10050358&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 
    
    [Web of Science](http://jpn.ca/lookup/external-ref?access_num=000077382400002&link_type=ISI) 

23. Bojanowski P, Grave E, Joulin A, et al. Enriching word vectors with subword information. Trans Assoc Comput Linguist 2017; 5:135–46.
    
    

24. Besche-Richard C, Iakimova G, Hardy-Baylé MC, et al. Behavioral and brain measures (N400) of semantic priming in patients with schizophrenia: test-retest effect in a longitudinal study. Psychiatry Clin Neurosci 2014;68:365–73.
    
    

25. Kiang M, Kutas M, Light GA, et al. Electrophysiological insights into conceptual disorganization in schizophrenia. Schizophr Res 2007;92:225–36.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1016/j.schres.2007.02.001&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=17383161&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

26. Lepock JR, Mizrahi R, Gerritsen CJ, et al. N400 event-related brain potential and functional outcome in persons at clinical high risk for psychosis: a longitudinal study. Psychiatry Clin Neurosci 2022;76:114–21.
    
    

27. Morgan SE, Diederen K, Vértes PE, et al. Natural language processing markers in first episode psychosis and people at clinical high-risk. Transl Psychiatry 2021;11:630.
    
    

28. Sabb FW, Van Erp TG, Hardt ME, et al. Language network dysfunction as a predictor of outcome in youth at clinical high risk for psychosis. Schizophr Res 2010;116:173–83.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1016/j.schres.2009.09.042&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=19861234&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 

29. Carrión RE, Goldberg TE, McLaughlin D, et al. Impact of neurocognition on social and role functioning in individuals at clinical high risk for psychosis. Am J Psychiatry 2011;168:806–13.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1176/appi.ajp.2011.10081209&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=21536691&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom) 
    
    [Web of Science](http://jpn.ca/lookup/external-ref?access_num=000293387600011&link_type=ISI) 

30. Tshitoyan V, Dagdelen J, Weston L, et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 2019;571:95–8.
    
    [CrossRef](http://jpn.ca/lookup/external-ref?access_num=10.1038/s41586-019-1335-8&link_type=DOI) 
    
    [PubMed](http://jpn.ca/lookup/external-ref?access_num=31270483&link_type=MED&atom=%2Fjpn%2F48%2F6%2FE414.atom)