M15 Hillcrest stats & predictions

<|repo_name|>KnowledgeEngineeringGroup/DKEW-Schizophrenia-Research<|file_sep|>/Report/Conclusions.tex % Conclusions chapter of this thesis % % For commands specific to the Analysis of Linguistic Detail (ALD) see % the pseudo-package ALD.sty % chapter{Conclusions}label{ch:conclusions} section{Summary of the thesis}label{sec:summary} This thesis investigated the use of natural language processing (NLP) techniques to support proneness to delusion for patients with schizophrenia that show early signs of the illness based on their spoken language. It consists of three parts: begin{enumerate} item A literature review of textsc{ald} and patient populations consisting of clinical and self-reported data. The review also analysed the use of machine learning approaches in order to prepare an evaluation baseline. item An extended analysis of the textsc{ald} features that have been applied to schizophrenia patients. item An implementation of an application that enables a clinician to query patients and receive an automated proneness evaluation based on the most up-to-date research knowledge. The application also comes with a browser-based data visualisation that is accessible by non-experts in NLP and machine learning. end{enumerate} The second part uses clinical data from the Southampton Neurocognition and Psychosis Study (SNAP). Altogether 202 patients were included in the second part of this thesis. The acoustic information from a subset of 88 patients was used as input for feature extraction using the textsc{ald} toolkit. From this subset 58 patients were diagnosed with psychosis and 30 patients were diagnosed with clinical high-risk (CHR). The goal of the second part was to find out whether information from the acoustic layer and the prosodic layer can provide an indication for future development of schizophrenia. This analysis will serve as a basis for the final system and its evaluation. There was a statistically significant difference between both groups in the lexical, syntactic, prosodic and acoustic groups. The most significant features were the textsc{ald} lexical features $textrm{simp1}$, $textrm{simp2}$ and $textrm{comp}$. All other features were statistically significant but not as significant as those three. In addition to these features other potential features were found that need to be further investigated in future research. The third part of this thesis outlines a prototype for a prediction model based on textsc{ald} features. The mode currently uses textsc{ald} features extracted from patient corpora to predict proneness in the SNAP dataset. It also has options to provide the user with feedback on available data and to train the model on new data. This is done using a browser-based implementation of a stacked generalization technique. The model currently considers four different classifiers (decision tree, SVM, random forest and naive Bayes) and combines them using another classifier. The model has been trained to reach an accuracy of $80%$. section{Evaluation}label{sec:evaluation} In this section we evaluate the conducted research studies on a theoretical level in terms of its scientific impact. We also assess how useful the application prototype is in practice. subsection{Literature review}label{sec:review} One of the systematic literature reviews performed in this thesis is very close to the state of the art in delusion research related to linguistic indicators. It is comparable to cite{barch2016speechworks} and cite{brown2016language}. The comparison shows that one of those works uses patients from the English-speaking countries, Australia and New Zealand but not from other countries as ours does. An interesting aspect to note is that we included studies on patients suffering from other illnesses besides schizophrenia. This makes it possible to compare this study with others that have been performed on patients who suffer from depression. This can be relevant if we want to merge those or other patient corpora in order to increase the sample size for training purposes. This will especially be helpful as cite{koellner2018listening} states that there are no large-scale studies performed due to the complexity of recording interviews from patients in different countries. The comparison between our literature review and cite{barch2016speechworks} also showed that we focused on studies that provide an indication about schizophrenia proneness. For example, cite{fusar2011word} investigates the use of semantic categories for differentiating between patients who have been diagnosed with schizophrenia as opposed to healthy controls. This means that they do not indicate an earlier stage as our study does in which case simple measures of proneness should be grasped. subsection{Acoustic analysis with textsc{ald}}label{sec:evaluation_acoustic} This part of the thesis can be seen as an extension of the work by citeauthor{ochs2004early478}cite{ochs2004early478}, who investigated language differences between patients with schizophrenia and healthy subjects using textsc{ald} features. It was found that there are indeed linguistic differences between patients with schizophrenia and healthy subjects. citeauthor{ochs2004early478}cite{ochs2004early478} did so at a very early stage using a relatively small sample of N = 40 subjects and therefore it was important to investigate whether these results can be reproduced for a prognostic schizophrenia population. As mentioned in the analysis chapter (Cref{ch:analysis}), we had access to data from a large-scale research project called SNAP. From that dataset we extracted 88 interviews of patients who were diagnosed with psychosis or healthy-controls or were at high risk of being diagnosed with psychosis (Cref{fig:ahnweshistogram}). These interviews were very similar in nature to the ones collected in the Chicago Consortium for Neuropsychiatric Phenomics; however, it is the first time that such a number of interdisciplinary studies was able to extract spoken language data for analysis from potential psychosis patients. These results are comparable to those of citeauthor{ochs2004early478}cite{ochs2004early478}, who also used textsc{ald} with similar outcomes. Another point that should be noted is that we did not perform feature selection but rather used several classifiers (Cref{sec:classification_acoustic}) trained on all features calculated by textsc{ald}. In contrast to our approach citeauthor{ochs2004early478}cite{ochs2004early478} used only lexical and syntactic information and outperformed us in terms of accuracy. However, they had less interviews available as they only analysed interviews from 40 patients split into two groups (20 healthy-controls and 20 patients with schizophrenia). Due to those facts it is difficult to compare their results with ours. For our acoustic analysis we followed existing studies cite{koellner2018listening} which have analysed acoustic patterns across patient populations based on their diagnosis groups using tools like Praat. A problem that needs further investigation is that it is still not clear which features should be used in order to discriminate between both groups. The significance scores in Cref{tab:lexical}, Cref{tab:syntactic}, Cref{tab:pronotional} and Cref{tab:acoustic} show that there are features which can be seen as potential indicators for psychosis proneness but still do not give any clues on what exactly should be used for future classification models. Another aspect that we lack is a comparison with healthy controls because our patient population consists of CHR patients and patients diagnosed with psychosis. Nevertheless, our results did give us an indication about important features which are considered as potential indicators for schizophrenia proneness; however, we still need to investigate if those or other features should be used in future classification models. subsection{Building a prediction model}label{sec:evaluation_prediction} The final part of this thesis was concerned with building a prediction model to support clinicians when diagnosing a patient with schizophrenia. There are two types of prediction models that have already been evaluated: begin{itemize} item One example is citeauthor{burdick2017first}, who build an automated measure for detecting early signs of psychosis using an odds ratio model based on statistical linguistic analysis. This model was able to achieve an area under the curve (AUC) of $0.78$ in terms of classifying whether a patient is at risk of psychotic relapse within the next six months or not using their written data. item The second example was introduced by citeauthor{xu2017predictive}, who used statistical machine learning approaches to predict risk of psychosis on an individual level using data collected from multiple sources like genetics to biomarkers. They built their classifier using clinical markers and structural imaging data like cortical thicknesses and geometrical properties of brain ventricles. The risk prediction achieved by their classifier was assessed through cross-validation using mean accuracy, sensitivity, specificity and positive likelihood ratio (PLR), with respectively accuracies for predicting conversion from HC-NP to CHR-P of 0.66 (0.57 - 0.74), and 0.74 (0.66 - 0.81), PLR values of 2.38 (1.35 - 4.20) and 3.01 (1.82 - 4.97) when predicting conversion from CHR-NP to CHR-P cite{xu2017predictive}. However, they did not present further information on other measures like false positive rate (FPR). end{