Communication is the process of expressing and sharing feelings, thoughts, and intentions with one another by verbal and various nonverbal means. Communication skills appear automatic but can pose severe challenges to individuals suffering from motor neuron disorders. The most devastating of motor neuron diseases is amyotrophic lateral sclerosis (ALS) , which is progressive and renders an individual motionless, severely affecting his or her communication ability .
Communication is the process of expressing and sharing feelings, thoughts, and intentions with one another by verbal and various nonverbal means. Communication skills appear automatic but can pose severe challenges to individuals suffering from motor neuron disorders. The most devastating of motor neuron diseases is amyotrophic lateral sclerosis (ALS) , which is progressive and renders an individual motionless, severely affecting his or her communication ability . As the disorder progresses, it destroys the respiratory and bulbar functions, forcing the individual to make vital decisions. If they opt for life and accept artificial respiration, they can no longer communicate verbally, and assistive communication devices that rely on nonverbal signals such as finger movement and gaze fixation are then used for communication . In ALS, the disorder progresses in most patients until the patient loses control of the last muscular response, usually the eye muscles, a condition known as completely locked-in state (CLIS) .
Brain–computer interface (BCI) represents a promising strategy to establish communication with paralyzed ALS patients [5–7], as it does not need motor control. BCI research includes invasive (implantable electrodes on or in the neocortex) [8–11] and noninvasive means, including electroencephalography [12,13], functional magnetic resonance imaging (fMRI) , and functional near-infrared spectroscopy (fNIRS) , to record brain activity for conveying the user’s intent to devices such as simple word-processing programs . The first BCI for communication in ALS patients with intact eye muscles was demonstrated by Birbaumer et al. (1999) . With at least intact eye muscles and the rest of the body paralyzed, the condition is known as locked-in state (LIS) . Since then, several invasive and noninvasive BCIs have been developed for communication in ALS patients. Noninvasive methods, namely slow cortical potential (SCP)-BCI [12,16,17], sensory motor rhythm SMR-BCI [18–20], and P300-BCI [21–24], have been utilized more frequently than invasive methods [25–27] for communication in people with ALS [6,12,27–30]. Irrespective of the types of BCI, during the BCI session patients selected letters or words after learning self-regulation of the particular brain signal or by focusing their attention to the desired letter or a letter matrix [21,23], and the attention-related brain potential selects the desired letter.
In a meta-analysis of the scientific literature of all ALS patients in CLIS , it was found that none of the existing techniques such as the P300 event-related brain potential (ERP), SCP, frequency analyses of various frequency bands of the electroencephalogram (EEG), and invasive electrocorticogram (ECoG) recordings  allowed reliable and meaningful communication with BCI. All BCI procedures mentioned above were based on effortful and explicit (conscious) voluntary control of a neuroelectric brain response such as learning with feedback and reward, during which patients learned to increase or decrease amplitudes of the SCP  to produce event-related desynchronization (ERD) of the central alpha-rhythm  to focus attention on a visually or auditorily presented sequence of letters in order to select a desired letter with the brain response. P300 [21,22,24] is also used in a similar manner to visually select a desired letter. The required activation of explicit-voluntary (controlled) attention in these BCI tasks, none of them resulting in stable learning of brain-based communication, prompted us to propose the theoretical psychophysiological notion of “extinction of goal directed cognition and thought”  in complete paralysis with otherwise intact cognitive processing. This theoretical account—certainly highly speculative in light of the complete lack of data about cognition and inner speech and motivational processing in CLIS—we substantiated with the failure to replicate initial positive reports about instrumental (volitional) [33,34] learning of autonomic responses in the curarized (paralysed) rat. The persistent incapability to replicate these experiments suggests that intact or partially intact motor functions and somatic-motor system mediation of autonomic functions (i.e., subtle postural or muscle tension changes to affect the desired physiological changes) is a mandatory requirement for instrumental learning and control of physiological functions. Theoretical views of this problem, like the one proposed, are not new but were expressed already in Greek philosophy by Aristotle  and by the philosophers of volition, particularly Arthur Schopenhauer in his monumental account of “Die Welt als Wille und Vorstellung”  (The World as Volition and Imagery) and during twentieth century learning theory [37,38]. Conscious of the fact that it is problematic to justify a theory on negative facts (lack of instrumental learning in the curarized rat and missing BCI control of BCIs requiring controlled attention in CLIS), we argue that  classical reflexive conditioning and learning might circumvent volitional effort in instrumental control. Thus, an experimental procedure involving processing of overlearned (“automatic”) questions (i.e., “Berlin is the capital of France,” “You are in pain”) asking for automatic cognitive processing only may fulfill this criterion. Thinking but not voluntary imagining affirmative “yes” and negative “no” to overlearned questions occurs effortlessly, such as automatic nodding of the head in a conversation: the extensive literature (mostly Russian) on semantic classical conditioning  and implicit attention and memory  provides ample support of this notion. However, for the case of patients in CLIS, one is faced with the dilemma that we cannot expect a learning curve characteristic of skill learning (usually exponential) or classical conditioning in a BCI task asking for overlearned “yes” and “no” responses as used on the present BCI system [29,30]. Patients were confronted in their lifetime with these questions (“Berlin is the capital of France”) before entering (or on the verge of) CLIS, and we can assume with certain confidence that no further learning at the time of assessment with the BCI is necessary and thus no learning curve can be expected. The same holds true for personal questions (“Your husband’s name is Joachim”). Thus, at an experimental level, it remains difficult to prove the speculation of intact classical conditioning but lack of instrumental voluntary learning in CLIS patients involved in a BCI task after entering CLIS. Only one patient in the literature  using an electrocorticographic-based BCI before and after transitioning from LIS to CLIS was published. This patient was unable to communicate with the BCI after entering CLIS. However, observation of a single case cannot serve as strong evidence comparable to the animal experiments [33,34] using curarization for the creation of reversible paralysis. We are also aware of the fact that a single case of a cognitively intact CLIS patient or curarized organism learning to instrumentally drive a BCI disproves our hypothesis. Experimental descriptions of patients with ALS or subcortical stroke in locked-in state (LIS) or who are severely paralysed using spiking frequency changes of motor neurons to move a robotic arm [10,11,41] cannot disprove our account. Patients in those studies [10,11,41] still had intact motor control of eye movements and some remaining muscles and thus could use the remaining muscular forces (somatomotor mediation) for instrumental learning and BCI control.
Because none of the BCI techniques outlined above are able to provide viable means of communication , the patients in CLIS due to ALS, without any muscular control, are rendered communicationless. We are then faced with the dilemma of defining communication in CLIS. Does it only mean to express one’s feelings, thoughts, and intentions in a fluent, automatized manner? Or, alternatively, does it mean to convey one’s intent or one’s feelings and thoughts to questions? As mentioned, all the existing BCIs rely on two elements: first, the neuroelectric signal (EEG or ECoG) control and second at least an intact eye muscle; the neuroelectric signal–based BCI did not work so far in patients in CLIS, in which eye movement control is lost.
A single case report by Gallegos-Ayala et al. (2014)  used fNIRS to measure and classify cortical oxygenation and deoxygenation following the “yes” or “no” thinking of the CLIS patient in response to true or false questions, respectively. The report described a CLIS patient with ALS achieving BCI control and “yes” and “no” communication to simple questions with known positive answers or negative answers and some open questions over an extensive time period. Although it was not spontaneous and voluntary, controlled communication, it at least enabled the individual without any means of communication to transmit “yes” and “no” to questions framed by family members and/or caregivers. The result opened a venue to provide at least some means of communication to individuals in CLIS who are otherwise left communicationless. Hence, an extensive study was performed on four ALS patients in CLIS to train them to communicate “yes” and “no.” In the present study, which is the first of its kind, fNIRS-based BCI was used for binary communication in four ALS patients in CLIS. The fNIRS-based BCI was employed successfully to train patients to regulate their frontocentral brain regions in response to auditorily presented questions. After training a classifier separating “yes” from “no” answers for several days, the patients were given feedback of their affirmative or negative response to questions with known answers and open questions over weeks.
The relative change in oxygenated hemoglobin (O2Hb) during the “true/yes” and “false/no” sentences’ interstimuli interval (ISI)—which corresponds to patients’ response interval over the frontocentral brain region of patients F, G, B, and W—are shown in Fig 1.
Fig 1. The averaged relative change in O2Hb corresponding to “yes” and “no” sentence interstimuli interval (ISI).
(A) Patient F, (B) patient G, (C) patient B, and (D) patient W. (E) Channel configuration: Eight sources and eight detectors placed on the frontocentral brain region translated into 20 channels, 10 on each side of each hemisphere. For clearly displaying the relative change in O2Hb, 10 channels on each side of hemisphere were further subdivided in groups of 5 channels—i.e., 20 channels were divided into four groups, each consisting of 5 channels. In each subplot, the x-axis is time in seconds and the y-axis is relative change in O2Hb, and the five different colored lines correspond to relative change in O2Hb across 5 different channels, as depicted in the channel configuration map. Fig 1 data is located at https://doi.org/10.5281/zenodo.192386; https://doi.org/10.5281/zenodo.192388; https://doi.org/10.5281/zenodo.192390; https://doi.org/10.5281/zenodo.192391. https://doi.org/10.1371/journal.pbio.1002593.g001
Fig 1 illustrates the change in O2Hb during “true/yes” sentences’ ISI—which is significantly different from the “false/no” sentences’ ISI—as corroborated by the t-test performed between the averages of true and false sentences’ ISI using the relative change in O2Hb across the four patients (p < 0.05), shown in Table 1, row D. The same analysis performed using the EEG signals in the time domain across all the training sessions showed no significant differences (p> 0.05) between the true and false sentences’ ISI across each patient, as shown in Table 1, row E. The eye movements measured with an electrooculogram (EOG) (vertical or horizontal; patients were free to use any direction) for patients F, G, B, and W while they were performing the “ja” (German word for yes) or “nein” (German word for no) thinking task showed no significant difference in the eye movements between the true and false sentences’ ISI for all patients, confirmed by the t-test (all p > 0.05), as shown in Table 1, row F, and S1B Fig.
Table 1. Statistics Results.
Total number of (A) training, (B) feedback, and (C) open question sessions performed by each patient. The total number of sessions averaged and degrees of freedom used to perform t-tests between the true and false sentences’ ISI corresponding to (D) O2Hb, (E) EEG, and (F) EOG signals. (G) ANOVA using support vector machine (SVM) classification accuracy (CA) of O2Hb, EEG, and EOG signals. Post hoc t-test performed between (H) O2Hb versus EEG, (I) O2Hb versus EOG, and (J) EEG versus EOG classification accuracy. Note that each session contains 20 questions: 10 asking for a “yes” and 10 semantically equivalent questions asking for a “no” answer. https://doi.org/10.1371/journal.pbio.1002593.t001
The support vector machine (SVM) classifier’s classification of true sentences as true, false sentences as false, true sentences as false, and false sentences as true was used to calculate the false positive rate (FPR) and true positive rate (TPR) for each training and feedback session and also for all the training sessions and feedback sessions separately for each patient. FPR and TPR was plotted to obtain the receiver operating characteristic (ROC) curve of the binary SVM classifier during training and feedback sessions for each patient, as shown in S2 Fig, S3 Fig, S4 Fig and S5 Fig for patients F, G, B, and W, respectively. The change in oxygenated hemoglobin (O2Hb), EOG, and EEG power spectrum in response to true and false questions, obtained from the frontocentral region of the brain, across all the sessions from each patient was used to determine the SVM classification accuracy of “true/yes” and “false/no” answers. Successively, the daywise CA (i.e., averaging CA of all sessions in a single day) of each patient was compared to the adjusted chance-level threshold (described in BCI effectiveness metric section), as shown in Figs 2, 3, 4 and 5 for patients F, G, B, and W, respectively, and Table 2. The offline CA is reported using O2Hb, EEG, and EOG signals for training sessions, while online CA is reported using only O2Hb for feedback and open question sessions because feedback was provided online only using the O2Hb signal. The answering concordance between semantically paired questions (“Paris is the capital of Germany,” “Paris is the capital of France”), expressed as the percentage of concordant answers over pairs’ repetition, was as follows: F, 68%; G, 67%; B, 67%; W, 70%. Thus, the semantic concordance rate (SCR) ranges from 67 to 78% (see S12 Table). Median values of SCR are significantly different from 50% (all p < 0.0001), in which 50% is the SCR expectation of a random classifier.
Fig 2. Classification accuracy of Patient F.
Linear SVM CA across “training sessions—offline CA” (histogram in grey), “feedback sessions—online CA” (green dot), and “open question session—online CA” (plus sign in red), obtained using (A) relative change in O2Hb, (B) EEG, and (C) EOG data. The classification accuracy reported here is daywise, as all the “training sessions” in a day were used to calculate the average classification accuracy of all the “training sessions” in a day. In the figure panels A, B, and C, the x-axis is the number of days and the y-axis is the classification accuracy. The solid black and dotted horizontal lines represent the chance-level threshold calculated using the metric described in the BCI effectiveness metric section for “training sessions” and “feedback sessions,” respectively. Since the feedback during the feedback and open question sessions was provided using the O2Hb, the online CA of the feedback and open question sessions is reported only for the fNIRS data. Fig 2 data is located at https://doi.org/10.5281/zenodo.191884. https://doi.org/10.1371/journal.pbio.1002593.g002
Fig 3. Classification accuracy of Patient G.
The description of this figure is the same as described in Fig 2. Fig 3 data is located at https://doi.org/10.5281/zenodo.191887. ttps://doi.org/10.1371/journal.pbio.1002593.g003
Fig 4. Classification accuracy of Patient B.
The description of this figure is the same as described in Fig 2. Fig 4 data is located at https://doi.org/10.5281/zenodo.191891. https://doi.org/10.1371/journal.pbio.1002593.g004
Fig 5. Classification accuracy of Patient W.
The description of this figure is the same as described in Fig 2. Fig 5 data is located at https://doi.org/10.5281/zenodo.191899. https://doi.org/10.1371/journal.pbio.1002593.g005
Table 2. Classification statistics.
(A) Number of days with CA above chance level; (B) number of delivered sentences with CA above chance level; (C) maximum chance-level upper limit, calculated using the metric described in BCI effectiveness metric section; and (D) mean and standard deviation of CA above chance level, obtained using the fNIRS, EEG, and EOG signal data for patients F, G, B, and W. For the training sessions, the CA is reported for fNIRS, EEG, and EOG signals. For feedback and open question sessions, CA is only reported for the fNIRS signal because the feedback was provided using only the fNIRS signal as a result of superior CA of the fNIRS signal during the training sessions as compared to EEG and EOG signals. Table 2 data is located at https://doi.org/10.5281/zenodo.191884; https://doi.org/10.5281/zenodo.191887; https://doi.org/10.5281/zenodo.191891; https://doi.org/10.5281/zenodo.191899. https://doi.org/10.1371/journal.pbio.1002593.t002
The results of ANOVA and post hoc t-test (see Table 1, row G, H, I, and J) further emphasize a significant difference between the classification accuracy of O2Hb versus EEG and O2Hb versus EOG, with no significant difference between EEG and EOG. There is one exception in the case of patient W, as no significant difference was found between the classification accuracy of O2Hb versus EOG. Patient W (23 y of age) suffering from juvenile ALS with an extremely rapid disease progression (2 y from diagnosis to CLIS) was not asked open questions because of the supposedly difficult emotional state at that early stage of BCI communication but continues to train the BCI at present.
The patients showed the following stable dominant frequencies: F, 6.75 Hz; G, 6.25 Hz; B, 7 Hz; and W, 8 Hz. Power spectrum density of electroencephalographic signals corresponding to “true/yes” and “false/no” sentences’ ISI acquired from channel FC6 is shown in S1 Fig. The middle-frequency bands’ (high-theta, low-alpha, and high-alpha) mean power comparison between “true/yes” and “false/no” sentences’ ISI revealed no main effects of conditions and channels in any patient (all p > 0.05). The middle-frequency bands’ (high-theta, low-alpha, and high-alpha) spectral features comparison between sentence presentation interval and sentences’ ISI revealed some main effects of the intervals factor, as reported in S1 Table, section A. In two (G and B) out of four patients, a smaller low-alpha band “power variability” in the sentences’ ISI compared to the sentence presentation interval was found (p < 0.05). In patient W, a higher high-theta, low-alpha, and high-alpha bands’ mean power in the sentences’ ISI compared to the sentence presentation interval was found (ISI was synchronized compared to sentence presentation). Patient F did not show any significant difference in the middle-frequency bands’ mean power and “power variability” (all p > 0.05). See Results section of S1 Text for details.
The correlation analysis between fNIRS classification accuracy and low-frequency bands’ (those more related to vigilance) mean power revealed some interesting results (see S1 Table, section B). In three (G, B, and W) out of four patients, the median of the negative averaged correlation between low-theta band mean power and fNIRS classification accuracy was significantly different from zero (patient G: r = –0.365; patient B: r = –0.264; patient W: r = –0.386; all p < 0.05). However, in patient F, who had the longest time period in CLIS, the median of the positive averaged correlation between delta and high-theta band mean power and fNIRS classification accuracy was significantly different from zero (delta: r = 0.233; high-theta: r = 0.213; all p < 0.05). The low-frequency bands’ mean power distribution medians of successful and unsuccessful days (i.e., days with classification accuracy above chance-level threshold were considered successful) was further investigated for each patient to ascertain the difference, if any. In patient G, the low-theta band mean power of successful days was significantly smaller than that of unsuccessful days (p < 0.05). In patient B, the high-theta band mean power of successful days was significantly smaller than that of unsuccessful days (p < 0.05). This strengthens the above results of lower-frequency bands’ dominance for more unsuccessful performance. Additional details are provided in the Results section of S1 Text and in S1 Table.
Four patients in CLIS communicated with frontocentral cortical oxygenation-based BCI with an above-chance-level correct response rate over 70% during a period of several weeks. The performance of the binary SVM classifier across all the patients, except a few training sessions of patient B, was above chance level. None of the sessions were eliminated in the analysis, and only very few sessions had to be interrupted because of life-saving measures such as sucking saliva; thus, no bias for selecting “successful” sessions incriminates the results. Correct response rate for feedback and open questions sessions, as judged by the criteria mentioned in the Material and Methods section (Experimental Procedures), exceeded 75% in three out of four patients (F: 78.6%; G: 78.8%; B: 75.8%). Patients F, G, and B answered open questions containing quality of life estimation repeatedly with a “yes” response, indicating a positive attitude towards the present situation and towards life in general, as reported in larger samples of ALS patients [43,44]. Repeated presentation of an open question is necessary to ascertain the validity of the answer. From the ROC curve of each patient, it can be deduced that if the patient answers a question seven out of ten times with the same answer, then we can be sufficiently certain of the answer if the questioning is repeated over a long time period as done here. Correct classification of “yes” and “no” answers given mentally through fNIRS exceeded classification of EEG oscillations from 0–30 Hz and vertical and horizontal EOG classification. However, despite the absence of reliable eye communication in all patients as the inclusion criteria in the study and by the definition of CLIS condition, EOG classification was at some sessions, albeit rarely, above chance, mainly in patient W. Nonetheless, the inability of the social environment to perceive them and their instability and the eye tracker’s failure to use them for communication  prevents the use of this physiological signal.
The unreliable discrimination between “true/yes” and “false/no” sentences’ ISI by means of EEG signals (see Results section of S1 Text for details) is consistent with the results of a very similar auditory paradigm used for discriminating delayed, conditioned brain responses and tested in fourteen healthy participants . However, with the exception of patient F, the comparison of middle-frequency bands’ spectral features (averaged across days) between sentence presentation interval and sentences’ ISI confirms the fact that two different states of arousal were present during sentence presentation and sentences’ ISI (see Results section of S1 Text for details). We cannot infer about the “qualia” of the two specific brain states occurring during listening to “yes–no” questions and executing the answers mentally, but at least we can state that the two mental states were different, meaning that differential cognitive processing occurred during the BCI task [47,48]. We do not have a physiologically plausible explanation as to why fNIRS responses to “yes” and “no” are different, as they are in three patients showing oxygenation increase during the answering interval (ISI) for “yes” responses with negligible topographical differences and oxygenation decrease during negative answers, again without topographical differences throughout the frontal cortical area. Only the very young patient W, with an extremely rapid course of the disease and a strong genetic variant, shows a variable fNIRS pattern with different slow oscillatory oxygenation changes between “yes” and “no” answering periods. In an animal study with nonhuman primates , we identified oxygenation increase as highly correlated with nearby recorded multiunit activity. Assuming a similar situation in the human brain, a “yes” answer may indicate a more coherent and more active brain state, probably supporting cellular associative binding [50–52] more readily than negative answering states. However, such a generalization remains highly speculative.
Three patients (G, B, and W) showed a negative averaged correlation between low-theta band mean power and fNIRS classification accuracy , meaning the smaller the low-theta band mean power, the higher the performance, except in patient F, who has been in CLIS for more than 4 y. The correlations analysis between daywise classified BCI performance and low-frequency bands across all intervals and all electrodes for each patient separately gave consistent and highly significant results (see Results section of S1 Text for details). The binary communication performance worsened with lower frequencies in two patients (G and B) as predicted. The number of days for patient W was limited (i.e., 6 d), thus it is quite unlikely to expect a significant difference in low-frequency bands’ mean power for successful and unsuccessful days (see S1 Table, section B). Decrease in vigilance reflected in slow frequencies impedes BCI performance and communication. Patient F, who had an extremely long history of CLIS without any communication over the years, showed a positive correlation in the delta and high-theta band with performance. She was the patient with very slow dominant frequency during rest, and it may be speculated that in such a deprived brain, superposition of delta and high-theta frequency represent a sign of increased attention and focus. For instance, low-theta band mean power can be used in future BCIs to stop a BCI session or to avoid the presentation of the sentences and/or questions during decreased vigilance.
For a robust validation of the BCI binary communication system in CLIS, two main unsolved questions remain: (i) the physiological identification of the cognitive processes underlying the listening to “yes–no” questions and the answerer’s mental state and (ii) the online identification of decreased vigilance states that are detrimental (lowering performance) for BCI binary communication purposes, such as decreased alertness, drowsiness, and sleeping.
Multielectrode EEG recordings used simultaneously with the fNIRS system and quantitative source analysis of the different frequency bands at different sites are necessary to clarify these questions. For the study reported here, only portable devices and a few EEG channels could be used in the interest of the bedside, home-based strategy selected. Thus, our interpretation of the EEG frequency bands’ variations remains speculative.
In patients completely motionless over a period of years with restricted vision because of eye muscle paralysis and compromised vision because of drying and reduced or absent afferent input from the sensorimotor system, reduced vigilance measured with EEG and an irregular sleep–wake cycle was documented by Ramos et al. (2011)  and Soekadar et al. (2013) . De Massari et al. (2013)  have shown that reduction of P300 amplitude across the BCI paradigm presentation predicted negative performance, again suggesting excessive loss and excessive variation of wakefulness and attention as a major limiting factor for BCI applications in such severely compromised patients. Thus, we modified the existing fNIRS–BCI–system in a hybrid EEG–fNIRS–BCI, with the EEG allowing online corrections of excessive reduction of vigilance indicated by appearance of delta and low-theta periods. This new hybrid system should allow further improvement of communication in CLIS.
The results on four CLIS patients reported here allow the following conclusions:
The Internal Review Board of the medical faculty of the University of Tubingen approved the experiment reported in this study, and the patients’ legal representative gave informed consent for the study with permission to publish the results and show the face of patients in the publication. The study was in full compliance with the ethical practice of medical faculty of the University of Tubingen. The clinical trial registration number is ClinicalTrials.gov Identifier: NCT02980380. At the time of this study, prospective clinical trial registration was not mandatory for nonpharmacological studies; it was therefore registered retrospectively.
A continuous wave (CW)-based fNIRS system, NIRSPORT (NIRX), which performs dual-wavelength (760 nm and 850 nm) CW near-infrared spectroscopic measurement at a sampling rate of 6.25 Hz, as shown in in Fig 6A, was used. The NIRS optodes were placed on the frontocentral regions as shown in Fig 6B. During the BCI sessions, the EEG was also recorded with a multichannel EEG amplifier (Brain Amp DC, Brain Products, Germany) from ten Ag/AgCl passive electrodes mounted on the head cap. Six electrodes (FC5, FC1, FC6, CP5, CP1, and CP6) were used to acquire EEG signals and four electrodes were used to acquire the vertical and horizontal EOGs. The signals were bandpass filtered using a finite impulse response filter with a bandpass of 0.5–30 Hz. The EOG was filtered with different bandpass filters (0.5–3.5 Hz, 0.5–10 Hz, and 0.5–30 Hz), but none of these filters led to significant differences of neurophysiological patterns related to the ocular activity. Question- or response-related eye movements were not detected in any of the patients over the whole time period of many weeks. Each EEG channel was referenced to an electrode on the right mastoid and grounded to the electrode placed at Fz location of the scalp. Electrode impedances were kept below 10 kΩ and the EEG signal was sampled at 500 Hz. During all BCI sessions, the spontaneous EEG was visually controlled by one of the authors (NB or BX) to avoid longer periods of slow-wave sleep during the BCI evaluation. A BCI session was initiated only if the EEG was free of high-amplitude slow activity below 4 Hz.
Fig 6. The procedure and flow diagram of the brain–computer interface (BCI) for communication in ALS patients.
(A) The continuous wave-based portable NIRX NIRSport instrument. The device consists of eight near-infrared light sources (highlighted in red), eight detectors (highlighted in green), and the NIRS data acquisition hardware is highlighted in yellow. (B) Depicts the placement of sources and detectors (optodes) on the frontocentral region of the scalp (blue). Four sources (highlighted in red) and four detectors (highlighted in green) were placed on each hemisphere to form a channel. https://doi.org/10.1371/journal.pbio.1002593.g006
Patient F (female, 68 y old, completely locked-in state) was diagnosed with bulbar sporadic ALS in May 2007, was diagnosed as locked-in in 2009, and was diagnosed as completely locked-in May 2010, based on the diagnoses of experienced neurologists. She has been artificially ventilated since September 2007, fed through a percutaneous endoscopic gastrostomy tube since October 2007, and is in home care. No communication with eye movements, other muscles, or assistive communication devices was possible since 2010. Further details of this patient are described in Gallegos-Ayala et al. (2014) .
Patient G (female, 76 y old, CLIS) was diagnosed with bulbar ALS in 2010. She lost speech and capability to walk by 2011. She has been fed through a percutaneous endoscopic gastrostomy tube since September 2011, artificially ventilated since March 2012, and is in home care. She started using assistive communication devices employing one finger for communication in February 2013. Later, she was diagnosed with degeneration of vision because of cornea defects in September 2013. After the failure of the finger-communication device, an attempt was made to communicate using eye tracking in early 2014. She stopped communicating with the eye in August 2014, before the BCI was introduced, and an attempt was made to communicate with the subtle twitch of an eye lid, which was not reliable. The husband and caretakers declared no communication with her since August 2014.
Patient B (male, 61 y old, CLIS) was diagnosed with nonbulbar ALS in May 2011. He has been artificially ventilated since August 2011, fed through a percutaneous endoscopic gastrostomy tube since October 2011, and is in home care. He started communicating with a speech device in his throat from December 2011, which ultimately failed, and he started using the MyTobii eye-tracking device in April 2012. He was able to communicate with MyTobii until December 2013, after which the family members attempted to communicate by training him to move his eyes to the right to answer “yes” and to the left to answer “no,” but the response was variable. No communication was possible since August 2014.
Patient W (female, 24 y old, locked-in state on the verge of CLIS) was diagnosed with juvenile ALS in December 2012. She was completely paralyzed within half a year after diagnosis and has been artificially ventilated since March 2013, fed through a percutaneous endoscopic gastrostomy tube since April 2013, and is in home care. She was able to communicate with eye tracking from early 2013 to August 2014 but was unable to use the eye-tracking device after the loss of eye control in August 2014. After August 2014, family members were able to communicate with her by training her to move her eyes to the right to answer “yes” and to the left to answer “no” questions until December 2014. In January 2015, eye control was completely lost, she tried to answer yes by twitching the right corner of her mouth, that too varied considerably, and parents lost reliable communication contact.
All four patients reported in this manuscript were enrolled consecutively. Patients’ family approached us to get enrolled in the study because of the past work and public appearance of the corresponding author. Patients were never screened and excluded for this study. The only criterion for the inclusion in this study was that the patient should be in completely locked-in state (CLIS) or on the verge of CLIS, and family members could not communicate with eye movements or any other response with the patient. The CLIS state was then verified with confirmation of the attending neurologist, EOG recordings, and video recordings of the families’ failures to achieve contact with the patient.
The schematic depicting the experimental procedure, acquisition, and analysis of fNIRS and EEG data during BCI sessions is shown in Fig 6.
An auditory paradigm was employed to (a) train patients on questions with known answers, termed as training sessions; (b) give feedback on questions with known answers, termed as feedback sessions (i.e., “Your husband’s name is Joachim,” and after classification during ISI: “your answer was recognized as ‘yes’/‘no’); and (c) answer open questions, termed as open question sessions (“You have back pain”). Known questions are personal questions based on patient’s biography. For every known question with a clear “yes” answer, a semantically related question with a clear “no” answer was constructed and vice versa; for example, “You were born in Berlin” and “You were born in Paris.” Patients were asked to think “yes” or “no” answers and, if possible, also to use their previously successful eye movements. They were explicitly instructed not to imagine the answer or visually or auditorily imagine the word (i.e., as a visual or sound form) “yes” or “no”. Open questions are general questions related to quality of life and questions of caretakers whose answers can only be known by the patient. A total of at least 200 known questions and 40 open questions were constructed for each patient with family members before the initiation of the BCI study. Each patient was visited for 4 to 5 d in a month, except patient W. Three to four sessions were performed each day depending upon the health condition reported by the caretakers of the patient. Every session lasted for 9 min, and a session in progress was terminated extremely rarely (i.e., if removal of saliva became urgent). In such a rare event, the session was started again. Since each session lasted for 9 min, the caretaker or the family member was always instructed to take care of the needs of the patient before the start of the session, and the session was always started with the permission of the caretaker or the family member. A session, once in progress, was never terminated for patients F, G, and W. For patient B, a session was terminated while in progress three times because of removal of saliva, and the data were not included in any kind of analysis.
Acoustically presented instructions about the procedure were given repetitively before each training, feedback, and open questions sessions, allowing patients to recall and consolidate the required task to listen and answer mentally. Each BCI session started with training sessions, during which the patients were instructed to listen to 20 personal questions (with known answers) consisting of 10 true and 10 semantically equivalent false sentences. The sentences were presented randomly in such a way that two semantically related questions never played one after another. Family members were always present throughout the BCI session, and they never prompted the patient to answer the question. Complete pin-drop silence was maintained during the session, and only the recorded sentences were presented via audio presentation software connected to sound box with the voice of a family member or caregiver. Patients were asked to think “ja, ja…” (German for “yes”) and “nein, nein…” (German for “no”) for 15 s during the ISI until they heard the next sentence after an interval of 5 s of rest, as shown in Fig 6. After the end of each session, the fNIRS feature necessary to differentiate between “yes” and “no” answers during ISI was extracted and classified. Only training sessions were performed during the first few days, and upon several successful training sessions (as described below in BCI effectiveness metric section), the online feedback session was performed. During training sessions, both the patient and the algorithm were trained. Patients learned to mentally answer the question, and the algorithm learned to classify the “yes” and “no” fNIRS pattern of a particular patient. This kind of “mutual learning” seems important to optimize the “yes” versus “no” classification outcome and to customize and/or adapt the BCI system to each individual patient. At the end of each training session with 20 sentences (questions), patients were told the average classification accuracy of the session (calculated using the SVM classifier) to motivate and help patients in learning. In the course of an online feedback session, patients were presented the known questions as described above, but now at the end of the 15 s ISI they were given auditory feedback of accuracy, during which the computer said, “Your answer was recognized as yes” or “Your answer was recognized as no” depending upon the question (all sessions were videotaped and are available on request). Feedback to strengthen the conditioned response was provided only if the classification accuracy was greater than the chance-level upper limit to guide the conditioned learning toward meaningful answers and to avoid frustration by negative feedback already at the beginning of a daily session. Feedback was driven by the fNIRS classifier, calculated using the data acquired during the training sessions. After successful training and feedback sessions, the patients were presented with open questions, during which they were always given the auditory feedback of their answer.
The validity of answers to open questions can only be estimated by (a) face validity (i.e., questions of pain in the presence of an open wound); (b) stability over time; (c) external validity, estimated by family members and caretakers; and (d) internal validity between questions (i.e., the concordance between the answer to “I love to live” with the answer to “I rarely feel sad” [presented to all patients—except W—regularly]). Table 1, rows A, B, and C enumerate the total number of training, feedback, and open questions sessions performed by each patient, respectively. Patient W received no open questions because of low classification accuracy, which we and the parents attributed to her emotional state distracting her from concentrating on the responses because of the short time period of adaptation to the CLIS.
The binary BCI system effectiveness and robustness depends on its capability of correctly classifying the neurophysiological correlates of “yes” and “no” answers to true and false questions. The proposed true and false questions have two possible outcomes only, which are equally distributed with a probability of 0.5. To ensure that the classification of “yes” and “no” answers is not at chance-level, a reliable metric has to be used. Based on binomial distribution theoretical background, Müller-Putz et al.  defined a metric for experimental procedures with a binary outcome and multiple repetitions to determine the chance-level threshold above which the classification accuracy results can be considered as not resulting from chance. Because type and number of questions (personal questions with known answers and open questions) are partly different over days (i.e., the experimental conditions were different) the chance-level threshold was calculated on a daily basis. The daily-based chance level was computed using the formulas described in Müller-Putz et al.  and by taking into account the number of true and false sentences presented in a single day to each patient.
The fNIRS data was acquired online throughout all the sessions, namely training, online feedback, and open question sessions. The fNIRS data acquired online was normalized, filtered using different bandpass filters (0.0016–0.3), (0.01–0.3) and (0.02–0.3) and processed using modified Beer–Lambert law [60,61] to calculate the relative change in concentration of oxyhemoglobin (O2Hb) and deoxyhemoglobin (RHb). The choice of bandpass filter had no effect on the waveforms of signal. The relative change in O2Hb computed online during each session was used to train a SVM classifier model. The mean of relative change in O2Hb across each channel was used as a feature to train the SVM model through a 5-fold cross-validation procedure. In this study, only the relative change in O2Hb was used, as after the end of sessions with known answers it was observed that O2Hb provided stable and higher cross-validation classification accuracy than RHb. In an invasive animal study with nonhuman primates, we have also measured a superior covariation of oxygenation changes compared to deoxygenation, with intracortically recorded neural activity  supporting this clinical observation. Since the classification accuracy achieved was higher for O2Hb, the SVM model generated using O2Hb was used to provide online feedback for known as well as open questions sessions. If the classification accuracies for at least three consecutive “training” sessions with questions with known answers were greater than the chance-level threshold, a new model was generated using the relative change in O2Hb across three training sessions to give online feedback. During an online feedback session, fNIRS data acquired online corresponding to each ISI was processed to obtain the relative change in O2Hb, as described above, across all the channels. The mean of the relative change in O2Hb across all the channels was used as test feature to map onto model space. Upon mapping of this test feature onto the model space, the SVM predicted (called predict label) the side of the hyperplane the test feature fell on. Depending on the value of the predict label, appropriate feedback was provided to the patient: if the predict label was 0, the patient was given feedback that his or her answer was recognized as “no,” and if the predict label was 1, the patient was given feedback that his or her answer was recognized as “yes.”
fNIRS provides three different signals: oxyhemoglobin (O2Hb), deoxyhemoglobin (RHb) and total hemoglobin (THb) [60,61]. As mentioned in the section Online data analysis, since the classification accuracy achieved was higher for O2Hb, only the results from the offline processing of O2Hb data will be shown along with the EEG and EOG data. The relative change in O2Hb, EEG, and EOG data were processed offline to determine:
To ascertain the difference between the averaged ISI of true and false sentences, t-tests were performed. t-test was performed separately for O2Hb, EEG, and EOG signals acquired from all the sessions and across all the channels in a session, averaged over many sessions varying slightly between patients. Furthermore, t-tests were also performed for each session between the ISI of all the ten true sentences and all the ten false sentences (“Berlin is the capital of France,” “Berlin is the capital of Germany”) across different channels in a session.
For the EEG, frequencies between 0 and 30 Hz, estimated by Welch’s method , were used for classification and statistical testing. ANOVA and post hoc t-test were used.
Frequency bands’ mean power and their “variability” were estimated using Welch’s method [44,58]. For each patient, middle-frequency bands’ (i.e., high-theta, low-alpha, and high-alpha) features of “true/yes” and “false/no” sentences’ ISI were compared, as well as middle-frequency bands’ features of sentence presentation and interstimulus intervals. Successively, the averaged correlation between each low-frequency band (i.e., delta, low-theta, and high-theta) mean power and fNIRS classification accuracy was computed to find relevant relationships of low EEG rhythms with the BCI experimental procedure outcome. Details are provided in the Methods section of S1 Text.
The performance of the binary SVM classifier was ascertained by plotting the ROC curve. The ROC curve was created by plotting the TPR against the FPR (obtained from the contingency table created for each session) and the average of all the sessions, separately for each patient, using the four possible outcomes of a binary SVM classifier. The formation of contingency table for training and feedback sessions for each participant is described in the Receiver operating characteristic curve section of S2 Text. Further chi-square test was performed to determine the statistical significance of the observed outcomes in the contingency table, also described in the Receiver operating characteristic curve section of S2 Text.
Semantic concordance rate (SCR) was calculated to ascertain the consistency and/or concordance of the answers between semantically equivalent but contrasting true and false sentences requiring “yes” and “no” answers, respectively. SCR (i.e., the percentage of concordant answers over pairs’ repetition) was calculated for all semantically related sentences presented to each patient. The method employed to calculate the semantic concordance rate is described in the Semantic concordance rate (SCR) section of S2 Text. This measure also provides indirect information about the intact cognitive processing of the presented sentences in a CLIS patient.
You must log in to like an article
Communication is the process of expressing and sharing feelings, thoughts, and intentions with one another by verbal and various nonverbal means. Communication skills appear automatic but can pose severe challenges to individuals suffering from motor neuron disorders. The most devastating of motor neuron diseases is amyotrophic lateral sclerosis (ALS) , which is progressive and renders an individual motionless, severely affecting his or her communication ability .