Predicting subsequent memory from single-trial EEG Eunho Noh1 , Grit Herzmann2 , Tim Curran3 , and Virginia R. de Sa4 1

Department of Electrical and Computer Engineering, University of California, San Diego [email protected] 2 Department of Psychology, The College of Wooster 3 Department of Psychology and Neuroscience, University of Colorado Boulder 4 Department of Cognitive Science, University of California, San Diego

Abstract. We show that it is possible to successfully predict subsequent memory performance based on single-trial EEG activity before and during item presentation in the study phase. Many studies have shown that the EEG signals during encoding of pictures or words are different between the later remembered vs. forgotten trials [11, 9]. In addition to brain activity during encoding, it has also been found that the signals preceding the stimulus presentation are different between the later remembered vs. forgotten trials [7, 8, 10, 3, 2]. These differences in brain activity between the subsequently remembered and forgotten trials are often referred to as subsequent memory effects (SMEs). EEG for this study was previously recorded during a visual memory task [4]. The experiment was divided into 8 blocks where each block consisted of a study phase and a recognition phase. In the study phases, the participants were given pictures of birds and cars in different blocks. In the recognition phases, they had to discriminate these target items from random new items using a rating scale with 5 options (recollect, definitely familiar, maybe familiar, maybe unfamiliar, and definitely unfamiliar ). The subjects were instructed to give recollect responses only when they had a conscious recollection of learning the item in the study phase. Two-class classification was conducted on the recollected vs. unfamiliar trials by combining the pre- and during-stimulus information in the EEG signal. The pre-stimulus classifier utilized the spectral information in multiple frequencies bands ranging from 4-40 Hz in the pre-stimulus period to predict good and bad brain states for memory encoding. The duringstimulus classifier combined the temporal and spectral information in the alpha band (7-12 Hz) during study item presentation to predict whether the encoding process was successful or not. The results from the individual classifiers were then combined to predict subsequent memory for each trial. By combining the pre- and during-stimulus classifier outputs, we were able to achieve an overall classification accuracy (calculated for all trials from the 18 subjects available for the classification analysis) of 59.6 %. The subject with the highest classification rate showed an accuracy of 71.1 % and the subject with the lowest classification rate showed an accuracy of 51.8 %. The combined classification results showed a 2 % increase in performance from the individual pre- and during-stimulus classifier results. The pre-stimulus and during-stimulus classifiers each gave individual classification results significantly over chance with p < 0.05 for 9 subjects (where the threshold for chance performance was defined based on the total number of trials for each subject [5]). The combined classification results gave significantly over chance results with p < 0.05 for 13 subjects out of the 18 subjects. A passive BCI system based on these classifiers could be developed to augment tutoring tools and tailor study scheduling to each individual’s brain dynamics. The system would measure the brain activity of a user in order to infer the user’s preparedness for learning and present study items at estimated optimal times. It would also monitor the brain activity during learning/encoding to assess whether the encoding process was successful or not. Items deemed unsuccessfully encoded could be re-presented for restudy purposes. As a long-term goal, we would like to explore the effects of long term use of our system. We hypothesize that the implicit neurofeedback users would get from being presented study items only during predicted good brain states (for memory encoding) could help them learn to get in and remain in a receptive brain state more often. Thus it is possible that the system could be used to train students to be more effective learners. As requested in the Workshop on Utilizing EEG Input in Intelligent Tutoring Systems call for papers, the majority of the work has been previously published [6].

2

ACKNOWLEDGEMENTS This research was funded by NSF grants # CBET-0756828 and # IIS-1219200, NIH Grant MH64812, NSF grants # SBE-0542013 and # SMA-1041755, and a James S. McDonnell Foundation grant, and the KIBM Innovative Research Grant. We would like to thank Dr. Marta Kutas and Dr. Tom Urbach for helpful comments.

References 1. Agresti, A., Caffo, B.: Simple and effective confidence intervals for proportions and differences of proportions result from adding two successes and two failures. The American Statistician 54(4), 280–288 (2000) 2. Fell, J., Ludowig, E., Staresina, B.P., Wagner, T., Kranz, T., Elger, C.E., Axmacher, N.: Medial temporal theta/alpha power enhancement precedes successful memory encoding: evidence based on intracranial EEG. Journal of Neuroscience 31(14), 5392–5397 (2011) 3. Guderian, S., Schott, B.H., Richardson-Klavehn, A., Duezel, E.: Medial temporal theta state before an event predicts episodic encoding success in humans. Proceedings of the National Academy of Sciences 106(13), 5365–5370 (2009) 4. Herzmann, G., Curran, T.: Experts’ memory: an ERP study of perceptual expertise effects on encoding and recognition. Memory & Cognition 39(3), 412–32 (2011) 5. M¨ uller-Putz, G., Scherer, R., Brunner, C., Leeb, R., Pfurtscheller, G.: Better than random? a closer look on BCI results. International Journal of Bioelectromagnetism 10(1), 52–55 (2008) 6. Noh, E., Herzmann, G., Curran, T., de Sa, V.R.: Using single-trial eeg to predict and analyze subsequent memory. NeuroImage 84, 712–723 (2014) 7. Otten, L.J., Quayle, A.H., Akram, S., Ditewig, T.A., Rugg, M.D.: Brain activity before an event predicts later recollection. Nature Neuroscience 9(4), 489–491 (2006) 8. Otten, L.J., Quayle, A.H., Puvaneswaran, B.: Prestimulus subsequent memory effects for auditory and visual events. Journal of Cognitive Neuroscience 22(6), 1212–1223 (2010) 9. Paller, K.A., Wagner, A.D.: Observing the transformation of experience into memory. Trends in Cognitive Sciences 6(2), 93–102 (2002) 10. Park, H., Rugg, M.D.: Neural correlates of encoding within- and across-domain inter-item associations. Journal of Cognitive Neuroscience 9, 2533–2543 (2010) 11. Sanquist, T.F., Rohrbaugh, J.W., Syndulko, K., Lindsley, D.B.: Electrocortical signs of levels of processing: perceptual analysis and recognition memory. Psychophysiology 17(6), 568–576 (1980)