12th GIST BCI Workshop  Date : 2014. 9. 26 (Friday), 15:30 ~  Place : SIC Bldg. B 206

 Organizer : Donghyeon Kim

1

Program ► Oral presentation time (40 min.) = Presentation (20min.) + Discussion (20min.)

Time

Schedule

Contents

15:30~15:40

Introduction

Greetings

15:40~16:20

Sangtae Ahn

Inter-brain synchronization

16:20 ~ 16:30

Coffee break

16:30 ~ 17:10

Yongwoo Lee

Android application for wireless BCI system

17:10 ~

Discussion

Announcement & Schedule for next workshop

2

Notice ► International conference ■ 7th International IEEE EMBS conference on Neural Engineering  Conference date, place : April 22-24, 2015 , Montpellier, France  Submission deadline : October 15, 2014

■ 3rd International Winter Workshop on Brain-Computer Interfaces XX  Conference date, place : January 12-14, 2015, High1 Resort, Korea  Submission deadline : October 15, 2014

■ 40th International Conference on Audio Speech and Signal Processing (ICASSP) 2015  Conference date, place : April 19-24, 2015, Brisbane, Australia  Submission deadline : October 5, 2014

■ HCI International 2015 (HCII)  Conference date, place : August 2-7, 2015, Los Angeles, CA, USA  Submission deadline : October 17, 2014 3

Notice ► Journal special issue ■ Multimodal Modeling and Analysis Informed by Brain Imaging  IEEE Transactions on Autonomous Mental Development  Manuscript Due: October 10, 2014

■ Advances in Brain-Computer Interface  Publisher : Hindawi Publishing Corporation  Manuscript Due : 7 Nov, 2014  Further information : http://www.hindawi.com/journals/tswj/si/387612/cfp/

4

Planning of next workshop ► Content changes ■ Research  Evgenii Kim: MSPOC  Thien: Correlation btw hemodynamic changes & neuro-activity

■ Discussions  None Order

Expected date

Organizer

Presenter

9th

2014-06-xx

Hohyun Cho

10th

2014-07-xx

Younghak Shin

Kyungae Yoon, Seungchan Lee

11th

2014-08-xx

Sangtae Ahn

Hohyun Cho, Donghyeon Kim

12th

2014-09-26

Donghyeon Kim

Yongwoo Lee, Sangtae Ahn

13th

2014-10-31

Seungchan Lee

Evgenii Kim, Thien

14th

2014-11-xx

Hohyun Cho

15th

2014-12-xx

Younghak Shin

16th

2015-01-xx

Sangtae Ahn

Minkyu Ahn, Younghak Shin

5

BioCompuing Lab.

Inter-brain synchronization

Sangtae Ahn Ph. D. Candidate [email protected] 14. 09. 26

BioCompuing Lab.

Contents  

Hyper-scanning History • • • •

   

fMRI hyper-scanning EEG NIRS MEG -

Synchronization Basic methods Neuromarker during interaction : phi complex and TPJ Conclusion and future direction

BioCompuing Lab.

HYPER-SCANNING

BioCompuing Lab.

Hyper-scanning 

What does hyper-scanning means? • Simultaneous recording of multiple subjects • Measurement of social interaction btw subjects http://labs.vtc.vt.edu/hnl/hyperScanFlash.html

MEG

BioCompuing Lab.

HISTORY

BioCompuing Lab.

History 

First hyper-scanning study using two fMRIs (Montague et al., Neuroimage 2002) • •

Synchronization Sender & Receiver

BioCompuing Lab.

History 

Hyper-scanning fMRI studies • • • • • • • •



(Montague et al., 2002) (King-Casas et al., 2005) (Tomlin et al., 2006) (Fliessbach et al., 2007) (Chiu et al., 2008) (Polosan et al., 2011) (Anders et al., 2011) (Spiegelhalder et al., 2014)

- Handy-dandy game - Trust game - Trust game - Trust game - Trust game - Stroop test - Facial expressions (joy, anger and etc) - Live verbal communication

Results? •

Which regions are precisely activated?  high spatial resolution

BioCompuing Lab.

History 

Hyper-scanning fMRI studies •

Activation during social interaction or coooperation

BioCompuing Lab.

History 

First hyper-scanning study using EEG (Babiloni et al., IEEE EMBS 2006) • •



Flexer and Makeig., 2007 •



Playing card game 4 subjects

Two-person game playing

Since then, more than 30 studies were published

BioCompuing Lab.

History 

Hyper-scanning EEG studies • • • • • • • • •



(Lindenberger et al., 2009) (Dumas et al., 2010) (De Vico Fallani et al., 2010) (Astolfi et al., 2010) (Babiloni et al., 2011) (Wang and Jung, 2011) (Sänger et al., 2012) (Gürkök et al., 2013) (Ménoret et al., 2014)

- Playing guitar - Hand movement - Prisoner’s Dilemma Game - Prisoner’s Dilemma Game - Playing saxophone together - Button press task - Face-to-face playing guitar - Mind the sheep game - Face-to-face interaction

Results? •

Which regions are related or correlated btw two or multiple subjects?  High temporal resolution

BioCompuing Lab.

History 

Hyper-scanning EEG studies

BioCompuing Lab.

History 

First study using NIRS (Suda et al., 2010)  Face-to-face conversation



Simultaneous NIRS at prefrontal (Funane et al., 2011)  Button press task

BioCompuing Lab.

History 

Hyper-scanning NIRS studies • • • • • • • •



(Suda et al., 2010) (Funane et al., 2011) (Cui et al., 2012) (Dommer et al., 2012) (Holper et al., 2012) (Jiang et al., 2012) (Duan et al., 2013) (Holper et al., 2013)

Results? •

Oxy-Hb changes and coherence

- Face-to-face conversation - Button press task - Button press task - Dual n-back test - Finger-tapping task - Face-to-face dialog - Tug-of-war game - Teacher-student

BioCompuing Lab.

History 

First study using MEG (Baess et al., 2012) • • •

Two MEGs apart from 5Km Audio recording Synchronization

BioCompuing Lab.

History 

Second study using MEG (Hirata et al., 2014)

• •

Two MEGs in a single room Mother-child facial expression



Only two MEG studies



Results? • •

No remarkable results Just experimental setup

BioCompuing Lab.

SYNCHRONIZATION

BioCompuing Lab.

Synchronization •

Synchronization •

Christiaan Huygens (1665)

•

Types of synchrony----------------------- A : in-phase ; reciprocal  B : external induced – cooperation

 C : driven – PDC

 D : not real (moment)

BioCompuing Lab.

Synchronization 

Inter-brain synchronization? • •

Correlation Coherence  measures the degree of linear dependency

• Phase synchronization • Mutual information • Connectivity between two brains (or multiple)

BioCompuing Lab.

METHODS

BioCompuing Lab.

Methods 

Brain-to-brain coupling • •



Partial directed coherence • •



Covariance in amplitude or power Event-related changes from FFT

Granger casuality Multivariate autoregressive

Phase synchrony •

Phase locking value (PLV)

BioCompuing Lab.

Methods 

PLV

• • • •

N : the number of trials Φ(𝑡,𝑛) : phase on trials n at time t Φ, 𝜓 : different channel 0 ≤ PLV ≤ 1

 1 : perfect phase locking  0 : no phase locking

BioCompuing Lab.

Phi complex and TPJ

NEUROMARKER

BioCompuing Lab.

Neuromarker

  

Tognoli et al., 2007 Proceedings of the National Academy of Sciences (PNAS) Dual-EEG, 60-ch

BioCompuing Lab.

Neuromarker 

Self-paced rhythmic finger movement • • •



Before-vision (opaque) During-vision (transparent) After-vision (opaque)

Phi (9.2~11.5Hz) : right centro (temporo)-parietal region

BioCompuing Lab.

Neuromarker 

Phi (9.2~11.5Hz) : right centro(temporo)-parietal



Parietal and superior temporal sulcus •

Human mirror-neuron system

BioCompuing Lab.

Temporo-Parietal Junction 

Temporo-Parietal Junction (TPJ) •

The TPJ is an area of the brain where the temporal and parietal lobes meet, at the posterior end of the Sylvian fissure (lateral sulcus)

BioCompuing Lab.

right TPJ 

rTPJ plays a pivotal role in analyzing signals from self-produced actions as well as with signals from the external environment



rTPJ plays a role in the way individuals observe and process information, thus impacting social interaction

Blue : rTPJ

Rebecca Saxe (2006)

PDC correlation , Dumas et al., (2010)

BioCompuing Lab.

left TPJ 

the left TPJ contains both Wernicke's area and the angular gyrus

• •

Language Conversation

Kawasaki et al., Scientific Reports (2013)

BioCompuing Lab.

CONCLUSION & FUTURE DIRECTION

BioCompuing Lab.

Conclusion and future direction 

Conclusion •

Inter-brain correlation and synchronization    



Hot topic I just think and interesting issue right TPJ Verbal communication : left TPJ Coherence, correlation, PLV and etc

Yonsei Hospital

Future direction • • •

Need of novel paradigm Need of novel method Need of MEG study  Technical problems → KRISS and Yonsei Hospital

KRISS

BioCompuing Lab.

Thank you Q&A

Android Application for Wireless BCI System Yongwoo Lee

Sep.26.2014 SIC Building B, 206 12th BCI Workshop

Contents I.

Demand for Mobile BCI

II.

Strong point of Android as Mobile BCI platform

III. INFONET BCI I.

Structure of INFONET BCI

II.

Strong point of this Application

III.

A demonstration

IV. A fast BCI keyboard for Android

Demand for Mobile BCI

1. BCI as a wearable Device •

Google glass with EEG

2. A great combination for wireless BCI

Google glass with EEG

• Google glass + MindWave Mobile NeuroSky • Taking picture by EEG Signal. • http://tvpot.daum.net/v/v8f6cMifWGWfPSP9wmPM LLw

A great combination for wireless BCI

• The wireless BCI should be used on wireless device. • Mobile devices are the most wireless, portable. • INFONET BCI is developed based on SeungChan`s wireless BCI device.

Strong point of Android as a Mobile BCI platform

1. Market share 2. Java – widely used language

Market share • Android is on Top of market share of Smartphone. • Also increasing.

Java – widely used language • Basically, android is based on Java. • Java is the most used language. • Lots of open libraries.

INFONET BCI

1.

Structure of INFONET BCI

2.

Strong point of this

Application 3.

A demonstration

Structure of INFONET BCI • Resembles bci2000. • Has main Operator. • Source Module, Signal processing Module, Application Module.

Structure of INFONET BCI(Cont.) • 3 important activities • Module selecting activity.

Structure of INFONET BCI(Cont.) • 3 important activities • Module selecting activity.

Structure of INFONET BCI(Cont.) • Application Activity

Structure of INFONET BCI(Cont.) • Application Activity – For Bluetooth.

Structure of INFONET BCI(Cont.) • Monitoring Activity

Structure of INFONET BCI(Cont.) • Monitoring Activity

Strong Point

1. High Compatibility 2. Easy to upgrade 3. Various Usage.

• It is designed for Seung-Chan`s device. • 100% Compatibility.

Strong Point

1. High Compatibility 2. Easy to upgrade 3. Various Usage.

• Modular programming • Develop additional module and adapt it. • If want, it is also possible develop a module for new wireless device.

Strong Point

1. High Compatibility 2. Easy to upgrade 3. Various Usage.

• Since it is easy to adapt new functions, It can have variable usage as updates.

Demonstration

A fast BCI keyboard for Android

A defect of Android for BCI key board

• Small display. • Not easy to use SSVP or P300 • Binary search is only way.

Binary selection is available.

• 6 selections/min -> to slow. • A new Algorithm for search is required.

Available choices.

• Weighted Tree of Characters. • Auto text. • Etc….

Thank You [email protected] for further questions