Acta Psychologica 131 (2009) 235–244

Contents lists available at ScienceDirect

Acta Psychologica journal homepage: www.elsevier.com/locate/actpsy

Glimpses of a one-speed mind: Focus-switching and search for verbal and visual, and easy and difficult items in working memory Yanmin Zhang a,*, Paul Verhaeghen b,* a b

Purdue University, Department of Psychological Sciences, United States School of Psychology, Georgia Institute of Technology, United States

a r t i c l e

i n f o

Article history: Received 28 April 2008 Received in revised form 7 May 2009 Accepted 25 May 2009 Available online 23 June 2009 PsycINFO classification: 2340 2343 Keywords: Short-term memory Working memory Theory of mind

a b s t r a c t We investigated focus-switching and search rates in an N-Back task for stimuli presumably encoded either in a phonological/semantic or an abstract–visual format. Experiment 1 used Chinese characters and tested Chinese speakers and non-Chinese speakers; character frequency and visual complexity were also manipulated. Experiment 2 presented Chinese characters and English words to non-Chinese English speakers. Effects of focus-switching on accuracy were larger for abstract–visual stimuli and for more difficult stimuli; effects on RT were larger for abstract–visual stimuli, but there was no effect of difficulty, with the exception of the most difficult stimulus set in Experiment 1. Search slopes outside the focus of attention did not covary with either type of code or item difficulty. The decline in accuracy over set-size was stronger for the items coded in abstract–visual format. This suggests that item availability is sensitive to robustness of the memory representations, but item accessibility is not. The data fit well with a model of STM in which a fixed number of ‘slots’ are searched at a constant rate, regardless of the slot’s contents. Published by Elsevier B.V.

1. Introduction Most theories of short-term memory posit a hierarchy of item accessibility. For instance, in his embedded-process account, Cowan (1995, 2001) proposed a hierarchical two-tier structure for working memory, distinguishing a zone of immediate access, labeled the focus of attention (typically considered to contain a magical number of 4 ± 1 elements; Cowan, 2001), from a larger, activated portion of long-term memory (which we will label ‘‘the outer store”) that holds information that is available but not immediately accessible. The size of the focus of attention appears to be malleable – under most circumstances the focus can hold 3–4 items (e.g., Cowan, 2001), while under other circumstances (viz., tasks of serial attention with unpracticed participants; e.g., Verhaeghen, Cerella, & Basak, 2004) it can accommodate only a single item (e.g., Garavan, 1998; McElree, 2001; Oberauer, 2002).

* Addresses: Purdue University, Department of Psychological Sciences, 703 Third Street, PRCE 389, West Lafayette, IN 47907-2081, United States (Y. Zhang), School of Psychology, Georgia Institute of Technology, 654 Cherry Street, Atlanta, GA 303320170, United States (P. Verhaeghen). Tel.: +1 765 494 8667; fax: +1 765 496 1264 (Y. Zhang). E-mail addresses: [email protected] (Y. Zhang), [email protected] (P. Verhaeghen). 0001-6918/$ - see front matter Published by Elsevier B.V. doi:10.1016/j.actpsy.2009.05.009

One logical consequence of this two-tier structure is the existence of a focus-switching process (McElree, 2001; as far as we can determine, the term ‘focus-switch’ was coined by Voigt & Hagendorf, 2002). When the number of items to be retained in working memory is smaller than or equal to the capacity of the focus of attention, they will be contained within the focus, where they are immediately retrievable. When the number of items to be retained exceeds the capacity of the focus, however, the excess items will be stored outside the focus of attention. In that case, accessing items for processing will necessitate a retrieval operation; this will slow down access time. Perhaps the strongest evidence for the existence of focusswitching comes from an identity judgment version of the N-Back task. In this task, participants are presented with a running series of stimuli on a computer screen, shown one at a time. The task is to judge whether the item currently presented is identical to the item presented N positions earlier, or not. Using a speed-accuracy tradeoff procedure, McElree (2001) found that access times were much faster for N = 1 than for either N = 2 or N = 3. Based on his observations, McElree posited a distinction between focus-switching’s effects on response time and those on accuracy. In his model, response time (RT) measures the speed of access of an item, considered an index of its ‘‘accessibility”. In the N = 1 version of the task, items are immediately accessible; in the N > 1 versions, an additional retrieval process is needed prior

236

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

to the comparison process. Likewise, while accuracy at N = 1 is typically near-perfect, accuracy dips below the ceiling for N > 1. The interpretation, again, is that the item inside the focus has special status, whereas items stored in the outer store are vulnerable to decay, interference, or both. The accuracy of retrieval is taken as an index of an item’s ‘‘availability”. Accessibility and availability are not necessarily based on the same mechanism. Many models of working memory, for instance, explicitly or implicitly posit a distinction between an item’s identity and the ‘‘spot” or ‘‘location” or ‘‘position” or ‘‘slot” it occupies (e.g., Averbach & Coriell’s, 1961; Conrad, 1965; Lewandowsky & Murdock, 1989; Shiffrin & Cook, 1978). This makes accessibility and availability dissociable. It is entirely feasible, for instance, that an item’s position might be accessed fast and flawlessly, but that the item’s representation has decayed or has been replaced, so that the item will be retrieved with low or zero accuracy. In the previous work, we (Verhaeghen & Basak, 2005; Verhaeghen et al., 2004) adapted McElree’s identity judgment N-Back task to a columnized and self-paced format – the task we will use in the present experiment. In this task, participants execute a key press to indicate their answer (‘‘Match” or ‘‘Mismatch”); this key press also advances them to the next stimulus. To minimize the control demand of keeping track of the items, items are projected on the computer screen in columns, the number of columns equaling N; additionally, each column receives its own color code. Stimuli are presented one at a time, the first in column 1 row 1, the second in column 2 row 1 (if N > 1), and so forth. Therefore, the participant always compares the current stimulus with the stimulus presented previously just above it in the same column. The results we obtained with this modified version are consistent with McElree’s predictions: Accuracy dropped smoothly over N, and response time increased for the transition from N = 1 to N = 2. Moreover, the response time curve stayed flat over the whole N = 2 to N = 5 range. This suggested to us, first, that working memory outside the focus of attention is content-addressable (i.e., an item is accessed by its ‘‘location”, without a need for search), and, second, that the focus-switching cost is not an artifact of memory load, but must be due to the focus-switching process itself (otherwise, each step in set-size would be associated with an identical cost). Our studies, so far, have used digits as stimuli (with the exception of Vaughan, Basak, Hartman, & Verhaeghen, 2008, where we also used three-digit numbers). Reliance on one particular type of stimulus obviously limits generalizability of findings. Digits are easy, overlearned, verbal stimuli of low complexity, and as such only speak to the phonological subsystem of working memory. The present study extends our previous research in that we, for the first time, directly compare the focus-switching costs for verbal, meaningful stimuli with those for visual, abstract stimuli (note that other researchers, e.g., Leonards, Ibanez, & Giannakopoulos, 2002, have compared N-Back performance on tasks with different types of visual stimuli). To do this, we exploited a characteristic of hanzi, the Chinese writing system: Non-Chinese speakers encode hanzi visually (and with great difficulty; Alvarez & Cavanagh, 2004; Awh, Barton, & Vogel, 2007; Eng, Chen, & Jiang, 2005), whereas Chinese speakers encode hanzi mostly phonologically (Chu-Chang & Lority, 1977), just like English speakers encode English words phonetically (Baddeley, 1966). Dependent on the subject group – readers or non-readers of hanzi – the same stimulus will then be processed in a different modality and at a different level of difficulty. This is the manipulation we used in our first experiment. In the second experiment, we used a similar manipulation in a within-subject comparison, contrasting memory for words with memory for hanzi within a group of non-Chinese speakers. Our experiments were not just inspired by curiosity (after all, the literature on focus-switching is still amazingly small, so any addition might be welcome), ours was mainly a desire to investi-

gate more closely the interplay, if any, between accessibility and availability of items stored in the outer store after a focus-switch. McElree has posited the independence between these two aspects of representations. Our manipulations have the potential to further illustrate this independence. More difficult items are likely to be associated with noisier or more degraded representations in working memory, and this, in turn, is likely to decrease accuracy. In a previous experiment using the modified N-Back task, we found decreased accuracy for three-digit numbers as compared to single-digit numbers, but no differential speed effects (Vaughan et al., 2008). In a very different paradigm, a change detection task, Alvarez and Cavanagh (2005) estimate working memory capacity, in non-readers of hanzi, to be 2.8 for (visually simple) hanzi, versus 3.7 for English letters; capacity for words is likely to be close to 4 (Cowan, 2001). The noisy and degraded representations for hanzi in non-Chinese readers in turn might slow down retrieval processes, just like visual search processes are slowed down when stimuli are degraded or noisy (e.g., Swensson & Judy, 1981), but this is far from certain. We note that in a different paradigm, Garavan’s (1998) running-count task, Voigt and Hagendorf (2002) obtained the finding that more complex stimuli lead to larger focus-switch costs in RT (but accuracy differences could not be investigated in that paradigm). As a manipulation check, Experiment 1 included two additional complexity/difficulty manipulations within the set of Chinese characters, aimed differentially at each of the two subject groups. The first manipulation was character frequency in the Chinese language. This manipulation should influence performance of the Chinese readers (word frequency effects on span, in English, have been noted by, among others, Hulme et al., 1997, and Gregg, Freedman, & Smith, 1989). The second manipulation was visual complexity of the characters. This should influence performance of the non-Chinese readers only. We are mainly interested in two aspects of the data. First, we are interested in examining the group and difficulty effects on focus-switch costs, that is, the jump in RT and/or the drop in accuracy due to the increase in set-size from N = 1 to N = 2 in the identity-judgment N-Back task. We note that it is not self-evident that such focus-switching effects will appear. After all, the aforementioned effects of difficulty or meaningfulness on accuracy or capacity could have other origins. One alternative hypothesis is that what is perceived as a focus-switching cost might simply be due to increasing working memory load (e.g., Voigt & Hagendorf, 2002, only examined focus-switching at a single load, viz. 2 items; in our paradigm, we can distinguish load effects from focusswitching by examining the effects operating on N > 2 from those operating exclusively between N = 1 and N = 2). Another is that these effects might only apply within the focus of attention (e.g., the capacity effects found in the Alvarez & Cavanagh, 2005, study and its replications might simply indicate more severe capacity limits of the focus of attention for more complex stimuli). Additionally, given that accessibility and availability of memory representations are not always correlated (e.g., McElree, 2001; Verhaeghen & Basak, 2005; Verhaeghen et al., 2004), it is uncertain whether effects will register in accuracy or RT alone, or in both. Second, we are interested in accessibility and availability of items stored in the outer store, as indexed, respectively, by RT and accuracy as a function of set-size (i.e., N) when set-size is equal to or larger than 2. More specifically, we are curious about type/ strength-of-encoding by set-size interactions. We found such interactions in previous data sets, but only in accuracy, not RT. For instance, older adults (who are likely to have weaker memory representations) show larger declines in accuracy over set-size than younger adults (but no concomitant increase in RT) (e.g., Verhaeghen & Basak, 2005), and the inclusion of more complex stimuli (e.g., three-digit numbers versus single digits) likewise

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

237

leads to a set-size by complexity interaction in accuracy but not RT (e.g., Vaughan et al., 2008). The imperviousness of RT to such complexity effects is curious, and could stand replication. 2. Experiment 1 2.1. Method 2.1.1. Participants Thirty-two native Chinese speakers/readers and 32 non-Chinese speakers/readers were tested. Chinese speakers were recruited from the graduate student population at Syracuse University; they received 15 US dollars for their participation. Non-Chinese speakers were recruited from an introductory psychology course in Syracuse University; they received course credits for their participation. No subject reported being colorblind. 2.1.2. Tasks and procedure The stimuli were 600 Chinese characters (as currently used in the People’s Republic of China), classified into four groups of 150 characters each, based on their frequency (high and low) and complexity (high and low). Character frequency was obtained from an online database, the Modern Chinese Character Frequency List (Da, 2004). We considered characters with a rank order between 1 and 700 as highly frequent, and characters with a rank order between 2000 and 4000 as characters of low frequency (see Table 1 for examples of each group). Complexity was measured by the number of strokes making up each character. The high-complexity characters have 8–10 strokes, and the low-complexity characters have 3– 6 strokes (Yu, Jing, & Simon, 1985). The size of the characters as shown on the screen was 2 cm  2 cm. A columnized identity-judgment N-Back paradigm was used (Verhaeghen & Basak, 2005). Characters were shown on a computer screen, distributed over N columns. Participants indicated whether the item currently presented on the screen was identical to the item presented N positions back (i.e., previously presented in the same column). Fig. 1 shows a black-and-white version of a sample block (in this case, N = 4 and high-frequency, low-complexity characters were used), as it would appear on the computer screen if all items remained visible. In practice, only one character was shown at any time; the order of presentation was the conventional reading pattern for the English language: left to right, top to bottom. Each column was projected in a different color. For the first row, a new character was presented every 2000 ms; from the second row on, participants pressed either of the two keys to indicate their answer. The ‘‘/” key stood for a match; it was masked with a piece of green tape. The ‘‘z” key stood for a mismatch; it was masked with a piece of red tape. Participants were instructed to be as fast and accurate as possible. As soon as the key was pressed, the next stimulus appeared. Participants were encouraged to choose a comfortable viewing distance from the screen. N varied from 1 to 5. Each stimulus set (a block) contained a total of 20 to-be-responded-items. After each block, the subject received feedback about both total accuracy and average RT over the run of 20 items. A total of 120 blocks (yielding a total of 2400 RTs) were presented Table 1 Stimulus examples. High frequency High complexity Example 1 Example 2

Low frequency Low complexity

High complexity

Low complexity

Fig. 1. An example of a trial in the 4-back version of the task, if all stimuli remained onscreen. In the experiment, stimuli were shown one at a time, in a reading pattern (left to right, then on the next line, etc.); each column was depicted in a different color. The first row was presented at a 2-s/item pace; presentation of subsequent stimuli was participant-paced. The response required was a judgment whether the character currently projected was identical to the character previously shown one row higher in the same column.

to each participant based on the following eight conditions: HHascending, HH-descending, HL-ascending, HL-descending, LHascending, LH-descending, LL-ascending, LL-descending (H stands for high, L for low; the first letter denotes frequency, the second complexity; ascending means that N varied from 1 to 5, and descending means that N varied from 5 to 1). Each of the eight conditions contained three blocks for each value of N. The eight conditions were presented in a Latin-square design. For each block, half of the stimuli were identical to the item N-position back, and the other half were not. Each block contained 10 different Chinese Characters. To limit interference, we allowed each character to be presented in only two blocks, one ascending, one descending. Participants were encouraged to take breaks between blocks. All participants were tested in a single session, lasting about 60 minutes for native Chinese speakers, and about 70–90 min for non-Chinese speakers. Chinese-speaking participants were tested by a native speaker of Mandarin; non-Chinese speakers by a native speaker of English. Significance level was set at p = .05. 2.2. Results 2.2.1. Response times Items within the focus. Data are presented in Fig. 2 (top panel). RTs were analyzed using repeated-measures ANOVA, with subject group (Chinese speakers vs. non-Chinese speakers) as the between-subject variable and character frequency and character complexity as the within-subject variables. There was a main effect of subject group, F(1, 62) = 5.81, p < .05 – Chinese speakers responded faster than non-Chinese speakers (692 ms vs. 751 ms). All other main effects and interactions were not significant, F < 1.90. Focus-switch costs. Focus-switch costs were defined as simple difference scores between RT at N = 2 and RT at N = 1. We found no significant effect of subject group, F(1, 62) = 2.01, ns, indicating equivalent focus-switch costs for Chinese speakers and non-Chinese speakers. The remainder of the effects were quite complex – there was a main effect of character frequency, F(1, 62) = 7.95, p < .01, an interaction between frequency and subject group, F(1, 62) = 4.11, p < .05, and an interaction between frequency and complexity, F(1, 62) = 5.15, p < .05. Fig. 2, middle panel, shows that these effects boil down to an increased focus-switch cost for the most difficult stimuli (i.e., high-complexity, low-frequency characters) in the non-Chinese speakers only.

238

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244 1300 Chinese speakers, high frequency, high complexity 1200

Chinese speakers, high frequency, low complexity

1100

Chinese speakers, low frequency, high complexity

1000

Chinese speakers, low frequency, low complexity Non-Chinese speakers, high frequency, high complexity

900

Non-Chinese speakers, high frequency, low complexity 800 Non-Chinese speakers, low frequency, high complexity 700

Non-Chinese speakers, low frequency, low complexity

600 500 0

1

2

3

4

5

6

N-Back value

300

Chinese speakers Non-Chinese speakers

Focus switch cost (ms)

250

200

150

100

50

0 High complexity

Low Complexity

High complexity

High frequency

Low Complexity

Low Frequency

1300 1200

Chinese speakers

1100 Non-Chinese speakers, high frequency, high complexity 1000 Non-Chinese speakers, high frequency, low complexity 900 Non-Chinese speakers, low frequency, high complexity 800 Non-Chinese speakers, low frequency, low complexity 700 600 500 0

1

2

3

4

5

6

N-Back value

Fig. 2. Experiment 1. Response time (RT) as a function of N in our N-Back task, separated by participant group and item difficulty. The top panel shows the data, along with standard errors; the middle panel shows focus-switch costs, along with standard errors; the bottom panel shows the fitted HLM regression lines.

On half of the trials, the item on the screen matched the item in memory; on the other half, it did not. In the former case, working memory did not need to be updated, in the latter it did. The RT difference between match and mismatch trials, therefore, gives some insight into the processes associated with working memory updating, that is, with encoding a new item.

To examine the influence of updating, we redid the ANOVA, adding a separate match/mismatch factor. Only the main effect of this factor was significant, F(1, 62) = 13.74, p < .001; none of its interactions with any of the other factors reached significance. This implies that the time to encode a new item was simply additive to all other processes.

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

Retrieval from the outer store. Fig. 1, bottom panel, shows that a linear model should fit the data well. This is borne out by a more formal analysis: In a repeated-measures ANOVA with subject group (Chinese speakers vs. non-Chinese speakers) as the between-subject variable and character frequency and character complexity as the within-subject variables, the partial g associated with the linear effect of N was .67, F(1, 62) = 124.66, p < .001. Consequently, the data were analyzed using a linear model, namely, hierarchic linear modeling (HLM). We started with a full model (1024 observations, that is, 64 subjects with 4 stimulus types at 4 levels of N) with random slopes that included all main effects (i.e., N, group, and character frequency and complexity) and all interactions. To enhance interpretability of the results, data were recentered by subtracting 2 from each value of N such that the intercepts indicate RT at N = 2 (i.e., after execution of the focusswitch), rather than at N = 0; group was coded with 0 being Chinese speakers and 1 non-Chinese speakers. The full model was then trimmed by deleting all non-significant parameters, at no significant cost to fit, LR v2 (6) = 7.44. This final model yielded the following parameters:

RT ¼ 843 ðSE ¼ 26Þ þ 47ðSE ¼ 4Þ ðN  2Þ þ 81 ðSE ¼ 37Þ group  22 ðSE ¼ 10Þ character complexity  36 ðSE ¼ 10Þ group by character frequency

239

plexity main effect, F(1, 62) = 15.91, p < .001, is qualified by a group by complexity interaction, F(1, 62) = 10.54, p < .01, indicating that the effect of complexity was larger in non-Chinese speakers than in Chinese speakers. None of the other effects reached significance. Retrieval from the outer store. As for RT, the accuracy data were modeled using HLM. A linear model fit the data well: In a repeatedmeasures ANOVA with subject group (Chinese speakers vs. nonChinese speakers) as the between-subject variable and character frequency and character complexity as the within-subject variables, the partial g2 associated with the linear effect of N was .883, F(1, 62) = 309.48, p < .001. We started with a full HLM model that included all main effects and all interactions. This model was trimmed by deleting all non-significant parameters, at no significant cost to fit, LR v2 (8) = 7.31. This final model yielded the following parameters:

Accuracy ¼ 96:8 ðSE ¼ 0:6Þ  2:4 ðSE ¼ 0:3Þ ðN  2Þ  9:0 ðSE ¼ 0:8Þ group  2:2 ðSE ¼ 0:7Þ character complexity  2:3 ðSE ¼ 0:4Þ group by ðN  2Þ  3:2 ðSE ¼ 0:9Þ group by character complexity þ 2:6 ðSE ¼ 0:6Þ character frequency by character complexity:

þ 87 ðSE ¼ 15Þ group by character complexity: This model describes five distinct regression lines, all parallel, with a common slope of 47 ms/N. These lines have the following intercept at N  2, in ascending order: 843 ms for Chinese speakers, regardless of condition; 887 ms for high-frequency, low-complexity characters in non-Chinese speakers; 924 ms for low-frequency, low-complexity characters in non-Chinese speakers; 953 ms for high-frequency, high-complexity characters in non-Chinese speakers; and 989 for low-frequency, high-complexity characters in nonChinese speakers. We also directly compared the focus-switch slope (i.e., the slope between N = 1 and N = 2, calculated as a difference score RT(N = 2)  RT(N = 1)) with the slope in the outer store (N > 2). For the slope at N > 2, we calculated the average difference score between adjacent set-sizes, that is, the RT difference N = 2 and N = 3, N = 3 and N = 4, and N = 4 and N = 5. The focus-switching slope (157 ms) was reliably larger than the slope in the outer store (47 ms), F(1, 62) = 79.80, p < .001. We analyzed the influence of updating/encoding by adding a dummy code for the type of trial (match vs. mismatch), as well as all possible interactions involving this variable, to the final equation described above. All interaction terms turned out to be nonsignificant, LR v2 (5) = 4.55, but the main effect of trial type was match trials were responded to 182 ms (SE = 6) faster than mismatch trials. This implies that the time to encode a new item was simply additive to all other processes, and did not influence the RT by N slope. 2.2.2. Accuracy Items within the focus. Accuracy data are presented in Fig. 3. Accuracy at N = 1 was near ceiling overall (97% correct). Consequently, we left these data unanalyzed. Focus-switch costs. Focus-switch costs were expressed as simple difference scores between accuracy at N = 1 and accuracy at N = 2. The data are shown in Fig. 3, middle panel. Chinese speakers showed smaller focus-switch costs, F(1, 62) = 122.28, p < .001. There was a significant main effect of character frequency, F(1, 62) = 12.25, p < .001, tempered by a frequency by complexity interaction, F(1, 62) = 9.60, p < .01: Low-frequency, high-complexity characters are less memorable than all other types. The com-

This result indicates that Chinese speakers are more accurate overall than non-Chinese speakers (a 9% difference at N = 2), and that the decline in accuracy over N is more precipitous on non-Chinese speakers than in Chinese speakers (slopes are 2.4%/N and 4.5%/N, resp.). More complex characters are associated with more errors, and this effect is larger for non-Chinese speakers. The character frequency by character complexity interaction signifies that the accuracy-decreasing effect of high character complexity is larger in the case of low-frequency characters than in the case of high- frequency characters. 2.3. Discussion We tested a group of Chinese speakers and non-Chinese speakers for their short-term memory for Chinese characters in a modified N-Back paradigm, under the assumption that Chinese speakers would store the characters in a verbal code, and non-Chinese speakers in a visual code. We were interested in two main aspects of the data: (a) the influence of type of encoding (as indexed by between-group differences) and within-group stimulus complexity on the focus-switch cost; and (b) accessibility (as indexed by RT) and availability (as indexed by accuracy) of items stored in the outer store as a function of these same characteristics. A first observation to make is that the jump in RT observed between N = 1 and N = 2 appears indeed to be due to focus-switching, and not merely to an increase in working memory load: There is a clear step in the data, also echoed by a precipitous drop in the accuracy data for the non-Chinese speakers. Turning to these focus-switch costs, then, we find an interesting dissociation between accessibility and availability. The effects of subject group (a proxy for type of encoding) are clear and outspoken for item availability (there is a massive drop in accuracy for non-Chinese speakers, compared to a much more modest drop for Chinese speakers), but more subtle (i.e., only noticeable for the most complex type of stimuli, low-frequency, high-complexity items) for accessibility. The decline in accuracy is not due to deficiencies in the comparison process for non-Chinese speakers: Both groups are essentially perfectly accurate at N = 1, showing that the items are equally (and perfectly) discriminable for both groups (although comparison RTs obviously differed). We note here that

240

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244 1 Chinese speakers, high frequency, high complexity 0.95 Chinese speakers, high frequency, low complexity 0.9 Chinese speakers, low frequency, high complexity 0.85 Chinese speakers, low frequency, low complexity 0.8 Non-Chinese speakers, high frequency, high complexity

0.75

Non-Chinese speakers, high frequency, low complexity

0.7

Non-Chinese speakers, low frequency, high complexity

0.65

Non-Chinese speakers, low frequency, low complexity

0.6 0.55 0.5 0

1

2

3

4

5

6

N-Back value

High frequency High complexity

Low Frequency

Low Complexity

High complexity

Low Complexity

Focus switch cost (proportion correct)

0

-0.02

-0.04

-0.06

-0.08

-0.1 Chinese speakers -0.12

Non-Chinese speakers

-0.14 1 0.95 Chinese speakers, low complexity 0.9 Chinese speakers, high frequency, high complexity 0.85 Chinese speakers, low frequency, high complexity

0.8

Non-Chinese speakers, low complexity

0.75

Non-Chinese speakers, high frequency, high complexity

0.7 0.65

Non-Chinese speakers, low frequency, high complexity

0.6 0.55 0.5 0

1

2

3

4

5

6

N-Back value

Fig. 3. Experiment 1. Accuracy (percent correct) as a function of N in our N-Back task, separated by participant group and item difficulty. The top panel shows the data, along with standard errors; the middle panel shows focus-switch costs, along with standard errors; the bottom panel shows the fitted HLM regression lines.

it is possible that one part of the reason for the high accuracy of the non-Chinese readers may be that their performance in the N = 1 condition may be partially based on iconic memory read-out, insofar as it survives the saccadic eye movements necessary to fixate the probe. This finding suggests that items that are stored in a more stable code (phonological/semantic) are not necessarily retrieved with

greater ease, or, conversely, that brittle representations (presumably visual representations of abstract stimuli) are retrieved at the same speed as very memorable representations. This in turn suggests that item availability is sensitive to robustness of the code, whereas accessibility is much less sensitive. We note that the apparent sensitivity of non-Chinese speakers to low-frequency (at least when combined with high complexity) stimuli is surprising;

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

after all, non-Chinese speakers do not have access to the meaning of or to frequency information about the characters to-be-remembered. Our suspicion is that low-frequency characters might be inherently more visually complex than high-frequency characters, even if the number of strokes is kept constant across the two categories (see also Table 1); unfortunately, we have no independent information to verify this assertion. The finding that non-Chinese speakers are sensitive to visual complexity in both RT and accuracy does yield independent evidence for the assertion of visual encoding in this subject group. The effect in RT is clearly located in retrieval, not encoding operations. If stimulus encoding were sensitive to the type of code or the difficulty/complexity of the stimulus, we would expect that on mismatch trials, where stimulus representations have to be updated, RT would slow down more for more complex/difficult stimuli or for stimuli commanding less stable representations than for less complex/difficult stimuli or for stimuli commanding more stable representations. There were, however, no significant interactions involving updating. This is remarkable in and of itself: The field is used to thinking of encoding as the locus of much of the action in memory (levels-of-processing, the encoding-specificity effect, etc.), but here it seems that encoding is not sensitive to differential characteristics of the stimuli – retrieval is. Inside the outer store, we find a similar dissociation: The slope of the RT by N function was not influenced by either participant group or item difficulty/complexity, indicating equivalent search rates of 47 ms/item, but non-Chinese speakers show a steeper drop in item accuracy as a function of N, losing about 4.5% accuracy per value of N inside the outer store, compared to 2.4% for the Chinese speakers. Again, this suggests a dissociation between item availability and item accessibility: Within the outer store, the more brittle representations created by the non-Chinese speakers become increasingly less available when set-size (and thereby retention time) increases (due to interference, decay, or some other mechanism), but they do not become less accessible than the more robust representations created by the Chinese speakers. The slope in RT is a surprise; in most of our previous work we obtained flat RTs for N > 1 (but see Vaughan et al., 2008). Slopes can be a sign of rehearsal (e.g., Basak & Verhaeghen, submitted for publication), but rehearsal is unlikely to occur with highly abstract–visual stimuli. The slope does not seem to be due to encoding processes: On those trials where the item on screen did not match the N-Back item in memory, an additional 180 ms was needed, but this cost did not increase with N, showing that encoding was independent of memory load. The most likely reason for a slope is that the present stimuli engender a search process. One difference between the present stimulus set and the set of digits we have used in the past is the size of the set, 600 versus 9. When only a limited number of stimuli are used, subjects might be more inclined to encode locations very strongly, perhaps forcing the system into content-addressable mode. Another possibility is that the item representations themselves are so complex (for Chinese speakers, the added complexity might be encoding the tonal properties of the phonological trace) that encoding their identity receives priority over encoding their location, perhaps necessitating a serial retrieval process. Regardless of the reason for search, the invariance of the search slopes across groups strongly suggests that search processes in short-term memory are not dependent on the type of code (verbal or visual) that is used to encode and store the stimuli.

3. Experiment 2 The critical manipulation in Experiment 1, type of encoding, was achieved in a between-subject contrast. One obvious criticism is that differences (and/or lack of differences) between the two

241

groups studied may be due to extraneous variables that covary with group membership. Therefore, Experiment 2 was set up as a control experiment, in which we manipulated type of encoding within subjects. The subjects were native speakers of English with no knowledge of Chinese; the stimuli were hanzi and English words. The expectation is that these subjects will code hanzi largely visually, and words largely phonologically. With this experiment, we simply aimed to replicate the main results of Experiment 1, now finding the relevant contrasts not between subjects, but between stimulus materials. 3.1. Method 3.1.1. Participants Twenty-four native speakers of English who were also non-Chinese speakers/readers were tested. They were recruited from an introductory psychology course at the Georgia Institute of Technology and received course credits for their participation. No subject reported being colorblind. 3.1.2. Tasks and Procedure Task, stimuli, and procedure were identical to that of Experiment 1, with the following exceptions. First, two types of stimuli were used: 150 different Chinese characters and 150 different English words. English words consisted of a single syllable and between three and five letters, and were highly frequent within the English language, as attested by a frequency score above 100 (Kucera-Francis Written Frequency) and familiarity score above 500. The Chinese characters were those used for the high-frequency, low-complexity condition in Experiment 1. Each stimulus set (a block) contained a total of 20 to-be-responded-items. A total of 60 blocks (yielding a total of 1200 RTs) were presented to each participant based on the following 4 conditions: ascending-word, descending-word, ascending-Chinese characters, descending-Chinese characters. Significance level was set at p = .05. 3.2. Results 3.2.1. Response times Items within the focus. Data are presented in Fig. 4 (top panel). RTs were analyzed using repeated-measures ANOVA, with stimulus type (hanzi vs. words) as the within-subject variable. There was no effect of stimulus type, F(1, 23) < 1. Focus-switch costs. Focus-switch costs were defined as simple difference scores between RT at N = 2 and RT at N = 1. There was an effect of stimulus type, F(1, 23) = 6.78, p < .05, indicating that the focus-switch cost was larger for Chinese characters than for words. To examine the influence of memory updating, we redid the ANOVA, adding a separate match/mismatch factor. Only the main effect of this factor was significant, F(1, 23) = 239.78, p < .001; none of its interactions with any of the other factors reached significance. This implies that the time to encode a new item was simply additive to all other processes. Retrieval from the outer store. Fig. 4 suggests that a linear model should fit the data well. This is borne out by a more formal analysis: In a repeated-measures ANOVA with stimulus type and setsize as the within-subject variables, the partial g associated with the linear effect of N was .36, F(1, 23) = 13.14, p < .001. As for Experiment 1, we first estimated the full HLM model (involving set-size, stimulus type, and their interaction); this model was trimmed by deleting the interaction term at no significant cost to fit, LR v2 (1) = 0.43. This suggests two parallel regression lines. These lines had a common slope of 18 ms/set-size, and the

242

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

1300

1 0.95

Chinese characters

1200

0.9

English words

1100

0.85 1000

0.8

900

0.75 0.7

800

Chinese characters

0.65 700

English words

0.6

600

0.55 0.5

500 0

1

2

3

4

5

0

6

1

2

3

4

5

6

N-Back value

N-Back value 1

1300

0.95 1200

Chinese characters

1100

0.9

English words

0.85 0.8

1000

0.75

900

0.7 800

Chinese characters

0.65 700

0.6

English words

0.55

600

0.5

500 0

1

2

3

4

5

6

N-Back value

Fig. 4. Experiment 2. Response time (RT) as a function of N in our N-Back task, separated by stimulus type. The top panel shows the data, along with standard errors; the bottom panel shows the fitted regression lines.

following intercepts at N  2: 884 ms for hanzi and 817 ms for words. As in Experiment 1, we directly compared the focus-switch slope with the slope in the outer store. The focus-switch slope (124 ms) was reliably larger than the slope in the outer store (16 ms), F(1, 23) = 54.26, p < .001. We analyzed the influence of updating/encoding by adding a dummy code for the type of trial (match vs. mismatch), as well as all possible interactions involving this variable, to the final equation described above. The main effect of trial type was significant: Match trials for words were responded to 124 ms (SE = 11) faster than mismatch trials. The trial type by stimulus type interaction was significant, implying that responding to mismatched words took an additional 76 ms (SE = 16). All other interaction terms involving updating turned out to be non-significant, LR v2 (2) = 3.08. 3.2.2. Accuracy Items within the focus. Accuracy data are presented in Fig. 5. Accuracy at N = 1 was near ceiling overall (97% correct). Consequently, we left these data unanalyzed. Focus-switch costs. Focus-switch costs were expressed as simple difference scores between accuracy at N = 1 and accuracy at N = 2.

0

1

2

3

4

5

6

N-Back value

Fig. 5. Experiment 2. Accuracy (percent correct) as a function of N in our N-Back task, separated by stimulus type. The top panel shows the data, along with standard errors; the bottom panel shows the fitted regression lines.

Hanzi led to reliably larger focus-switch costs than words, F(1, 23) = 38.08, p < .001. Retrieval from the outer store. As for RT, the accuracy data were modeled using HLM. A linear model fit the data well: In a repeated-measures ANOVA with stimulus type and set-size as the within-subject variables, the partial g2 associated with the linear effect of N was .79, F(1, 23) = 88.14, p < .001. As for Experiment 1, we first estimated the full HLM model (involving set-size, stimulus type, and their interaction); this model was trimmed by deleting the interaction term at no significant cost to fit, LR v2 (1) = 0.18. This model suggests two parallel regression lines. These lines had a common slope of 4%/N, and the following intercepts at N = 2: 87% correct for hanzi and 97% correct for words. Our linear analysis obscures a more complex pattern, however: Fig. 5 shows that the decline is monotonic and more or less linear for words; for characters, the decline seems more precipitous for N = 2 to N = 4, with stabilization at around 75% accuracy for N = 5. This suspicion was confirmed in a repeated-measures ANOVA: There was a significant quadratic trend in both the main effect of set-size and the set-size by stimulus type interaction, F(1, 23) = 12.92, p < .01, partial g2 = .36, and F(1, 23) = 33.14, p < .001, partial g2 = .59, respectively.

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

243

3.3. Discussion

4. General discussion

In Experiment 1, we investigated the role of type of encoding (verbal vs. visual) on focus-switching and memory search in a between-subject contrast – Chinese characters as memorized by Chinese readers and English readers with no knowledge of Chinese. In Experiment 2, we examined the same contrast in a within-subject design, contrasting memory for Chinese characters and English words within a group of English readers with no knowledge of Chinese. Three of the key findings of Experiment 1 did replicate. First, there is clear evidence of a focus-switching process, as indicated by a clear jump in RT from N = 1 to N = 2, accompanied by a strong decrease in accuracy. Second, we again obtained a significant (but smaller) RT by set-size slope for N = 2–5, and this slope was invariant across stimulus type, suggesting a memory search process proceeding at identical rates for the two types of memory code. In the Discussion of Experiment 1, we put forward one interpretation for the existence of a slope that can now be ruled out, namely, the complexity of the stimulus: The word stimuli we used were short words of high familiarity – not likely to be more complex in their representations than digits. This suggests that the emergence of a slope is likely due to the relatively larger emphasis on the encoding of stimulus identity at the detriment of stimulus locations in a large (quasi-open) stimulus set as compared to a small, closed stimulus set, where positional coding is more critical. Third, we found a steeper decline for characters than for words, at least for the N = 2 to N = 4 range; accuracy for characters stabilized at around 75% accuracy for N = 5. This pattern seems to be consistent with the pattern observed for the more difficult conditions in Experiment 1, which also appears to be negatively accelerating towards a horizontal asymptote, located at around 70% correct. These values for the asymptote suggest an average capacity of about 3.5– 3.75 items, perfectly in line with Cowan’s (2001) estimate of the size limits of the active part of working memory. A few of the findings from Experiment 1, however, did not replicate, and should therefore be ascribed to sampling error. First, we now find significant differences in focus-switch costs in RT for the two types of materials, with Chinese characters yielding a larger cost than English words. This suggests that retrieval from the outer store is indeed differentially impacted by the type of code used for storage. (The effect is situated in retrieval, not updating, as testified by the absence of stimulus type by match/mismatch trials in focusswitch costs.) A second discrepant finding is that Experiment 2 yields no difference in either RT or accuracy between the two types of stimuli in the N = 1 condition, that is, when items are held in the focus of attention. This finding is important for two reasons. First, it emphasizes yet again the special status of items held in the focus of attention: Outside the focus, the two types diverge in both accuracy and RT. E finding of near-perfect accuracy for N = 1 reaffirms our view that interference likely plays no role for the items held within the focus, even with trials of quite substantial length (20 successive items within a run). Second, these results reassure us that any difference between the fate of the two types of items held outside the focus of attention cannot be due to differential discriminability: Subjects are clearly capable of comparing both types with equal accuracy and equal speed when they are held within the focus of attention. A final difference between the two experiments pertains to updating: Mismatched words took 76 ms longer to respond to than mismatched Chinese characters. It seems likely that this difference is tied to a working memory updating process that operates on words, but not Chinese characters. One such process is phonetic coding. Our 76 ms might not be an unreasonable estimate of the duration of a phonetic coding process, given evidence that phonological priming does not occur with primes exposed for 50 ms or less (Peterson & Savoy, 1998).

The main findings of the present set of two experiments can be summarized as follows: First, we find again clear evidence for the existence of a focus-switching process, as testified by a stark discontinuity in both RT and accuracy from N = 1 to N = 2. Our Experiment 2 is the first to observe a significant influence of type of stimulus – verbal and meaningful versus visual and abstract – on the focus-switch cost: English speakers showed a larger focus-switch cost for Chinese characters than for English words. Experiment 1 demonstrated that within these modalities, however, focus-switch costs in RT stayed constant, with the exception of the most difficult condition – lowfrequency, high-complexity Chinese characters viewed by non-Chinese speakers. In contrast, focus-switch costs in accuracy did covary with frequency and complexity as well as type of stimulus; the latter was true even in Experiment 1, where we found no stimulus type effect on RT, and only a weak effect of difficulty. Second, in both experiments we found evidence for an RT by set-size slope in the N = 2 to N = 5 range. No such RT by set-size increase was found for the difference between match and mismatch trials, which must at least partially reflect the cost associated with updating the representation of the mismatched item. This implies that the observed RT by N slope is tied to retrieval processes, likely some form of memory search. In both experiments, we found the rate of this search process to be identical for both types of stimuli. Experiment 1 additionally investigated the role of different types of difficulty – visual complexity and character frequency – within each type of representation, and found again no rate differences between types. The accuracy data tell a different story: The decrease in performance over N was always steeper for the visual–abstract stimuli than for the verbal-meaningful. Taken together, these results suggest a clear dissociation between an item’s availability and its accessibility, both in the focusswitching process and in search inside the outer store. Thus, search and retrieval processes in short-term memory are relatively independent of both the intactness of the stimulus representation and the type of code used to encode and store the stimuli into the outer store. This in turn has implications for theories about the representation of elements in working memory. The finding that it is possible for a subject to access an item’s ‘‘representation” even when the item has disappeared from memory insinuates a model in which item representations are sorted into an integer number of slots (e.g., Vogel & Machizawa, 2004) rather than a model in which items are stored as distributed representations limited by a maximum level of the sum of the item’s activations (e.g., Alvarez & Cavanagh, 2004). This conclusion is further augmented by the finding that stimulus difficulty had no discernible effect on retrieval rates within the outer store: All four stimulus types (obtained by crossing high and low visual complexity with high and low character frequency) yielded a statistically equivalent slope over the N > 1 range. A distributed-representations account would expect search rates to be slower for more difficult items, which would yield simplified, that is, degraded or noisier representations. Such slowing is not observed. Therefore, it appears that what the subject accesses during memory retrieval is first and foremost an item’s memory slot; what may have disappeared is the slot’s content. Acknowledgement This research was supported by a grant from the National Institute on Aging (AG-16201). References Awh, E., Barton, B., & Vogel, E. K. (2007). Visual working memory represents a fixed number of items, regardless of complexity. Psychological Science, 18, 622–628.

244

Y. Zhang, P. Verhaeghen / Acta Psychologica 131 (2009) 235–244

Alvarez, G. A., & Cavanagh, P. (2004). The capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological Science, 15, 106–111. Alvarez, G. A., & Cavanagh, P. (2005). Independent resources for attentional tracking in the left and right visual fields. Psychological Science, 16, 637–643. Averbach, E., & Coriell, A. S. (1961). Short-term memory in vision. Bell Systems Technical Journal, 40, 309–328. Baddeley, A. D. (1966). Short-term memory for word sequences as a function of acoustic, semantic and formal similarity. Quarterly Journal of Experimental Psychology, 18, 362–365. Basak, C., & Verhaeghen, P. (submitted for publication). Three layers of working memory: Focus-switch cost and retrieval dynamics as revealed by the N-Count task. Chu-Chang, M., & Lority, D. L. (1977). Phonological encoding of Chinese ideographs in short-term memory. Language Learning, 27, 341–348. Conrad, R. (1965). Order error in immediate recall of sequences. Journal of Verbal Learning and Verbal Behavior, 4, 161–169. Cowan, N. (1995). Attention and memory: An integrated framework. New York, NY: Oxford University Press. Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87–185. Da, J. (2004). Modern Chinese character frequency list. Retrieved from Middle Tennessee University Web site: . Eng, H. Y., Chen, D., & Jiang, Y. (2005). Visual working memory for complex and simple visual stimuli. Psychonomic Bulletin and Review, 12, 1127–1133. Garavan, H. (1998). Serial attention within working memory. Memory and Cognition, 26, 263–276. Gregg, V. H., Freedman, C. M., & Smith, D. K. (1989). Word frequency, articulatory suppression and memory span. British Journal of Psychology, 80, 363–374. Hulme, C., Roodenrys, S., Schweikert, R., Brown, G. D. A., Martin, S., & Stuart, G. (1997). Word-frequency effects on short-term memory tasks: Evidence for a redintegration process in immediate serial recall. Journal of Experimental Psychology: Learning Memory and Cognition, 23, 1217–1232.

Leonards, U., Ibanez, V., & Giannakopoulos, P. (2002). The role of stimulus type in age-related changes of visual working memory. Experimental Brain Research, 146, 172–183. Lewandowsky, S., & Murdock, B. (1989). Memory for serial order. Psychological Review, 96, 25–57. McElree, B. (2001). Working memory and focal attention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 817–835. Oberauer, K. (2002). Access to information in working memory: Exploring the focus of attention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 411–421. Peterson, R. R., & Savoy, P. (1998). Lexical selection and phonological encoding during language production: Evidence for cascaded processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 539–557. Shiffrin, R., & Cook, J. (1978). Short-term forgetting of item and order information. Journal of Verbal Learning and Verbal Behavior, 17, 189–218. Swensson, R. G., & Judy, P. F. (1981). Detection of noisy visual targets: Models for the effects of spatial uncertainty and signal-to-noise ratio. Perception and Psychophysics, 29, 521–534. Vaughan, L., Basak, C., Hartman, M., & Verhaeghen, P. (2008). Aging and working memory inside and outside the focus of attention: Dissociations of availability and accessibility. Aging, Neuropsychology, and Cognition, 15, 703–724. Verhaeghen, P., & Basak, C. (2005). Aging and switching of the focus of attention in working memory: Results from a modified N-Back task. Quarterly Journal of Experimental Psychology (A), 58, 134–154. Verhaeghen, P., Cerella, J., & Basak, C. (2004). A working memory workout: How to change to size of the focus of attention from one to four in ten hours or less. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 1322–1337. Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428, 748–751. Voigt, S., & Hagendorf, H. (2002). The role of task context for component processes in focus switching. Psychologische Beiträge, 44, 248–274. Yu, B., Jing, Q., & Simon, H. A. (1985). STM span for Chinese words and phrases. Acta Psychologica Sinica, 17, 361–368 [in Chinese].