Working memory (WM) is the cognitive capacity available for online processing and short-term storage of information (Baddeley, 2012; Ma, Husain, & Bays, 2014). It is limited to three or four items (Cowan, 2001), except when encoding can take place in relation to representations that are already established in long-term memory (Hulme, Maughan, & Brown, 1991). Indeed, the short-term store can accommodate as many as nine familiar words (Miller, 1956)—that is, items with preexisting representations in the mental lexicon—but considerably fewer nonwords (Hulme et al., 1991) or items that cannot be verbalized (Luck & Vogel, 1997). Long-term representations also influence short-term storage of nonwords, such that nonwords with a common phonological structure are more robustly represented than those that are more unusual (Gathercole, Frankish, Pickering, & Peaker, 1999). However, it is not known whether these semantic and phonological effects pertain exclusively to speech-based representations in the auditory domain, or whether they can be generalized to sign-based representations in the visuospatial domain. The main purpose of the present study was to investigate this.

Sign languages are natural languages in the visuospatial domain used by deaf communities (Sutton-Spence & Woll, 1999). They develop independently of the spoken languages that surround them and have a different grammatical structure (Emmorey, 2002). However, the sublexical structure of signed languages can be understood in terms similar to those used to describe the phonology of spoken languages (Sandler & Lillo-Martin, 2006). Spoken language phonology relates to a largely sequential set of contrasts, manifest in the notion of minimal pairs—in which two words contrast in a single phonological element, such as the final consonants in words like bag and bad, or in rhyme. In signed languages, the less sequential phonological elements comprising the shape, movement, and location of the signing hands (Sandler & Lillo-Martin, 2006) give rise to minimal pairs consisting of two signs differing, for instance, in location only, such as in British Sign Language (BSL) NAME and AFTERNOON; see Fig. 1. Phonological processing tasks generate similar patterns of performance across the language modalities of sign and speech (Andin, Rönnberg, & Rudner, 2014) and activate similar neural networks, suggesting at least some degree of amodal representation of phonology (MacSweeney, Waters, Brammer, Woll, & Goswami, 2008).

Fig. 1
figure 1

British Sign Language (BSL) minimal pair. The BSL minimal pair NAME (left panel) and AFTERNOON (right panel) share a hand shape and movement, but differ in location

Full size image

The Ease of Language Understanding (ELU) model of WM (Rönnberg et al., 2013) proposes that WM in the service of communication is multimodal. Input to the system can be in any language modality, transmitted by any or several sensory modalities, and enters an episodic buffer (Rudner & Rönnberg, 2008b) whose function is the rapid automatic multimodal binding of phonology. When the input can be smoothly matched to existing representations in long-term memory, language understanding is implicit and experienced as effortless. However, when there is a mismatch, language understanding becomes explicit and, depending on the individual cognitive capacity, may be experienced as effortful. Mismatch may arise either due to a range of problems with input to the cognitive system, including structural distortion and semantic distraction (Mattys, Davis, Bradlow, & Scott, 2012; Rudner & Lunner, 2014; Zekveld et al., 2011), or to nonexistent or degraded representations (Classon, Rudner, & Rönnberg, 2013; Molander et al., 2013) in long-term memory. When explicit processing is brought into play, limited cognitive resources are devoted to processing, and thus storage limits become critical. This means that preexisting representation improves performance in two ways, by avoiding mismatch and by reducing the load involved in maintaining items without preexisting representations in WM. Evidence is accumulating to support the ELU model in the auditory/speech domain, and because this model accepts multimodal input, it is likely that similar phenomena may be observable for sign language (for a discussion, see Rudner, Toscano, & Holmer, 2015).

Indeed, previous research has shown, in support of the multimodal nature of the ELU model, that signers and speakers perform at similar levels on WM tasks presented either in their preferred language modality or in a format that is language modality neutral (Andin et al., 2013; Boutla, Supalla, Newport, & Bavelier, 2004; Rudner, Fransson, Ingvar, Nyberg, & Rönnberg, 2007). However, there are differences between the neural organizations of WM for sign and speech, suggesting that at least partially different underlying mechanisms come into play when explicit WM processing is engendered—for example, when executive functions are engaged (Rudner et al., 2007) or load is high (Rönnberg, Rudner, & Ingvar, 2004; for a review, see Rudner, Andin, & Rönnberg, 2009). The main goal of the present study was to determine whether preexisting semantic and phonological representations in the sign-based mental lexicon improve WM performance in the visuospatial domain and whether such representations mitigate the effect of increasing memory load, in line with the predictions of the ELU model (Rönnberg et al., 2013).

In order to achieve this goal, we manipulated preexisting representations using different materials and groups. Three groups took part in the experiment: two groups who were native users of BSL—deaf and hearing—and one hearing, sign-naïve group. We recruited both deaf and hearing signers to control for the effect of auditory deprivation, which has been shown to influence neural organization (Bavelier, Dye, & Hauser, 2006; Cardin et al., 2013). Because BSL users were recruited for the present study, the signs of BSL served as the familiar signs. Swedish Sign Language (SSL) is another well-documented European sign language that is mutually unintelligible with BSL. Thus, SSL signs were used as unfamiliar signs. Nonsigns were created by combining sign components in a manner that contravenes the principles of signed language phonology. Because there is evidence that nonsigners are sensitive to regularities in nonsigns (Wilson & Fox, 2007), we included a fourth kind of material that consisted of meaningless nonlinguistic manual actions, in the form of ball-catching events. Other work has shown that such items can be successfully processed in WM by hearing nonsigners, despite the limited diversity of the motoric gestures involved (Rudner, 2015).

Because we wished to test WM for items with and without preexisting representations, we chose to use an n-back paradigm (Rudner, 2015). The n-back procedure avoids the need for articulation, which is likely to be better for items with preexisting representations than for those without, and this procedure has previously been used successfully to study WM for both sign language (Rudner et al., 2007; Rudner, Karlsson, Gunnarsson, & Rönnberg, 2013) and gestures (Rudner, 2015). The n-back paradigm also allows for the parametric manipulation of WM load (Barch et al., 1997), enabling investigation of the potential interactions between load, material, and group.

We reasoned that sign language users have preexisting representations, comprising semantic and phonological information relating to their own sign language, that may bear phonological similarity to an unfamiliar sign language. Nonsigners, on the other hand, have no preexisting representations, with or without semantic or phonological information, relating to sign language. Thus, by comparing WM for familiar and unfamiliar sign languages in sign language users, we could isolate the effect of semantic information in preexisting representations, while no such effect should be found for nonsigners. Similarly, by comparing WM for unfamiliar signs and nonsigns in signers, we could isolate the potential effect of the phonological information in preexisting representations, and again no such effect should be found for nonsigners. Indeed, in nonsigners we should find no difference in WM performance between the two categories of lexical signs (familiar and unfamiliar), or between signs and nonsigns, since nonsigners have no preexisting representations with information concerning either the semantics or phonology of any of these categories of items. However, we also reasoned that the differences in motoric diversity relating to hand shape, position, and movement between nonsigns and nonlinguistic manual actions would lead to differences in WM performance for all three groups of participants, based on differences in the richness of representation and mutual salience. Furthermore, by definition, signers are expert at processing signs, and thus we expected them to have better WM performance than nonsigners with all three sign-related materials (Ericsson & Kintsch, 1995). On the basis of previous work showing better performance by deaf signers than by hearing nonsigners on a nonverbal visuospatial task (Corsi blocks: Geraci, Gozzi, Papagno, & Cecchetto, 2008; Orsini et al., 1987), we expected this effect, attributed to experience of sign language, to generalize in the present study to the nonlinguistic manual actions.

The main aim of the present study was to test whether the enhancement of WM capacity due to semantic and phonological representations in the mental lexicon in long-term memory can be generalized to sign-based representations in the visuospatial domain. We also investigated whether sign language experience generally improves WM for manual gestures, irrespective of semantic content or phonological structure. Furthermore, we studied whether sign language experience mitigates the effect of increasing WM load, as is predicted by the ELU model, and if so, whether any such interaction is influenced by preexisting semantic or phonological representations.

Specifically, we predicted that signers would perform better on the n-back task with familiar than with unfamiliar signs (semantic representation), and better with unfamiliar signs than with nonsigns (phonological representation), as well as better with nonsigns than with nonlinguistic manual actions (motoric diversity). We predicted no difference in performance between the different sign-based materials for nonsigners, but we did predict that they would perform better with sign-based materials than with nonlinguistic manual actions (motoric diversity). At the same time, we predicted better performance for signers than for nonsigners on all materials, due to their experience with visuospatial information. We did not predict differences in performance between the two signing groups. Furthermore, we predicted that increasing memory load would reduce n-back performance for all groups, but that this effect would be mitigated by sign language experience, preexisting representation, and motoric diversity.

Method

Participants

The 68 participants belonged to three groups: deaf signers (DS), hearing signers (HS), and hearing nonsigners (HN). Both HS and DS groups were included to control for any effect of auditory deprivation. Group size was estimated on the basis of previous experience with mixed repeated measures designs—for instance, in Rudner, Davidsson, and Rönnberg (2010). Details of the groups are shown in Table 1. The three groups did not differ in terms of age and nonverbal intelligence, measured using the t score of the block design scale from the WASI battery (Wechsler, 1999). All participants had completed secondary education. All of the HS had at least one deaf parent with whom they communicated in sign language and had been exposed to BSL before the age of 3 years. All but two of the DS had at least one deaf parent. One deaf signer with hearing parents had been exposed to BSL before the age of 3, and the other before the age of 5. The sign language fluency of the two signing groups was assessed using the BSL Grammaticality Judgment Test (Cormier, Schembri, Vinson, & Orfanidou, 2012). The signers had native or near-native proficiency in BSL—see Table 1. Because we were using SSL materials as semantically inaccessible but phonologically well-formed items (see below), we ensured that none of the participants was familiar with SSL. All of the participants gave their written informed consent, and this study was approved by the UCL ethics committee.

Table 1 Participant information (standard deviations in parentheses)
Full size table

Materials

The stimulus set included four different types of material. We created three types of sign-based materials: lexical signs in BSL, lexical signs in SSL, and nonsigns. The fourth type of material consisted of images of the model catching a ball (nonlinguistic manual actions). These materials were constructed as follows.

BSL

An initial set of about 100 signs that potentially fulfilled the criteria for BSL stimuli were selected from Vinson, Cormier, Denmark, Schembri, and Vigliocco (2008), which provides an inventory of BSL signs ranked with respect to age of acquisition (AoA), familiarity, and iconicity on the basis of average ratings obtained from 30 deaf BSL signers. Rankings were used for stimulus matching. In addition, complexity ratings were obtained from two deaf native BSL signers. The raters were asked to look at videos of the candidate signs, concentrating on the movements of the model’s hands, and then to rate complexity on a scale of 0 to 4, based on first impressions. Each sign was viewed twice. Pearson’s correlation was computed to determine the interrater reliability (IRR), r = .49, p < .001. Thus, the BSL material consisted of items that we have every reason to believe should correspond to existing semantic and phonological representations stored in the long-term memories of DS and HS, but not of HN.

SSL

An initial set of about 100 SSL signs was selected from the Swedish Sign Language Dictionary (Hedberg et al. 2005). The inventories of contrastive hand shapes and locations differ somewhat between signed languages. However, only a small number of BSL hand shapes are not found in SSL, and vice versa, and these tend to be rarely occurring hand shapes only found in a small number of signs. For example, a BSL hand shape made with the index and little fingers extended from the fist does not occur in SSL. However, only three signs with this hand shape are to be found in Brien’s (1992) dictionary of BSL. This can be compared to 292 entries for the fist hand shape in BSL, and 213 in SSL. SSL was chosen for this study because, although the inventories of contrastive hand shapes, locations, and movements in SSL are highly similar to those of BSL, SSL is not generally familiar to BSL users, and the lexical similarity between the two sign languages is only 35% (Mesch, 2006), a figure indicating two historically unrelated sign languages (Woll, 1984).

Two deaf native signers of SSL ranked all items for AoA (IRR: r = .80, p < .001), familiarity (IRR: r = .81, p < .001), iconicity (IRR: r = .89, p < .001), and complexity (IRR: r = .75, p < .001), according to the principles used for the BSL sign ratings; two deaf native signers of BSL provided additional complexity ratings (IRR: r = .77, p < .001) and were asked whether any of the signs could be considered BSL signs. If a sign was considered to be a BSL sign by any of the judges, it was removed from the set. The remaining SSL signs were not lexical signs in BSL, and their semantic content was not transparent. Thus, the SSL material consisted of items that we have every reason to believe should correspond to existing phonological but not semantic representations stored in the long-term memories of DS and HS, but not of HN (i.e., they were possible signs of BSL).

Nonsigns

About 100 nonsigns were generated by deaf native BSL signers. Most of these nonsigns had previously been used in behavioral studies (Orfanidou, Adam, McQueen, & Morgan, 2009; Orfanidou, Adam, Morgan, & McQueen, 2010), but additional nonsigns were created specifically for the present study. The nonsigns were constructed so as to violate the phonological rules of BSL, and therefore were not phonologically well-formed (i.e., they were impossible signs). For example, some nonsigns had movements of both hands, but the hands had different hand shapes, or there was a change of location on the body with movement from a lower to a higher location (well-formed BSL signs that involve a change of location height must move from a higher to a lower location). Other nonsigns included those with an unusual place of contact on the signer’s body—for example, a nonsign could occlude the signer’s eye—or with an unusual place of contact on the signer’s hand—for example, a hand shape with the index and middle fingers extended, but contact only between the tip of the middle finger and a location on the body. Complexity ratings were again obtained from native BSL signers, as above (IRR: r = .32, p = .03). Although statistically significant, the IRR coefficient for nonsign complexity is low. This may reflect the fact that the characteristics of the nonsigns were unusual. Thus, the nonsign material consisted of items that we have every reason to believe included existing phonological components, although they had neither semantic representations nor phonologically permissible combinations of the components (i.e., they were without a phonological representations), stored in the long-term memories of the DS and HS, but not of the HN.

Nonlinguistic manual actions

This type of material consisted of the model catching a soft, bright green ball about 15 cm in diameter, thrown by an assistant to different locations proximal to the model’s torso. This provided a control condition that included movements of the hands and arms to a range of locations but with limited variation in hand shape. These stimuli were non-sign-based and nonlinguistic, being generated in a bottom-up manner in response to an external stimulus. Thus, we have no reason to believe that any of these items would correspond to linguistic representations stored in the long-term memories of any of the participants.

Stimulus set

A final set of 45 unique items was selected for each of the four types of material—that is, 180 items in all. The three categories of sign-based material were selected for similar AoA, familiarity, iconicity (lexical signs only), and complexity (based on the BSL signers’ ratings). A univariate analysis of variance (ANOVA), in which Stimulus Type (BSL and SSL, plus nonsigns only for the complexity analysis) was entered as the fixed factor, and familiarity, iconicity, AoA, and complexity were entered as the dependent variables, showed no significant differences between the different materials [familiarity, F(1, 88) = 2.9, p = .09; iconicity, F(1, 88) = 3.1, p = .08; AoA, F < 1; complexity, F < 1]. Importantly, there was no difference in rated complexity, despite the low IRR for nonsigns. Table 2 summarizes the characteristics of the sign-based materials, Appendix A lists the BSL and SSL signs, and Appendix B lists the nonsigns. The selection ensured that a wide range of hand shapes, movements, and locations were represented in a balanced manner over sign-based categories and that the nonlinguistic manual actions were performed over a broad range of locations.

Table 2 Material information (mean ratings, with standard deviations in parentheses)
Full size table

The final set of stimulus items was recorded in a studio environment using a digital high-definition camera. The signing was produced by a male deaf native signer of German Sign Language who was unfamiliar with either BSL or SSL. He was dressed in black and visible from the hips to above the head, against a blue background. All items were signed with comparable ease, speed, and fluency; no mouthing was used. The items were modeled individually, and thus there were no transitional movements between forms. The videos of the individual items were between 2 and 3 s long. The mean durations of the stimuli were as follows: BSL, 2.77 s; SSL, 2.68 s; nonsigns, 2.75 s; nonlinguistic manual actions, 2.55 s. A univariate ANOVA in which Material was entered as fixed factor and duration as the dependent variable showed a significant effect of material on duration, F(3, 180) = 4.481, p = .005. Pairwise comparisons showed that the duration of the nonlinguistic manual actions was significantly shorter than the durations of both the BSL signs, p = .001, and the nonsigns, p = .004, and that the nonlinguistic actions tended to be shorter than the SSL signs, p = .053. We found no other significant differences in duration between the material types, all ps > .16. Since the model was not a native user of either BSL or SSL, all of the sign-based materials were equally “accented.”

Task and design

We used an n-back task, in which WM load was systematically varied by manipulating n (one, two, three). All tasks were administered using the DMDX software (Forster & Forster, 2003). Two different lists of each type of material were constructed for each of the three versions of the task (one-back, two-back, three-back). Each list included 45 items that were arranged so that there would be 16 or 17 correct “yes” responses, in accordance with the task description, but no more than four correct “yes” responses or six correct “no” responses in a row. Each item could be repeated up to three times, and five lures were also included in each of the lists.

The participants were instructed to make a “yes” response when the video currently being shown exactly matched the last video in the sequence (one-back), the last-but-one video in the sequence (two-back), or the video three steps back in the sequence (three-back). Otherwise, a “no” response was required. The responses were given by pressing the appropriate button on a two-button box. The “yes” responses were given with the participant’s preferred hand. All of the participants performed all three versions of the task (n back: one, two, three) with one list of each of the materials. Lists and task order were balanced across participants within groups, and material order was randomized within each task. Responses were collected by buttonpress, and d' (Stanislaw & Todorov, 1999) was calculated. Because of the near-ceiling performance for the one-back task with sign-based stimuli, these d' scores were arcsine-transformed into radians to provide for a more normal distribution (Studebaker, 1985). The arcsine-transformed scores were used in all analyses. The time between stimulus onsets was 4 s, and the participants were given 3.5 s to respond.

Results

The overall pattern of performance on the n-back task is shown in Table 3.

Table 3 Mean d’ scores and arcsine-transformed scores and standard deviations for all groups under all conditions
Full size table

Effect of semantic representation and interaction with load

The effect of semantic representation and its interaction with load were determined by computing a 2 × 3 × 3 mixed repeated measures ANOVA, with two within-participants factors, Type of Material (BSL, SSL) and Load (one-back, two-back, three-back), and one between-participants factor, Group (DS, HS, HN). The analysis revealed main effects of all three factors: material, F(1, 65) = 6.07, MSE = .05, p = .016, partial eta-squared (η p 2) = .09; load, F(2, 130) = 49.43, MSE = .09, p < .001, η p 2 = .43; and group, F(2, 65) = 9.97, MSE = .22, p < .001, η p 2 = .24. The predicted two-way interaction between material and group was marginally significant, F(2, 65) = 2.87, p = .06 (see Fig. 2), as was the predicted three-way interaction, F(4, 130) = 1.55, p = .19. None of the interactions was statistically significant.

Fig. 2
figure 2

Interaction between material (BSL, Swedish Sign Language [SSL]) and group (deaf signers [DS], hearing signers [HS], hearing nonsigners [HN]). Error bars show standard errors for the individual conditions and groups. ** p < .01, *** p < .001

Full size image

The predicted interactions were investigated by computing separate ANOVAs for each of the groups. Contrary to our prediction, no statistically significant main effect of material emerged for DS, F(1, 23) = .57, MSE = .04, p = .46. However, there was a statistically significant main effect of load for this group, F(2, 46) = 13.27, MSE = .09, p < .001, as well as a statistically significant interaction between material and load, F(2, 46) = 3.52, MSE = .09, p = .04. Separate ANOVAs for each of the materials showed a significant main effect of load with SSL, F(2, 46) = 23.41, MSE = .06, p < .001, η p 2 = .50, but not for BSL, F(2, 46) = 1.23, MSE = .12, p = .30, η p 2 = .05. Further investigation of the Material × Load interaction using paired-samples two-tailed t tests, adjusted for multiple comparisons, showed significantly better performance with BSL than with SSL when WM load was high, at n = 3, t(23) = 3.03, p = .02, but no difference at n = 1, t(23) = 1.47, p = .46, or n = 2, t(23) = 0.08, p = 1; see Fig. 3.

Fig. 3
figure 3

Statistically significant interaction between load and material (BSL, SSL) for deaf signers. Error bars show standard errors for the individual conditions. * p < .05

Full size image

For HS, we found a statistically significant main effect of material, revealing significantly better performance with BSL than with SSL, F(1, 19) = 11.38, MSE = .04, p = .003, in line with our prediction. A statistically significant main effect of load also emerged, F(2, 38) = 27.05, MSE = .06, p < .001, but no statistically significant interaction, F(2, 38) = 0.16, MSE = .05, p = .85. For HN, there was no statistically significant main effect of material, F(1, 23) = 0.03, MSE = .06, p = .87, in line with our prediction. We did observe a statistically significant main effect of load for HN, F(2, 46) = 15.00, MSE = .11, p < .001, but no statistically significant interaction with material, F(2, 46) = 0.08, MSE = .08, p = .92.

Further investigation of the predicted two-way interaction between material and group, computing separate ANOVAs for BSL and SSL, revealed significant main effects of group for both BSL, F(2, 65) = 10.77, MSE = .13, p < .001, and SSL, F(2, 65) = 6.79, MSE = .14, p = .002. With BSL, the performance of DS was significantly higher than that of HN, mean difference (MD) = .25, p < .001, and the performance of HS was also significantly higher than that of HN, MD = .25, p < .001, but we found no difference in performance between DS and HS, MD = .01, p = 1. This pattern of between-group differences was as predicted. With SSL, the performance of DS was significantly higher than that of HN, MD = .23, p = .001, as predicted. However, while there was no difference in performance between DS and HS, MD = .10, p = .41, the difference in performance between HS and HN, MD = .13, p = .15, also did not reach significance.

Effect of phonological representation and interaction with load

The effect of phonological representation and its interaction with load were determined by computing a 2 × 3 × 3 mixed repeated measures ANOVA, with two within-participants factors, Material (SSL, nonsigns) and Load (one-back, two-back, three-back), and one between-participants factor, Group (DS, HS, HN). The analysis revealed main effects of all three factors: material, F(1, 65) = 4.71, MSE = .06, p = .034, η p 2 = .07; load, F(2, 130) = 77.07, MSE = .08, p < .001, η p 2 = .54; and group, F(2, 65) = 7.04, MSE = .20, p = .002, η p 2 = .18. The predicted two-way interaction between material and group was not significant, F(2, 65) = 0.61, p = .55, nor was the predicted three-way interaction, F(4, 130) = 0.48, p = .75.

The predicted two-way interaction between material and group was investigated by computing separate ANOVAs for each of the groups. Contrary to our prediction, we observed no statistically significant main effect of material for DS, F(1, 23) = 0.19, p = .67, or HS, F(1, 19) = 2.15, p = .16, and the tendency observed for HN, F(1, 23) = 2.95, p = .10, showed marginally better performance with nonsigns than with SSL. Further investigation of the interaction, computing a separate ANOVA for nonsigns, revealed a statistically significant main effect of group, F(2, 65) = 4.44, MSE = .55, p = .016. Bonferroni-adjusted pairwise comparisons showed a statistically significant difference in performance with nonsigns between DS and HN, MD = .17, p = .015, but not between HS and HN, MD = .12, p = .16, or between DS and HS, MD = .05, p = 1. Investigation of the three-way interaction, computing separate ANOVAs for nonsigns for each of the three groups, showed significant main effects of load for all three groups (ps < .001 for all tests).

Effect of motoric diversity and interaction with load

The effect of motoric diversity and its interaction with load were determined by computing a 2 × 3 × 3 mixed repeated measures ANOVA, with two within-participants factors, Material (nonsigns, nonlinguistic manual actions) and Load (one-back, two-back, three-back), and one between-participants factor, Group (DS, HS, HN). The analysis revealed statistically significant main effects of material, F(1, 65) = 511.69, MSE = .06, p < .001, η p 2 = .89, and load, F(2, 130) = 102.40, MSE = .05, p < .001, η p 2 = .61, but the effect of group was only marginal, F(2, 65) = 2.97, MSE = .13, p = .059, η p 2 = .18. The two-way interaction between material and load was significant, F(2, 130) = 3.81, p = .03, reflecting the fact that the negative effect on performance of increasing load was greater for nonsigns than for nonlinguistic manual actions, probably due to a floor effect at high load with nonlinguistic manual actions, despite significant differences between all levels of load (all ps < .001); see Fig. 4.

Fig. 4
figure 4

Two-way interaction between material (nonsigns, nonlinguistic manual actions) and load. Error bars show standard errors for the individual conditions. *** p < .001

Full size image

The predicted two-way interaction between material and group was marginally significant, F(2, 65) = 3.02, p = .06; see Fig. 5. Investigation of this interaction, with an ANOVA including nonlinguistic manual actions only, showed no significant main effect of group, F(2, 65) = 0.39, p = .68, reflecting the fact that the effect of group found for nonsigns did not generalize to nonlinguistic manual actions. The two-way interaction between group and load was not significant, F(4, 130) = 1.08, p = .37, and neither was the three-way interaction, F(4, 130) = 1.20, p = .32.

Fig. 5
figure 5

Interaction between material (nonsigns, nonlinguistic manual actions) and group. Error bars show standard errors for the individual conditions and groups. * p < .05

Full size image

Discussion

The main aim of the present study was to investigate whether WM in the visuospatial domain is improved by preexisting semantic and phonological representations in long-term memory in a manner similar to WM for speech-based language (Gathercole et al., 1999; Hulme et al., 1991). We also investigated whether differences in motoric diversity influence WM for manual gestures. Furthermore, we investigated whether sign language experience generally improves WM for manual gestures and whether sign language experience, preexisting representation, and motoric diversity mitigate the effect of increasing WM load, as is predicted by the ELU model.

Effect of preexisting semantic representation

HS performed better with BSL than with SSL stimuli, in line with our prediction, supporting the notion that preexisting semantic representation improves WM performance in the visuospatial domain. We found evidence of a similar effect for DS, but only when WM load was high. Thus, the effect of preexisting semantic representation seems to play out differently for the two signing groups, possibly indicating the use of different strategies. HS have access to representations in two language modalities, sign and speech. Hall and Bavelier (2011) showed that the short-term recall performance of sign–speech bilinguals increases when they are instructed to silently mouth the spoken equivalents of to-be-remembered items presented in sign language. This applied even with signed recall. Thus, for individuals who have well-established speech-based representations, it may be more efficient to recode signs they know into their spoken equivalents, in order to retain them in WM, than to process sign-based representations. However, it is possible that this strategy is less effective, or even counterproductive, for unfamiliar signs that do not have an existing semantic representation.

Deafness restricts access to spoken language and makes it hard to develop speech-based representations. Thus, as compared to HS, DS are likely to be more reliant on sign-based representations during WM processing. The results of the present study indicate that DS process familiar and unfamiliar signs just as successfully in WM when load is low or moderate, but also suggest that when load is high, preexisting semantic representation facilitates WM processing also for DS. This finding is in line with flexible-resource models of WM that propose that the quality rather than the quantity of WM representations determines performance (Ma et al., 2014). We suggest that for DS, preexisting semantic representations enhance the quality of the representations temporarily maintained in WM, thus releasing WM resources to deal with increased load. This may become particularly important when the quantity of items is large. Such an interpretation is in agreement with the ELU model (Rönnberg et al., 2013), which states that when preexisting representations cannot be activated due to a mismatch with the input, explicit processing demands increase. Here we see the opposite effect: When the matching process is enhanced because preexisting semantic representations are available, the effect of load is decreased. This supports the notion that the ELU model can explain phenomena related to sign language processing, and thus has cross-modal validity. Because DS performed relatively well even at the highest load level tested in the present study, future work should investigate the effect of preexisting semantic representation at even higher levels of WM load.

We found no significant difference in performance between DS and HS with any of the materials, suggesting that even if different strategies were used, they did not differ in efficiency. However, the findings of the present study also suggest that the representational benefit of recoding familiar signs as words, identified by Hall and Bavelier (2011), is restricted to the population they tested, HS, but can be generalized across speech–sign pairs from American English–American Sign Language, tested in their study, to British English–BSL, tested here.

No effect of preexisting phonological representation

Because the forms of signs are sometimes visually motivated (iconic) in sign language (Thompson, Vinson, Woll, & Vigliocco, 2012), the formally contrastive elements in phonology often carry meaning. For example, signs may depict the perceptual features of an object, such as an airplane’s wings; action-based features, such as drinking; or action location, such as the head for thinking (BSL examples; Thompson et al., 2012). This means that the signs of an unfamiliar sign language that are not lexicalized in a particular signer’s own language, or even nonsigns, may nonetheless bear semantic information. Thus, the comparison of WM for familiar versus unfamiliar signs in the present study is a conservative test of the influence of semantic information on WM processing. By the same token, any semantic influence at play during phonological processing would have tended to enhance performance with unfamiliar signs relative to nonsigns, rendering the comparison of SSL to nonsigns a liberal test of the effect of preexisting phonological representation. Because we observed no difference in performance between SSL and nonsigns for either of the signing groups in the present study, we found no evidence of an effect of preexisting phonological representation. The absence of a phonology-related effect in the present results was all the more surprising because a wealth of evidence has suggested that phonological representation is an important factor in WM processing. Indeed, WM capacity has been shown to be influenced by a range of factors relating to phonology. These include not only phonological similarity, but also the length of to-be-remembered items, as well as articulatory suppression (Baddeley, 2012), and some evidence suggests similar effects for sign language (for a review, see Wilson, 2001). Effects of formational similarity have also been found for nonsigns (Wilson & Fox, 2007) and for meaningless gestures (Rudner, 2015).

However, other work has shown that the effects of phonological similarity on WM for sign language can be elusive (Rudner & Rönnberg, 2008a), despite effects of semantic category (Rudner et al., 2010; Rudner & Rönnberg, 2008a). Indeed, in a recent study, the researchers showed that although deaf users of SSL displayed an effect of phonological similarity on the short-term store, as measured by digit span, this effect did not generalize to digit-based WM, as measured by operation span, and when the same experiment was performed with deaf users of BSL, no clear effect of phonological similarity was discernible for either the short-term store or WM (Andin et al., 2013). Since the versions of both digit span and operation span used in Andin et al.’s (2013) study required the recoding of printed stimuli to the preferred language modality, it was argued that the difference in the patterns of effects between the users of these two sign languages could be explained by a greater emphasis on sign-based deaf education in Sweden versus a bias toward oral education for deaf children in the UK. This explanation is supported by evidence that speech-based phonology influences memory performance in British deaf individuals (Conrad, 1972; MacSweeney, Campbell, & Donlan, 1996), whereas we know of no evidence of phonological similarity relating to BSL influencing recall.

Despite the lack of any previous evidence of a sign-phonology effect on memory performance in BSL users, this group has been shown to display an awareness of the phonological structure of their language (MacSweeney et al., 2008), and because all items were presented as manual actions in the present study, the phonological structure of the SSL signs was clearly visible. It is possible that in the present study the nonsigns were more perceptually salient than the SSL signs, supporting WM encoding and thus counteracting any phonological benefit. This interpretation receives some support from the tendency for our nonsigners to perform better with nonsigns than with SSL. However, because there was no statistically significant difference in the rated complexities of the different sign-based manual gestures, this is not our preferred interpretation. Instead, we suggest that a parsimonious explanation of the significant effect of semantic representation on n-back WM performance, combined with no effect of phonological representation, is that semantic, but not phonological, information was used in determining the n-back match. Although previous work has shown an effect of speech-based phonological similarity on performance on an n-back task, imaging results suggested that the phonological similarity among items presented during an n-back task led to strategic disengagement of executive and language functions in the face of distracting information (Sweet et al., 2008), possibly leading to less distinct representations of items in terms of their phonological content (Rudner, 2015) when this information was not explicitly required for solving the task (Rudner et al., 2013). It is possible that phonological information is systematically suppressed during n-back processing when it does not specifically contribute to the task solution, which in this case required determining whether items were identical. Another possible explanation that should be entertained is the specific lack of a form-based effect for sign language processing. Future work should investigate this by manipulating the type of task and phonological demands.

Effect of sign language experience

We predicted better performance overall for signers than for nonsigners, due to experience with visuospatial information. We found that DS performed better than HN with all of the sign-based materials, but HS only performed better than HN with BSL. The relatively high performance of nonsigners overall is in line with other recent work showing that individuals with no experience of sign language can successfully perform an n-back WM task on the basis of lexical signs (Rudner et al., 2015). This could be explained by an ad-hoc, quasi-phonological processing strategy capitalizing on existing motor representations. Indeed, such an interpretation is in line with results showing an effect of formational similarity on WM for nonsigns (Wilson & Fox, 2007). At any rate, the pattern of results in the present study does not support the notion that sign language experience alone facilitates WM processing of sign-based materials. However, it does indicate that a reliance on visual information due to deafness, combined with sign language experience, facilitates WM processing of sign-based materials. It also suggests that when HS have preexisting semantic representations of sign-based items, they may be able to adopt a mnemonic strategy that allows them to outperform HN. This further supports the notion that HS, who have ready access to speech-based representations, may use these strategically during WM processing (Hall & Bavelier, 2011).

Sign language experience does not enhance WM for nonlinguistic manual actions

Our results showed the predicted poorer n-back performance with nonlinguistic manual actions than with nonsigns across the groups. Our prediction was based on motoric diversity in relation to hand shape, position, and movement, allowing for richer and better-differentiated manual representations. The shorter duration of the nonlinguistic stimuli possibly also reflected reduced information for these items. However, it should be noted that stimulus length did not influence the timing of the WM task, and thus did not confound the effect of load. We predicted that the effect of load would be smaller for nonsigns than for nonlinguistic manual actions, but this was not the case.

Furthermore, we did not find the predicted effect of sign language experience facilitating WM performance with nonlinguistic manual actions for either of the signing groups. This finding suggests that the better visuospatial processing for DS than for HN (Geraci et al., 2008) with the Corsi block task does not generalize to nonlinguistic manual actions when the task does not require spatial processing. However, it does support the notion that nonsigners capitalize on existing motor representations during a gesture-based WM task, even when the to-be-remembered items are nonlinguistic manual actions, in line with the findings of Rudner (2015). We suggest that WM is adapted to the storage and processing of linguistic items, even when those items are gesture-based and in the visuospatial modality. This may be due to the systematic rhythmic motor patterns inherent in those items activating aspects of existing phonological representations at an abstract level that transcends modality, or simply to the mutual distinctiveness between the motor patterns of linguistic items, but nonetheless it supports the notion of multimodal models of WM such as the ELU (Rönnberg et al., 2013).

Conclusion

We found no evidence that preexisting phonological representation improves WM in the visuospatial domain. However, we did find some evidence that preexisting semantic representation improves visuospatial WM. In particular, performance was better with BSL than with SSL for hearing BSL signers, demonstrating a facilitatory effect of preexisting semantic representation. Performance was also better with BSL than with SSL for deaf BSL signers, but only when WM load was high. This suggests that preexisting semantic representation mitigated the effect of increasing WM load for this group, possibly by enhancing the quality of the gesture-based representations temporarily maintained in WM, thereby releasing WM resources to deal with increased load, in line with the ELU model (Rönnberg et al., 2013). The difference in the effects of preexisting semantic representation for DS and HS suggests different underlying mechanisms, possibly reflecting reliance on visuospatial processing in DS and automatic access to speech-based representations in HS. Furthermore, the DS performed better than hearing nonsigners with all sign-based materials, although this effect did not generalize to nonlinguistic manual actions. We argue that DS, who are highly reliant on visual information for communication, develop expertise in processing sign-based items, even when those items do not have preexisting semantic or phonological representations.