The impact of visual cross-modal conflict with semantic and nonsemantic distractors on working memory task: A functional near-infrared spectroscopy study

Cross-modal conflicts arise when information from multisensory modalities is incongruent. Most previous studies investigating audiovisual cross-modal conflicts have focused on visual targets with auditory distractors, and only a few studies have focused on auditory targets with visual distractors. Moreover, no study has investigated the differences in the impact of visual cross-modal conflict with semantic and nonsemantic competition and its neural basis. This cross-sectional study aimed to characterize the impact of 2 types of visual cross-modal conflicts with semantic and nonsemantic distractors through a working memory task and associated brain activities. The participants were 33 healthy, right-handed, young male adults. The paced auditory serial addition test was performed under 3 conditions: no-distractor and 2 types of visual distractor conditions (nonsemantic and semantic distractor conditions). Symbols and numbers were used as nonsemantic and semantic distractors, respectively. The oxygenated hemoglobin (Oxy-Hb) concentration in the frontoparietal regions, bilateral ventrolateral prefrontal cortex (VLPFC), dorsolateral prefrontal cortex, and inferior parietal cortex (IPC) were measured during the task under each condition. The results showed significantly lower paced auditory serial addition test performances in both distractor conditions than in the no-distractor condition, but no significant difference between the 2 distractor conditions. For brain activity, a significantly increased Oxy-Hb concentration in the right VLPFC was only observed in the nonsemantic distractor condition (corrected P = .015; Cohen d = .46). The changes in Oxy-Hb in the bilateral IPC were positively correlated with changes in task performance for both types of visual cross-modal distractor conditions. Visual cross-modal conflict significantly impairs auditory working memory task performance, regardless of the presence of semantic or nonsemantic distractors. The right VLPFC may be a crucial region to inhibit visual nonsemantic information in cross-modal conflict situations, and bilateral IPC may be closely linked with the inhibition of visual cross-modal distractor, regardless of the presence of semantic or nonsemantic distractors.


Introduction
Information is received via multisensory modalities.We focus or select the necessary information, while we ignore or inhibit unnecessary information for adaptive responses.Cross-modal conflict arises when multisensory information is incongruent.Of the modalities, visual and auditory stimuli are the primary ones. [1]revious studies investigating audiovisual cross-modal conflicts have focused on visual targets with auditory distractors, and only a few have focused on auditory targets with visual distractors. [1][4] Using stimuli of the same nature across modalities creates semantic competition for maintaining and updating information because numbers have a well-defined semantic organization and are approximately equal in their degree of activating semantic information. [5,6]Semantic distractors generally have a higher interference effect than nonsemantic distractors across categories.However, no study has examined the differences in the impact of visual cross-modal conflict with semantic and nonsemantic competition.
[9] The VLPFC shows hemispheric lateralization that depends on the stimulus modality.The right VLPFC is mainly involved in general inhibition of task-irrelevant distractors and is associated with visuospatial information inhibition, [7,8] while the left VLPFC is mainly linked to semantic and verbal information inhibition. [7,9]Accordingly, we determined these areas as regions of interest (ROIs).The ROI approach is often used to assess the activity of the frontoparietal brain regions, and functional near-infrared spectroscopy (fNIRS) is a neuroimaging technique, in which the ROI approach is used.
fNIRS is a noninvasive device for measuring cerebral cortical hemodynamic activity and can do so in an environment closer to daily life.fNIRS is also much quieter than fMRI and less affected by electrical or magnetic interference from auditory devices.Thus, fNIRS is an ideal imaging technique for auditory research. [10,11]his study aimed to characterize the impacts of semantic and nonsemantic distractors on visual cross-modal conflicts through a working memory (WM) task and assess the correlation between task performance and associated brain activities.Our hypotheses were as follows: first, both distractor conditions would lower task performance, especially in the semantic distractor condition.Second, higher ROI activity would be shown in both distractor conditions than in the no-distractor condition.Additionally, right and left VLPFC activities would be higher in the nonsemantic and in the semantic distractor condition, respectively.Third, brain activity changes in ROIs would correlate with changes in WM task performance.Furthering the basic understanding of brain activity during cross-modal conflict is important for elucidating the neural mechanisms underlying distractibility and the potential to contribute toward future clinical assessment and treatment of distracted patients with brain disorders.

Participants
This cross-sectional study, conducted between February 2021 and July 2021, included consecutively recruited right-handed healthy young men from the Health Sciences University of Hokkaido, Tobetsu, Japan.The sample size was calculated a priori using G*power (Universitat Dusseldorf, Dusseldorf, Germany), and the sample size to achieve a.90 statistical power level, based on the results of a previous study, [12] was 27.To be conservative, 15% was added to account for dropouts and outliers, and the determined sample size was 33.
The inclusion criteria were as follows: age range between 20 and 30 years; male sex, considering the sex differences in the WM network [13] ; and right-handedness.The exclusion criteria were as follows: medical history of neurological and psychiatric disorders; and visual or auditory impairments that affect task performance.Of the 33 participants, 3 were excluded owing to excessive artifacts and device malfunction.Therefore, a total of 30 participants (age range, 20-27 years; mean age,= 22.87 ± 1.94 years; mean score of FLANDERS handedness questionnaire, [14] 9.73 ± .93)were included in the final analysis.The study protocol was approved by the Ethics Committee of the Health Sciences University of Hokkaido (approval number: 20R138128), and written informed consent was obtained from all participants.

Experimental task
The paced auditory serial addition test (PASAT) is a complex WM test, in which a single-digit number that presents aurally is added to the previously presented number, and is answered orally in a continuous manner.This test was used as a target task.All participants performed the PASAT in 3 conditions: no-distractor, nonsemantic distractor, and semantic distractor conditions (Fig. 1).The no-distractor condition required participants to gaze at the center of the screen and perform the PASAT.In the nonsemantic distractor condition, a set of symbols referring to the WAIS-III Digit Symbol-Coding were used as visual distractors.The auditory target stimuli and visual distractor stimuli were simultaneously presented on the sound speaker and computer monitor using DMDX display software, [15] respectively.In the semantic distractor condition, number stimuli were used as visual distractors.The stimulus sound in the PASAT was set at 70 dB, and fixation or visual distractor stimuli was presented at the center of the screen (stimulus size: 4 cm × 4 cm, visual angle: 3.27°).Each stimulus was presented for 500 ms, followed by a 1500 ms interstimulus interval.The experiment was conducted in a block design, in which a 61.0 seconds PASAT was interleaved with 30.0 seconds rest.The participants completed 9 blocks comprising 3 repeated runs randomly sorted into the 3 task conditions to control for learning effect and fatigue.
The fNIRS data were analyzed based on the modified Beer-Lambert law to quantify changes in oxygenated hemoglobin (Oxy-Hb).We focused on Oxy-Hb concentration, which is the most representative indication of brain activity. [17]In this study, the baseline period comprised the 6-second period before task onset, and the average Oxy-Hb value of the baseline period was set to zero.The changes in Oxy-Hb concentration during the task were calculated as the difference from the baseline value. [18]In addition, a bandpass filter was applied between 0.01 and 0.3 Hz.Task-related changes in Oxy-Hb concentration in each ROI were averaged over the task period.

Experimental procedure
The experimental procedure consisted of 3 phases.In the practice phase, the participants sat in front of a computer monitor and were instructed to avoid eye, head, and body motion, as well as deep breathing during the NIRS measurements.Subsequently, all participants received instructions for performing the 3 task conditions and practiced each task.During the NIRS recording phase, the Oxy-Hb concentrations were continuously measured.The fNIRS recording phase consisted of 9 blocks for a total of 819.0 seconds.The self-rating phase was the final part of the procedure, where the participants were required to answer their degree of alertness using the Stanford Sleepiness Scale, a self-rating scale that takes approximately 1 minute to complete.

Near-infrared spectroscopy activation
The Shapiro-Wilk test showed that the assumption of the normality was met for Oxy-Hb concentration in each of 3 task conditions in the bilateral VLPFC and IPC (P > .05).

Relationship between the changes in task performance and functional near-infrared spectroscopy activation in the ROIs
An overview of the correlations between the Δpercentage of correct responses in the PASAT and their ΔOxy-Hb concentrations is presented in Figure 4. Spearman rank-order correlation analyses were only performed for Δpercentage of correct responses and ΔOxy-Hb in the bilateral DLPFC in nonsemantic distractor condition, as well as for Δpercentage of correct responses and right DLPFC in semantic distractor condition because of significant departure from normality.Significant positive correlations were observed between Δpercentage of correct responses and ΔOxy-Hb in the bilateral IPC in both nonsemantic distractor (left: R = 0.37, uncorrected P = .047;right: R = 0.36, uncorrected P = .049)and semantic distractor condition (left: R = 0.39, uncorrected P = .032;right: R = 0.44, uncorrected P = .015).

Discussion
The aim of this study was to characterize the 2 types of visual cross-modal conflict with semantic and nonsemantic distractors.To our knowledge, this study is the first to examine the differences between the 2 types of visual cross-modal conflict, focusing on both WM performance and brain activity.In line with our hypothesis, the present study showed that WM task performance was interrupted in both distractor conditions, but no significance was observed between the 2. Regarding brain activity, right VLPFC activity was higher only in the nonsemantic distractor condition, while no significant higher activity was observed in the left VLPFC in the semantic distractor condition.Correlations between changes in task performance and changes in brain activity were found in both distractor conditions only in the bilateral IPC, but not in other regions.
For the PASAT task performance, a significantly lower performance was observed in both distractor conditions than that in the no-distractor condition, as hypothesized.This suggests that visual cross-modal distractors significantly impair auditory task performance regardless of the semantic nature, as previously shown. [3,12]In contrast, no significant differences were observed between the 2 types of conditions, suggesting that visual cross-modal semantic competition does not significantly affect auditory WM performance compared to visual crossmodal nonsemantic competition.This may reflect the characteristics of the number of stimuli selected as target stimuli for this study.Compared to the lexical-semantic competition, the number-derived semantic competition of this study characterized a highly automated way, in which number semantics are accessed, and may result in less effort to actively capture the information. [6,7]In addition, we focused only on task accuracy.Processing speed can be a more sensitive measure to detect the differences between the 2 types of distractors. [19]or brain activity, a significantly increased activation in the right VLPFC was only observed in the nonsemantic distractor condition, in line with our hypothesis.This result suggests that right VLPFC is closely linked with the inhibition of visual object distractor and supports the preference along with right-object laterality trend of VLPFC. [7]Additionally, the lack of increase in left VLPFC activity in the semantic distractor condition may also support the possibility of insufficient or weak semantic competition.
Correlation analyses revealed significant positive correlations between Δpercentage of correct responses and ΔOxy-Hb in bilateral IPC in both distractor conditions.This indicates that increased bilateral IPC activity decreases the interference effect of visual cross-modal conflict, regardless of semantic or nonsemantic distractors.It has been reported that the bilateral IPC lacks hemispheric specialization for inhibiting distractors [19,20] and plays a crucial role in inhibiting visual distractors in neuromodulation [21] and lesional studies. [22]his study has some limitations.First, we investigated only young men.The present study deliberately omitted the effects of sex.The second limitation pertains to the characteristics of the target stimuli.The selection of a target task that causes stronger semantic competition would allow us for more insight into the differences between the 2 types of distractors.The third limitation pertains to the low spatial resolution of the NIRS.Therefore, precise brain areas could not be defined and subcortical areas could not be examined.Future studies should investigate the influence of the population factor, including female individuals and the older population, on the visual cross-modal interfering effect of semantic or nonsemantic distractors after considering the strength of target and distractor stimuli.Furthermore, future research should investigate how each brain region inhibits cross-modal distractors and what functional differences exist between the hemispheres in these brain regions, using neuromodulation techniques.
In conclusion, the visual cross-modal interfering effect of semantic or nonsemantic distractors significantly impaired auditory task performance.Significantly increased activation in the right VLPFC was only observed in the nonsemantic distractor condition, and bilateral IPC was positively correlated with task performance in both types of visual cross-modal conflict.These results suggest that the right VLPFC is a crucial brain area for specialized visual cross-modal conflict with nonsemantic distractors, and bilateral IPC plays a pivotal role in decreasing the interference effect of both types of visual cross-modal distractors.

Figure 1 .
Figure 1.Three types of experiment tasks.The right matrixes indicate all 9 types of distractors.PSAT = Paced Serial Addition Test.

Figure 2 .
Figure 2. Channel configuration.(A) Gray and white squares show illuminators and detectors; yellow squares show near-infrared spectroscopy channels.Red, blue, and green frames show VLPFC, DLPFC, and IPC, respectively.(B) All channels are placed on the surface of the standard human cortex.DLPFC = dorsolateral prefrontal cortex, IPC = inferior parietal cortex, VLPFC = ventrolateral prefrontal cortex.

A 1 -
way repeated-measures analysis of variance (ANOVA) or Friedman test was performed to determine the percentage of correct responses in the PASAT and the changes in Oxy-Hb concentration in each ROI according to the normality of the data distribution checked by the Shapiro-Wilk test.The Bonferroni method was used for multiple comparisons.In addition, correlation analyses were performed to examine the relationship between the changes in the percentage of correct responses (Δpercentage of correct responses) and the changes in Oxy-Hb concentration (ΔOxy-Hb) in ROIs in each distractor condition compared to those in the no-distractor condition.We performed Pearson product-moment correlation analysis or Spearman rank-order correlation analysis according to the normality of data distribution.The Δpercentage of correct responses and Δ Oxy-Hb were calculated by subtracting the value in the no-distractor condition from the value in each distractor condition.Statistical analyses were performed using SPSS (version 25.0;IBM Corp., Armonk, NY), and the α level was set at 0.05.

Figure 4 .
Figure 4. Relationships between the changes in task performance and functional near-infrared spectroscopy activation in each ROI.(A) Overall correlation matrixes.The bold flames show significant correlations.The color bar indicates Pearson correlation coefficient.(B) Scatterplot illustrating the relationship between Δpercentage of correct responses and ΔOxy-Hb in the bilateral IPC in nonsemantic distractor condition.(C) Scatterplot illustrating the relationship between Δpercentage of correct responses and ΔOxy-Hb in the bilateral IPC in semantic distractor condition.DLPFC = dorsolateral prefrontal cortex, IPC = inferior parietal cortex, Oxy-Hb = oxygenated hemoglobin, PASAT = paced auditory serial addition test, ROI = region of interest, VLPFC = ventrolateral prefrontal cortex.*Uncorrected P < .05. a Spearman rank-order correlation analysis.