Next Article in Journal
Effect of Blood Transfusion on Cerebral Hemodynamics and Vascular Topology Described by Computational Fluid Dynamics in Sickle Cell Disease Patients
Next Article in Special Issue
Intermodulation from Unisensory to Multisensory Perception: A Review
Previous Article in Journal
Inter-Individual Variability in Motor Output Is Driven by Recruitment Gain in the Corticospinal Tract Rather Than Motor Threshold
Previous Article in Special Issue
Visual-Based Spatial Coordinate Dominates Probabilistic Multisensory Inference in Macaque MST-d Disparity Encoding
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Brain Sciences
Volume 12
Issue 10
10.3390/brainsci12101400
brainsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Graphemic and Semantic Pathways of Number–Color Synesthesia: A Dissociation of Conceptual Synesthesia Mechanisms

by
Shimeng Yue
1 and
Lihan Chen
1,2,3,4,*
1
School of Psychological and Cognitive Sciences, Peking University, Beijing 100871, China
2
Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing 100871, China
3
National Engineering Laboratory for Big Data Analysis and Applications, Peking University, Beijing 100871, China
4
Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing 100871, China
*
Author to whom correspondence should be addressed.
Brain Sci. 2022, 12(10), 1400; https://doi.org/10.3390/brainsci12101400
Submission received: 24 August 2022 / Revised: 15 October 2022 / Accepted: 15 October 2022 / Published: 17 October 2022
(This article belongs to the Special Issue The Neural Basis of Multisensory Plasticity)
Browse Figures
Review Reports Versions Notes

Abstract

:
Number–color synesthesia is a condition in which synesthetes perceive numbers with concurrent experience of specific, corresponding colors. It has been proposed that synesthetic association exists primarily between representations of Arabic digit graphemes and colors, and a secondary, semantic connection between numerosity and colors is built via repeated co-activation. However, this distinction between the graphemic and semantic pathways of synesthetic number–color connection has not been empirically tested. The current study aims to dissociate graphemic and semantic aspects of color activations in number–color synesthesia by comparing their time courses. We adopted a synesthetic priming paradigm with varied stimuli onset asynchronies (SOAs). A number (2–6, prime) was presented in one of three notations: digit, dice, or non-canonical dot pattern, and a color patch (target) appeared with an SOA of 0, 100, 300, 400, or 800 ms. Participants reported the color as quickly as possible. Using the congruency effect (i.e., shorter reaction time when target color matched the synesthetic color of number prime) as an index of synesthetic color activation level, we revealed that the effect from the graphemic pathway is quick and relatively persistent, while the effect from the semantic pathway unfolds at a later stage and is more transient. The dissociation between the graphemic and semantic pathways of synesthesia implies further functional distinction within “conceptual synesthesia”, which has been originally discussed as a unitary phenomenon. This distinction has been demonstrated by the differential time courses of synesthetic color activations, and suggested that a presumed, single type of synesthesia could involve multiple mechanisms.

1. Introduction

Synesthesia is a condition in which individuals (synesthetes) automatically perceive certain stimuli with experiences of additional properties, such as seeing achromatic words with certain colors. The physically present stimulus that elicits synesthesia (e.g., word) is termed the “inducer”, and the additional experience (e.g., color) is termed the “concurrent” [1]. In particular, in general grapheme–color synesthesia, inducers are graphemes (e.g., letters or digits), and concurrents are colors, while in number–color synesthesia, inducers are numbers and concurrents are colors.
To quantitatively measure synesthesia, common behavioral paradigms have been developed, including the synesthetic priming paradigm [2,3]. In a typical priming task, the prime and the target were presented consecutively, and the participant responded upon the target (e.g., pressing keys to report whether a number is larger or smaller than 5 [4,5]). The performance could be influenced by the task-irrelevant, preceding stimulus (prime). When certain attributes of a prime (e.g., its shape) provide congruent information with that associated with the target, the participant’s response is facilitated, indexed by a higher accuracy and a shorter reaction time (RT). On the contrary, when the prime conveys conflicting information, response is interrupted, resulting in a lower accuracy and a longer RT. This paradigm has demonstrated an automatic processing of the manipulated attribute of the prime and, hence, its interference with the processing of a given target. This interference can be quantified by calculating the difference in RTs between congruent and incongruent conditions (hereafter termed the congruency effect). Specifically, in priming paradigms for grapheme– and number–color synesthesia, a grapheme or number prime facilitates or interrupts a synesthete’s response to a color target. For instance, for a synesthete who associates the number 5 with color red, a “5” prime will lead to quicker and more accurate color-naming of a red color patch than a green one. In contrast, naïve non-synesthetes (people who do not experience synesthesia and who have not been trained with inducer–color parings for specific research purposes) show no congruency effect in such tasks.
Studies on number–color synesthesia have found that numbers automatically elicit color processing when presented in various notations, such as Arabic digits, dot patterns, and finger counts [2,6,7], demonstrating that color concurrents have been evoked regardless of variances in perceptual forms of inducers. Mathematical operations (e.g., “5 + 2”) also facilitated response to colors corresponding to the result (e.g., 7) [8,9], which means that an inducer (“7”) does not even need to be physically present to elicit the concurrent color. Together, these findings suggest that a high-level, conceptual representation of numbers is linked to color in the synesthetes’ brains (though this may not be the only source of synesthetic connection). Early theorizations of synesthesia distinguished between “lower” and “higher” types of synesthesia. The former was thought to depend on low-level perceptual properties of the inducer, and the latter was based on semantic or conceptual attributes [10,11]. Under this framework, number–color synesthesia underscores the “higher” aspect of synesthesia, since it exists regardless of how the number is presented [6,7,12,13]. More recently, Chiou and Rich [14] proposed a conceptual-mediation model as the unifying mechanism for synesthesia, suggesting that inducers and concurrents are linked on the conceptual (“higher”) level. Temporal aspects in synesthesia have been addressed by neurophysiological studies to tease apart the underlying “perceptual” vs. “conceptual” mechanisms. For instance, Brang et al. [15,16] found early (100–200 ms) event-related potential (ERP) differences in synesthesia and other mental process such as imagery, favoring the “perceptual” nature of synesthesia, while Teichmann et al. [17] argued for a “conceptual” nature of synesthesia, given that they had decoded colors from late (300–400 ms) magnetoencephalogram (MEG) signals.
However, most number–color synesthetes report no subjective color experience for numbers in less common or non-graphemic forms (e.g., dots) [6,7,18]. Accounting for this, Berteletti et al. [7] proposed a cognitive model of number–color synesthesia, and Gertner et al. [18] extended it to other types of synesthesia with number as inducers. They suggested that in the first place, Arabic digits, as symbolic graphemes, induce color experiences. When synesthetes observe a digit, they extract its semantic meaning (a numerosity) and meanwhile experience a color. Repeated simultaneous activations of paired numerosity and color representations establish the connections between them through associative learning mechanisms. The primary-digit–color connection is “synesthesia”, and the secondary, learned numerosity–color connection is termed “pseudo-synesthesia”. Pseudo-synesthesia activates color representations and produces synesthesia-like behavioral results, but this activation is usually too weak to evoke conscious color experiences. Note that graphemes (with emphasis to their identities rather than low-level features) are considered conceptual entities in the theorization of synesthesia [14], and semantic processing also operates on the conceptual level. Therefore, the proposed distinction between “synesthesia” and “pseudo-synesthesia” is indeed a distinction of processes under the umbrella of conceptual/higher synesthesia. To avoid confusion, we hereafter use the terms “graphemic pathway” and “semantic pathway” of synesthesia to refer to “synesthesia” and “pseudo-synesthesia” in the model by Berteletti et al [7], and use the term “synesthesia” for the phenomenon as a whole (In this article, we mainly discuss “pathways” as part of a cognitive model, i.e., as connections between representations. Still, it is likely that actual neuronal connections underlie these pathways.)
Our present study aims to dissociate the hypothesized graphemic and semantic pathways by comparing the time course of synesthetic activation through them. Different from previous studies which distinguished between perceptual and conceptual synesthesia, we are making this graphemic/semantic dissociation within conceptual synesthesia. Empirically, number–color synesthesia is consensually considered as conceptual/higher synesthesia, and the dissociation of graphemic and semantic pathways is a distinction of number–color synesthesia mechanisms. Theoretically, grapheme identities are considered as concepts in theorizations of synesthesia [14], and semantics is clearly conceptual rather than perceptual. Therefore, from both empirical and theoretical points of view, the graphemic/semantic dissociation is a dissociation within conceptual synesthesia. To the best of our knowledge, there have not been empirical studies on this dissociation. We hereby used numbers in different notations to dissociate the graphemic and semantic aspects of the synesthetic inducer. Specifically, numbers in graphemic notations (e.g., Arabic digits) activate synesthetic colors via both graphemic and semantic pathways, while numbers in non-graphemic notations (e.g., dots) activate colors via the semantic pathway alone. By varying the interval between onsets of the prime and target (i.e., the stimulus onset asynchrony, SOA) in a priming paradigm, we are able to measure synesthetic color activation (quantified by congruency effect) at different time points following the presentation of a number prime.
In our experiment, we used numbers in different notations (digits, dice, and non-canonical (NC) dot patterns) as the prime and manipulated the SOAs between the prime and target. Synesthetes and non-synesthetes reacted to a color patch target as quickly as possible. The SOA at which the congruency effect occurs, which we hereafter refer to as the latency of congruency effect, reflects the speed of synesthetic color activation. The maximum congruency effect size across SOA, which we hereafter refer to as the magnitude of congruency effect, reflects the strength of number–color association. Since graphemic processing precedes semantic processing [19,20,21], we expected that digits would elicit synesthetic color faster than dot patterns. In accordance with most synesthetes’ differential conscious experiences (digits are consciously experienced as colored, while dot patterns are not), we also expected that digits would activate synesthetic colors more strongly. Taken together, congruency effect yielded by digits should have a shorter latency and a larger magnitude than either dot notations. Additionally, participants may recognize (i.e., access the semantic numerosity of) dice patterns more rapidly than NC patterns due to more familiarity with dice, but both notations should lead to color activation via the semantic pathway. Therefore, we expect dice patterns to yield congruency effects with a shorter latency and similar magnitudes when compared with those in NC patterns.

2. Materials and Methods

2.1. Participants

Eleven synesthetes (4 males and 7 females, age range 18–26 years, M = 20.91, SD = 2.39) and eleven gender-matched non-synesthetes (age range 18–26 years, M = 20.91, SD = 2.66) participated in the experiment. Synesthete participants were then assessed and screened with a customized eight-item questionnaire. With this questionnaire, they reported several key features of synesthetic experience (see Appendix A for details). They were considered genuine synesthetes by meeting the following criteria: (1) they had a general feeling that numbers are colored; (2) their experiences of concurrent colors were involuntary; (3) their number–color pairings were consistent across time. They also gave verbal descriptions of their number–color pairings, which matched with their color choice in the experiment (see Section 2.4). This demonstration of consistency supported the genuineness of synesthesia. Non-synesthete participants denied having similar experiences as those of the synesthetes. All participants were right-handed.

2.2. Stimuli and Apparatus

We used two kinds of stimuli in the experiment, the numbers and the color patches. A number stimulus was always colored black and occupied a 2° × 2° area at the center of the screen. We used three notations (Arabic digit, dice pattern, and NC dot pattern) for five numbers (2–6), the stimulus for a given number × notation was fixed, e.g., NC dot configuration for “4” did not change between trials (Figure 1). A color stimulus was a 2° × 2° square color patch appearing at the center of the screen. Specific colors differ between participants (see Section 2.4 for color selection procedure, and Appendix B for list of colors used).
The experiment was programmed with the Psychophysics Toolbox-3 [22,23,24] implemented on MATLAB 2018b (MathWorks Inc., Natick, MA, USA), on a 27-inch LCD computer screen. The refresh rate was 60 Hz. Participants rested their head on a chin rest, which kept their eyes aligned to the screen center at an 80 cm distance.

2.3. Design

We adopted a 3 (number notation) × 5 (SOA) × 2 (congruency) factorial design, yielding 30 experimental conditions. Number notations include Arabic digit, dice pattern, and NC pattern. SOAs were picked by a pilot test and set with five levels: 0, 100, 300, 400, and 800 ms. A trial was defined as “congruent” if the target color matched the corresponding color of the number prime and “incongruent” if they did not match.
We arranged the trials in 8 blocks. Only two numbers and their corresponding colors appeared in each block. Thus, a color (e.g., red or green) always had an equal probability (50%) of being either congruent or incongruent to its preceding number prime. Considering that numerical range (small or large numbers, usually taking 4 as the cut point) would affect the speed and accuracy of object numerosity processing [25] and may influence behavioral effects of synesthesia [7], we used numbers 2, 3, 5, and 6. We arranged them in a way to balance the numerical ranges (Table 1; except for the cases of four synesthetes and four non-synesthetes, in which number 4 was substituted for 3 or 5 in their experiments. See Section 3). Specifically, we included an equal number of trials with small and larger numbers, as well as an equal number of blocks where two numbers were in the same or different numerical ranges. Since some of our synesthetes reported associating numbers or colors to specific spatial layouts (e.g., feeling that certain colors belong to the left or right side), we therefore reversed the correspondence of left- and right-hand responses to colors in repetitions of identical blocks.

2.4. Procedure

Participants completed the experiment in two days, with four blocks (taking approximately 90 min) each day. Upon arrival at the laboratory on Day 1, they chose colors for numbers 2–6 from the custom font color palette in Microsoft Word (Microsoft Inc.). Synesthetes chose their synesthetic colors, and non-synesthetes assigned any color they preferred to the given numbers. In this way, though color sets were not physically identical across participant groups, they were functionally identical in that they were chosen by participants themselves.
Each block contained a color response practice session and a main experiment session. The practice session familiarized the participant with the correspondence of response keys (left and right “shift” on the keyboard) to colors. Participants were instructed to press the left “shift” key when seeing one color, and the right “shift” key when seeing the other. This response should be as quick as possible. Experimental stimuli were presented against a white background. Each trail started with a 300 ms central fixation cross (diameter: 0.6°, RGB: (128, 128, 128)), and a color stimulus was presented at screen center until response (maximum waiting time: 2000 ms). Immediately after response, a 500 ms “correct/incorrect” feedback was given. The next trial started after an inter-trial interval (ITI) of 1000 ms. There were 80 randomized trials (40 for each color).
The main experiment followed the practice session. Participants were instructed to respond to colors with the same keys as in the practice session, and then report numbers by pressing “Y” and “B” keys (the keys were labeled with numbers; “Y/B” always corresponded to the smaller/larger number in the block). The response to color (hereafter referred to as the color response) should be as quick and accurate as possible, and the response to number (hereafter referred to as the number response) could be at their own pace. In each trial, a central fixation cross (diameter: 0.6°, RGB: (128, 128, 128)) was presented for 300 ms, and a number stimulus was presented at screen center for 50 ms. After a delay (for SOA > 0 conditions), a color stimulus was presented at screen center until color response (for SOA = 0 ms, the number stimulus was superimposed on the color patch; maximum waiting time: 2000 ms). Immediately following color response, a prompt for number response was presented until the response was made (maximum waiting time: 5000 ms). No feedback was given throughout the trial. The next trial started after a random ITI of 2000–2500 ms (Figure 2).
Within a block, identical trials (i.e., trials with same condition and number) were repeated 3 times. Thus, each block contained 30 (conditions) × 2 (numbers) × 3 (repetition) = 180 trials. These trials were randomized. In total, the experiment contained 8 × 180 = 1440 trials.

3. Results

The color patches for each individual are listed in Appendix B (Table A1). There were four synesthetes who either had similar colors associated with two numbers or had black or white concurrent colors for 3 or 5. To ensure that the corresponding colors were easily distinguishable and could be seen against a white background and that a black number stimulus could be seen when superimposed on the color patch, 3 or 5 was substituted for 4 in their experiments. Correspondingly, the same number changes were made in four non-synesthetes’ experiments.
Four blocks were excluded due to technical failure. Two more were excluded due to number response accuracy lower than 3 SDs below average, which was thought to indicate inattentiveness. Trials with incorrect responses for either color or number were excluded from analysis. Within each participant, we removed outlier trials (where RT of color response was more than 3 SDs deviant from the mean RT). In all, 7.66% of trials in the remaining blocks were excluded. To check for the congruency effect, using SPSS 25.0 (IBM Corporation), we ran a repeated measures ANOVA on RT with participant group (i.e., synesthete or non-synesthete) as the between-subject factor, and congruency, number notation, and SOA as within-subject factors. Overall RTs of synesthetes (M ± SE: 550 ± 28 ms) and non-synesthetes (593 ± 28 ms) were not significantly different (MD = 43 ms, F(1,20) = 1.212, p = 0.284). The RTs for congruent trials (551 ± 18 ms) were significantly shorter than incongruent trials (592 ± 22 ms; MD = 41 ms, F(1,20) = 10.868, p = 0.004, partial η2 = 0.352), i.e., there was a congruency effect. There was an interaction between participant group and congruency (F(1,20) = 6.590, p = 0.018, partial η2 = 0.248). Congruency effects were significant in synesthetes (Bonferroni-corrected post-hoc comparison: MD = 73 ms, p < 0.001) but not in non-synesthetes (MD = 9 ms, p = 0.612). These results indicate a synesthetic congruency effect only in synesthetes, typical in synesthetic priming paradigms (Figure 3).
For each participant and under each condition, we calculated congruency effects by subtracting the average RT of congruent trials from the average RT of incongruent trials. We ran a repeated measures ANOVA on congruency effect with participant group as the between-subject factor, and number notation and SOA as within-subject factors. We found a significant main effect of participant group (synesthete (73 ± 18 ms) > non-synesthete (9 ± 18 ms), MD = 64 ms, F(1,20) = 6.590, p = 0.018, partial η2 = 0.248). SOA had a main effect (F(4,80) = 4.944, p = 0.001, partial η2 = 0.198) due to a larger congruency effect at 300 ms (60 ± 16 ms) than 800 ms (20 ± 8 ms) SOA (Bonferroni-corrected post-hoc comparison: MD = 40 ms, p = 0.016). The interaction between SOA and participant group was also significant (F(4,80) = 2.500, p = 0.049, partial η2 = 0.111). Bonferroni-corrected post-hoc comparison indicated that synesthetes’ congruency effects were larger at SOA of 100 ms (83 ± 21 ms; MD = 46 ms, p = 0.028) and 300 ms (104 ± 23 ms; MD = 67 ms, p = 0.003) than at 800 ms (37 ± 11 ms), whereas non-synesthetes’ congruency effects did not differ across SOAs. Number notation exhibited no main effect (F(2,40) = 2.095, p = 0.136) but interacted with SOA (F(8,160) = 2.178, p = 0.032, partial η2 = 0.098). At 400 ms SOA, congruency effect for digits (60 ± 17 ms) were larger than for dice (33 ± 13 ms; MD = 26 ms, p = 0.032), and at 800 ms SOA, congruency effect for NC patterns (38 ± 9 ms) were larger than for dice (5 ± 10 ms; MD = 0.33 ms, p = 0.007).
To determine the origin of interactions, we ran repeated measures ANOVAs with number notation and SOA as within-subject factors, separately in synesthetes and non-synesthetes. In synesthetes (Figure 4a), SOA had a main effect (F(4,40) = 5.450, p = 0.001, partial η2 = 0.353) due to larger congruency effect at 300 ms (73 ± 29 ms) than 800 ms (37 ± 13 ms) (Bonferroni-corrected post-hoc comparison: MD = 67 ms, p = 0.038). This is in line with our expectation that synesthetic interference would occur at short SOAs and later decline. Number notation had a marginally significant main effect (Greenhouse–Geissler-corrected for violation of sphericity assumption, F(1.287,12.873) = 4.332, p = 0.050, partial η2 = 0.302) due to larger congruency effect for digits (87 ± 27 ms) than for dice (64 ± 25 ms; Bonferroni-corrected post-hoc comparison: MD = 23 ms, p = 0.005). SOA × number notation interaction was marginally significant (F(8,80) = 1.960, p = 0.062, partial η2 = 0.164). Bonferroni-corrected simple main effect analysis revealed that at 0 ms SOA, congruency effect for digits (109 ± 26 ms) was larger than for NC patterns (44 ± 25 ms; MD = 0.65 ms, p = 0.018); at 400 ms SOA, congruency effect for digits (93 ± 32 ms) was larger than for dice (54 ± 24 ms; MD = 39 ms, p = 0.045); and at 800 ms SOA, congruency effect for NC patterns (56 ± 11 ms) was larger than for dice (17 ± 17 ms; MD = 0.39 ms, p = 0.033). Taken together, digits tend to yield larger congruency effects than dot patterns at 0 and 400 ms SOAs, while at SOA levels in between (100 and 300 ms), effect sizes were similar across number notations. The congruency effect elicited by digits was persistent throughout the initial 400 ms following stimulus onset, while the congruency effect elicited by dot patterns increased through 0–300 ms SOAs, but decreased from 300 ms onwards, i.e., exhibited a pattern of “rise-and-fall”. In non-synesthetes (Figure 4b, none of the effects were significant (number notation: F(2,20) = 0.124, p = 0.884; SOA: F(4,40) = 1.280, p = 0.294; interaction: F(8,80) = 0.823, p = 0.0585).

4. Discussion

In order to dissociate graphemic and semantic components in number–color synesthesia, we investigated the time course of color activations elicited by numbers in three notations. We implemented a priming paradigm with varying SOAs and compared number–color congruency effects. Throughout the 0–400 ms time window post-stimulus onset when differences between notations were expected to appear, we found differences between digits and dot patterns but not between types of dot patterns (i.e., dice and NC). Specifically, congruency effect elicited by digits had a shorter latency (i.e., an immediate onset) and was persistent, while that elicited by both dot patterns exhibited a “rise-and-fall” pattern. At 800 ms SOA, NC yielded an unexpected larger effect than dice. The three number notations yielded similar congruency effect sizes (i.e., maximum congruency effect sizes across SOA). Our findings revealed a distinction between effects of graphemic and semantic pathways of synesthesia on the temporal dimension. In contrast to previous studies distinguishing “perceptual vs conceptual” synesthesia, we highlighted that this distinction is within “conceptual” synesthesia. Moreover, the associated two pathways (as well as multiple working mechanisms) could co-exist in individual synesthetes, rather than a between-individual difference or a single mechanism, which could explain the current results (see more details below).
As we have expected, digits did show superiority over dot patterns in eliciting synesthesia, but this superiority was reflected in a quicker and more persistent time course rather than a larger magnitude. Thus, the graphemic and semantic pathways differ in time courses in their effects: effect of the former was quick and persistent, while the latter had actions of a relatively slow and transient manner. Previous studies have measured number–color synesthetic interference using either simultaneous stimuli presentation or a fixed SOA [2,6,7,8,9], providing information only at a single time point. Therefore, congruency effect sizes measured in these studies were interfered with by different latencies across conditions and may not accurately reflect “strengths” of synesthetic connections. Our study teased apart magnitudes and latencies of congruency effects and provided a dynamic and more comprehensive profile of synesthetic activation. The result of comparison between dice and NC patterns differed from our expectation. One possible explanation is that synesthetic color activation elicited by NC patterns had a longer latency, due to slower number recognition [Indeed, we tested the same group of participants for number recognition speed. Participants were slower in recognizing numbers from NC patterns than from dice patterns, and both were slower than that from digits (see Appendix C for details). Note that the difference between digit and dot patterns cannot be explained by different recognition speed alone. Faster recognition speed may account for the quick onset of congruency effect for digits, but not for the persistence. The observed difference in patterns (rather than mere latencies) of congruency effects across SOAs suggests a mechanism difference]. At 800 ms SOA, the congruency effect elicited by NC patterns had not completely decayed, while that by dice patterns had, resulting in a larger congruency effect of NC patterns at 800 ms SOA. It is noteworthy that congruency effects of digit (which activates colors via both pathways) and dot patterns (which activate colors via the semantic pathway alone) at 300 ms SOA were equivalent, suggesting that effects from the graphemic and semantic pathways are non-additive. A possible explanation of this non-additivity is that both pathways activate the same color representation (e.g., a common group of neurons, instead of different groups). When a digit is presented to a synesthete, this color representation is fully activated under the input from the graphemic pathway (e.g., causing maximum firing rate in the neurons) [26]. Additional input from the semantic pathway cannot further increase the activation level and, thus, do not contribute more to the behavioral outcome, namely the congruency effect.
Our distinction of pathways differs from previous studies in two major aspects. First, models of synesthesia typically distinguish between perceptual and conceptual levels of stimulus processing, and either argue for the occurrence of synesthesia on one of the levels (e.g., the conceptual mediation model [14]), or identify types of synesthesia based on its level of occurrence (e.g., lower and higher synesthesia [11,27]). Here, we are not making a parallel distinction; rather, we propose the distinction between graphemic and semantic aspects as a further distinction within conceptual synesthesia. Second, studies typically discussed multiple mechanisms of synesthesia as individual differences, possibly underlying phenomenological differences in synesthetic experiences (e.g., the spatial location where a color concurrent appears [12,28,29]). Here, by distinguishing the graphemic and semantic pathways, we are not proposing a further division of conceptual synesthesia into more detailed subtypes. Rather, we propose that the pathways are different mechanisms underlying a single type of synesthesia (i.e., conceptual synesthesia) and are co-existent in individual synesthetes. For the line of research on neural substrates of synesthesia, this implies that multiple pathways may be involved. For instance, the cross-activation model suggests that synesthesia arises from direct, local anatomical links between different modality-specific brain areas, and these links are unique to synesthetes [10,27,30,31]. Meanwhile, the hyperbinding account proposes that synesthesia results from global changes in the functioning of common brain structures. Specifically, when synesthetes’ brains process inducers, feedback from higher-level areas was less inhibited, resulting in additional signals corresponding to concurrents [1]. We suggest these accounts may not be mutually exclusive. Instead, different pathways can co-exist in individual synesthete brains, and may be accounted for by different neural mechanisms. Indeed, neuroimaging studies have provided both structural [32] and functional [33] findings supporting this view of pathway co-existence.
In sum, our study did not aim to separate different subtypes of number–color synesthesia but to establish a distinction between the graphemic and semantic aspects of number–color synesthesia on the temporal dimension. We revealed the temporal dynamics of the behavioral outcomes in number–color synesthesia, in which the graphemic pathway was quick and relatively persistent in sub-serving the color activation, while the semantic pathway had a slower onset and is more transient. Moreover, effects from the pathways appeared non-additive, possibly implying a common neuronal representation of the synesthetic concurrent. In contrast to previous studies which discussed the heterogeneity of synesthesia by distinguishing between its perceptual vs. conceptual nature and considered each type of synesthesia as a unitary whole, our distinction between graphemic and semantic pathways is a distinction within conceptual synesthesia. Moreover, the two pathways are co-existing mechanisms within individuals, rather than between-individual differences. For future studies, our study has demonstrated that investigating the time courses of synesthetic activation can be an effective methodology of distinguishing between different mechanisms in synesthesia; theoretically, we call attention to the possibility that a single type of synesthesia may involve multiple mechanisms.

Author Contributions

Conceptualization, L.C. and S.Y.; methodology, L.C.; formal analysis, S.Y.; data curation, S.Y.; writing—original draft preparation, S.Y.; writing—review and editing, L.C. and S.Y.; funding acquisition, L.C. and S.Y. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Ministry of Science and Technology of China (2021ZD0202601) and the Natural Science Foundation of China (grant numbers T2192932, 62061136001, and 31861133012) to L.C., and the CURE fund at Peking University for young talents to S.Y.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Academic Affairs Committee of School of Psychological and Cognitive Sciences at Peking University (protocol code MSI-2021-10-18 and approval date 21 October 2021).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data will be granted access to: https://osf.io/9z54d/ (accessed on 24 August 2022).

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Questionnaire for Screening Synesthetes

Participants read the following information from the advertisement prior to volunteering and taking the questionnaire:
“Some people have these experiences: they see graphemes as having colors, or feel that music is accompanied by colors, or experience tastes for words… Synesthesia is a condition in which one attribute of a stimulus evokes experiences of another unrelated attribute. People having this experience are called synesthetes. The examples above are common types of synesthesia: grapheme-color, music-color, and lexical-gustatory synesthesia. Synesthetes may experience multiple types of synesthesia. Our research focuses on number-color synesthesia”.
The questionnaire contained the following items:
  • Synesthetes are typically aware that they see numbers/letters/words as having colors. Do you have such feelings?
    (A) Yes.                          (B) No.                         (C) Uncertain (please explain).
  • Do you feel that the colors exist in external space (e.g., on the paper where numbers are written) or in your mind’s eye?
    (A) In external space.   (B) In my mind’s eye. (C) Uncertain (please explain).
  • Is the color experience voluntarily imagined, or does it occur automatically?
    (A) It is voluntarily imagined.                          (B) It occurs automatically.
    (C) Uncertain (please explain).
  • Are the colors for each number changeable or fixed?
    (A) Changeable.            (B) Fixed.                      (C) Uncertain (please explain).
  • Do you assign colors to numbers yourself, or do colors seem intrinsic for the numbers and not decided by yourself?
    (A) I decide the colors. (B) The colors seem intrinsic for the numbers.
    (C) Uncertain (please explain).
  • Will you be able to choose the colors corresponding to numbers if asked to?
    (A) Yes.                           (B) No.                         (C) Uncertain (please explain).
  • Do synesthetic colors hamper you from seeing physical colors of numbers (e.g., the ink color or display color)?
    (A) Yes.                           (B) No.                         (C) Uncertain (please explain).
  • Please give a verbal description of colors for 2~6. You can also give more examples of your synesthetic experience (not limited to number-color synesthesia).
Participants were considered genuine synesthetes if they made the following choices for items: 1. A, 3. B, 4. B, 5. B, 6. A, 7. B, and were able to give color descriptions in item 8. However, they might choose C on certain items due to ambiguity in understanding, and subsequently described experiences corresponding to the above-mentioned choice under clarification. All screenings were done before any further information about synesthesia were given to the participants.

Appendix B. List of Numbers and Colors for Each Participant

Table
Table A1. List of numbers and colors used in the experiment for participant.
Table A1. List of numbers and colors used in the experiment for participant.
Numbers Colors and RGB Triplet for Numbers (R, G, B)
123456
Synesthetes
CJX2, 4, 5, 6 (255, 5, 5) (244, 249, 31) (255, 255, 255) (178, 178, 178) (255, 0, 0) (91, 155, 213)
CWR2, 4, 5, 6 (0, 0, 0) (255, 153, 204) (190, 32, 47) (46, 139, 172) (211, 136, 61) (213, 13, 13)
CYS2, 3, 4, 6 (0, 0, 0) (192, 0, 0) (51, 204, 51) (255, 255, 0) (0, 0, 0) (91, 155, 213)
HYQ2, 3, 5, 6 (0, 0, 0) (253, 57, 57) (135, 84, 81) (83, 171, 59) (135, 15, 121) (240, 216, 70)
KJQ2, 3, 5, 6 (255, 255, 255) (248, 213, 86) (146, 208, 80) (252, 29, 24) (22, 22, 132) (243, 102, 45)
LM2, 3, 5, 6 (255, 255, 255) (41, 169, 227) (235, 240, 40) (28, 134, 48) (255, 0, 0) (123, 242, 64)
LXY2, 3, 5, 6 (255, 37, 37) (95, 255, 87) (10, 141, 250) (245, 255, 109) (30, 0, 84) (182, 169, 2)
MXD2, 3, 5, 6 (216, 14, 14) (255, 192, 0) (97, 126, 209) (137, 90, 232) (241, 138, 61) (131, 210, 92)
SZY2, 3, 5, 6 (216, 44, 10) (211, 207, 31) (40, 160, 198) (30, 63, 226) (58, 206, 93) (160, 219, 238)
YNJ2, 3, 5, 6 (203, 203, 203) (235, 33, 96) (86, 157, 236) (234, 159, 58) (111, 88, 208) (223, 196, 33)
YTC2, 4, 5, 6 (0, 0, 0) (255, 102, 204) (255, 153, 0) (0, 204, 102) (0, 102, 204) (255, 102, 0)
Non-synesthetes
HBH2, 4, 5, 6 (255, 255, 0) (0, 176, 240) (255, 192, 0) (112, 48, 160) (112, 173, 71) (255, 0, 0)
LX2, 4, 5, 6 (255, 0, 0) (0, 176, 240) (255, 255, 0) (0, 176, 80) (112, 48, 160) (255, 126, 29)
MJT2, 3, 5, 6 (255, 0, 0) (255, 255, 0) (146, 208, 80) (0, 176, 240) (68, 114, 196) (112, 48, 160)
MM2, 3, 4, 6 (146, 208, 80) (0, 176, 240) (255, 204, 255) (0, 128, 0) (237, 125, 49) (112, 48, 160)
MZ2, 3, 5, 6 (192, 0, 0) (255, 192, 0) (112, 48, 160) (112, 173, 71) (255, 102, 153) (91, 155, 213)
WQS2, 4, 5, 6 (110, 228, 94) (236, 251, 95) (92, 151, 246) (165, 165, 165) (255, 0, 0) (253, 89, 206)
WZN2, 3, 5, 6 (237, 125, 49) (91, 155, 213) (68, 114, 196) (112, 173, 71) (112, 48, 160) (255, 255, 0)
YY2, 3, 5, 6 (255, 0, 0) (237, 125, 49) (255, 255, 0) (146, 208, 80) (0, 176, 240) (112, 48, 160)
ZJH2, 3, 5, 6 (51, 153, 255) (255, 255, 102) (255, 153, 51) (0, 0, 0) (112, 173, 71) (102, 102, 255)
ZMY2, 3, 5, 6 (68, 114, 196) (237, 125, 49) (255, 0, 0) (112, 173, 71) (112, 48, 160) (165, 165, 165)
ZYJ2, 3, 5, 6 (197, 208, 60) (60, 201, 208) (75, 43, 213) (130, 204, 148) (232, 136, 211) (250, 71, 52)
In each participant × number cell, the color patch shows the color which the participant chose for the number, and the array shows its RGB values in format: (R, G, B).

Appendix C. Number Recognition Speed Test

All participants in the main experiment (11 synesthetes and 11 non-synesthetes) completed the number recognition speed test. Stimuli were identical to the number stimuli in the main experiment and were presented with the same apparatus. There were three conditions with different number notations (digit, dice, and NC). Trials were arranged into 8 blocks in the same way as in the main experiment. Each trail started with a 300 ms central fixation cross (diameter: 0.6°, RGB: (128, 128, 128)), and a number stimulus was presented at screen center (maximum waiting time: 2000 ms). Participants chose the number by pressing “Y” and “B” keys (same as the number response in the main experiment) as quickly as possible. The number stimulus disappeared immediately after response. The next trial started after a random ITI of 2000–2500 ms. Each block contained 18 randomized trials (6 for each number notation), and there were 144 trials in total (48 for each number notation).
For each participant, we calculated the average RT of correct trials for each number notation. We then ran a repeated measures ANOVA with number notation as the within-subject factor and participant group as the between-subject factor. Number notation had a significant main effect on RT (M ± SE: digits: 476 ± 12 ms, dice: 506 ± 12 ms, NC: 553 ± 12 ms; F(2,40) = 44.426, p < 0.001, partial η2 = 0.690). RT increased from digit to dice, and to NC patters (Bonferroni-corrected post-hoc comparison: dice − digit: MD = 30 ms, p = 0.004; NC − dice: MD = 47 ms, p < 0.001). Participant group displayed neither main effect (M ± SE: synesthetes: 499 ± 16 ms; non-synesthetes: 524 ± 16 ms; F(1,20) = 1.203, p = 0.286) nor interaction with number notation (F(2,40) = 1.775, p = 0.183). These results showed that, for both synesthetes and non-synesthetes, digits were recognized faster than dice patterns and dice faster than NC patterns.

References

  1. Grossenbacher, P.G.; Lovelace, C.T. Mechanisms of Synesthesia: Cognitive and Physiological Constraints. Trends Cogn. Sci. 2001, 5, 36–41. [Google Scholar] [CrossRef]
  2. Elias, L.J.; Saucier, D.M.; Hardie, C.; Sarty, G.E. Dissociating Semantic and Perceptual Components of Synaesthesia: Behavioural and Functional Neuroanatomical Investigations. Cogn. Brain Res. 2003, 16, 232–237. [Google Scholar] [CrossRef]
  3. Mattingley, J.B.; Rich, A.N.; Yelland, G.; Bradshaw, J.L. Unconscious Priming Eliminates Automatic Binding of Colour and Alphanumeric Form in Synaesthesia. Nature 2001, 410, 580–582. [Google Scholar] [CrossRef] [PubMed]
  4. Herrera, A.; Macizo, P. Semantic Processing in the Production of Numerals across Notations. J. Exp. Psychol. Learn. Mem. Cogn. 2012, 38, 40–51. [Google Scholar] [CrossRef]
  5. Herrera, A.; Macizo, P. Cross-Notational Semantic Priming between Symbolic and Nonsymbolic Numerosity. Q. J. Exp. Psychol. 2008, 61, 1538–1552. [Google Scholar] [CrossRef]
  6. Ward, J.; Sagiv, N. Synaesthesia for Finger Counting and Dice Patterns: A Case of Higher Synaesthesia? Neurocase 2007, 13, 86–93. [Google Scholar] [CrossRef]
  7. Berteletti, I.; Hubbard, E.M.; Zorzi, M. Implicit versus Explicit Interference Effects in a Number-Color Synesthete. Cortex 2010, 46, 170–177. [Google Scholar] [CrossRef] [PubMed]
  8. Smilek, D.; Dixon, M.J.; Cudahy, C.; Merikle, P.M. Concept Driven Color Experiences in Digit-Color Synesthesia. Brain Cogn. 2002, 48, 570–573. [Google Scholar] [CrossRef]
  9. Dixon, M.J.; Smilek, D.; Cudahy, C.; Merikle, P.M. Five plus Two Equals Yellow: Mental Arithmetic in People with Synaesthesia Is Not Coloured by Visual Experience. Nature 2000, 406, 365. [Google Scholar] [CrossRef] [PubMed]
  10. Ramachandran, V.S.; Hubbard, E.M. Synaesthesia—A Window into Perception, Thought and Language. J. Conscious. Stud. 2001, 8, 3–34. [Google Scholar]
  11. Hubbard, E.M.; Ramachandran, V.S. Neurocognitive Mechanisms of Synesthesia. Neuron 2005, 48, 509–520. [Google Scholar] [CrossRef] [PubMed]
  12. Ward, J.; Li, R.; Salih, S.; Sagiv, N. Varieties of Grapheme-Colour Synaesthesia: A New Theory of Phenomenological and Behavioural Differences. Conscious. Cogn. 2007, 16, 913–931. [Google Scholar] [CrossRef] [PubMed]
  13. Ranzini, M.; Girelli, L. Colours + Numbers Differs from Colours of Numbers: Cognitive and Visual Illusions in Grapheme-Colour Synaesthesia. Atten. Percept. Psychophys. 2019, 81, 1500–1511. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Chiou, R.; Rich, A.N. The Role of Conceptual Knowledge in Understanding Synesthesia: Evaluating Contemporary Findings from a “Hub-and-Spokes” Perspective. Front. Psychol. 2014, 5, 105. [Google Scholar] [CrossRef] [Green Version]
  15. Brang, D.; Kanai, S.; Ramachandran, V.S.; Coulson, S. Contextual Priming in Grapheme-Color Synesthetes and Yoked Controls: 400 Msec in the Life of a Synesthete. J. Cogn. Neurosci. 2011, 23, 1681–1696. [Google Scholar] [CrossRef]
  16. Brang, D.; Edwards, L.; Ramachandran, V.S.; Coulson, S. Is the Sky 2? Contextual Priming in Grapheme-Color Synaesthesia. Psychol. Sci. 2008, 19, 421–428. [Google Scholar] [CrossRef] [Green Version]
  17. Teichmann, L.; Grootswagers, T.; Moerel, D.; Carlson, T.A.; Rich, A.N. Temporal Dissociation of Neural Activity Underlying Synesthetic and Perceptual Colors. Proc. Natl. Acad. Sci. USA 2021, 118, e2020434118. [Google Scholar] [CrossRef]
  18. Gertner, L.; Arend, I.; Henik, A. Numerical Synesthesia Is More than Just a Symbol-Induced Phenomenon. Front. Psychol. 2013, 4, 806. [Google Scholar] [CrossRef] [Green Version]
  19. Liu, B.; Jin, Z.; Qing, Z.; Wang, Z. The Processing of Phonological, Orthographical, and Lexical Information of Chinese Characters in Sentence Contexts: An ERP Study. Brain Res. 2011, 1372, 81–91. [Google Scholar] [CrossRef]
  20. Paap, K.R.; Noel, R.W. Dual-Route Models of Print to Sound: Still a Good Horse Race. Psychol. Res. 1991, 53, 13–24. [Google Scholar] [CrossRef]
  21. Coltheart, M.; Curtis, B.; Atkins, P.; Haller, M. Models of Reading Aloud: Dual-Route and Parallel-Distributed-Processing Approaches. Psychol. Rev. 1993, 100, 589–608. [Google Scholar] [CrossRef]
  22. Pelli, D.G. The VideoToolbox Software for Visual Psychophysics: Transforming Numbers into Movies. Spat. Vis. 1997, 10, 437–442. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Kleiner, M.; Brainard, D.H.; Pelli, D.G.; Broussard, C.; Wolf, T.; Niehorster, D. What’s New in Psychtoolbox-3? Perception 2007, 36, 1–16. [Google Scholar] [CrossRef]
  24. Brainard, D.H. The Psychophysics Toolbox. Spat. Vis. 1997, 10, 433–436. [Google Scholar] [CrossRef] [Green Version]
  25. Dehaene, S. The Number Sense: How the Mind Creates Mathmatics; Revised and Updated; Oxford University Press: New York, NY, USA, 2011. [Google Scholar]
  26. Goldstein, E.B. Sensation and Perception, 9th ed.; Cengage Learning: Wadsworth, OH, USA, 2013. [Google Scholar]
  27. Ramachandran, V.S.; Hubbard, E.M. Psychophysical Investigations into the Neural Basis of Synaesthesia. Proc. R. Soc. B Boil. Sci. 2001, 268, 979–983. [Google Scholar] [CrossRef] [PubMed]
  28. van Leeuwen, T.M.; Petersson, K.M.; Hagoort, P. Synaesthetic Colour in the Brain: Beyond Colour Areas. A Functional Magnetic Resonance Imaging Study of Synaesthetes and Matched Controls. PLoS ONE 2010, 5, e12074. [Google Scholar] [CrossRef] [Green Version]
  29. Dixon, M.J.; Smilek, D.; Merikle, P.M. Not All Synaesthetes Are Created Equal: Projector versus Associator Synaesthetes. Cogn. Affect. Behav. Neurosci. 2004, 4, 335–343. [Google Scholar] [CrossRef] [Green Version]
  30. Hubbard, E.M.; Arman, A.C.; Ramachandran, V.S.; Boynton, G.M. Individual Differences among Grapheme-Color Synesthetes: Brain-Behavior Correlations. Neuron 2005, 45, 975–985. [Google Scholar] [CrossRef] [Green Version]
  31. Ramachandran, V.S.; Hubbard, E.M. Hearing Colors, Tasting Shapes. Sci. Am. 2003, 288, 52–59. [Google Scholar] [CrossRef]
  32. Hänggi, J.; Wotruba, D.; Jäncke, L. Globally Altered Structural Brain Network Topology in Grapheme-Color Synesthesia. J. Neurosci. 2011, 31, 5816–5828. [Google Scholar] [CrossRef] [Green Version]
  33. Yokoyama, T.; Noguchi, Y.; Koga, H.; Tachibana, R.; Saiki, J.; Kakigi, R.; Kita, S. Multiple Neural Mechanisms for Coloring Words in Synesthesia. Neuroimage 2014, 94, 360–371. [Google Scholar] [CrossRef]
Brainsci 12 01400 g001 550
Figure 1. Three types of number stimuli for numbers 2–5. Grey lines delineate borders of stimuli area (2° × 2°, for illustration only), and were not present in the experiment.
Figure 1. Three types of number stimuli for numbers 2–5. Grey lines delineate borders of stimuli area (2° × 2°, for illustration only), and were not present in the experiment.
Brainsci 12 01400 g001
Brainsci 12 01400 g002 550
Figure 2. Schematic diagram of a main experiment trial.
Figure 2. Schematic diagram of a main experiment trial.
Brainsci 12 01400 g002
Brainsci 12 01400 g003 550
Figure 3. RTs of synesthetes and non-synesthetes in congruent and incongruent trails. Error bars denote one standard error. ** p < 0.01, *** p < 0.001.
Figure 3. RTs of synesthetes and non-synesthetes in congruent and incongruent trails. Error bars denote one standard error. ** p < 0.01, *** p < 0.001.
Brainsci 12 01400 g003
Brainsci 12 01400 g004 550
Figure 4. Congruency effect of three number notations as a function of SOA, for (a) synesthetes; (b) non-synesthetes. Error bars denote one standard error. The stars (*) indicate p < 0.05 for the test of simple main effects. The conditions where congruency effects were significantly different were marked with enlarged symbols.
Figure 4. Congruency effect of three number notations as a function of SOA, for (a) synesthetes; (b) non-synesthetes. Error bars denote one standard error. The stars (*) indicate p < 0.05 for the test of simple main effects. The conditions where congruency effects were significantly different were marked with enlarged symbols.
Brainsci 12 01400 g004
Table
Table 1. Arrangement of numbers and hands for responses to colors in each block (a number in a color column denotes the color corresponding to that number). For participants who used other number sets (2, 3, 4, 6 or 2, 4, 5, 6), 4 took the place of 5 or 3.
Table 1. Arrangement of numbers and hands for responses to colors in each block (a number in a color column denotes the color corresponding to that number). For participants who used other number sets (2, 3, 4, 6 or 2, 4, 5, 6), 4 took the place of 5 or 3.
BlockNumbersColors to Which Participants Responded with Left- or Right-Hand Key-Press
Left HandRight Hand
Day 112, 323
25, 665
32, 552
43, 636
Day 253, 663
62, 525
75, 656
82, 332
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Yue, S.; Chen, L. Graphemic and Semantic Pathways of Number–Color Synesthesia: A Dissociation of Conceptual Synesthesia Mechanisms. Brain Sci. 2022, 12, 1400. https://doi.org/10.3390/brainsci12101400

AMA Style

Yue S, Chen L. Graphemic and Semantic Pathways of Number–Color Synesthesia: A Dissociation of Conceptual Synesthesia Mechanisms. Brain Sciences. 2022; 12(10):1400. https://doi.org/10.3390/brainsci12101400

Chicago/Turabian Style

Yue, Shimeng, and Lihan Chen. 2022. "Graphemic and Semantic Pathways of Number–Color Synesthesia: A Dissociation of Conceptual Synesthesia Mechanisms" Brain Sciences 12, no. 10: 1400. https://doi.org/10.3390/brainsci12101400

APA Style

Yue, S., & Chen, L. (2022). Graphemic and Semantic Pathways of Number–Color Synesthesia: A Dissociation of Conceptual Synesthesia Mechanisms. Brain Sciences, 12(10), 1400. https://doi.org/10.3390/brainsci12101400

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop