Evaluation of the Effectiveness of Sonification for Time-series Data Exploration

Data sonification is described as "the transformation of data relations into perceived relations in an acoustic signal for the purpose of facilitating communication or interpretation" (Kramer et al. 1999). Simple examples of sonification involve binary messages, such as the bell sound we hear in modern cars when driving without seatbelts. However, more sophisticated sonifications, that transform more complex data such as numerical quantities into sound, have been under development for decades. Data sonification is a valuable tool in scientific research, facilitating a deeper understanding of the intrinsic narratives present in sets of data, a benefit for both the scientific community and the general public (Sawe et al. 2020). Sound can offer benefits in data analysis thanks to its multidimensional nature—pitch, loudness, duration, and timbre, among others—harnessing the sophisticated capabilities of the human auditory system. Our ability to process multiple sound streams and recognize patterns (Bregman 1990; Hermann et al. 2011; Sawe et al. 2020), decode melodic contours, and notice slight variations in frequency, enhances the potential to uncover significant patterns embedded in the data (Kramer et al. 1999). These qualities make sonification particularly interesting for conveying time-dependent information, such as those collected through monitoring (e.g., see the sonification of electroencephalograms of epileptic seizures; Supper 2012; Matinfar et al. 2023). Traditionally, astronomy was studied and disseminated through visual means, such as images, animations, and graphs. Yet, most of the Universe's content (dark matter and dark energy) does not emit or absorb light. Even the matter that does often emits light at frequencies outside the human visible range or is too faint to be seen directly. Finally, data collected at the telescope are not collected as images, but rather as digital numbers that researchers turn into visuals (e.g., Guiotto Nai Fovino et al. 2023a; Varano & Zanella 2023). Existing reviews outlined the usefulness of astronomical sonification in both scientific and educational contexts (Hermann et al. 2011; Zanella et al. 2022). The astronomical community is increasingly turning to sound as a medium for data representation, as evidenced by the numerous contributions to the Data Sonification Archive (Lenzi & Ciuccarelli 2020; Harrison et al. 2021; Lenzi et al. 2021; Zanella et al. 2022). Sonification efforts aimed to enhance accessibility in public engagement, education, and research activities, especially for those with visual impairments (Hyman et al. 2019; Bardelli et al. 2022; García-Benito & Pérez-Montero 2022; Harrison et al. 2022). It also proved to be a promising tool for exploring complex (Cooke et al. 2019; Sawe et al. 2020) and multidimensional data sets and for monitoring time-series data without the need for visual attention (Guttman et al. 2005). One promising application of sonification in time-series data is the analysis of light curves. For example, Diaz-Merced (2013) showed that sonification, when added to the visual display of noisy unidimensional data, increased sensitivity to events that might otherwise go unnoticed by the human eye. A common technique to sonify light curves is the so-called parameter-mapping sonification. In parameter-mapping sonification, it is necessary to establish a correspondence between the dimensions of the data and auditory features. Although it provides greater flexibility compared to other simpler types of sonification (e.g., the bell sound that we hear in cars when someone is not wearing a seatbelt is always the same, and we cannot identify who is not wearing the seatbelt), the design of individual mappings needs careful consideration. The most common sound dimension mapped to data is pitch (Dubus & Bresin 2013). This applies to the current literature on parameter-mapping sonification in astronomy as well (Cooke et al. 2019). Several studies on the sonification of light curves and other time-series data have been performed in recent years (Diaz-Merced et al. 2008; Diaz-Merced 2013; Cooke et al. 2019). The most recent experimental study used the default Astronify ³ sonification algorithm, which maps brightness to pitch within a one-octave frequency range (440–880 Hz, Tucker Brown et al. 2022). In the experiment of Tucker Brown et al. (2022), the star's brightness was mapped as a function of time. The success rate in identifying signals in visual plots, sonification, and a combination of both was analyzed. Participants had to identify the presence of transits of different signal-to-noise ratios (S/Ns). The results showed a consistently better performance for visual and plots combined with sonification in comparison to sonification only, especially for medium and low S/Ns. Astronomers performed almost at ceiling for visual plots and were generally better than nonastronomers. Interestingly, sonification enabled the detection of signals with high S/N also for users with little to no experience in sonification (Tucker Brown et al. 2022). Building upon the investigation of Tucker Brown et al. (2022), we investigated the ability of astronomers and nonastronomers to detect light curves of various S/Ns. We tested three different sound mappings (pitch, duration, and duration and pitch) and compared them to the classic visual representation of data. Previous studies (Diaz-Merced et al. 2008; Cooke et al. 2019; Tucker Brown et al. 2022) have represented time-series data with sound by mapping brightness to pitch so that a drop in brightness (i.e., the dip of a transient-like light curve) is associated with low-frequency sounds. We tested the reversed pitch association so that a decrease in brightness (i.e., the dip in a light curve) coincides with high-frequency sounds that are usually perceived as more salient than low-pitch ones (Haas & Edworthy 1996). We also tested whether sound duration could be used as an alternative to pitch. Human sensitivity to sounds duration is high, and hearing is, among the senses, the one with the highest sensitivity for duration (Keetels & Vroomen 2012). Finally, we tested a redundant sonification with the simultaneous manipulation of sound frequency and sound duration. Unlike previous works (Tucker Brown et al. 2022), we did not investigate the performance only, but also the way sonification and visualization affected the response bias of the participants. The response bias reveals whether the participants produce more "yes, there is a transit" or "no, there is no transit" responses regardless of the presence (or absence) of the signal. It is the spontaneous tendency of the participant to use one (or the other) response. This is routinely used in the psychoacoustics domain when analyzing signal-in-noise detection experiments (Green & Swets 1966).

The paper is structured as follows: in Section 2 we present the method that we adopted to recruit the participant, build the experiment and sonify the data; in Section 3 we report the statistical results of the experiment; in Section 4, we discuss the results and their implications; finally, in Section 5 we summarize and conclude.

2.1. Participants and Apparatus

2.2. Synthetic Light Curves and Adopted Mappings

2.3. Procedure

We calculated the proportion of affirmative responses ("yes, the signal is present") for each participant and, separately, for each type of stimulus (i.e., signal present/signal absent), modality of the stimulus (visual or auditory, which comprised pitch, duration, and duration and pitch), and S/N ratio (Figure 3). These proportions were the "hits" (H, the participant responded "yes, there is a transit" and the light curve included a flux dip) and the "false alarms" (FAs; the participant responded "yes, there is a transit" and the light curve did not include a dip). We calculated two indexes to represent two aspects of the participants response. These indexes were inspired by the works on the signal detection theory (Green & Swets 1966). The first way to represent the participants responses is referred to as "performance," which is the difference, for each given stimulus, between the hit and the false alarm rate of the participant: P = H − FA. Performance is the net accuracy of the participant over his/her false alarm rate. Theoretically, performance can range from P = − 1 and P = 1. When P = − 1 the participant made a perfect, but reversed, performance, such as a participant that reversed the meaning of the instructions received before the experiment. When P = 1 the participant produced hits only and no false alarms. Finally, P = 0 indicates that the participant cannot distinguish the stimuli including a signal (light curve dip) from those that do not include it. An alternative way to represent the participants responses is referred to as "bias" (B, Figure 4). This is the mean of the hits and the false alarms rate of the participant. The bias lets emerge the tendency of the participant to use more the "yes, there is a transit" response or the "no, there is no transit" response regardless of the performance. Bias ranges from 0 to 1. For B = 0 the participants responded always "no, there is no transit," while for B = 1 the participants responded always "yes, there is a transit." B = 0.5 means that the participant used both responses with a similar frequency. In our experiment, an ideal respondent should have a high performance but a bias of 0.5, revealing that s/he distributed evenly the "yes" and the "no" responses (the experiment included 50% signal trials and 50% noise trials). In other words, they would produce many hits but no false alarms. Usually, stimuli that are easy to detect are associated with no response bias (B = 0.5). Bias becomes an interesting way to represent the responses in the case of stimuli that are difficult to detect (e.g., low S/N). For these stimuli it is possible to understand whether the participant is more prone to produce a "yes" or a "no" response, in other words, whether the participant had a liberal or a conservatory response criterion. We calculated an analysis of variance (ANOVA) on performance considering the following within factors: type of stimulus (duration, duration and pitch, pitch, visual) and S/N. The group (astronomers/nonastronomers) was also included as a between factor. This analysis investigates whether performance changed as a function of S/N, mapping used to represent the light curve data, and the group of participants, or subsets of these variables when combined in pairs or triplets. We show the results in Figure 3. The ability of the participants to detect a transit-like feature increased as a function of S/N: F(3, 168) = 51.19, p < 0.0001. In general, the performance with the visual stimuli was better than with the auditory stimuli for any level of S/N: F(3, 168) = 95.49, p < 0.0001. An interaction between stimulus type and S/N was also observed: F(9, 504) = 14.21, p < 0.0001. This result revealed that performance increased differently for the various types of stimuli as a function of S/N. In the case of visual stimuli, the growth of performance was abrupt whereas in the case of sound stimuli, the growth of performance was more gradual. Astronomers had a performance averagely higher than nonastronomers: F(1, 56) = 4.78, p = 0.03. No other statistical significance was observed. These findings are summarized in Table 2.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We repeated the same calculations on the response bias, as summarized in Table 3. No overall difference was observed between astronomers and nonastronomers: F(1, 56) = 2.43, p > 0.05. Bias changed as a function of the S/N: F(3, 168) = 51.19, p < 0.0001. Bias also changed for stimulus type: F(3, 168) = 30.19, p < 0.0001. Interestingly, bias was different for astronomers and nonastronomers as a function of the type of stimuli: F(3, 168) = 6.36, p < 0.00004. A visual inspection of the responses when represented as bias revealed that, for astronomers, the auditory conditions and the visual conditions elicited different biases (Figure 4). With visual stimuli astronomers showed no bias (and ceiling performance) for high S/Ns. In contrast, for the lowest S/N they showed a positive performance but a tendency to be reluctant to produce a "yes" response: the performance was the result of several correct rejections and relatively few hits. In contrast, for almost all auditory stimuli, regardless of the performance, astronomers were more prone to use the "yes" response, also for the lowest S/N stimuli. For nonastronomers, the auditory stimuli using the duration as a mapping tended to gather a "no" response, whereas the rest of the stimuli (except one) tended to gather a "yes" response. Interestingly, when presented with visual stimuli with low S/N, both astronomers and nonastronomers showed a conservative response use and rarely responded "yes." Noticeably, auditory stimuli with low S/N are less affected by this behavior, and for these stimuli participants use the "yes" response more often (except for the responses to duration stimuli for nonastronomers). However, this more abundant use of the "yes" response did not lead to an increment in performance, and these responses are likely distributed equally among hits and false alarms.

In the current study, we investigated the effectiveness of sonification in conveying information contained in time-series data and whether different parameter mappings yielded different performances in the recognition of events. We also compared sonification with visualization and examined whether the visual or auditory nature of the stimuli affected the participants' response bias. Our analysis revealed that the participants performed better with visual stimuli than with sonified stimuli at all S/N levels, albeit the difference decreased at low S/N both for astronomers and nonastronomers. Detection increased with higher S/N for all groups and for all conditions.

4.1. Performance Analysis for Astronomers and Nonastronomers

4.2. Bias Analysis in Response Strategies

4.3. Evaluation of Sonification Parameters

Data sonification leverages the human auditory system's ability to finely process and recognize sound patterns. Sonification can be used to enhance existing data sets or to represent additional dimensions in complex multidimensional data sets, making it particularly interesting for astronomy (Diaz-Merced et al. 2008; Diaz-Merced 2013; Cooke et al. 2019; Tucker Brown et al. 2022). Our experiment tested three sonification mappings and compared participants' performance and response bias in identifying transits with those obtained using visual plots. 58 astronomers and nonastronomers took part in our study. We generated synthetic light curves and visualized them as plots or sonified them as streams of sounds, adopting different data-to-sound mappings (duration, pitch, or both). Our results revealed that:

• 1.  
  Both astronomers and nonastronomers were able to exploit sonification to detect transit-like features. However, performance with visual stimuli was higher compared to sonified stimuli.
• 2.  
  Astronomers outperformed nonastronomers in all conditions of the experiment.
• 3.  
  Visualization and sonification yielded different response biases: visualization yielded no bias (for S/N ≥ 10) whereas it yielded a conservative bias for S/N < 10. In contrast, sonification gathered a more liberal response bias. The only exception was the response bias for duration stimuli in nonastronomers that yielded a conservative bias at all S/N.

Future developments of this project will need more controlled experimental conditions (e.g., all users use headphones). We will also test the impact of training participants about sonification and include people who are already sound trained (e.g., musicians) in studies.

The authors thank Martina Barbieri for data collection, and the reviewer and data editor of the journal for the constructive feedback that improved the clarity of the manuscript. The stimuli, analysis scripts, and data sets used in this study are publicly available at doi:10.17605/OSF.IO/K7W5H (Guiotto Nai Fovino et al. 2023b).