An approach to STFT and CWT learning through music hands-on labs

This work presents a series of lectures and activities for Digital Signal Processing (DSP) teaching, based upon music and their principal elements such as melody, pitch, timbre, beat, and metric, to explain the time-frequency analysis and its repercussions in other areas of engineering. DSP courses are difficult to the students due to a high mathematical content and high level of abstraction, for this reason, the primary objective of this work is to create a pedagogical tool that allows the improvement on the teaching process in Signal Processing courses. There are two popular approaches: Short-Time Fourier Transform (STFT) and Continuous Wavelet Transform (CWT) which have been used to find time-frequency maps. The student roll was designed for active participation by creating melodies from real instruments as well as synthetic generators, and by implementing the codes of STFT and CWT in MATLAB. The results demonstrate that in general terms, after carrying out these laboratory activities the students are more motivated to learn Signal Processing theory, and some of them become interested in the new research line on Music Information Retrieval (MIR), which would benefit the continuity of the Accreditation Board for Engineering and Technologies ABET, obtained for the Automation Engineering program at Universidad Autónoma de Querétaro (UAQ) since 2016.

meaning could be strongly supported using hands-on experience [12], the interaction between the person and environment (real things, practice, or external stimulus) gives relations to known sensations while construct meanings about the new concepts. In other words, knowledge is constructed by the particular context in which the cognizing individual is operating [1,15]. When many different concepts are shown in the classroom, brain processes are developed by the students in their minds, at that point, the students are relating the new concepts to the know concepts, something known as "the appropriation of knowledge" in the Piaget and Vygotsky theories [3,16,19,25]. The constructivism recognizes that the construction of new understanding is a combination of prior learning, new information, mental association, and readiness to learn [30].
Based on these ideas, Signal Processing courses can be improved combining the new and abstracts mathematical concepts for the students, with tangible and intuitive examples [13]. The students require for the concepts learned at school to be as close to real-life and daily situations in which they become immersed (associated knowledge) as much as possible [29]. Metacognitive experiences can have significant effects on cognitive goals or tasks; if the student assimilates better these observations, it can help to develop strategies and course methodologies, to attract the attention from the student continuously during the classes, variables that are related to the goals orientation and the intrinsic motivation as an additional factor that helps to the teachers efforts in focus the learning, engage the students in their objectives and continue with the student interest for learning. These self-student's objectives should be included in the course objectives, making the learning process easy, useful and conscious by the student (conscious learning) [24,33]. The student self-determination improves the quality of motivation to the course, according to this, the student hardly gives up his goals and can obtain an autonomous motivation, that makes it persist in finding something new, creating and innovating [5,31].
The objective of this work is to identify a tool for teaching Signal Processing that can be related: 1) for attaining the theoretical and mathematical concepts involved in the courses; 2) to acquire the contemporary tools such as software and hardware environments; 3) in order to simply understand; and 4) to be familiar with the use. There have been reports of effective hands-on techniques for teaching Signal Processing. For example: the Harvard-MIT Division of Health Sciences and Technology (HST) [11] used biomedical signals to teach Signal Processing concepts about deterministic and random signals as well about biomedical images. In ref. [14] there is a description of an undergraduate level laboratory course on DSP with the primary purpose of giving students a better understanding of the theoretical concepts of DSP, and to let the students experiment with in real-time using real devices. In ref. [23] a learning tool was proposed for basic acoustics, combining sound and image to explain the dependence of timbre and acoustic spectrum, of pitch and frequency. In ref. [21], was reported a work based on music, the main idea is the creation of sounds and music instead of numbers, the programming challenges include structured programming, graphing, file manipulation and simple analysis of sound waves. A learning tool for the musical scale, which uses generated sounds in an electric organ connected to an oscilloscope in order to identify the frequencies [22]. However, these hands-on activities are not focused on time-frequency analysis based on STFT and CWT, which is fundamental for non-stationary phenomena, and not in musical scales and their origins.
Bamberger [2] has shown that a study of coherent music structure can provide contexts for better learning of basic and intermediate math skills, and for that reason music can be considered a good candidate for teaching Signal Processing. Furthermore, music is part of the daily life of people, providing a strong association that this generates with the individual's senses and emotions. According to our best knowledge there are very few reported hands-on laboratory activities that use music as pedagogical tool in Signal Processing courses. Some hands-on approaches that use music one way to another as didactic element are for example: 1) a scheme for teaching Science, Technology, Engineering and Mathematics (STEM) through music technology and digital Signal Processing is reported in ref. [17]; 2) the implementation of a virtual laboratory to teach signals and systems, through audio examples and music synthesis [27]; 3) a sound demonstration software developed in MATLAB to teach DSP antialiasing filters [34]. Nonetheless, these works have not considered music concepts such as timbre, melody, harmony, pitch, tempo, and beat which are the main pedagogical resources inherent in music focused on teaching Signal Processing.
This work presents a hands-on approach to laboratory activities for complementing the Signal Processing courses, taking music as the interactive element and the link between theory and practice. In this manner it is expected for the students to correlate their audition and vision senses, the listening of melodies, observing the graphs generated by time domain, frequency domain, and time-frequency domain and therefore, achieve an improvement in the understanding and motivation. In that sense, this proposal contains a degree of novelty in the teaching approach, complementing the theoretical courses with attractive activities for engineering students, and also for music students who wish to learn the mathematical and physical foundations behind theory of music.
Finally, this set of laboratory activities are part of an educational innovation aimed at helping the Automation Engineering at the Universidad Autónoma de Querétaro (UAQ). A previous work was recently published by this faculty [9] focused on the multidisciplinary industrial robot for teaching mechatronics. Such works help significantly to preserve the Accreditation Board for Engineering and Technology (ABET), which has been awarded to the program since 2016. This accreditation is important for undergraduate students since it guarantees the international standard requirements in the disciplines of applied science, computing, engineering and engineering technology both at the associated bachelor and master degree levels. This paper is organized as follows: section 2 enlists fundamental concepts related to the course background, section III describes in detail the activities methodology, section 4 explains the evaluation of the course according to surveys and opinions taken from teachers and students. Conclusions after applying and evaluating the results of the proposed methodology are given.

| Music concepts
Some important elements of the music are described below: Tuning systems: Western music has been evolving in different tuning systems; however, the most relevant are the Pythagorean and the 12-Tone Equal Tempered systems (12-TET). The Pythagorean tuning could be obtained from where F 0 is the reference frequency and n are an integer number. In this equation, when the frequency obtained exceeds the octave, it must be multiplied by two. Pythagorean tuning has a problem that arises since no stack of 3:2 intervals (perfect fifths) will fit exactly into any stack of 2:1 intervals (octaves). The (12-TET) was extended in 1721 by Johan Sebastian Bach in "The Well-Tempered Clavier" in order to demonstrate the music possibilities of this system and solve the problem generated by Pythagorean tuning [18]. The 12-TET system has the seven natural notes (C-D-E-F-G-A-B), and five alterations obtained from a circle of fifths. The complete chromatic scale with twelve equal steps (semitones) could be expressed as According to Equation (2), the frequencies generated for eight octaves are shown in Table 1.
Sheet notation: The most common music notation is the sheet music that is the basic form in which classical Western music is written. This notation uses symbols to indicate melodies, time duration, harmonies and rhythmic patterns.
Sheet music is composed of five principal lines, and their respective spaces indicate the pitch of the note.
Regarding Signal Processing, sheet music is a very highresolution time-frequency map because music symbols represent both, time duration and pitch at that time, while the rhythmic elements are defined by beat (the basic unit of time). The time duration of each note and their silences are represented according to Table 2.
With this symbolic notation the time duration and pitch of each note in a musical composition could be written.

| Frequency and time-frequency concepts
Fourier Transform (FT): In continuous signals, frequency transform is calculated using FT expressed by Equation (3) F where f(t) represents the signal in time domain, and F(ω) is the respective projection in the frequency domain. The FT could be implemented using the Fast Fourier Transform (FFT) when the signals are discretized. FT provides frequency information difficult to analyze in time representation. In the case of where W(t − τ) is the time window that is moved along the signal finding the FT in each fragment of time. When the window is a Gaussian function, the STFT is called a Gabor Transform. The main disadvantage with STFT is the uncertainty that it generates in frequency when the window period is too short, and in time, when the window period is increased. This trade-off is very similar to the well-known uncertainty principle which is present in the Quantum Mechanics (QM) [32]. Continuous Wavelet Transform (CWT): The continuous wavelet transform is a multi-resolution analysis and is expressed by Equation (5) where ψ is called the mother wavelet and is expressed by In Equation (6), the frequency is scaled by a factor and temporally shifted by b factor. The instantaneous frequency could be recovered using the complex analytic wavelet. When mother wavelet is a complex-valued function, their coefficients obtained can be analyzed in modulus and phase [4]. The main advantage in CWT in comparison with STFT which is due to its multi-resolution analysis; thus, it is possible to obtain a better information in abrupt frequency changes.

| ACTIVITIES
The course is organized in five lectures and lab activities performed in MATLAB, as follows:

| Activity one: Introductory session
Objective: To familiarize the students with the musical scales, nature of sound and audio signals. In the first lecture, the origins of Western music scales and its relation with physical and mathematical fundamentals are explained, as well as music elements such as pitch, amplitude, dynamics, tone color, tempo, and beat. In activity one, the students implement the Pythagorean and 12-TET tuning systems, from the Equations 1 and 2 in order to compare in a graphic and auditive way differences. In the MATLAB code the parameters associated with the timbre can be modified by students, experimenting for themselves auditive and graphical variations. Figure 1a shows a comparison between Pythagorean and 12-TET scales and 1b a fragment of the waveform for a generated melody.

| Activity two: Create different melodies
Objective: To introduce musical figures, sheet notation and some basic elements in musical compositions. In a second lecture some fundamentals of sheet music notation and musical figures through solfeggio activity are explained. Here the students can correlate intuitively the relationship between musical figures and time and frequency concepts. Melodies could be recorded from real instruments, as well as a melody generator implemented in MATLAB according Equation 2, where the students will be able to generate their own melodies. In Table 3 three example melodies are proposed.
For the generation of the melodies in MATLAB, two numeric vector was created, notas, an tempo, where 12-TET scale and musical figures respectively are represented as integer numbers in the code.

| Activity three: Fourier transform
Objective: Implement Fourier Transform algorithm using the FFT and analyzing different audio files. In the third lecture, the Fourier Series and transform are explained, including continuous and discrete cases. Different stationary signals are shown graphically in time domain and frequency domain, with a simultaneously reproduction of the sound associate to each waveform. In this sense, the students can strongly correlate properties of sound, with their respective harmonics components.
In time domain, the students could easily identify time duration of the notes in the melodies generated in last activity. However, frequency components are not visible in this representation. In the lab session 3, the same melodies are taken in order to analyze their frequency response using the FFT function. Timbre concept is reinforced once more, changing spectral intensities of harmonics components within the melody generator. In Figure 2a, is shown the time domain for melody of example two, and Figure 2b is their respective frequency domain representation.
Principal aim of this activity, is to show the limitation inherent in the Fourier representation of non-stationary signals (as the music) and the need for simultaneous information in time and frequency domains. The students may notice in Figure 2b the frequency components of the melody, but it is not possible to identify the order in which the notes were played, as well as the time duration of each of them.

| Activity four: Short-time fourier transform
Objective: To introduce time-frequency analysis. In the fourth lecture, there are time-frequency concepts showing the limitations of time-domain and frequency-domain analysis by themselves, and therefore, the need to create another approach that combines the two domains is evident. Short-Time Fourier Transform (STFT) equations are deduced from FT, including the windowed concept. In the activity four, STFT is implemented using the spectrogram function of MATLAB. The adjustment parameters are the kind of windowing (Rectangular, Hamming, Hann, Blackman, Triangular, Bartlett), the number of windows for FFT, the overlap. Once the parameters are selected, time-frequency maps are obtained from previous melodies. An example of time-frequency map obtained from STFT is shown in Figure 3.
In this experimentation, the students can observe the uncertainly principle entailing in the STFT transform. However, using the appropriate parameters, it is possible to obtain the necessary information to reconstruct the musical notes that compose a certain melody. The next part of this activity consist in obtaining the main features from STFT, in such a way as to enable estimates the vectors tempo and notas.

| Activity five: Continuous wavelet transform
Objective: Introduce CWT algorithm with the aim to compare with previous results obtained with STFT. In the fifth lecture, the uncertainty time-frequency problem is shown using some non-stationary signals as examples, the Continuous Wavelet Transform (CWT) is introduced, showing the advantages and issues of this multiresolution approach. In the activity five, the cwt MATLAB function is used to obtain a time-frequency representation for the same studied melodies. The adjustment parameters are the mother wavelet selection, and the scale factor. When parameters were adjusted, then the automatic melody detection is carry  out using CWT instead STFT. Finally, a comparison between both, STFT and CWT is proposed, by taking into account the time processing and accuracy of detection. An example of time-frequency map obtained from CWT is shown in Figure 4.

| PROGRAM EVALUATION
Qualitative and quantitative analysis were made in order to evaluating the effectiveness of the proposed activities and lectures.

| Evaluation based on survey
Qualitative analysis was based on a survey applied to 30 undergraduate engineering students, enrolled in first half of 2017 Signal Analysis Lab program. Four teachers participated as assistants to the lectures in order to obtain a feedback. The survey was conducted after the completion of the entire practice sessions. A 5-point Likert scale [10], with response scales: 1-strongly disagree; 2-disagree; 3-neither agree nor disagree; 4-agree and 5-strongly agree, was used in this study. The questionnaire and corresponding average scores are provided in Table 4. The questionnaire was examined by two faculty members with more than 10 years' experience in undergraduate teaching Signal Processing courses. The total average value of the survey was 4.57, indicating that in general terms, the proposed method to teach signal processing concepts and algorithms is successful. By comparing the score of each question, it can be noted that the high score in question 1, indicates that both, teachers and students have found the relationship between the music and signal processing. The second highest average is shared between questions 8, 12, and 13, indicating that the majority of teachers and students have found the workshop exciting and they would recommend this method to other engineering student. The third highest score was shared between questions 5, 7, and 11, where it is suggested that the workshop could be implemented as a complement to signals and systems class, given that the theoretical concepts including non-stationary signals could be more easily internalized using the music sounds and graphs as examples.
The opinions of the students and teachers are similar in most of the items, except in questions 2 and 3 in Table 4, where the teachers thought that the activities order could be reorganized to achieve a better understanding of the musical concepts. In the student's case, some issues were detected, the most relevant facts are: The transition from theory to implementation of Pythagorean and 12-TET scales, generated some confusion; therefore, there is a decrease in motivation for that particular practice.
In general, the students possess a reasonable programming level; however some of them were not familiarized with MATLAB environment. The time proposed of two hours by each lecture was not enough for some topics.
Based on the above, it is expected to reorganize the activities, making an introduction to MATLAB codes, complementing the workshop with an additional lecture about musical notation, and increasing the time for certain lectures. The proposed modifications are depicted in Table 5.

| Evaluation based on specific indicators
Based upon ABET criteria, five specific indicators for DSP courses in Engineering Faculty at Universidad Autónoma de Querétaro was proposed in Table 6.
Three Digital System courses were used in the analysis, where the proposed method was applied in first half of 2017.  I-1 and I-3 were evaluated using the test scores from a written exam. I-2 and I-4 were obtained from laboratories activities and I-5 in their final reports. In Table 7 are shown the results of the quantitative analysis.
Analyzing the results, it can be observed that in first half of 2017, ABET indicators are indeed higher than the two preceding periods, particularly the I-3 and I-4 related to the understanding of nature of signals and algorithms implementation respectively, both of which had a significant improvement.
The report card of students in a (0-10) scale from Signal Processing course is shown in the histogram of Figure 5, where it can be seen an improvement of their marks regarding the first half of 2017 versus the two previous scholar semesters. The grades between 7 and 9 were increased compared to previous semesters, while there were no grades below 7. In the same way, the maximum grade 10 was achieved by a larger number of students.

| CONCLUSIONS
Considering the suggested outcomes by ABET, the first experience with the proposed hands-on approach to teach Signal Processing through the music has been motivating for further implementations in the undergraduate student's classes. According to the objectives of this work, music has turned out to be a great tool to explain intensity, time, frequency, and time-frequency concepts in an intuitive way, due the association of both, visual and auditive senses. The evaluation results indicate that the students achieved a better interpretation of Signal Processing techniques through the musical waves, understanding the harmonic concepts visually and audibly, differentiating stationary and non-stationary signals and generalizing the learned concepts in other real-life applications, such as vibration analysis, image processing, biomedical diagnostics among others. In second half of 2017, after the first test of proposed method, the students enrolled in Digital Systems courses proposing a major number of projects related to musical applications, hence, improving the correlation between different areas of knowledge. The current project also led to the creation of a new research line in our University, focused on Musical Information Retrieval (MIR), It should be noted that five students continued to attend more Signal Processing topics, initially motivated by the present work and with the aim of addressing future works such as speech recognition and biomedical signal processing.   Average of the course notes 7. | 2033