A Sound-Image Coding Method Inspired by an Acousto-Optic Electronic Piano

This paper firstly brings up a point that there seems to be a lack of a direct mapping bridge method for interconversion between sound and image, because many methods invented for scientific research on sound visualization are based on the sound characteristics of pitch, duration, loudness and timbre. Secondly, it designs an acousto-optic electronic piano which can possess both sound and color effects with the application of music visualization, and presents a sound-image coding method inspired by the previous design which can transform music note to picture pixel. Thirdly, it manages to transform a simple piece of music called “I am a painting master” into a corresponding picture with the mapping bridge method using Matlab software. More coding bits of the method should be added for complex music. The sound-image coding method exhibits a mapping bridge role for sound-image interactive design, which may help to find essential relationship between music and picture in mentality.


Introduction
Music visualization can present music information through a visual way of graphic images to enhance the performance of the music [1,2]. Many methods have been used to show the enhanced performance effects such as a system of notation which applies different colors and three-dimensional space to present music [3], a visual interface for browsing music collections which counts on a graphical metaphor [4], a Music Visualization on Robot (MVR) prototype system that automatically integrates flashlight, colors and emotion of a robot through music [5].
However, most methods invented are based on the characteristics of pitch, duration, loudness and timbre of sound, there seems to be a lack of a direct link mapping method to present music visualization. This paper firstly designs an acousto-optic electronic piano that has a similar model of the direct link mapping method, further generalize the method of a sound-image coding to show the direct link mapping bridge for sound-image interactive design.

An Acousto-Optic Electronic Piano
The acousto-optic electronic piano is designed based on RGB LED and MCU [6,7]. It can extend the multi-level playing effects of the electronic piano combined with text display. The design provides a possibility that a direct link mapping bridge can be found for sound-image interactive design.

Design Overview
The structure of the design is shown in figure 1 and its performance effects are shown in figure 2. From figures 1 and 2, STC89C52RC is chosen as MCU to control whole components, "I love You" is the text display module with different colors. MCU will detect the situation, send corresponding instructions to RGB led driver and speakers simultaneously when the piano keys are pressed.

Theoretical Analysis
In the design, the crystal vibration frequency of MCU is 12MHz, so the machine cycle can be set as 12×[1 ∕ (12×10 6 )]=10 -6 s, i.e. 1μs. Also, it is assumed that mixed effects are based on the condition that R, G, B use the same light intensity. Because scale steps correspond to different frequencies, MCU will utilize the timer to produce square waves which have the same corresponding frequencies, the mathematical relation between note frequency and T in figure 3 is T=65536-[(1 ∕ F)×10 6 ] ∕ 2. The "Total" in figure 3 represents total time length per period set by timer of MCU for each color pin of RGB led. In the design, the total time per period is set 100 times the machine cycle, i.e. 100μs, far less than visual retention time 0.1s ~ 0.4s.
When the timer is on, the luminous time length of R pin, G pin and B pin will be controlled by MCU separately, so the mixed effects of colors will then be determined by each color pin combined. Through this control way, the theoretical whole color kinds will be 100 3 , just change the values of one color pin within 100, the synthesised color will be changed.

Inspiration Illustration
It is commonly known that Red, Green and Blue are three primary colors of light which can construct a pixel of an image. Coincidentally, an RGB led can be seen as a pixel in the text display module, MCU can be seen as a bridge that connects sound and image pixel in a system. To be more precise, each color pin of RGB led is linked by MCU with a pitch frequency of note.
However, the performance effects of audio-visual combination in the design may be constrained by hardware conditions themselves. If a new method which can simulate the mapping bridge function of MCU and principles of implementation without hardware, some conveniences for sound-image interactive design may be brought by the method.

A Sound-Image Coding Method
The key implementation of the design above mostly adopts digital method, such as part A and part B of figure 3. If systematic representation of the sound-image performance is needed without relying on hardware, coding can be applied to help achieve this. Therefore, a sound-image coding method is introduced to enhance performance effects.

Method Analysis
A complete electronic piano has 88 keys, one key to one scale step, a group of 12 scale steps. So, at least 88 scale steps should be considered. In addition to the basic scale steps, metaphone mark, liaison mark, etc. should also be taken into consideration in the sound-image coding method. On account of the specificity of each scale step, the sound-image coding method adopts one to one mapping way. Figure 4 provides a basic coding method that arranges note in stave for a small range of use. From figure 4, it can be seen that the number of bits for each note encoded is 24, every eight into 3 groups, respectively represent R, G and B. As shown in figure 4, R group mainly indicates the scale steps, G group mainly shows the speed and structure of note and B group mainly provide some auxiliary help for note. The meaning of each bit is illustrated in figure 4. In this coding way, each note corresponds to a pixel in an image with 24-bits RGB. However, the coding method cannot contain all conditions of music note. If difficult pieces of music wanted to be covered in this coding method, more coding bits should be added, such as sound-image coding representation of 48-bits RGB.

Method Application
Through this coding method, the music can be presented in a form of a specific image, and an image can also be transformed into music. The application of the method generally needs to follow 4 steps in order from music to picture.
• Identify each note in stave and encoding them following the approach shown in figure 4.
• Enter the code of each note into a professional software, such as Matlab.
• Arrange the order of note code with scanning technique. • Combine these note code which corresponds to pixels into a picture. From picture to music, the steps will be the inverse of the 4 steps above. For a detailed description, the paper takes a song "I am a painting master" as an example shown in figure 5 to with this method.  figure 6. In figure 6, it shows that white represents right-hand note code and grey shows left-hand note code. Secondly, enter the note code into Matlab. Thirdly, arrange the note code order in the transverse manner of scanning. Figure 7 is the detailed Matlab program code for the second and third steps, it will construct an image of 80*80 pixels. Lastly, form a corresponding image, the image result is shown in figure 8. From figure 8, it can be imagined that regular "bricks" of different colors are lining up, like a simple work art by a painting master. This example adopts transverse scanning manner to construct the figure 8, if more vivid picture should be formed, the effort should be on the precise and complex scanning manner.  In summary, this sound-image coding method can enhance music performance effects. Also, the image which contains the whole information of music can be presented in a short real time, it is not necessary to spend much time getting the whole information from a music. Furthermore, if the music needs to be adjusted, just change the values of coding bits. At last, the coding method provides an evaluation of the vitality of music performance, a delightful piece of music should be associated with a vivid picture.

Conclusion
Base on the design of an acousto-optic electronic piano, a sound-image coding method is presented for sound-image interactive design. Although a sound-image coding method is only based on electronic piano theory, it exhibits a mapping bridge role between sound and image. More research on other instruments with different timbres should be done to improve this method. Also, the number of bits used for coding should be integrated for complex music and accurate scanning manner should be adopted for vivid pictures. More detailed analysis on the sound-image coding method will be included in a latter paper.