Efficient demodulation scheme for rolling-shutter-patterning of CMOS image sensor based visible light communications

Recently even the low-end mobile-phones are equipped with a high-resolution complementary-metal-oxide-semiconductor (CMOS) image sensor. This motivates using a CMOS image sensor for visible light communication (VLC). Here we propose and demonstrate an efficient demodulation scheme to synchronize and demodulate the rolling shutter pattern in image sensor based VLC. The implementation algorithm is discussed. The bit-error-rate (BER) performance and processing latency are evaluated and compared with other thresholding schemes. © 2017 Optical Society of America OCIS codes: (230.3670) Light-emitting diodes; (060.4510) Optical communications; (060.4080) Modulation. References and links 1. S. Wu, H. Wang, and C. H. Youn, “Visible light communications for 5G wireless networking systems: from fixed to mobile communications,” IEEE Netw. 28(6), 41–45 (2014). 2. C. W. Chow, C. H. Yeh, Y. Liu, and Y. F. Liu, “Digital signal processing for light emitting diode based visible light communication,” IEEE Photon. Soc. Newslett. 26, 9–13 (2012). 3. J. R. D. Retamal, H. M. Oubei, B. Janjua, Y. C. Chi, H. Y. Wang, C. T. Tsai, T. K. Ng, D. H. Hsieh, H. C. Kuo, M. S. Alouini, J. H. He, G. R. Lin, and B. S. Ooi, “4-Gbit/s visible light communication link based on 16-QAM OFDM transmission over remote phosphor-film converted white light by using blue laser diode,” Opt. Express 23(26), 33656–33666 (2015). 4. H. H. Lu, Y. P. Lin, P. Y. Wu, C. Y. Chen, M. C. Chen, and T. W. Jhang, “A multiple-input-multiple-output visible light communication system based on VCSELs and spatial light modulators,” Opt. Express 22(3), 3468– 3474 (2014). 5. C. H. Chang, C. Y. Li, H. H. Lu, C. Y. Lin, J. H. Chen, Z. W. Wan, and C. J. Cheng, “A 100-Gb/s multiple-input multiple-output visible laser light communication system,” J. Lightwave Technol. 32(24), 4723–4729 (2014). 6. Z. Wang, C. Yu, W. D. Zhong, J. Chen, and W. Chen, “Performance of a novel LED lamp arrangement to reduce SNR fluctuation for multi-user visible light communication systems,” Opt. Express 20(4), 4564–4573 (2012). 7. C. W. Hsu, C. W. Chow, I. C. Lu, Y. L. Liu, C. H. Yeh, and Y. Liu, “High speed imaging 3× 3 MIMO phosphor white-light LED based visible light communication system,” IEEE Photonics J. 8(6), 7907406 (2016). 8. R. Boubezari, H. L. Minh, Z. Ghassemlooy, and A. Bouridane, “Data detection for smartphone visible light communications,” Proc. CSNDSP’14, 1034–1038. 9. P. Luo, M. Zhang, Z. Ghassemlooy, H. L. Minh, H. M. Tsai, X. Tang, L. C. Png, and D. Han, “Experimental demonstration of RGB LED-based optical camera communications,” IEEE Photonics J. 7, 7904242 (2015). 10. C. Danakis, M. Afgani, G. Povey, I. Underwood, and H. Haas, “Using a CMOS camera sensor for visible light communication,” Proc. OWC’12, 1244–1248. 11. C. W. Chow, C. Y. Chen, and S. H. Chen, “Visible light communication using mobile-phone camera with data rate higher than frame rate,” Opt. Express 23(20), 26080–26085 (2015). 12. K. Liang, C. W. Chow, and Y. Liu, “Mobile-phone based visible light communication using region-grow light source tracking for unstable light source,” Opt. Express 24(15), 17505–17510 (2016). 13. W. C. Wang, C. W. Chow, L. Y. Wei, Y. Liu, and C. H. Yeh, “Long distance non-line-of-sight (NLOS) visible light signal detection based on rolling-shutter-patterning of mobile-phone camera,” Opt. Express 25(9), 10103– 10108 (2017). Vol. 25, No. 20 | 2 Oct 2017 | OPTICS EXPRESS 24362 #301898 https://doi.org/10.1364/OE.25.024362 Journal © 2017 Received 7 Jul 2017; revised 26 Jul 2017; accepted 26 Jul 2017; published 25 Sep 2017


Introduction
Visible light communication (VLC) has attracted much interest recently because it can provide wireless communication using the visible light spectrum to relieve the exhausted traditional radio-frequency (RF) communication spectrum.Hence, it has been considered as one of the promising technologies for the 5G and beyond mobile communication systems [1].Different high speed VLC systems have been demonstrated [2][3][4][5][6][7].Laser diode (LD) and light emitting diode (LED) are two main optical transmitters (Txs) in VLC.Being a coherent light source, LD based VLC can achieve a much higher data rate and longer distance transmission [4,5].As the LD based VLC has a high directionality, the offset tolerance between the LD Tx and the receiver (Rx) is usually low.On the other hand, LED is lower cost than LD.The blue LED chip covered with yellow phosphor layer can provide white-light illumination and VLC simultaneously [6,7].As the yellow phosphor has a long relaxation time, the white-light LED based VLC has lower data rate.Photo-diode (PD) and image sensors are two main optical Rxs in VLC.Recently even the low-end mobile-phones are equipped with a high resolution complementary-metal-oxide-semiconductor (CMOS) image sensor; supporting highdefinition (HD) moive capture at frame rate even up to 60 frame-per-second (fps).Hence, this motivates using CMOS image sensor for the VLC systems [8].
One approach of CMOS image sensor based VLC depends on the global shutter effect of the image sensor.However, this approach uses a frame or multiple frames to represent one logic bit; and the data rate is usually limited; and a data rate of only 150 bit/s (3 x 50 bit/s when using red-green-blue LEDs) based on this approach was demonstrated [9].Another approach depends on the rolling shutter effect, in which the detection data rate can be significantly enhanced [10][11][12].Previously, we demonstrated that a net data rate of 5.76 kbit/s can be achieved by using commercially available mobile phone CMOS image sensor [12].
In this work, we propose and demonstrate an efficient demodulation scheme to synchronize and demodulate the rolling shutter pattern.As the demodulation performance is usually limited by the number of pixel row per bit, linear interpolation is proposed and employed to increase the effective sampling points.Then, a novel thresholding scheme, named extreme value averaging (EVA) is proposed and demonstrated.This scheme is particularly suitable for high fluctuation data pattern likes the rolling shutter pattern.Experimental results show that by using the proposed scheme, extinction ratio (ER) enhancement required in previous thresholding schemes [12] is not needed and the processing latency can be significantly improved.A recorded data of 7.68 kbit/s can be achieved.The implementation algorithm of EVA is discussed.The bit-error-rate (BER) performance and processing latency are evaluated and compared with other thresholding schemes.

Experiment and algorithm
Figure 1 shows the experimental setup of the CMOS image sensor based VLC system.Matlab program in a computer is used to construct the data packets, which are then stored in an arbitrary waveform generator (AWG, Tektronix AFG3252C) with sampling rate of 2 GSample/s and bandwidth of 240 MHz.The AWG acts as a digital-to-analog convertor (DAC) providing the electrical waveform to drive a phosphor-based white-light LED chips (Cree XR-E) with color temperature of 5500 K.After a free-space propagation, the VLC signal is received by a mobile-phone CMOS image sensor (Iphone6) with resolution of 1920 x 1080 pixels and frame rate of 60 fps.In the proof-of-concept demonstrate, the movie captured by the mobile-phone are transferred to a computer; and Maltab program is used for the demodulation.Application program in the mobile-phone could be used for the demodulation.
When the CMOS image sensor is operated, it will not capture the whole image at the same time (called the global shuttering); but each pixel row is activated sequentially (called the rolling shuttering [10][11]).As the LED is modulated on and off, bright and dark rows will be captured in each image frame.Then, by demodulating them, the transmitted VLC data can be retrieved.Besides, after all pixel rows in the CMOS image sensor are activated in one frame period, there will be a processing time-gap; which can be about 30% of one frame period.During this processing time-gap, the CMOS image sensor cannot detect any signal.Hence the CMOS image sensor is in "burst-mode"; and it is difficult to synchronize it with the LED Tx.Because of the processing time-gap, in order to guarantee a complete VLC packet sent by the Tx can be captured in one image frame, the length of each data packet is 1/180 s and each packet is transmitted three times.The structure of VLC data packet contains a 16-bit header and different bit-length payload.Non-return-to-zero (NRZ) on-off keying (OOK) format is used.It is possible to use other data patterns as the header as long as they can be differentiated from the payload.In this work, we use all logic one.Using all logic zero or using periodic Manchester coded data [11] are also possible.Then, the image (1080 x 1920) shown in Fig. 2(b) can be observed.Due to the uneven light exposure of the CMOS image sensor, "blooming" (a bright spot can be observed at the image screen due to pixel saturation) will be observed.Then, blooming mitigation is used [11], and a 1 x 1080 matrix of grayscale levels can be obtained as shown in Fig. 2(c).Even the "blooming" mitigation is applied; there is still a high data fluctuation owing to different illumination falling into different parts of the CMOS image sensor, as can be observed in Fig. 2(c).In this work, two novel processes are employed as indicated in the "Linear interpolation module" and "thresholding module" boxes in Fig. 2(a).Firstly, in order to increase the effective sampling points for the synchronization and demodulation, a linear interpolation is proposed and applied.Hence, the 1 x 1080 matrix is expanded to 1 x 2159 matrix after the proposed linear interpolation.Secondly, in order to properly defining the logic 1 and 0 in this highly fluctuated grayscale value pattern, we propose and demonstrate the efficient EVA thresholding scheme.The algorithm of EVA is to find the local maximum points and local minimum points of the grayscale value pattern in turn.The average value between the local maximum and local minimum is the threshold.After the thresholding, grayscale levels turn into data logics.Then, determining the location of header and calculating the sampling rate are needed for sampling.Finally, sampled data is examined for BER calculation.Besides the linear interpolation, another key contribution is the EVA scheme as shown in Fig. 3. How to determine the correct extreme value is the most crucial part in this thresholding scheme.As shown in Fig. 3, we first find the maximum point of grayscale pattern; where P max and G max represent the pixel row position and the corresponding grayscale value.Next step is to find the local minimum; hence the cursor is shifted to find the local minimum.This is done by increasing the pixel row position G cursor ; and the corresponding grayscale value at that pixel row position is P cursor.Because of the noise from ambient light; as well as the thermal noise and shot noise of the CMOS image sensor, there are many "spikes" observed in the grayscale pattern as indicated in Fig. 2(c).These noises can cause bit errors during the thresholding.Because of these noises, two conditions are used to identify the correct extreme value but not the noise.First, the fluctuation caused by noises would be smaller than the signal.The solution is by setting a reference value G ref to distinguish the signal from noises.The second condition is the width between two nearby extreme values must be at least one sampling rate.If a local maximum or minimum detected is smaller than one sampling rate, this local maximum and minimum are noise and can be neglected.When the local minimum is determined, the threshold value is the average of local maximum and local minimum; its pixel row location and grayscale value are G threshold and P respectively.The subsequence procedure is almost the same but finding local maximum instead of local maximum.This loop will stop until the cursor is shifted to the end of all data points.Since we start from the maximum of grayscale pattern; and this will divide the grayscale pattern into two parts.As a result, we should do the loop again, but by decreasing the cursor location.Figure 4(a) shows the experimental grayscale value pattern and the applied EVA thresholding scheme (red curve).It is worth to point out that since linear interpolation is applied; the x-axis is increased from 1080 (in Fig. 2(c)) to 2159 (in Fig. 4(a)); and a "zoom-in" grayscale pattern is shown in Fig. 4(b).

Results and discussions
Other thresholding schemes, such as quick adaptive (QA) [12] and Bradley adaptive (BA) [13] used in previous works are compared in terms of BER performance and processing latency.The BER performances without and with the linear interpolation at a low illumination of 250 lux are shown in Figs.5(a) and 5(b) respectively.Since the amount of data sent from the Tx is limited to 10 6 , the error-free condition is considered when BER = 1 x 10 −6 .The net data rates after the header and redundant packets removal can be calculated by the bit-length in payload per image frame times the frame rate.Hence, the net data rates are 1.92 kbit/s, 3.84 kbit/s, 5.76 kbit/s and 7.68 kbit/s when the payload bit-lengths are 32-bit, 64bit, 96-bit and 128-bit, respectively.The corresponding sampling rates are 10.1 pixels/bit, 6.08 pixels/bit, 4.34pixels/bit and 3.38 pixels/bit before the linear interpolation.It is worth to point out that when payload bit-length is increased, fewer pixel rows are used to represent one logic bit.Hence, the BER becomes poor, as illustrated in Figs.5(a) and 5(b).Because of this, linear interpolation is employed to increase the sample points of the original grayscale signal. 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03  As shown in Fig. 5(a), at the payload bit-lengths of 32 bits and 64 bits, the BER performances of all thresholding schemes are error-free.When payload bit-lengths are increased to 96 bits and 128 bits, BER performances of QA and BA cannot satisfy the 7% forward error correction (FEC) limit without the ER enhancement.QA thresholding is to calculate the weighted moving average of grayscale values; while BA thresholding is to calculate the moving average in different sections.Both thresholding schemes can adaptively finding the threshold of the grayscale value pattern; but their adaptability is determined by the section length section.It is hard to select optimum section length for different kinds of situations.In order to improve the performance, ER enhancement is required for both the QA and BA thresholding schemes to increase the contrast ratio.Because EVA thresholding scheme can track the data fluctuation more precisely, ER enhancement is not needed.As a result, EVA reveals a good BER performance before and after the linear interpolation is used.Without the use of linear interpolation, the EVA thresholding scheme can achieve error-free transmission at data rate of 5.76 kbit/s.When the linear interpolation is used, the EVA thresholding scheme can achieve a net data rate of 7.68 kbit/s, satisfying the FEC.Finally, the processing latencies are evaluated as shown in Fig. 6.The computer used has an Intel i5-2410M processor @ 2.3 GHz with 4 GB random access memory (RAM).The processing times required for QA, QA with ER enhancement, BA, BA with ER enhancement, and the proposed EVA schemes are 218 ms, 226 ms, 1.3 ms, 8.5 ms, 5.1 ms, respectively.QA requires the longest processing time since it needs to calculate the weighted moving average of grayscale values.BA without ER enhance requires the shortest time, but the BER performance is not good enough when compared with that of the EVA.When BA with ER enhancement is used, the BER performance is similar to that in EVA; but is not as efficient as the EVA thresholding.

Summary
A novel thresholding scheme, named EVA is proposed and experimentally demonstrated to synchronize and demodulate the rolling shutter pattern in CMOS image sensor based VLC.The implementation algorithm of EVA is discussed in detail.This scheme is particularly suitable for high fluctuation data pattern likes the rolling shutter pattern.A recorded net data of 7.68 kbit/s can be achieved.Experimental results show that by using the proposed scheme, ER enhancement required in previous thresholding schemes is not needed and the processing latency can be significantly improved.A recorded net data of 7.68 kbit/s can be achieved.

Fig. 1 .Fig. 2 .
Fig. 1.A proof-of-concept experimental setup of the VLC using LED Tx and mobile-phone CMOS image sensor Rx.

Figure 2 (
Figure 2(a) shows the flow diagram of the synchronization and demodulation processes.First, the raw movie file captured by the mobile-phone CMOS image sensor is converted into grayscale format, in which 255 and 0 represent full brightness and darkness respectively.Then, the image (1080 x 1920) shown in Fig.2(b) can be observed.Due to the uneven light exposure of the CMOS image sensor, "blooming" (a bright spot can be observed at the image screen due to pixel saturation) will be observed.Then, blooming mitigation is used[11], and a 1 x 1080 matrix of grayscale levels can be obtained as shown in Fig.2(c).Even the "blooming" mitigation is applied; there is still a high data fluctuation owing to different illumination falling into different parts of the CMOS image sensor, as can be observed in Fig.2(c).In this work, two novel processes are employed as indicated in the "Linear interpolation module" and "thresholding module" boxes in Fig.2(a).Firstly, in order to increase the effective sampling points for the synchronization and demodulation, a linear interpolation is proposed and applied.Hence, the 1 x 1080 matrix is expanded to 1 x 2159 matrix after the

Fig. 3 .
Fig. 3. Flow diagram showing the working principle of EVA thresholding scheme.

Fig. 4 .
Fig. 4. (a) Experimental grayscale value pattern and the applied EVA thresholding scheme (red curve), (b) a "zoom-in" grayscale value pattern after the two headers are removed.

FECFig. 5 .
Fig. 5. Measured BER performances against different payload bit length (a) without and (b) with the linear interpolation at a low illumination of 250 lux.