Dual Mode High Speed Uncooled Short Wave Infrared Camera Based on FPGA

This paper introduces a short-wave infrared camera logic design based on FPGA. The hardware design adopts the architecture of Sofradir Tecless SWIR detector SW640+Xilinx FPGA xc4vsx55, which has a dual working mode of 640×512 at 25 FPS and 64×48 at 4000 FPS. The function of high precision target tracking can be realized by high frequency capture of laser signal. In this design, the non-uniform correction coefficient table in SDRAM cache Flash is controlled by FPGA to correct the image of full resolution mode, and the table of 64×48 is cached by Bram in FPGA to achieve fast image correction and processing. The exposure time is controlled by the external synchronous signal, and the 14 bit precision image data are collected by four ADCs for restoration. The output data is processed by image bad point correction, non-uniform correction, median filtering, histogram equalization, and then the image is packaged and sent by the specified protocol.


Introduction
Snake SW640 Tecless is the first shortwave infrared detector without a cooler that Sofradir has commercialized. It's a very low noise and high speed InGaAs sensor designed for the most demanding SW applications from 0.9 to 1.7µm [1]. The resolution of SW640 is 640×512, and the pixel size is 15µm. It's well-adapted to a large range of applications such as surveillance, scientific applications and laser detect [2]. The wavelength of a single-wavelength laser is about 1.5µm, and Maximum transmission distance greater than 20km. The SWIR Camera can be used for laser signal detection.
In this design, the camera supports two modes of image output, and the full resolution mode of 640×512 at 25 FPS can be used for large-scale target search. When the target is identified, the 64×48 at 4000 FPS high frame frequency windowing mode can be switched to real-time lock the target unit to achieve the purpose of high-precision laser tracking.

The hardware environment
The hardware environment of the camera is shown in Figure 1. The FPGA input 60MHz clock, and the 9MHz master clock of the Sensor is generated by the DCM division inside the FPGA. ADCs converts four analog signals output by Sensor into 14bit precision digital signals with frequency of 18MHz. NOR Flash stores the calculated non-uniformity correction (NUC) coefficient  -uniform correction coefficient table, and participate in real-time image processing in full resolution mode. Exposure time, gain and working state are controlled by UART command. The external synchronous trigger signal is used to control the starting exposure time of Sensor. The processed image data is output through 4LVDS channels.

Software System Design
System has three working modes: real-time calculation of NUC coefficient, image output full mode and windowing mode.

Real-time calculation of NUC coefficient table
Two-point correction method is currently the most widely used correction method in infrared systems [3]. FPGA software needs to count more than 128 image sequences in full resolution mode in real time. Sw640 has three gains that can be switched, and it needs to calculate and generate the NUC coefficient table in different mode. The calculation method of coefficient table under each gain is the same. Insert a tungsten filament lamp in the light path so that the detector can image uniformly. By adjusting the brightness of tungsten filament lamp, under the same exposure time and gain, 128 groups of highillumination images and low-illumination images are collected. The following data of this group of images are counted. The coefficient table of G and O is calculated according to the following formula.
V : The mean value of pixel response of all sequences of high illumination image. V : The mean value of pixel response of all sequences in low illumination image. V ıȷ (∅ ): The mean value of the response value of each pixel (i, j) in high illumination. V ıȷ (∅ ): The mean value of the response value of each pixel (i, j) in low illumination. The coefficients of each pixel x (∅) occupy 4 Byte, 2 Byte is used to represent the signed number of G , 16 bits and the complement is used to represent. The high 2 bits are integer parts, and the low 14 bits are decimal parts. 2 Byte is used to represent O . The highest bit marks the blind element, and the signed number of 15 bits is represented by the complement. The 15th bit is the integer, the low 14 bits is the decimal part. When the calculated gain coefficient exceeds the range, the pixel is marked as 3 a defective pixel. The storage space occupied by each set of correction coefficients is: 640 * 512 * 4 Byte=1280KB. Three gains need to occupy 3.75MB of memory, and use three-mode redundancy to store in FLASH, which occupies 11.25MB of storage space. In the image processing mode, the image input x (∅) is corrected by two points according to the formula to obtain y .
: the gain of pixel (i, j) in the two-point correction.
: the offset of (i, j) pixels in two-point correction.
: corrected output. Figure 2. The structures of software procedure.
After the system is reset and initialized, FPGA reads the NUC coefficient table stored in Flash into SDRAM. Since the coefficient is the key data that directly affects the image quality, the design method of triple modular redundancy (TMR) is adopted [4]. In order to ensure the validity of the data, three copies of the same correction coefficient are saved, three copies of data are read each time online and compared with SDRAM. If one of the three data is different, then rewrite it into FLASH, covering all three data. Update the correct data in SDRAM at the same time, store the 64×48 coefficient table required by windowing mode in Bram according to windowing position, ready to use in image output mode.

Implementation of full mode and windowing mode
After the correction coefficient is verified, the camera enters the working state. Determine whether the current working state is full mode or windowing mode. Then the corresponding Sensor register configuration. The integral modes of image sensors are divided into IWR and ITR. In IWR mode, there will be another non-uniformity problem due to integral, and ITR mode will not produce this problem. The reason is that during the exposure process, the read-out data will interfere with the readout circuit.  The boundary corresponds to a rising edge of the integration signal. By changing the integration time can change the position of the boundary. One way to solve this problem is to record corrections tables for the different integration times and apply these corrections. Another way is to use ITR mode for shorter time. IWR mode can be used without this boundary when integration time is greater than 11ms (because the max read out time is 11ms).
Therefore, under windowing mode, it is necessary to avoid the emergence of IWR mode and use ITR mode. The improved method is to improve the reading speed of data and obtain more time for ITR mode. Since the clock frequency of the sensor is fixed at 9MHz, the readout rate can only be improved by selecting more readout channels. Sensor supports 2 / 4 / 8 output mode. However, too many readout channels mean that there are many AD interface circuits and IO. After balance, the final selection of 4 output mode, using 4 channels 14 bit AD acquisition. Windowing mode 64×48. According to the read-out timing of the sensor needs159.323us, 4000FPS, each frame time is 250us. So the exposure time in ITR mode is 250-159.323=90.77us. The minimum exposure time of Sensor is 4us.  Figure 5. Readout timing of 4000 FPS mode.

Image processing
In order to improve the processing speed, two separate image processing processes are carried out from the data entered by A/D. Because the full mode resolution is greater than windowing mode, SDRAM is used for non-uniformity correction. In windowing mode, the correction coefficient is stored in 64 × 48 × 16bit × 3 Bram. This not only simplifies the design, but also further improves the processing speed of windowing mode, reduces the processing delay, and is conducive to the later image tracking. Image processing flow is shown in Figure 6. ADC sends image data to the corresponding processing flow according to the current mode after receiving digital signals. Except that the resolutions of the two processes are different, other algorithms are consistent, including nonuniformity correction, defect pixel correction, 3×3 median filter and histogram equalization. The processed image is sent through LVDS interface for calculation by the tracking algorithm. Parallel pipeline processing method is adopted, no external image cache is needed. Thus further improve the data transmission speed. The project is designed and implemented using VHDL language. After the implementation of Xilinx ISE, the device utilization of xc4vsx55 is shown in Table 1.