Detecting abnormality in medical images with thermal and RGB images

- Medical abnormalities in human body are often reflected by raise in temperature at various areas in the body. With the requirement of reliable non-invasive on the increase Infrared Thermal Image is an effective aiding in monitoring and diagnosing medical abnormalities. Existing research has applied Infrared Thermal Image effectively for various medical conditions like breast cancer screening, diabetes and peripheral vascular disorder, Risk Assessment and Treatment Monitoring. Thermal Image cameras are capable of capturing the body temperature variations, these temperature variations can lead to significant diagnosis in several areas ranging from simple flu caused by influenza virus to several conditions like diabetes, eye syndrome and thyroid to name a few. Heat distribution captured from Infrared Thermal Image by thermal cameras like Forward Looking Infrared Imaging (FLIR) with a sensitivity range of 0.10C and wide temperature ranging from - 100C to +1000C can produce good thermal images. This research suggests a non-expensive and non-obtrusive diagnostic procedure which utilizes thermal imaging for unexplored areas of applying thermal imaging and the possibility of extracting thermal variations with RGB images. To achieve the objective various image processing techniques like image preprocessing, selecting the Region of Interest (ROI), extraction by region segmentation, selective feature extraction and finally suitable classification of the relevant application selection are adopted. Results of the proposed method for detecting abnormality have been validated based on the temperature map histogram comparison from thermal image.


Introduction
Several research articles have utilized temperature as a critical factor and indicator for abnormality in various living organisms including the humans. Invasive techniques like blood sampling, tissue sampling etc., for medical diagnosis are currently being replaced with noninvasive techniques like ultrasound imaging, magnetic resonance, infra-red temperature measurement etc., for critical medical diagnosis also. Human beings are homoeothermic and their body temperature is uniform and interestingly it differs for the areas encompassing them. Considering the uniformity in temperature in normal humans if there are temperature variations at specific areas of the body then it can be an effective indicator of a possible disease condition. All human beings have a regulatory mechanism which ensures there is temperature control inside the body. It is a well-known fact that normal body temperature is around 36⁰C and 37.5⁰C and any measurement beyond this is a clear indication of abnormality or sickness. Sir Herschel in 1800 was the first person to indicate radiations can happen due to temperature which was also substantiated by a capture of an infrared image which further paved the way for devising a mechanism to measure various layers of temperature in the area of interest. Later these measurements led to a new avenue of study called infrared thermography (IRT) which was applied in many industrial and medical applications. The commonly used device for IRT are Infrared cameras with variable feature setting like the IR radiation, facilitating capturing the temperature values from the reflected radiation intensity. This conversion or mapping of reflected radiation to corresponding temperature variations is named as thermography or thermal imaging.
Any physical object which has a temperature more than 0⁰K by nature will radiate electromagnetic waves out of which the spectrum of infrared radiation and its recording is names as Thermography. It is also observed that electromagnetic waves corresponding to the values above the ambient temperature are dominantly in the infrared spectrum band. By devising good sensors, the IR radiations can be converted to equivalent electrical voltages and manipulated to get useful information to display and process. For medical applications wavelengths ranging from [3][4][5][6][7][8][9][10][11][12][13][14][15] are helpful indicators of human temperature variations. Infrared cameras come with varying resolutions and models for various applications. Thermal sensitivity which is indicative of the accuracy of capture is the smallest temperature difference which can be detected by the camera. E.g., For an infrared camera with uncooled micro bolometer detector the temperature sensitivity could even be 0.01⁰C @ 30⁰C whereas for an infrared camera with semiconductor kind of detector the temperature sensitivity could even be 25mK @ 30 ⁰C. For health care thermal imaging these cameras can capture temperature variations from the heat emitted from human body. The thyroid gland is an important hormone gland which plays a major role in regulating the metabolism, growth and development of the human body. It plays a major role in regulate many body functions by constantly releasing a consistent number of thyroid hormones into the bloodstream, which can be sensed by reflection in thermal cameras. IR thermal cameras with sensitivity of 0.01C with temperature range from -100⁰C to +1000⁰C are effective in capturing thermal image. Currently all research utilize instantaneous human temperature indication in this research for more specific and small change detection the differential temperature values are proposed to be considered. For dynamically varying thermo graphic images Fast Fourier Transform (FFT) applied on the static temperature are used quantitative investigation. The frame of reference for which thermal analysis is to be done depends on various factors like the surrounded air flow, humidity, ambient temperature and distance of observance. These constraints pave to the solution that if the temperature variation to be observed is less than the conditions for capturing thermography images can be executed in restricted surroundings. For correct identification certain conditions for imaging like the controlled illumination of the imaging room the distance of the subject etc., must be predefined. The bio heat transfer model can very well be applied to understand the temperature distribution on the human body surface. This modeling helps in assuring that a specific condition of disorder identified by thermal imaging in the region of interest is accurate. Equations (1), (2) and (3) gives a simplified model for Pennes bio heat equation where q m -Metabolic heat source The purpose of an Infrared thermal camera is to effectively represent the thermal abnormalities and if minimal temperature changes can be observed at earlier stages of disease diagnosis, this can lead to early detection of abnormality in any human organ which becomes an interesting medical application of infrared images. Although that as stated thermal imaging has to be done in specified background conditions and it is highly operator dependent, due to its low cost, ease of use, being a nonradiation technique and its remote handling capability, it is widely accepted in medical applications also. IRT can be done quickly as well as it can be synchronized for conceiving and monitor larger body areas. Diagnosing a thermography image is much simpler and faster and requires lesser skills compared to other imaging techniques like ultrasound and MRI. Thermal imaging also has the advantage that it is easy to deploy, can be repeatedly used without any wait time or preparation and it is free from negative emission impacts since it tracks only the differences extracted from the specific regions. With the big list of advantages and favorable conditions thermography has been deployed as an effective medical diagnostic tool in monitoring thermal regulation of the human system. For this research, an IR camera with FLIR tools which can customize Areas of Interest (AOIs), allow for quick analysis, continuous or intermittent; automate data recording and archiving for abnormality detection has been used. They can also record and display statistical measures like the maximum, average minimum and deviation of temperature of a selected region. The images are also linked with MATLAB software for color range extraction so that it can be mapped for temperature variations.

Existing Literature
For the past two decades, thermal imaging systems with IR imaging has a paradigm shift in industrial applications, arising due to their wide usage in defense, military and most interestingly in medical applications [1]. Due to the fact that industries and equipment manufacturers are investing heavily towards newer techniques for thermal imaging the scope of application research and development has also increased. A sample thermal image is shown in Figure 1. As far as medical applications are concerned thermal imaging has paved the way for new application domains like analyzing various conditions and symptoms of the human biological system based on the variations and distribution of the body temperature [2]. The temperature mapping is also useful technique which can extend the research towards mapping RGB images with IR images with the help of sophisticated mapping algorithms [3]. Several standards have been documented in the ISO regarding Infrared thermal imaging for measurement by remotely capturing human body temperature with proper implementation and operating procedures for mass screening. In medical applications currently there are numerous researches on identifying and atomizing diagnosis with image manipulation and image analysis.
To infer thermal mapping in materials one may refer to "iron" where an iron rod turns to yellow when it is heated up and when the temperature is further increased it appears red. Researchers have also used ultrasound Bmode scanning to find the temperature mapping in underlying tissues Non-invasive temperature measurement of tissues deep inside the body has great potential for early diagnosis by monitoring the internal temperature mapping [4]. Middle and long range IR imaging has been used to identify Stress in human imaging, based on which emotional classification can be applied [5]. Shortwave IR imaging can be helpful for applications like peripheral blood perfusion and blood vessels imaging. The near infrared waves have been researched for blood perfusion, cerebral blood flow characterization, to find oxygen saturation, monitoring brain injury and also as biomarkers for images. IR imaging has wide spread applications in temperature, metabolism monitoring and respiration detection in NICU [6]. These discussions and applications give an idea that thermal imaging can be applied to variety of problems in diagnosis wherever the body exhibits thermal variations as biomarkers to indicate abnormalities.
The concept of embedding contactless imaging inside Incubator [7] for monitoring several physiological parameters using imaging techniques inside the neonatal incubator using IR imaging and Magnetic impedance tomography (MIT) has been ideated by combined imaging techniques. Another interesting application involves classification of optical images into cataract, corneal haze and iridocyclitis used specificity and sensitivity as features and the back propagation algorithm for error correction. The study utilized 670 normal IR images for performing the feature extraction and classification. The subjected images were characterized by mean ocular temperature and the temperature variation displayed distinct variations in surface temperatures [8]. Thermal imaging was proposed as an easy alternative to differentiate normal and abnormal eye based on the irregular distribution of temperature based on temperature variance using the features specificity, susceptivity and ROC [9].
IR digital imaging can be used to identify breast cancer based on the principle that metabolic activity and vascular circulation in precancerous tissues and areas surrounding developing cancer are almost always higher than in normal tissues. Interestingly IR imaging can predict 10 years before a similar condition be predicted with mammograms with appropriate algorithms. IR imaging also has the advantage that it does not require physical contact for examination, free from ion radiations and eliminates the requirement for experienced radiologists [10].
Thermography may be a suitable alternate modality to identify joint inflammation in humans using static and dynamic protocols The static protocol measures steadystate conditions while the dynamic includes cooling the subject and increasing the surround temperature, then multiple captures of thermo-grams provides useful tissue information [11].
A study on the utilization of infrared thermal imaging for identifying musculoskeletal injuries with the perspective of facture diagnosis reveals that thermo-grams are very useful in diagnosis of fractures [12]. Thermal imaging applied on eye diagnosis and malfunctioning observing the activities of ciliaris muscle which influencing the bend of the eye lens is advantageous compared to the traditional dioptre measurement [13]. Another area where thermal imaging has been applied is in the diagnosis of breast images. Several variants using the normalized bi-spectrum features and texture analysis are found effective in classifying normal and benign tumour classification [14] [15].
With the noticeable contribution of thermal imaging and due to the fact that it can be applied to various other medical condition diagnosis this research shall study the possibility of applying this technique to for various disease conditions like Hyperthermia, Hyperthyroid, Hypothyroid, Diabetic, influenza (H1N1) and eye syndrome.
The steps involved in thermal image classification is shown in Figure 2.

Image Data set, modeling and processing
Digital Infrared Thermal Imaging has been providing open-source thermal images which have been used for various analyses in this research, these images are widely recognized by doctors and medical institutions and medical ethics and protocols have been followed for the images available in this database. The specifications for the camera used the FLIR-E30 for image capture is capable of handling temperature ranging from -20 to 250o C with indicated accuracies of ±2% and good sensitivity of 0.10 oC, the pixel specification for images are 160 x 120 pixels. Most of the image classification techniques utilize several preprocessing steps like enhancing the image contrast, removal of noise depends, resizing images and segmentation for capturing the ROI so that effective best feature extraction is achieved. Contrast enhancement can be done linearly or non-linearly, several filtering techniques like median filter, diffusion filters etc., can be used. This research applied the wavelet based denoising algorithm referred in [16]. Some techniques which are widely used for segmentation are thresholding, edge detection and region-based segmentation.

Fig. 2: Steps in Image Processing
These techniques could not yield expected results for segmentation and hence could not be considered useful. The several investigations applied, gives some useful insights for segmentation approaches that may be most suitable for thermal images and consequent ROI extraction of malignant lesions. For this research the unsupervised K-Means clustering algorithm is used to segment the area of interest. This algorithm divides the data into K-clusters based on the recognized centers named K-centroids based on similar color sensing in this case similar temperature regions exhibit same color. The separated data objects represent the sub regions. As shown in Equation (4), the algorithm works on the principle of forming clusters to minimize the sum of squared distances between all points with reference to the cluster center.
where J is the objective function k is the number of cluster c j is the centroid Fig. 3: K-mean segmented image based on color �x n − c j � 2 is the distance function To begin with the algorithm selects the value of k the number of clusters and performs the k-mean clustering. The processing of any thermal images mostly depends on the temperature distribution of normal person's thermal image of eye which is symmetrical to temperature range which could be confirmed by many clinical reports. For this research the first case considered is to identify Thyroid Eye Disease (TED), also referred to as Graves' Eye Disease or Graves' ophthalmopathy, which is a condition where the eye muscles, eyelids, tear glands and fatty tissues behind the eye become inflamed due to abnormal thyroid levels with the inspection of normal and abnormal patients. The image is first preprocessed for enhancement, the image gives various features for abnormality identification. The next process is to separate the colorbased parts of the image. This step is essential since the red component is the indicating color for temperature variations. Figure 3 indicates a color-based segmentation based on K-mean clustering. In the subject of interest, the ROI is around the eyes in the red color-based segmentation. The classification is based on the intensity level dividing it into low level pixels and high-level pixels [17]. As expected, the inflamed parts of the eye and its surroundings will have higher temperatures from the human eye as considered with a normal eye [21][22].
After segmentation feature selection is very important, the feature extraction technique that is applied after the segmentation is implemented based on statistical texture features namely the gray-level co-occurrence matrix (GLCM) which has the feature vector mean, standard deviation, smoothness, entropy, kurtosis, skewness, variance, contrast, correlation and energy level. Once the features are extraction further it is validated with its corresponding histograms. The classification was done well with the extracted features for classifying normal and thyroid affected eye. The histogram indicated in Figure 4 is corresponding to a   Figure 5 is the histogram generated from image corresponding to an eye affected with thyroid inflammation. In both the figures the X axis corresponds to the temperature and Y axis number of pixels observed in that particular temperature or intensity.

Dedicated Imaging tool
Forward-looking infrared imaging systems (FLIR) cameras is a handy tool which offer a unique view of explosive volcanism by providing an image of calibrated temperatures. FLIR Tools is a very powerful and free to use software solution which can be used to quickly import, edit, and analyze images, and is capable of generating professional PDF inspection reports. It's can effectively show users or decision-support the problems identified with FLIR thermal imager. This application allows thermal level tuning, span correction, change the color palette, adjust parameters such as reflective temperature, emissivity etc. The same case of thyroid abnormality with reference to another area namely the neck portion has been presented which is captured and generated results by FLIR are also presented. FLIR cameras features marking of ROI which is observed as a square in the reference Figures 6 and 7 representing normal and abnormal thyroid images. The camera can generate temperature distribution mapping in the specified area as a vertical bar. FLIR imager has a dedicated panel where indications of some statistical parameters like minimum, maximum and average observed temperature is the region selected are indicated. Users can also get information like camera model, infrared resolution, details about the file loaded etc. The model used for these images was FLIR E30. To substantiate the results obtained for thyroid detection, further images for various other diseases were also tested. The Table 1 below shows various disease conditions and it also reveals that thermal images are capable of capturing useful data which can help physicians is diagnosis and

Temperature sensing with RGB Image
With reference to the discussion of FLIR based images and the possibility of detecting abnormalities it can be noted that it requires additional equipment to get the thermal images. In most cases it is easy to get RGB images and this section shall focus on the useful inferences that can be obtained from. RGB image given that the pixel values do not give the temperature. Given an RGB image extracting temperature related information cannot be done directly. This research attempts to form a method a methodology which can be applied to any pseudo-colored image or RGB image for getting thermal images. As an example, Figure  9a shows an RGB image of a patient affected with Thyroid Eye Disease in comparison with another Figure 9b with a normal person.
The condition of Graves' Eye Disease as discussed previously radiates inflammation around the eye region causing irritable uneasiness for human beings [18] [19], which affects the patients' eye with auto immune thyroid illness [20]. The given image is segmented for the affected eye area which is the region of interest in identifying TED disease as indicated in Figure 10. The MATLAB software offers an inbuilt 'colourbar type' function which can be used to generate the color-bar for the ROI extracted. The colormap applied for this image is autumn. The values associated with the selected colors indicate the intensity of the color in the image. The obtained color mapping cannot be directly considered as representing the temperature since it's only a pixel intensity mapped with shades of red.  The mapping from RGB image to temperature values can be done by imaging an RGB image with varying temperatures and the same object observed under a FLIR camera. Then the values of temperature obtained from the FLIR camera is mapped to the corresponding color maps observed from the RGB images. This condition for RGB images can be solved by mapping the intensity values with a corresponding temperature map. Figure 11 shows the image after mapping by a rule-based method. After the mapping the range of temperature that was observed for normal eye ranges from 27 ℃ to 35 ℃ . While that for abnormal eye ranges from the research carried out shows that the temperature range for normal eye is the range [28 ℃ -34 ℃ ]. The overall average temperature of the cornea area of the eye ranges are 34.5 ± 0.8℃ Now for the mapped image we can project that red color represents more temperature and as the color moves towards yellow it represents regions with comparatively less temperature. These discussions about the differences in temperature at various regions can also be easily visualized from the figure. For the given image red color represents the hotter region, as the temperature decreases the color moves towards yellow indicating cooler regions. The difference in temperature is clearly visible by comparing both images. For the currently taken sample image the inner regions of the abnormal eye are mapped to be between 32℃ -36℃. Therefore using the rule-based methods it is possible to classify normal and TED image with thermal and RGB images.

Conclusion
This research can be concluded with the note that thermal images can be effectively used for disease diagnosis. The cases considered for this conclusion are drawn from the fact that FLIR cameras can reveal the temperature variations for various conditions indicated in Table 1. The next useful observation made is that RGB images which are easily captured by normal cameras can also be effectively used for temperature difference identification after proper colour mapping. Few cases that were tested are hyperthyroid, hypothyroid, diabetic, influenza (H1N1) and Thyroid eye syndrome. It is also interesting to note there are markable differences to show the diseases properly, this method can be extended to several applications like neurology, vascular, arthritis, dermatological disorders, neonatal, ophthalmology etc., for abnormalities, testing of temperature in electronic gadgets and printed circuit boards and several other several other industrial applications. With technology up gradation powerful algorithms can be developed to improve resolutions that are suitable for medical applications. Predictions algorithms and disease classification can be built with the sensing cameras to ease out diagnostic procedures. The introduction about mapping RGB images to its equivalent temperature values shows interesting leads in devising new algorithms that can be developed such that RGB images can prove effective in generating thermal equivalent images and eliminate the need for separate IR cameras.