Learning Thermographic Models for Optimal Image Processing of Decorated Surfaces ^†

Abstract

The use of infrared thermography presents unique perspectives in imaging of artifacts to help interrogate their surface and subsurface characteristics, highlight deviations and detect contrast. This research capitalizes on active and passive thermal imagery along with advanced machine learning-based algorithms for pre- and post-processing of acquired scans. Such codes operate efficiently (compress data) to help link the observed temperature variations and the thermophysical parameters of targeted samples. One such processing modality is dictionary learning, which infers a “frame dictionary” to help represent the scans as linear combinations of a small set of features, thus training data to show a sparse representation. This technique (along factorization and component analysis-based methods) was used in current research on ancient polychrome marquetries aimed at detecting aging anomalies. The presented research is unique in terms of the targeted samples and the applied approaches and should provide specific guidance to similar domains.

1. Introduction

The structures with decorated surfaces due to their very nature and historical context (material, assembly, and aging), present a number of challenges specific to their diagnosis and restoration, which limit the adoption of standard methods and known techniques borrowed from other applications. As the strict fulfilment of established guidelines is necessary to ensure both rational analyses and repair approaches, which are appropriate to the cultural context. Consequently, the formation of the multidisciplinary team of experts, is not trivial, as it should be guided by the domains, the scale, and the sciences involved in each assessment and restoration step. This typically starts by the visual inspection/evaluation of the work of art under analysis [1]. Thermographic analyses offer imaging perspectives across an electromagnetic spectrum (i.e., the infrared band) that is beyond the visible, thus furnishing new information and exposing new material structures (delaminating, splits, inclusions, etc.), otherwise undetectable by the naked eye [2]. Furthermore, thermographic images are coupled nowadays to efficient pre and post processing routines, to help increase the contrast and defects’ detectability in real-time and at larger frames (with higher resolutions). For the current application on the samples under test (SUT), these techniques will be adjusted to highlight the variations of the SUT constituting materials’ via their thermophysical properties and structural integrity. Thus, providing a visual representation of the SUT structure and its unique aging, in passive [3] and active [4] modes of thermography. Several techniques based on physics (of heat conduction), mathematics, and/or statistics have recently been proposed within the thermographic research field, such as principal components thermography [5], thermographic signal reconstruction [6], dynamic thermal tomography [7], higher order statistics thermography [8], among many other [9,10,11]. On the other hand, recent advances in machine learning can help process, discern and interpret images thoroughly and in real-time. When applied to image processing, artificial intelligence (AI) may, e.g., detect and recognize objects and patterns in images. Most effective machine learning models for image processing use neural networks and deep learning. Recently, machine learning techniques found greater adoption in the infrared thermography processing context [12]. The contribution from current work can be considered novel as a data mining search on Scopus^® website [13], via the combined keywords of “machine learning” + “thermography” + “cultural heritage”, resulted in “No documents were found”. Thus, the current research can present a pioneering approach to obtaining clear defect maps of ancient marquetries using thermography, specifically via a controlled heating regime [14]. Furthermore, the unique application of dictionary learning (DL) to raw thermal images is compared against other advanced algorithms described in the next section [15]. For the sake of brevity, only the main DL results will be shown and discussed in Section 3; the remaining ones will be illustrated during the virtual presentation.

2. Algorithms for Thermal Image Processing

A test campaign, in reflection mode, used a halogen lamp rated at 230 V and 2.5 kW synchronized to an Indium Antimonide InSb thermal camera (commercial name GF309 from FLIR^®) to produce 250 thermal images in total, at a frame rate of 25 Hz. The following algorithms were then applied to the acquired scans.

2.1. Dictionary Learning (DL)

It is a recent modality of signal processing and machine learning that infers a sparse representation of the input thermal images using a DL approach. The sparser the representation, the better the dictionary [15]. A common setup for the dictionary learning problem starts with access to a training set, a collection of training vectors, each of length N. This training set may be finite, and then the training vectors are usually collected as columns in a matrix X of size NxL, or of infinite size. For the finite case, the aim of DL is to find both a dictionary, D, of size NxK, and a corresponding coefficient matrix W of size KxL such that the representation error, R = X − DW, is minimized and W fulfill some sparseness criterion. The DL problem can be formulated as an optimization problem with respect to W and D.

2.2. Non-Negative Matrix Factorization (NMF)

By combining attributes, NMF can produce, e.g., meaningful patterns. Each feature created by NMF is a linear combination of the original attribute set. Thus, each feature will have a set of coefficients that measure the weightage of each attribute on the feature. Such that, there is a separate coefficient for each numerical attribute and for each distinct value of each categorical attribute. The coefficients are all non-negative, hence the name [16].

2.3. Fast Independent Component Analysis (ICA)

The ICA approach uses statistical independence of the source input to solve the blind signal separation problems. The fast ICA algorithm improves efficiency of ICA. Here, the independent components can be estimated one by one, which is roughly equivalent to conducting projection pursuit [17].

2.4. Mini Batch Sparse Principal Component Analysis (PCA)

Sparse PCA is a variant of PCA. Consequently, Mini Batch Sparse PCA is a variant of Sparse PCA that is faster yet less accurate. The increased speed is reached by iterating over small chunks of the set of features, for a defined number of iterations. Therefore, Mini Batch Sparse PCA performs the dimensionality reduction process using a part of the sample features and at a given number of iterations, to help solve the problem of slow feature decomposition in large samples. It may require tuning for the L1 regularization parameters [18].

2.5. Factor Analysis (FA)

The FA model allows information reduction from a larger set of variables into a smaller one. Such variables are termed “latent variables”. FA is based on the common factor model. FA measures the impact of one specific factor on the measured variables. In short, the mathematical difference between PCA and FA is the use of specific factors for each original variable [19].

3. Description of the Marquetry Samples and Main DL Results Obtained

The thickness of the marquetries (19th century) called, for simplicity, Boy and Girl was ~0.5 mm. Multicolored tesserae were applied by means of an adhesive (presumably, a protein glue) on a solid, planar and visible support realized in fruit wood (Figure 1). In the Boy sample (Figure 1a), large detachments can be revealed along with a deformation inherent to tesserae of the ground and of the arm. As for the Girl sample, a lack of tesserae that make up the marquetry is evident on the feet, ground, and left arm (Figure 1b). More information regarding the marquetry samples can be found in [14].

In both cases, DL was able to detect several splitting between the support layer and tesserae, as well as highlighting the areas in which tesserae are missing (Figure 2a,b). In particular, by comparing the marquetries, the Girl sample has more critical splits across the border of the oval frame. It should also be noted that the detachment at the bottom, depicted in the Boy sample, appears larger relative to the rest of the image. The restorer ruled the results an added value and sufficiently accurate for moving to the next step, i.e., the restoration.

4. Conclusions

Several advanced algorithms were applied to raw thermal images acquired on two ancient marquetries with the aim to detect different types of defects. For the sake of brevity, here the authors reported and discussed only the DL results; the remaining presented techniques and algorithms (from Section 2) will be shown during the virtual presentation. The results obtained will help the restoration efforts in repairing similar cultural artifacts. Supervised methods will be used in the future for additional comparative analyses.