Applications of visible spectral imaging technology for pigment identification of colored relics

Identifying pigments from colored relics is essential for their color restoration and for facsimile creation. A work‑ flow for identifying pigment information is constructed based on visible spectral imaging technology, aligned with the drawing process of colored relics. This workflow includes three steps: boundary extraction, material identifi‑ cation and prediction of mixture proportions. The methods for segmenting visible spectral images, identifying chemi‑ cal compositions, and predicting mixture proportions of pigments are extensively reviewed. Future research trends of these methods are also analyzed. The influence of the pigment particle size is currently underexplored but can be accomplished by multidisciplinary research.


Introduction
Colored relics offer invaluable insights into past cultures, societies, and artistic practices.They act as tangible links to historical events, religious beliefs, and daily life, showcasing the technological advancements, materials, and techniques of ancient civilizations.These relics also enable historians and archaeologists trace the evolution of artistic styles and cross-cultural influences across history.Unfortunately, many of these historically significant relics have suffered from severe decolorization, fading, and discoloration due to factors such as human activities, environmental changes, and natural disasters.To preserve their integrity and enhance their aesthetic appeal, it is crucial to provide accurate and scientific pigment information for color restoration and for facsimile creation.This information is expected to encompass details such as the application boundary, chemical composition, physical properties, and mixture proportions of the pigments, reflecting the original painting techniques used.
Advancements in digital technologies have provided effective methods for pigment identification, aiding in the color restoration and facsimile creation of colored relics.Initially, researchers employed techniques such as scanning electron microscopy, energy spectrum analysis [1][2][3], X-ray diffraction [4][5][6] and Fourier transform infrared spectroscopy [7][8][9] to identify the chemical composition of pigments in colored relics.However, these methods are invasive or minimally invasive, as they require collecting small samples from the colored relics.Recently, many non-contact analysis techniques have been developed.Their devices such as handheld Raman spectrometers [10,11], spectroradiometers [12,13] and optical fiber reflectance spectrometers [14][15][16] are now available, offering portability and ease of use, making them popular among scholars and conservationists.Despite the advantages of these techniques, they only provide point-by-point analysis, identifying pigment information at specific locations without offering spatial information that reflects spatial distribution of the pigments.As a result, they fall short of fully meeting the needs for guiding color restoration and facsimile creation of colored relics.
To capture the spatial information of colored relics, scanners or digital cameras with high spatial resolution have been widely used.These devices rely on trichromatic theory, meaning the color data (i.e., RGB values) they capture reflects the specific lighting and observation conditions at the time of capture.Let S( ) denote the spectral power distribution of the illumination, ρ( ) the visible spectral reflectance of the pigment, and r( ) , g( ) and b( ) the three-channel response characteristics (observation condition) of the devices.The RGB values can then be expressed as follows: It is clear that the color data represents the photometric integration of light reflected from pigments back to the imaging device after being illuminated by a light source.This color data depends on the device and illumination conditions, and any changes in illumination or the acquisition device can alter the recorded color information [17].Because the color data reflects various factors, including illumination condition, the device used, and the pigments of the relics, there is no direct relationship between the color data and the actual pigment materials.This can lead to the phenomenon of metamerism [18], which can cause inaccuracies when using this color data for color restoration and facsimile creation of the relics.
Visible spectral reflectance is a crucial physical property of materials that characterizes their color regardless of illumination and observation conditions.In recent years, visible spectral imaging technology has been extensively researched for recording both the visible spectral reflectance and spatial information of colored relics [19].Compared to the previously mentioned techniques, the imaging system used in visible spectral imaging is simple, cost-effective, and noninvasive, making it ideal for analyzing colored relics.Furthermore, this technology provides high-resolution spatial information and accurate color data for colored relics.The high spatial resolution can be used to identify the boundaries of pigments applied to colored relics, while the color information can help deduce the chemical composition, physical properties, and mixture proportions of the pigments.Since visible spectral reflectance acts as a "fingerprint" for an object and is independent of device and lighting conditions, visible spectral imaging technology can offer systematic guidance for pigment identification in the color restoration and reproduction of colored relics.This capability relies on fully utilizing the features of visible (1) spectral images and accurately establishing the connections between these features and pigment information.

Visible spectral imaging and framework for pigment identification
A variety of visible spectral imaging techniques and systems have been documented in the literature, which can be categorized into four main types [19]: wavelengthscan methods, spatial-scan methods, time-scan methods, and simultaneous acquisition methods.Wavelengthscan methods capture images one wavelength at a time.Spatial-scan methods acquire the entire visible spectrum for a specific region of the image at a time, scanning the image sequentially, such as single-pixel detection [20] or line by line [21].These methods are advantageous for capturing the visible spectrum of linearly moving objects but are relatively slow in acquisition speed.Time-scan methods collect a series of images, each representing a combination of spectral or spatial information.The final data is then processed to generate the complete spectral image, such as using Fourier methods [22,23] or Hadamard transform [24].Time-scan methods allows for the collection of intensity at each wavelength throughout the entire measurement period.However, even if only a few points along the spectral range are needed, the entire spectrum must still be gathered.Simultaneous acquisition methods such as color filter array [25,26] capture the entire spectral image at once but may trade off in terms of spectral resolution, field of view, or spatial resolution.
Wavelength-scan methods are frequently used in the digital preservation of colored relics due to their advantages in acquisition speed, field of view, and spatial resolution.A conventional method for capturing visible spectral images involves placing a set of optical filters in a filter wheel in front of a monochrome camera [27].Each optical filter transmits a narrow band of wavelengths, and images are sequentially captured through each filter.A more efficient method is to use tunable filters, such as liquid-crystal tunable filter (LCTF) [28,29] or acoustooptical tunable filter (AOTF) [30,31], in front of a monochrome camera.An LCTF can transmit a narrow-band wavelength by applying a varying voltage to a polarizable liquid crystal situated between two linear polarizers.An AOTF deforms into a grating with a specific period at each frequency of the acoustic waves, allowing it to transmit different wavelengths in a given direction.Tunable filters are compact and robust due to their lack of moving parts, but they tend to be expensive.Using RGB cameras and optical filters with a wide band of wavelengths significantly enhances acquisition speed [32][33][34].With the aid of spectral reconstruction algorithms [35], using two filters with two shots to obtain six channels can achieve relatively high-precision visible spectral images.Another promising method is based on LED illumination.Initially, a set of LEDs are selected, each LED is illuminated in sequence, and a monochrome camera captures an image under each LED, thus producing a visible spectral image by spectral reconstruction [36,37].To increase the acquisition speed, an RGB camera and three LEDs are optimally combined to acquire nine channels with three shots [38,39].The visible spectral image is then obtained through reconstruction.For a more comprehensive assessment of the performance and quality of the various imaging systems, including both qualitative and quantitative evaluations, please refer to references [40][41][42][43].
A detailed analysis of colored relics is crucial for fully unlocking the potential of visible spectral imaging in pigment identification.The drawing process of Chinese colored relics typically involves three stages: drafting, outlining, and coloring, as illustrated in Fig. 1.Drafting involves determining the position and proportion of patterns using long horizontal lines; outlining defines the boundaries for pigment application; and coloring involves filling in pigments within these boundaries to assign colors.To effectively guide color restoration and facsimile creation of colored relics, research on visible spectral images should focus on several key aspects: segmenting the images based on the spatial distribution of pigments to define boundaries for pigment application; developing models to correlate visible spectral reflectance with pigment material information for accurate identification; and establishing models to predict pigment mixture proportions from visible spectral reflectance for precise color matching.
A framework for pigment identification using visible spectral imaging technology is presented, as depicted in the upper part of Fig. 2. The framework begins by examining the spatial correlation between pigment distribution and image pixels.It details the pixel features used to assess pigment similarity and establishes an image segmentation method to extract pigment boundaries.To provide material information of the pigments, a standard visible spectral reflectance database with known chemical composition and physical properties is created for reference.The framework identifies the chemical composition of the pigments by preprocessing visible spectral reflectance to eliminate noise in the original data, constructing a spectral feature space to characterize the chemical composition information, and defining matching criteria to differentiate between chemical compositions.Based on the identified chemical composition, physical properties such as pigment particle size are determined by extracting spectral features, modeling the relationship between physical properties and spectral features, and fitting data to these models.The framework also explores the light propagation mechanism in the pigment layer and the optical properties of mixed multicomponent pigments compared to their monocomponent counterparts to understand color rendering in colored relics.To achieve this, a forward spectral prediction model is first established to map the optical properties and proportions of monocomponent pigments to the visible spectral reflectance of mixed pigments.Then, a corresponding reverse proportion prediction model is created to map the relationship back.The optimal pigment mixture proportion is obtained as the optimal solution to the reverse proportion prediction model using optimization methods.Additionally, the framework is designed for continuous improvement through effective verification and evaluation.

Segmentations of visible spectral image for boundary extraction
Visible spectral images capture high-spatial resolution, two-dimensional details of colored relics and preserve the boundaries of pigments.These boundaries can be extracted through segmentation of the visible spectral images.Current segmentation methods for visible spectral images can be categorized into three types: methods based on single-pixel [44][45][46][47], methods based on spectral feature similarity between pixels [48][49][50], and methods based on superpixels.

Methods based on single-pixel
Methods based on single-pixel consider each pixel as a fundamental segmentation unit to map the spatial distribution of pigments in colored relics.These methods are convenient for subsequent pigment identification and therefore are widely used in early studies.For example, researchers have analyzed the visible spectral images of paintings such as Van Gogh's "The Starry Night," [51] Seurat's "A Sunday Afternoon on La Grande Jatte," [52] and Munch's "The Scream."[51] When the pigment particles are small enough, the original pattern of the colored relics can be treated as a continuous-tone image.This approach preserves as much detail as possible and allows for accurate pigment identification in regions where colors transition.However, if the particle size of the pigments is larger than the size of each pixel in the visible spectral image, the identified pigment information in the pixels where different pigments overlap may become distorted.Furthermore, these methods face challenges such as large data processing demands and a lack of boundary information for the pigments.

Methods based on spectral feature similarity between pixels
These methods focus on the similarity of spectral features between pixels in visible spectral images.They group pixels with similar spectral properties together, regardless of their spatial location, to represent the spatial distribution of pigments.Many traditional image segmentation techniques have been used to organize pixels for boundary  [53] and the frescoes of Gautama Temple [54].While these methods effectively extract pigment boundaries and reduce data processing needs, they often struggle with accurately delineating pigment contours in connected areas and providing precise color details.As illustrated in Fig. 3, this limitation can result in errors when identifying and interpreting pigments in colored relics.

Superpixel methods
In colored relics, the same pigment often forms spatially connected blocks that span multiple neighboring pixels with similar spectral features.Previous segmentation methods have not accounted for this spatial organization, leading to inaccuracies in extracting pigment boundaries.To address this, superpixel methods that integrate both spectral and spatial similarities between pixels have been introduced to better capture the painting characteristics of colored relics [55][56][57].As demonstrated in Fig. 4, these segmented superpixels are irregular clusters of neighboring pixels with similar spectral features, offering a more precise depiction of pigment boundaries.
Given the emerging trend in boundary extraction from visible spectral images, it is crucial to further explore the spatial relationship between image pixels and pigment distribution in colored relics.To accurately represent both the color details and the boundaries of pigment application, new segmentation methods should be developed.These methods should organize the pixels of visible spectral images in a way that effectively characterizes the pigment distribution in colored relics while preserving as much color detail as possible.

Identification of pigment material information
The chemical composition and physical properties of pigments determine their visible spectral reflectance.Conversely, the visible spectral reflectance provides insights into the chemical composition and physical properties of the pigments, allowing for the inference of material information.Identifying the chemical composition of pigments is a frequent subject in the literature.Currently, methods for analyzing the chemical composition of pigments can be categorized into two types: manual identification and automatic identification.

Manual identification
Manual Identification is an early method for identifying the chemical composition of pigments.It primarily relied on the human eye to visually compare features such as the shape of visible spectral reflectance and the similarity of derivative characteristics between the standard sample and the test sample to determine the pigment composition of the test sample.By comparing the shape of visible spectral reflectance, Cavaleri et al. [58] identified the pigments of the fresco in an ex-church in Piedmont.Similarly, Vitorino et al. [59] identified the red lake pigments in fourteenth to sixteenth centuries paintings.Wang et al. [60] identified the pigments from Tang Dynasty tombs by analyzing the shape of visible spectral reflectance combined with its first derivative characteristic peaks.However, relying on visual comparison is inefficient and highly dependent on experience, making it unsuitable for pixel-level batch processing of visible spectral images of colored relics.

Automatic identification
Each pigment has a distinct interaction with electromagnetic radiation, absorbing and reflecting certain wavelengths of light.This interaction results in a unique feature in its visible spectral reflectance, which can be used to identify the pigment.Automatic identification involves using various algorithms to analyze visible spectral reflectance to automatically identify the chemical composition of pigments.This process typically consists of at least three key steps: preprocessing, feature extraction, and identification modeling.Preprocessing prepares the raw visible spectral reflectance data for analysis, which can involve tasks such as data cleaning, normalization and scaling to remove noise or correct for environmental factors.Feature extraction focuses on extracting the relevant features from the preprocessed data.The selection of appropriate features is crucial, as it can significantly influence the performance of the identification model.Commonly used features include the visible spectral reflectance data, the first and second derivatives of the data, spectral angle, spectral correlation coefficients and principal component analysis [61][62][63].In identification modeling step, a model is built to identify pigments based on the extracted features.This could involve using various machine learning algorithms such as decision trees, support vector machines or neural networks [48,[64][65][66].

Reference database
Pigment material information is identified through comparison or training, which require the preparation of standard samples to create a reference database of visible spectral reflectance.The accuracy of this identification is significantly influenced by the variety of pigment types used, as well as the methods employed for preparing and measuring the standard samples.To ensure precise identification, standard samples should be prepared following the drawing process of the specific identification target, and their visible spectral reflectance should be measured under the same geometric conditions as those used for acquiring visible spectral images of colored relics.Additionally, the standard database should include a wide range of pigment types to cover those used in the identification target.Cosentino [67] has established a comprehensive standard reference database for identifying pigments in oil paintings.In China, Chai [68] and Ding [69] have established reference databases that include 29 and 31 color blocks, respectively.
In addition to chemical composition, physical properties such as particle size are another important aspect of pigment material information [70][71][72].As shown in Fig. 5, the particle size of mineral pigments significantly influences their color, with saturation and lightness being strongly determined by particle size.Some qualitative and quantitative analyses of the relationship between pigment particle size and color have been discussed in reference [73][74][75].Painters have long mastered the techniques of grinding and rinsing mineral pigments to effectively control particle size and enhance color richness.This phenomenon is commonly observed in colored relics [55].Figure 6 illustrates the identification results of wall paintings in the Mogao Grottoes, where both color blocks 1 and 2 are lapis lazuli, but their colors differ due to variations in particle size.These findings highlight the significance of particle size as a key technical parameter for identifying pigment material information, crucial for guiding the scientific color restoration and facsimile creation of colored relics.Despite its significance, this topic has not been extensively studied, and few reports focus on identifying the pigment particle size of colored relics based on visible spectral reflectance.In our previous research, we introduced three methods for this purpose: the image texture-based identification method [76], the graph-based identification method, and the parametric identification method [77].Zou et al. proposed a model that uses principal component analysis and nonlinear curve fitting to identify the particle size of various malachite samples used in ancient Chinese colored relics [78].It is important to point out that the relationship between visible spectral reflectance, chemical composition, and particle size of pigments remains unclear.Current methods for identifying chemical composition do not fully account for the interaction between chemical composition and particle size on visible spectral reflectance.Therefore, the effectiveness of these methods needs further investigation.Moreover, the reference database must also consider the particle size and the aging properties of the pigments.

Prediction of pigment mixture proportions
A mixture of monocomponent pigments with different colors can create new colors.This phenomenon is commonly seen in colored relics to achieve a variety of colors.When restoring colored relics, the color of the restored area should match the surrounding area, regardless of lighting and viewing conditions.This means the restored area must have the same visible spectral reflectance as the surrounding area.The visible spectral reflectance of a multicomponent pigment depends on the proportions of its monocomponent pigments, so determining these proportions based on the visible spectral reflectance of the multicomponent pigment is necessary for color restoration of colored relics.This can be accomplished by constructing color prediction models, which include a forward spectral prediction model and a reverse proportion prediction model.

Forward spectral prediction model
The reverse proportion prediction model is derived from the forward spectral prediction model, making the accuracy of the forward model crucial for accurate proportion prediction.The forward spectral prediction model involves a mapping relationship F of the visible spectral reflectance from the proportion of monocomponent pigments to the mixed multicomponent pigment, as follows: (2) r mix, = F c 1 , c 2 , . . ., c n , r 1, , r 2, , . . ., r n, where n represents the number of monocomponent pig- ments mixed, is the wavelength of the visible spectral reflectance, r i, is the visible spectral reflectance of each monocomponent pigment, c i is the corresponding pro- portion, and r mix, is the visible spectral reflectance of the multicomponent pigment.
Although many linear unmixing techniques from remote sensing [79][80][81] have been used to model pigment mapping relationships [82][83][84], these methods differ fundamentally from the pigment mapping process.In remote sensing, each pixel samples a larger spatial area, and the radiance measured is influenced by various materials within that area, making linear unmixing effective for modeling its mapping relationships.In contrast, pigment mixtures often consist of small particles evenly distributed in a binding medium, known as intimate mixtures.Consequently, the overall visible spectral reflectance is not simply a linear combination of the visible spectral reflectance of the monocomponent pigments [42].To accurately develop the forward spectral prediction model and model the mapping relationships, it is crucial to understand the mechanism of light propagation within the pigment layer and the optical relationship between the multicomponent pigment and its monocomponent constituents.
The Kubelka-Munk (K-M) theory is commonly employed to describe the light propagation mechanism in the pigment layer [85][86][87].According to this theory, light is considered to diffuse through a homogeneous and isotropic medium, where it is absorbed and scattered in both upward and downward directions.The absorbing and scattering property of a pigment, denoted as k s , are linked to the visible spectral reflectance r of the pigment layer.Specifically, k s is defined as follows: Duncan [88] was the first to report on the relationship between the optical properties of multicomponent pigments and their monocomponent pigments.The absorbing and scattering properties k s mix, of a mul- ticomponent pigment can be viewed as a linear combination of the absorbing and scattering properties k s i, of its monocomponent pigments, with the corresponding proportions serving as coefficients.This relationship is defined as follows: K-M theory combined with Duncan theory forms the forward spectral prediction model.This model assumes that the interior of the pigment layer is homogeneous and isotropic.However, in colored relics, the particle size of mineral pigments varies from a few microns to several hundred microns, and the particles are irregularly shaped, leading to the formation of voids in the pigment layer.The current model does not account for these effects, and a previous study [89] has shown that particle size significantly impacts the model's prediction accuracy.It is crucial to determine how this effect manifests in colored relics, re-examine the light propagation mechanism in mineral pigment layers, and develop new models to accurately map the relationship between visible spectral reflectance, mixing proportions, and particle size of the pigments.

Reverse proportion prediction model
The inverse proportion prediction model aims to map the visible spectral reflectance of a multicomponent pigment to the proportion of each monocomponent pigment.Ideally, this model serves as the inverse function F −1 of the forward spectral prediction model F .However, the for- ward spectral prediction model is often complex, making it difficult to derive an analytical equation for the inverse function directly.Consequently, the inverse proportion prediction process is typically treated as a constrained optimization problem, as described below: where r targeted represents the visible spectral reflectance of the colored relics to be restored.(c 1 ′, c 2 ′, .., c n ′) is the predicted optimal proportion of the monocomponent pigments, such that the visible spectral reflectance of the multicomponent pigment matches the visible spectral reflectance of the colored relics as closely as possible.
Considerable research has been conducted on predicting pigment proportions based on visible spectral images by inverting the K-M theory [86,90,91].These studies primarily focus on identifying pigments by referencing an existing database of monocomponent pigments.For example, Zhao et al. [92] conducted an in-depth analysis of Vincent van Gogh's "The Starry Night", where they utilized a preexisting database of pure pigments to determine the closest match.With limited or no prior knowledge of the pigments present, Taufique et al. [93] proposed a new method for analyzing the Selden Map.They estimated abundances and performed classification in the abundance space, which can significantly reduce the workload of the model.In earlier studies, color tristimulus values were frequently used as the matching target for reverse proportion prediction models, with linear iterative optimization employed to predict the proportions of monocomponent pigments [94].As shown in Eq. ( 5), visible spectral reflectance is used as the matching target for predicting the proportions of monocomponent pigments.The traditional linear optimization strategy cannot provide a solution in this case.Thus, exploring optimal nonlinear optimization methods is necessary for accurate pigment proportion predictions.Additionally, unlike models based on optical principles, some methods predict proportions directly using nonlinear spectral unmixing techniques [95][96][97].However, the effectiveness of these methods in predicting the proportions of pigment mixtures in colored relics requires further validation.

Conclusion
Visible spectral imaging technology offers a non-destructive way to capture both the spatial and spectral information of colored relics.This technology reveals the chemical composition and physical properties of the pigments used, as well as the true color characteristics of the relics.By understanding and modeling the relationships between image characteristics and the spatial distribution, chemical composition, particle size, and mixture proportion of pigments, challenges in pigment selection for color restoration and facsimile creation can be effectively addressed.Currently, the use of visible spectral images for pigment identification is gaining traction across various fields such as analytical chemistry, color science, and cultural heritage conservation.While researchers have developed several models with promising results, a comprehensive theoretical and technical framework has yet to be fully established, largely because visible spectral imaging technology has only recently advanced.Some significant challenges remain, particularly regarding how pigment particle size affects boundary extraction, material identification, and mixture proportion prediction.This research area is inherently multidisciplinary, involving applied optics, imaging science, image processing, analytical chemistry, and color science.Although interdisciplinary research has made some progress, only through deeper collaboration across multiple disciplines can these issues be thoroughly resolved.Given the low cost and simplicity of visible spectral imaging devices, this technology holds the potential to systematically identify pigments, facilitating pigment selection for color restoration and facsimile creation.Additionally, these methods can be extended to industries such as textiles, printing, dyeing, and coating, as well as agriculture for applications such as seed screening and plant disease detection.When combined with spectroscopic techniques like Raman and Fourier transform infrared spectroscopy, visible spectral imaging can provide more comprehensive and accurate analyses.

Fig. 1
Fig. 1 Painting process of colored relics

Fig. 2
Fig. 2 Flowchart of the pigment identification of the colored relics based on the visible spectral images

Fig. 3 Fig. 4
Fig. 3 Segmentation based on spectral feature similarity between pixels result in lost details.The left half is the original image; the right half is the segmented image

Fig. 5 Fig. 6
Fig. 5 Changes in malachite color resulting from variations in pigment particle size, with sizes decreasing from left to right