Increasing the reuse of wood in bulky waste using artificial intelligence and imaging in the VIS, IR, and terahertz ranges

Bulky waste contains valuable raw materials, especially wood, which accounts for around 50% of the volume. Sorting is very time-consuming in view of the volume and variety of bulky waste and is often still done manually. Therefore, only about half of the available wood is used as a material, while the rest is burned with unsorted waste. In order to improve the material recycling of wood from bulky waste, the project ASKIVIT aims to develop a solution for the automated sorting of bulky waste. For that, a multi-sensor approach is proposed including: (i) Conventional imaging in the visible spectral range; (ii) Near-infrared hyperspectral imaging; (iii) Active heat flow thermography; (iv) Terahertz imaging. This paper presents a demonstrator used to obtain images with the aforementioned sensors. Differences between the imaging systems are discussed and promising results on common problems like painted materials or black plastic are presented. Besides that, pre-examinations show the importance of near-infrared hyperspectral imaging for the characterization of bulky waste.


Introduction
The increased use of wood is a key to achieve national and international goals in the fight against climate change and minimize the CO 2 footprint [1].In this situation, the use of waste wood as a substitute for fresh wood is an interesting way to reduce the scarcity of wood.Waste wood for use as a material has meanwhile become a scarce commodity itself in Germany [2].This is also because, according to national legislation, it can only be reused as raw material if it is free of wood preservatives and other contaminants such as PVC.The development of new sources for "clean" waste wood is therefore gaining importance.Although half of the bulky waste consists of wood, only about half of it has been used as a recycling material so far [3].Reasons for that are the difficult separation of impurities from wood and a huge variety of materials.
Established methods for sorting bulky waste are manual picking and automatic waste sorting based on heavily shredded materials, with the cost of shredding worsening the ecological balance.A concept similar to the system proposed here was presented in [4], but for the sorting of building rubble that is not as homogeneous as bulky waste.
Thus, the project ASKIVIT (Altholzgewinnung aus Sperrm üll durch k ünstliche Intelligenz und Bildverarbeitung im VIS-, IR-und Terahertz-Bereich) aims at developing a solution for the automated sorting of bulky waste.The goal is to extract wood, wood-based materials, and non-ferrous metals based on a multi-sensor approach combined with artificial intelligence.Conventional RGB, near-infrared hyperspectral, and thermographic cameras, as well as a developed terahertz imaging system, are used in this work.In the first step, the different sensors are described and the fusion approach based on a convolutional neural network (CNN) is motivated.Preliminary investigations are carried out to determine the potential of near-infrared hyperspectral material characterization using machine learning.Moreover, the benefit of a multi-sensor approach is discussed and verified with sample images.

Material and methods
In this section, the different imaging systems are described and the fusion approach based on a CNN is motivated.

Visible imaging
Humans can characterize material from bulky waste very accurately only by its appearance in the visible spectral range.Therefore, images from conventional RGB cameras, that imitate the human eye, include highly relevant information.Furthermore, RGB cameras are available in high resolution and often by one order of magnitude more costeffective compared to other sensors used for material characterization [5].
In the course of this study, a prism-based RGB line scan camera (SW-4000T-10GE) was chosen.The built-in prism of the camera splits incoming light onto three spatial separated chips, each measuring one color channel.The frame rate was set to 625 Hz.Halogen lamps were used as a light source for visible as well as near-infrared radiation.The later was utilized for the near-infrared imaging system.
By moving the samples on a conveyor belt, images with two spatial axes were constructed using the push-broom method.The complete setup including all imaging systems presented in this paper can be seen in Figure 1.

Near-infrared hyperspectral imaging
Near infrared (NIR) hyperspectral imaging is another sensor principle that is used in this work to characterize bulky waste.It is particularly suitable for the detection of organic products and thus also for the identification of wood.Whereas color cameras can only view the superficial appearance, spectral information provided by NIR hyperspectral cameras shows the physical-chemical composition of the material.
As a measuring device, the camera FX17e from SPECIM is chosen.The camera collects hyperspectral images with 224 bands ranging from 900 nm to 1700 nm.The frame rate was chosen to be 104.17Hz, such that the resolution was equal in both spatial axes of the image.

Active heat flow thermography
Like the recording of RGB and NIR hyperspectral images, thermography is a camera-based sensor technology.In contrast to the first two methods mentioned, the samples in thermography do not have to be illuminated during the measurement, but are heated in advance.A detector that is sensitive in the thermal infrared range (wavelength: approx.3 µm to 14 mm) records the thermal radiation that the samples emit on the basis of Planck's law.The radiation intensity depends on the temperature of the samples and their emissivity.In order to be able to make statements about material parameters beyond the emissivity, the samples are heated with infrared radiators as they were transported by the conveyor belt.The infrared camera is a Geminis 327k ML from IRCAM (Erlangen, Germany) having a dual-band HgCdTe detector (the 1st sensitivity band: 3.7 -5 µm; the 2nd sensitivity band: 8 -9.4 µm) with 640 x 512 pixel.Only the 2nd band was used in order to avoid parasitic signals from direct irradiation by the infrared heater into the camera.A frame rate of 100 Hz and a 25 mm lens were used.The camera was arranged in such a way that the width of the conveyor belt filled the image along the long edge.The distance between the camera and the heater amounted to 0.6 m.
The infrared heater consists of two Carbon Twin-Tube Emitters from Heraeus Noblelight having a length of 0.7 m and a power of 6000 W/m each.The peak wavelength of their radiation spectrum was 2 µm.The heaters were placed about 0.28 m above the conveyor belt.Given the velocity of the conveyor belt of 0.108 m/s, the energy per area deposited in the samples is The increase in temperature on the sample surface as a result of heating by the radiant heater depends on the underlying thermophysical parameters.Therefore, structured samples can obtain a characteristic temperature pattern that allows a look underneath the sample surface.

Terahertz imaging
Terahertz radiation is electromagnetic radiation between far infrared and millimeter waves.Due to the capability of terahertz waves to penetrate through most of the dielectric materials, such as plastics, paper, foams, or upholstery, the differences in the refractive index may be observed in 3D [6].Opposed to X-ray radiation, terahertz is non-ionizing.Therefore, it enables safe 3D imaging on complex structures, which are common for bulky waste.
For this application, a terahertz camera was developed as a line scan camera with 12 emitters and 12 receivers, which operate in the W-band (75 -110 GHz or approx.2.7 -4 mm wavelength).A synthetic aperture radar (SAR) design for the terahertz imaging system was chosen [7].The received signal (amplitude and phase) depends on the refractive index and spatial position of the sample structure.The aim of this system is to provide additional 3D information on overlapping and complex features of pre-crashed bulky waste.
144 effective aperture elements (12 emitter and 12 receiver combinations) are scanned for all frequencies Nf that are used to scan the scene within the W-band.The data acquisition algorithm obtains measured reference, receiver, and encoder signals.The data acquisition time as well as the resolution depends on the number of frequency points Nf and the covered bandwidth, respectively.Each set of complex Nf x144 data has to be reconstructed in a defined reconstruction volume in order to obtain a 3D image, which can be later observed at each reconstruction plane (referenced as a distance to the imaging array).
The used reconstruction algorithm is based on matched filter approach [8].For the given sample shown in the photograph on the left of Figure 2, the reconstruction volume of 80 x 40 x 10 cm was chosen with corresponding 800 x 400 x 50 voxels.The reconstruction was made from 134 line scans, i.e. on average, one picture was taken every 6 mm with a speed of 0.30 m/s.
The reconstructed images show good results from the reflection of the objects (middle) as well as from the shadow image (right).The metal reflects most of the radiation, whereas shaped metals show prominent shapes due to scattering from the surfaces which are not parallel to the scanner imaging plane.A piece of a CD (as well as metallic markers) shows the strongest reflection due to conductive materials and a parallel face toward the scanner.Wood and cardboard reflect part of the radiation.The chosen rubber mat has a stripped structure, which reflects a big part of terahertz radiation giving a good contrast for shadow images of the wood.Upholstery and plastics are the most transparent in the terahertz range, therefore only tiny changes in the image can be recognized.This is important for the characterization of material composites, as terahertz radiation enables the detection of wood and metal underneath upholstery or plastic.
The terahertz images in Chapter 3.2 were obtained using 0.108 m/s conveyor belt speed.
The line scans were obtained every 6 ms.The chosen reconstruction volume was 80 x 55 x 18 cm with 1600 x 550 x 80 voxels in x, y, z directions correspondingly.

Sensor data fusion approach
The characterization of materials can be solved by a broad variety of classification methods, including classical and machine learning methods [9,10].Senecal et al. showed that using a CNN optimized for multispectral data can result in very high classification accuracy if the data set is large enough [11].However, multispectral datasets are often very limited in size.Therefore, it is a key point in our project to enable fast data recording to capture a dataset sufficient in size.This is done by using the setup described in the previous sections.The benefit of CNN architectures is that they can use much of the spatial and all spectral information at the same time, and therefore make use of the spectral differences between the materials early.The relevant spatial and spectral features are learned by the network automatically and simultaneously, which is hard to reproduce by a classical feature design.
To combine the information of the proposed sensor modalities, a fusion technique together with a registration is necessary.In this way, the strength of each imaging system can be used to achieve a classification result better than using one technology individually.Lately, early fusion methods based on deep learning e.g.CNNs show very promising results on multispectral datasets like EuroSAT [12].In early fusion, data from various sensors is registered and merged before classification [13].
In our project, the registration is done by using a marker-based registration approach.For the registration of RGB, NIR, and thermographic cameras, AruCo markers [14] are introduced supported by a similar marker for the Terahertz spectrum.With this marker-based approach, the image registration is robust and accurate, even if sensors show significantly different intensities on the same object.After registration, the preprocessed data from all sensors will be given into a CNN, which is currently under development.The CNN will implicitly perform an early fusion and classify the material perceived by the sensors.

Results and discussion
After describing the setup, preliminary results of NIR hyperspectral imaging will be presented.Moreover, recordings from all imaging systems will be shown and discussed.

Preliminary results of NIR hyperspectral imaging
Hyperspectral image analysis is state of the art for material characterization used for sorting applications.Therefore, pre-examinations have been carried out based on NIR hyperspectral data combined with a common classifier, namely partial least squares discriminant analysis (PLS-DA).The samples to be analyzed are different objects appearing in bulky waste.The objects were divided into six classes, namely wood, upholstery, rubber, plastic, metal, and ceramic.Each class can include slightly different types of material.The class wood for example included particle board, old varnished window scantlings, high-density fiberboard, and plywood.
Hyperspectral images of the samples were acquired using the FX17e camera and the setup described in section 2.2.Eight images of different sample collections were chosen for training from which 10 5 pixels were randomly selected.From another eight images, 10 4 pixels were extracted for testing.A single pixel contains 224 values, each representing the reflectance of the material at a different wavelength.As a preprocessing step, standard normal variate (SNV) correction was performed [15].Additionally, outliers that differ more than five standard deviations from the mean have been removed from training data in order to improve the classification model.
In the spectral plot of Figure 3 the intensity over wavelengths for different materials is visualized.The intensity values can be negative due to SNV correction.Several spectra are drawn on top of each other for each class, making the variance of the data visible.It can be seen that the spectral data varies very little within each class and, by looking at the course of the spectra, the classes are visually distinguishable from each other.
The classification performance of the PLS-DA model is evaluated on test data with a confusion matrix (on the right of Figure 3).The overall accuracy on test data is 0.64.In the confusion matrix, it can be seen that plastic is falsely classified as upholstery in most cases.A reason for that might be that the two materials are not linearly separable.However, the material wood (including particle board, varnished wood, fiberboard, and plywood) is classified correctly with a probability of 0.79.This confirms the assumption, that NIR hyperspectral imaging gives highly relevant information for detecting waste wood in bulky waste.

Comparison of sensor modalities
After showing the potential of hyperspectral material characterization in the near-infrared range, this section will focus on the comparison of the presented imaging systems.Therefore, four sample quantities were chosen and images were recorded using the setup shown in Figure 1.
The results can be seen in Figure 4. Sample 1 contains old varnished window scantlings, and Sample 2 are pieces of red and black rubber mats.Samples 3 and 4 are wood chips partially covered with foam and metal pieces, correspondingly.RGB and NIR hyperspectral data contain multiple channels, each representing a different wavelength.The corresponding images are in color or rather false color in the case of NIR hyperspectral data (selected wavelengths are 1100 nm, 1300 nm, and 1500 nm).In the terahertz pictures, the given number defines the visualized plane from the whole reconstruction volume by the distance of the plane to the imaging array.The distance is chosen such that the features relevant to the underlying comparison are visible.The sample carrier is approximately 680 mm away from the terahertz imaging array.
The RGB image of Sample 1 shows the paint color and surface but does not reveal the wood structure.The same applies to the NIR pseudo-RGB image, but it is less affected by the paint.The thermographic and terahertz images show the wood texture with its characteristic annual ring pattern under the paint so that this sample can be clearly identified as wood with help of thermography or terahertz.The terahertz image shows the upper plane that is 599 mm below the imaging array, which leads to a sample thickness of approx.8 cm.
Sample 2 shows a common problem of sorting black polymers.It is not readily recycled in conventional plastic sorting facilities due to the high absorption of black pigments to radiation in NIR or visible wavelength range [16].The red rubber chips in Figure 4 are clearly visible in the RGB image, while the black ones are hardly recognizable on the background of the black sample carrier.This also applies to the NIR pseudo-RGB image.In thermography, however, red and black rubber both have a significantly improved sensitivity and can therefore be easily distinguished from the background.The terahertz image contains information about the height of the visible mats encoded in the reconstructed volume.The image is blurred out due to the scattering of the texture of the black mats.
Samples 3 and 4 show foam and metal on wood chips, respectively.NIR pseudo-RGB images are again less influenced by the paint color of the material in comparison to the RGB images.Foam and metal are distinguishable from wood chips in almost all images.Terahertz images show strong reflection from metals, whereas wood chips absorb most of the radiation.In thermography, metal appears darker than wood because it absorbs the radiation from the radiant heater less and has a higher heat capacity and lower emissivity than wood.In contrast to that, foam appears very bright due to its low thermal capacity.

Conclusions and outlook
A novel approach for bulky waste material characterization has been presented.Different sensor modalities including visible, NIR hyperspectral, thermography, and terahertz imaging are exploited to achieve a better classification result than using a single technology individually.Regarding terahertz imaging, a synthetic aperture radar system was developed, which is specifically designed for sorting applications.The system aims to provide additional 3D information on overlapping and complex features of pre-crashed bulky waste.
All four imaging systems were brought together to build a demonstrator acquiring data using RGB, NIR, thermography, and terahertz imaging techniques in one attempt.The recorded and post-processed images showed promising results on common problems like painted materials or black plastic.The presented thermography and terahertz images reveal the wood texture with its characteristic annual ring pattern under the paint.Besides that, thermography showed good sensitivity for plastic regardless of color.
Pre-examinations on NIR hyperspectral data have shown that waste wood is distinguishable from plastic and upholstery.Furthermore, using a PLS-DA six different materials from the used set of bulky waste samples were classified with an accuracy of 0.64.
Whereas the PLS-DA estimated the class of each pixel separately, a CNN is able to make use of the spatial and spectral information at the same time.Therefore, a CNN performing a patch-wise classification on all sensor modalities will be part of future work.With an even larger dataset, the goal is to reach a high classification accuracy on a huge variety of different materials from bulky waste.With thermographic and terahertz imaging it might be even possible to look underneath overlapping material.

Figure 2 :
Figure 2: Terahertz measurement on a sample with various materials (left), and two reconstructed terahertz images at various distances to the array to obtain reflection and shadow images, respectively.

Figure 3 :
Figure 3: Measured spectra (left) of different materials after SNV correction and outlier removal.And confusion matrix (right) of PLS-DA classifier trained and tested on NIR hyperspectral data.

Figure 4 :
Figure 4: Various samples acquired by various sensor modalities.Each row shows a corresponding imaging technology from top to bottom: RGB, NIR hyperspectral, thermographic, and terahertz imaging.