Haptic input devices with intelligent signal processing ensuring process stability and quality management

As part of the Cluster of Excellence Merge, a complete process chain was developed for the production of a hybrid laminate with sensory function for continuous production processes. An interior surface of the VW UP! is a good example of this. In this work, the forming processes of the centre console and the parameters influencing quality are discussed. An important parameter for the polarisation of the sensor layer is the thickness of the piezoceramic foil after forming. The maximum signal quality can only be achieved by an exact prediction of the thinning of the foil during the forming process. In addition, the electrical characterisation, especially the capacitance, of the sensor areas is used to determine the foil thickness within the sensor areas in the complex-shaped centre console. Furthermore, a practicable polarisation strategy is deducted taking into consideration the thickness and electrical characteristics of the piezoceramic foil and the process parameters of the forming process. For the evaluation, a novel impact localisation method based on machine learning is presented. Special focus is put on the independence of the impact intensity in order to guarantee a user-independent operation. In this respect, the suitability of various intensity-independent localisation methods will be discussed and subsequently empirically evaluated.


Introduction
The combination of metal sheets with fibre-reinforced plastic (FRP) composites to form hybrid layered composites allows the range of properties of the basic components to be significantly expanded. These composites are characterised in particular by low weight combined with high specific stiffness and strength as well as high damping and crack resistance. The insertion of the metal layers also improves the damage tolerance of the FRP, making the hybrid laminates predestined for use in the aviation industry. A well-known representative is GLARE, a glass fibre reinforced epoxy resin aluminium foil laminate, which represents a further development of the well-known hybrid laminate ARALL (aramid fibre reinforced epoxy resin aluminium foil laminate). The layer structure of GLARE consists of glass fibre reinforced prepreg layers combined with aluminium sheets. The duroplastic matrix material of the laminate, however, requires a long curing time and complex plant technology, which makes the production of GLARE-based hybrid components cost-intensive [1]. Hybrid laminates with thermoplastic matrices represent a significant advantage here. Their production is characterised by short cycle times with moderate investment in plant and equipment. Two important representatives of such laminates are CAPAAL© (Carbon Fibre Reinforced Polyamide Aluminium Laminate) and CAPET© (Carbon Fibre Reinforced PEEK Titanium Laminate) [2].
A further approach to increase the lightweight construction potential of hybrid laminates is the integration of additional functions into structural components. For example, piezoceramic elements can be contacted with electrically conductive polymers by injection moulding and used, for example, for structure monitoring in fibre-plastic composites [3]. Similarly, piezoceramic fibres inserted and joined into micro cavities produced by forming allow the monitoring of metal components [4]. An efficient and large-scale production of hybrid laminates with large-area integration of sensors for the detection and localisation of impact damage in hybrid sheet metal structural components is not yet known. This results in the overriding objective of researching a process suitable for mass production for the production of active composite materials from metal sheets and thermoplastic sensor foils. The schematic process sequence is shown in Figure 1.
The focus of the research work is the production of active film sheets with extrusion technology suitable for mass production on the basis of electromechanically functionalised plastic granulate (1). Piezoceramic particles inside the film generate self-sufficient voltage signals due to mechanical loads, which can be used as a basis for component monitoring. In a continuous thermal joining process, an aluminium sheet and copper electrodes are then connected to the piezo foil on one side (2). For the active sensor compound formed in this way, the forming into a structural component (3) and the subsequent imprinting of an electrical preferred direction by polarisation (4) takes place. For the evaluation of the generated signals, a cost-efficient embedded system will be developed that allows the localisation and characterisation (5) of local deformations on the basis of characteristic stress patterns and their propagation time behaviour and can adapt the expected parameter variations of the series process. For demonstration purposes, the compound material is integrated into the centre console of a vehicle, where it is used as an input system [5]. Another possible application is the localisation and classification of damage in metallic structural components. 2 Materials and methods

Production and polarisation process
The forming of hybrid laminates poses a special challenge, in particular to avoid failure of the individual layers, delamination and wrinkling. Failure-free forming can only be achieved by sufficient temperature control [7]. With the aid of non-destructive testing methods for hybrid laminates, defects can also be detected which have a direct effect on the part properties [8].
In preliminary tests, the process parameters for forming by V-bending and deep drawing were investigated. The most important influencing parameter was the forming temperature. It was found that the piezoceramic compound cracked brittle at room temperature. On the other hand, when the temperature is too high, the thermoplastic matrix flows out of the forming zone [9], [10].
Based on these results, a tool concept for the forming of a cover for the centre console of an automobile was developed. The special feature of this concept is the variothermal temperature control of the active tool parts. This allows the workpiece to be heated to forming temperature in one stroke and cooled in the closed tool after forming. This has a further practical benefit: this concept is expected to reduce springback. The exact process sequence and the function can be read up on in [6].
This work was continued by a parameter variation. The main influences are the preheating temperature, the forming temperature, the composite thickness and the penetration depth of the punch. These parameters are compared to the effects of springback and thinning of the composite. This is particularly important for the subsequent polarisation process. To imprint a preferred electrical direction into the piezoceramic foil, a strong electrical field was applied between the aluminium sheet and the copper electrodes. For an effective polarisation, the electric field strength has to be set to 4 kV/mm. Thus, the required polarisation voltage depends on the foil thickness. If the applied voltage is too low, the required electric field strength cannot be reached. However, if the voltage is too high, the dielectric strength is exceeded and the sensor is damaged. Therefore, the thickness of each sensor area must be known exactly. For this purpose, the correlation between capacitance and thickness of the sensor areas was used to deduce the required polarisation voltage. The polarisation process takes place in a tempered oil bath at 125°C. The polarisation voltage was maintained for 5 mins. Afterwards, the polarised parts were taken from the oil bath and cooled down to ambient temperature.

Localisation
A reliable method for evaluating of the laminate functionality is represented by impact localisation. In [6], we have already demonstrated how the laminate described can be used to detect the impact using machine learning methods that ensure the correct functionality of the hybrid laminate. In detail, it has been shown that a Support Vector Machine (SVM) with a polynomial kernel achieves a detection rate of 84%.
The used system is illustrated in Figure 2. The system was developed with the objective to be used in mobile scenarios and to provide real-time processing capabilities. For this reason, an embedded solution on the basis of an energy-efficient embedded System on Chip (SoC) architecture was used, which allows high parallelisation capabilities of the signal processing through the combination of an FPGA and ARM processor. At the same time, it guarantees flexibility regarding possible interfaces used. As a first step of the signal processing, an analogue circuit was designed that serves as an energy injector for the signal to increase the signal energy. Additionally, the electric circuit centralises the signal in the ADC range of 0 and 1 V and limits the voltage to the specified range securing the upper and lower limits of the ADC. In the embedded system, the signal processing processes were implemented in the FPGA, where the Xillinx FPGA's integrated XADC was used to sample the sensor data. A Moving Average Filter was used to smooth all signals in parallel aiming to eliminate slight interference and noise. Based on this, the signal features used to classify the impact position are determined. The features used are based on the determination of the summed gradient, which in turn is used to extract the time differences over the gradient. The determined time differences are then transferred via the AXI interface to the ARM processor. The ARM processor is used to classify the impact positions by SVM usage. The classification results are transferred to further process units via standardised interfaces such as CAN, Ethernet or UART. Figure 2: System architecture for real-time processing [6] Technologies for Lightweight Structures 3 (1) (2019) As a result, the system guarantees a sufficient localisation accuracy of 84 %. However, the associated classification process has significant disadvantages. The use of the summed gradient results in a dependency on the impact intensity, since the increase of the gradient changes depending on the intensity of the taps of the user on the metal surface. Furthermore, a fixed threshold is used to extract the temporal differences in order to determine the time of difference. Likewise, this additionally causes the method to depend on the impact intensity, since this threshold can be reached earlier or later depending on the impact intensity, in the worst case the threshold won't be reached. This is directly related to the increase of the curve of the summed gradient, which leads to a lack of generalisability of the results further functional instability. Thus, the system presented in [6] represents a very good individual solution, but only works in this form if the SVM model has been developed for the corresponding user. Due to this reason new features are analysed in chapter 3.3, which serve as input for the SVM ensuring a user and intensity independent impact localisation.

Process stability
A series of form samples were optically measured using the GOM Atos Core system. The measured parts are based on forming with different parameter sets and are used to compare the influence on the maximum deviation from the target (equal to springback), the maximum thinning, and the process reliability.
Process safety is calculated using the process safety index CPK. It is the minimum of the lower process capability index CPK L and the upper process capability index CPK U (3). The two values are calculated from the difference of the upper (U) or lower (L) tolerance with the position of the process Xmid (mean value) and in relation to the variation range of the process ΔU,L (1), (2) [11]. If the value is above 1, the process is considered safe. The upper and lower limits are defined as medium according to the general tolerance DIN ISO 2768. Figure 3 shows the dependence of the three parameters' drawing depth, initial composite thickness and forming temperature on springback. To compare the values, the other parameters were left constant in one set. The diagramme clearly shows that the drawing depth has the greatest influence on the springback. This is due to the increase in plastic deformation as the drawing depth increases, which in turn creates a greater resistance to the elastic stresses. The composite thickness has only a small influence on the springback, with increasing thickness the springback also increases slightly. As known from the preliminary tests [10], a higher forming temperature also reduces springback.

Figure 3: Comparison of the springback behaviour in relation to the parameters
The thinning was calculated as technical strain over the thickness using equation (4) where the measured thickness s1 is set in relation to the initial thickness s0.
For better differentiation, the evaluation of the thinning was divided into the corner areas and the regions of the later function area (Figure 4, right). The corner areas show the highest thinning to be expected in the transition zone to the bottom due to the typical deep-drawing behaviour. In the functional area it is important to keep the thinning as low as possible.

Figure 4: Comparison of the thinning behaviour in relation to the parameters
The results in Figure 4 show that, as expected, the thinning in the corners is greater than on the function area. Increasing the drawing depth always leads to greater thinning and at low drawing depths the thinning on the function area is zero. With the increase in composite thickness, a reduction of the thinning can also be proven. The increase in forming temperature also significantly increases thinning in both areas. This is due to the low yield stress at high temperatures of the plastic.
The analysis of the process capability shows a differentiated picture ( Figure 5). On the one hand, it becomes clear that the CPK value is greater than 1, especially in the edge areas and parts of the functional area. The edge areas are particularly important, since these are later decisive for the exact assembly of the centre console. The areas with a CPK value of less than 1 deviate from the required tolerance due to springback. However, these deviations can be compensated by joining a veneer together.

Quality management
The thickness of the piezoceramic foil is an important parameter for the polarisation of the sensor layer. The polarisation voltage depends on the foil thickness because of the required electric field strength of 4 kV/mm for an optimal polarisation. Optical measurement was used to determine the thickness at the location of the later sensors. In the digital image of the centre console, sections were made through the areas of the later sensors to determine the corresponding composite thickness. For an in-line determination of the foil thickness during manufacturing, the correlation between the thickness and the sensor capacitance was used. Based on the measured capacitance between the copper electrodes and the aluminium sheet, the foil thickness can be calculated using the sensor area and the permittivity of the piezoceramic material of the foil. The measured capacitances and foil thicknesses in relation to the calculated foil thickness depending on the capacitance are shown in Figure 6.

Figure 6: Measured and calculated foil thickness depending on sensor capacitance
Due to the good correspondence between the measured and calculated foil thickness, the capacitance of the sensor areas can be used to define the polarisation voltage. For the measured capacitances between 197 and 158 pF and the corresponding thicknesses between 120 and 150 µm, polarisation voltages between 490 and 610 V were used. These process parameters led to successful polarisations of the piezoceramic foil in the sensor areas.

Digital signal processing
To ensure the functional stability of the localisation process, new signal features are analysed to reduce or even neutralise the dependence on the intensity of the impact. In this context, phase difference, cross correlation, meantime, mean frequency and short-term energy ratios were analysed as possible input features for SVM training.
The data acquisition was analogous to the data acquisition in [6]. The same centre console was used to ensure the comparability of the results. For the same reason, the same class division of 20 fields (4 columns, 5 rows) was chosen. When recording the data, special attention was paid to gaining different impact strengths. In sum, 25 measured values of 1s length were recorded for each field at a sampling rate of 500 kHz. The obtained data were subdivided into measurement windows of length N. For the following analyses, N is set to 32768 unless otherwise specified. This value corresponds to approx. 65 ms representing a window length covering the whole impact. Additionally, this value represents the maximum window length of the Xilinx FFT-IP Core enabling the FFT calculation in parallel using the FPGA.
An intensity independent method for impact detection is based on the determination of the temporal displacement of the signals due to the local separation of the sensors and the propagation velocity of the wave in the medium. In this context, the cross correlation and the phase difference are identified as possible features. The cross correlation is a measurement method for the temporal difference in the time domain, whereas the phase difference represents the same in the frequency domain. However, there is a considerable difference between the two methods in that the accuracy of the cross correlation depends on the sample rate of the signal since the time difference can only be determined in the number of samples. The phase difference on the other hand is not subject to this restriction, but the signal must be transformed into the frequency domain using an FFT before the phase difference can be calculated. The discrete cross correlation [ ] is determined by the following equation: In this context, ( ) and ( ) represent two discrete sensor signals sampled synchronously at time and window size . The phase difference is determined directly by definition. In the first step, the signal is transformed via the FFT into the frequency domain. The phase is then determined as the angle between real and the imaginary parts of the FFT result of the respective signals. Afterwards, the difference is determined by subtracting the phases from the corresponding signals.
The results show an unexpectedly behaviour. Contrary to the expectation that the phase difference changes with the signal over time, this did not occur. Instead, a significant pattern was found in all recordings at the time of impact. In each case, a global maximum of the phase difference follows a global minimum, which is shown in Figure 7 for better illustration. In further analyses, this property could not show suitable localisation results but served as a trigger for the identification of the measurement windows with the impact.
Similarly, the results of the cross correlation provide no significant localisation results. This can be explained by two possible approaches. On the one hand, the contact of the laminate leads to a capacitive change whereby the level of the voltage rises abruptly falsifying the measured values. On the other hand, it is possible that the speed of propagation of the wave within the medium is changed by the deformation processes, which would also explain this effect.

Figure 7: Example of phase difference results showing a significant pattern at the time of impact
A further approach to obtaining independence from impact strength is to calculate the mean time and the mean frequency. These two values represent a measure of the time and frequency at which the energy is concentrated. The calculation of the mean time of the input signal is done by following equation: In this context, represents the amount of samples per measurement window, time describes the measurement time of the sample depending on the corresponding measurement window start time and represents the short-term energy calculated by: Accordingly, the mean frequency is calculated by: With representing the SFFT transformed input signal and the corresponding frequency bin. A further property results from the short-term energy. The short-time energy is determined in the time domain according to equation (7), but it is also possible to determine the energy in the frequency domain according to Parseval's relation leading to the definition by: Of course, the energy of a signal is directly dependent on the impact strength, which contradicts the objective of independence form the impact intensity. For this reason, only the relationships of the energies to each other are analysed. Since the mean time and the mean frequency also depend on the signal energy, only the relationships to each other are examined in order to neutralise the influence of the energy level.
An SVM was trained to analyse the results, whereby 3 kernel versions were analysed analogously to [6]. A distinction was made between the polynomial kernel, the radial kernel and the sigmoid kernel. For evaluation, 25 random recordings per field were chosen for training, and 7 recordings for testing. The results are depicted in Table 1.
The results show that the ratio of mean frequency has no capability for localisation, this can be explained by the propagation of the wave which does not change its frequency. Thus, the frequency with the highest energy density is the same or at least similar for all measuring channels leading to these results. Compared to the mean frequency, the mean time performs much better, but with an accuracy of only 63% it is still insufficient. On the other hand, the ratio of the short-term energy offers a detection rate of over 70% showing the possibility of successful localisation at this level.

Discussion and conclusion
It has been shown that the parameters of drawing depth and forming temperature have a decisive influence on springback when forming hybrid laminates. Furthermore, thinning is also influenced by these parameters. It can be seen that the largest thinning always takes place in the corner area. The process stability is particularly high in the edge areas of the centre console. Only in the middle areas does the index deviate from the target. Here an improvement is expected by the subsequent joining of the veneer.
The measured foil thickness after the forming process shows a good compliance with the calculated foil thickness based on the measured sensor capacitance and the materials permittivity. Based on these results, applicable parameters for the polarisation process could be defined and a successful polarisation of the piezoceramic foil was performed.
The signal processing results demonstrate the possibility of an impact intensity independent detection, but provide insufficient results. For this reason, it was analysed whether there are systematic erroneous regions identifying possibilities for improvement. For the analysis of these error sources, a two-stage evaluation was performed.
In the first stage, the classification results were analysed by determining the error rate for each field separately. All fields with an error rate of more than 50 % were excluded, leading to an exclusion of 5 fields in total. With this setting, an accuracy of 90 % could be achieved using the ratio of the short-term energy. Considering the excluded fields, it was remarkable that they have a local proximity of sensor positions and impact point. For this reason, the second step analyses whether the proximity of the impact point and the sensor had significant influences. For this purpose, 5 fields were excluded, which are directly adjacent to the sensor positions, regardless of the error rate of the individual fields. By using the same function as in step one, an accuracy of 82 % was achieved. This leads to the conclusion that the proximity of sensors and impact positions show a significant correlation since most of the errors occur close to the sensor position. A possible explanation for this can be found in the propagation of the wave in the metal, since an impact directly on the sensor does not represent a measurement of the wave itself, but rather of the impact as such. However, the features used are based on the measurement of specific wave characteristics leading to the degradation of the results and causing the error susceptibility of the identified regions.
In summary, the presented results demonstrate that the novel laminate can be used for impact detection and thus as a haptic input interface. According to the results presented, the highest detection rate depending on the area coverage is still achieved by the models published in [6]. However, the disadvantage is a specialised model developed only for a specific user. In this work it could be shown that a generalisation is also possible if impact points close to the sensor are avoided, which lead to a partial area use. Furthermore, the results show a significant correlation between the positioning of the sensors and the impact position, which is the starting point for further work analysing the influence of the sensor positions in order to ensure an overall area-wide use.