In-water handwriting in multi-medium using electrical impedance imaging

A novel approach to human–computer interaction, in water handwriting is presented in this work. The novel handwriting method is based on tomographic imaging of electrical resistivity changes. The new method allows the users to write inside of a water ball in a natural, unconstrained way. In-water handwriting suffers no friction and allows handwriting or drawing in a more relaxed way. This work shows how an electrical impedance tomography (EIT) system and sensor/phantom can be used to create a robust handwriting in water. It is well known that EIT has low special resolution but offers a very good functional imaging with high temporal resolution. The work shows an example of a good quality functional imaging aspect of the EIT in a novel human computer interface (HCI) application. Although it is almost impossible to produce images of letters or complex shapes using EIT sensor, it is totally possible to produce the same using many EIT frames. The system used in this study has a frame rate of 15 frames per second so the actual handwriting of letters and shapes can be done without noticing any delays. Analysis of image similarity between actual handwriting and the one follow the EIT scheme shows a good correlation.


Introduction
New and low-cost sensing methods are being used as user-friendly human-computer interaction (HCI) techniques. There has been a great interest in using inertial sensing methods, such as gesture recognition, activity recognition, motion tracking, and handwriting recognition. Compared to standard keyboard and touch screen input methods, handwriting character recognition in alternative medium provides a much more intuitive, convenient and comfortable interaction experience. Many new works are based on vision based sensors in hand gesture recognition applications [1][2][3]. Advances in computing technology and user interfaces have led to notable developments in interactive applications, such as gestural interfaces in virtual environments. In everyday situation the use of gestural devices, which use hand motion data to control interactive interfaces, has increased over the years [4][5][6].
The paper shows a new type of HCI using electrical impedance tomography (EIT) demonstrating in water handwriting. In fact, the EIT imaging has been proposed in HCI context recently using wearable sensor for hand gesture set up or in fabric based planar sensor [7][8][9][10]. The EIT is gaining new applications in medical domain for human lung monitoring [11] and in many industrial applications for measurement of flow of oil/gas water in pipeline [12]. There are newer interests in EIT in robotics and in pressure sensing [13,14]. This work demonstrates the EIT device as a new HCI tool allowing a robust handwriting in water.

EIT system
EIT has three main components of sensors, the data acquisition (DAQ) system and a computational software. EIT sensors are essentially a series of boundary electrodes, via which currents are injected into the domain and measurements can therefore be taken by the DAQ. The DAQ system used in this study is shown in figure 1(a). Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
It is a 16 channel, low-cost EIT device which can generate images at the speed of 15 frames per second. The computational models involve field electrical modelling for forward model and inversion algorithms to recover conductivities from voltage measurements.

The forward problem
For the appropriate formulation of the system, the complete electrode model (CEM) is used to constrain the boundary conditions. An electrical conductive field model can be used to describe the EIT main problem as below: It describes the potential field u within the domain W, with the internal electrical conductivity distribution σ and the voltage measurement at the electrode U k is described by the complete electrode model: where u is the potential field, Z k is the contact impedance, n is the outward unit normal vector and e k is the electrodes. To be able to image a domain, it is essential to turn the continuous problem into a discrete problem.  The finite element method (FEM) is a numerical discretizing method commonly used in EIT, and it discretizes the domain of interest into piecewise smooth elements to solve the forward model [2,5].

The inverse problem
The inverse problem is the method which produces images of electrical conductivity changes. We linearized the relation between measured changes in boundary measurements Du as a function of changes in conductivity s D which can be done through Taylor expansion to produce: where J is the Jacobian which is essentially a 'sensitivity distributions' within the domain. The Tikhonov Regularization solves for s D with the following equation [15]: where α is the regularization parameter, and I is the identity matrix. This is what will be used to reconstruct all permittivity distributions and each frame of EIT data is typically reconstructed independently of the others. This is referred to as a 'non-scanning' method.

Temporal reconstruction
A dynamic Tikhonov based method can be used in the applications where conductivity changes are very fast with respect to the EIT frame rate [16]. It is formulated in the way that one sequence of data is treated as one single inverse problem with a regularization prior which accounts for both spatial and temporal correlations between image elements. Specifically, the data frame sequence is concatenated as D = D ¼ D ¼ D [ ] Therefore, the temporal inverse problem can be rewritten as where J assumed to be constant, so a linear but time aware reconstruction is used.

Normalization
Due to the inherent ill-posed nature of the EIT problem, the signal responses to the changes occurring near the boundary of the domain are stronger than those in the middle of the medium. Handwriting trajectories can be across the entire phantom. Hence, further normality is needed so that the uniform resolution in the imaging region among all the frames can be achieved. Here two steps of normalization are concerned: 1. Difference data normalization: adjust the difference of background and inclusion data to the scale between −1 and 1; 2. Reconstructed distribution normalization: adjust the reconstructed conductivity changes to the scale between 0 and 1.

Data segmentation
A data segmentation process is considered to identify the beginning and the end of a writing. The norm of difference between background and inclusion data is used here to allow separation between letters. The time window within which the writing is happening can then be marked by the norm above a predefined threshold. The threshold is set empirically and in the manner that any changes caused by handwriting motion would be captured whilst background noises would be discarded. Data segmentation allows us to extract the frames that contain the writing information.

Experiments and the results
The experiment is set up in a water tank as shown in figure 2, and a plastic rod can be used as a pen. The plastic rod will change the electrical resistivity distribution of the domain which can be then converted to images. Measurements are taken using the 50 kHz 16 channel EIT device, which injects currents of 8 mA and works based on simple adjacent driving and measuring strategy. A set of background measured data is collected and used as a reference data using such experiment set up. Figure 3(a) shows a frame of 208 measurements, averaged over 200 sets of data. The Signal to noise ratio plot of the background measurements under one injection is plotted in figure 3(b).

Single letter
The EIT based handwriting is implemented by the combination of many low spatial resolution imaging creating letter using points on path of trajectory of the plastic bar. Individual EIT images are synthesized to produce letters. Figure 4(a) shows images of frame 85 to 109 forming a letter L ( figure 4(b)). Each frame acts as a point in letter L. A total of 65 informative frames can be extracted by applying data segmentation method to the graph in figure 5, with a threshold of 0.1, and superimposed to form the letter.

Multiple letters
More challenging applications of multiple letter recognition are investigated in this section. Figure 6 shows the letters of word EIT, XMAS, DAN, HAN, manuch, soleimani.
In the example of the word 'XMAS', a total 640 frames of EIT images were considered and the norm difference is plotted in figure 7. When the norm is smaller than 0.1, it indicates that the rod is out of the water. Hence, one can see the distinct frame window of each letter in the word 'XMAS'. It is possible to use various letter recognition methods to further enhance these handwriting, but this is beyond scope of this work. Using finger pressure point on a fabric based EIT [13,14] using the same method presented here can give rise to a fabric-based handwriting and use of a dielectric (plastic rod) with electrical capacitive tomography [7] enables in air handwriting.

Correlation analysis
To evaluate the performance of the EIT handwritings, a Pearson correlation coefficient is used, which compares the similarity between the real and reconstructed images over the entire view region. For two grayscale images X, Y, the correlation coefficient can be defined by: where cov is covariance, and s , X s Y are standard deviations of the pixel values. The closer it gets to 1, the stronger the association between the two images is.
In the example another version of the word 'EIT' is used. A printed word 'EIT' with the thickness of lines the same as the diameter of the plastic rod is created, as shown in figure 8(a). Then, the in-water handwriting was accomplished by making the plastic rod follow the trajectory of printed word. Individual letter in the reconstructed word, presented in figure 8(b), is compared to the printed written letters accordingly. The correlation coefficients of letters 'E', 'I', and 'T' are 0.647, 0.712 and 0.548 respectively.  It is also worth noting that letters with complex trajectories poses more difficulty on EIT reconstruction and this can be confirmed with the lower coefficients. Overall, the coefficients are all above 0.5, which reveals a satisfactory correlation of the EIT reconstructed letters with the printed ones.

Conclusions
This paper presents a new type of user input for handwriting using electrical imaging. The start and end of letters are recognized by analyzing measured data from electrical imaging in comparison with the background data, so this is entirely automatic. The proposed new interface system could also be used in other natural interaction techniques, such as in early year education or in games. It may also be used as real-time user activity recognition. EIT sensor used in this study is low cost and therefore can find new applications in human computer interaction. Based on speed of our imaging system the writer does not need to speed down their handwriting, allowing for natural handwriting. The correlation between real handwriting and the one exactly followed by EIT shows satisfactory output. Demonstrating a functional performance for EIT despite its compromised static imaging ability may lead to better understanding of the EIT in its traditional medical and industrial applications.

Data availability statement
'The data that support the findings of this study are available upon request from the author.