Multiradar Data Fusion for Respiratory Measurement of Multiple People

This study proposes a data fusion method for multiradar systems to enable measurement of the respiration of multiple people located at arbitrary positions. Using the proposed method, the individual respiration rates of multiple people can be measured, even when echoes from some of these people cannot be received by one of the radar systems because of shadowing. In addition, the proposed method does not require information about the positions and orientations of the radar systems used because the method can estimate the layout of these radar systems by identifying multiple human targets that can be measured from different angles using multiple radar systems. When a single target person can be measured using multiple radar systems simultaneously, the proposed method selects an accurate signal from among the multiple signals based on the spectral characteristics. To verify the effectiveness of the proposed method, we performed experiments based on two scenarios with different layouts that involved seven participants and two radar systems. Through these experiments, the proposed method was demonstrated to be capable of measuring the respiration of all seven people by overcoming the shadowing issue. In the two scenarios, the average errors of the proposed method in estimating the respiration rates were 0.33 and 1.24 respirations per minute (rpm), respectively, thus demonstrating accurate and simultaneous respiratory measurements of multiple people using the multiradar system.


I. Introduction
R ESPIRATORY measurements have become increasingly important in a variety of applications, including healthcare, home medication, medical alert systems for babies, and monitoring systems used for infants and senior citizens. A variety of sensors and systems have been developed for these applications to monitor patient respiration [1]- [3] and among these systems, radar-based systems have been attracting attention as a promising approach because they are suitable for performing longterm measurements without patient discomfort since there is no need for the patient to wear a device. A number of existing studies have been reported on the measurement of people using radar techniques, including systems for estimation of the locations of people [4]- [9], tracking of people in motion [10]- [17], and detection of patient vital signs, including their heartbeat and respiration [18]- [38].
Another advantage of radar-based respiratory measurement is the possibility of performing simultaneous measurements of the respiration of multiple people using only a single device. There have been numerous studies in which a single radar system has been used to measure the respiration of multiple people [18]- [36]. For example, Nosrati et al. [33], Islam et al. [34], Su et al. [35], and Koda et al. [36] used single radar systems to demonstrate simultaneous measurement of the respiration of two, three, three, and seven people, respectively. These studies, however, did not consider the effect of the shadowing problem, in which an echo from one target person is blocked by another person located between the first person and the radar system; this results in the radar system being unable to detect a person (or some people) when multiple people are present in the scene.
To overcome the shadowing problem, this study introduces simultaneous use of multiple radar systems to perform respiratory measurements of multiple people. There have been previous studies that used multiple radar systems to measure the respiration of multiple people: Shang et al. [37] used two radar systems to measure two people in the presence of body motion, and Yang et al. [38] also used two radar systems to measure five people. In these studies, it was assumed that the relative positions and orientations of the radar systems were known in advance, and that the echoes from all target people could be measured directly by each of the radar systems without the shadowing problem. These assumptions, however, cannot always be satisfied in practice, which hinders the use of the radarbased respiratory measurement approach in actual daily environments.
As discussed above, none of the existing studies have accounted for the effect of the shadowing problem, which prevents the echo from one person from being detected by one of the radar systems. In this study, we propose a novel multiradar data fusion method to perform respiratory measurements that can measure multiple people simultaneously. The proposed method can mitigate the effect of the shadowing problem using multiple radar systems without knowing the relative locations and orientations of these radar systems. The proposed method generates multiple radar images using the multiradar data, and these radar images are then converted into multiple sets of twodimensional point clouds. To align these point clouds, the proposed method then estimates the relative positions and orientations of the multiple radar systems, which allows for data fusion of the multiradar system and thus overcomes the shadowing problem. The proposed method even works when the positional relationships among the multiple radar systems are unknown, which means that there is no requirement for calibration to be performed beforehand, and also no restriction on the physical installation of the radar systems. The performance of the proposed method is evaluated quantitatively by performing radar measurements involving seven participants and a pair of radar systems.

II. Imaging and Clustering of Human Targets
Using Multiple Radar Systems We assume use of a multiradar system comprising M radar systems with positions and angles that are unknown. Each radar system contains a K-element array (or virtual array). Let s i (t, r) denote the signal received by the i-th element (i = 0, 1, · · · , K −1) of the array, where t is the slow time and the range r is expressed as r = ct ′ /2 using the fast time t ′ and the speed of light c. A signal vector s(t, r) is defined as s(t, r) = [s 0 (t, r), s 1 (t, r), · · · , s K−1 (t, r)] T , where the superscripted T represents a transpose operator. We assume here that all elements of the array antennas are calibrated and that static clutter is suppressed by subtracting the time-averaged signal.
Using the m-th radar system (m = 1, · · · , M ), a radar image is generated and is expressed in the twodimensional Cartesian coordinate system (x m , y m ), where the x m axis is aligned with the array baseline of the m-th radar system. A complex radar image I m (t, r m ) is generated from the m-th radar system's data as I m (t, r m ) = w(φ m ) H s(t, ρ m ), where r m is the position vector that can be expressed as (x m , y m ) in Cartesian coordinates and as (ρ m , φ m ) in polar coordinates. The weight vector is w(θ) = [w 0 , w 1 , · · · , w K−1 ] T , where the choice of w i depends on the imaging algorithm being used. In this study, we use a straightforward beamformer (BF) method for simplicity, in which the weights are set as w i (θ) = α i e −j(2πxi/λ) sin θ (i = 0, 1, · · · , K − 1). Here, we use the Taylor window coefficient α i . Fig. 1 shows a system model containing multiple radar systems for measurement of multiple human targets, in which we see that each radar system has its own coordinate system.
Next, a point cloud is generated from each radar image; the m-th point cloud is generated randomly to follow a density distribution that is proportional to ρ m |I m (t, r m )| 2 , where ρ m is introduced to compensate for propagation losses. The X-means clustering technique [36], [39] is then applied to the point cloud and the clusters c 1 , c 2 , · · · , c N are generated; these clusters correspond to the N targets. For presentation purposes, we generated a radar cluster image C m (t, r m ) that takes the values {0, 1, 2, · · · , N }, which correspond to the indices of the clusters c 1 , c 2 , · · · , c N except for 0; C m (t, r m ) = 0 indicates that no cluster is associated with r m . Finally, the representative position r positions are then processed in the proposed data fusion algorithm, which combines multiple radar images.

III. Multiradar Data Fusion for Respiratory Measurement of Multiple People
The main issue when performing radar measurements of multiple human targets indoors is that not all of these human targets can be measured directly using only a single radar system because of shadowing, i.e., where one human echo is blocked by another human target or by obstacles; this occurs frequently when multiple people are located together and are densely spaced. To mitigate this issue, we use a multiradar system to increase the probability that at least one of these radar systems can detect all human targets in the scene. Given the radar images I m (t, r m ) and the cluster positions r (i) m for m = 1, · · · , M , we propose a data fusion method here that combines the multiple radar images. Here, we assume a special case with M = 2 for simplicity. The first step in the method is to find a rigid transformation that can align the coordinates from the x 2 -y 2 coordinate system to those of the x 1 -y 1 coordinate system.

A. Procrustes Analysis for Alignment of Associated Target Clusters
We propose a data fusion method that uses Procrustes analysis [40] to find the rigid transformation parameters required to describe the relationship between two sets of points in a plane. Let us assume here that N pairs of two-dimensional vectors (r 2 ) correspond to N pairs of points. The purpose of this section is to find a rigid transformation that adjusts the points r (n) 2 (n = 1, · · · , N ) with respect to r (n) 1 (n = 1, · · · , N ). If appropriate transformation parameters are selected, the points are then transformed to satisfy r  2 . Here, we define 2 × N matrices S 1 and S 2 as: and we estimate a rotation matrix R * as: where the subscripted F denotes the Frobenius norm and represents the square root of the sum of squares of all components.
In the Procrustes analysis, rather than solve for the optimization problem in Eq. (3) directly, singular value decomposition is used to estimate R * ; S 1 S T 2 is decomposed using the form S 1 S T 2 = U ΣV T , where U and V are 2 × 2 orthogonal matrices, and Σ represents a 2 × 2 diagonal matrix with non-negative real numbers. We then obtain the rotation matrix R * = U V T . Using R * , the position vectorr (n) 2 is transformed to give R * r (n) 2 +r 1 , which is a mapping from the coordinate system x 2 -y 2 to the coordinate system x 1 -y 1 . Although we have discussed the special case where M = 2 above, a similar procedure can be performed in a general case, including cases where M > 2.

B. Proposed Data Fusion Method Using Respiratory Correlation and Procrustes Analysis
To apply Procrustes analysis to transformation of the coordinate systems of a multiradar system, the cluster positions can be used to form the input matrices S 1 and S 2 . We should note here that Procrustes analysis requires at least two pairs of corresponding points, e.g., (r 2 ). In this section, we propose a method to associate the cluster positions that uses the respiratory features of the human targets. In the proposed method, the similarity of the respiratory displacement waveforms is used to associate multiple echoes. Although we explain the procedure for the case where M = 2 for simplicity, the procedure can be extended to a general case with M > 2. The proposed method comprises the following five steps.
2) Procrustes analysis is applied to the two pairs of points (n * 1 , n * 2 ) and (n * * 1 , n * * 2 ), and the rigid transformation parameters R * and t * are then obtained.
3) Using R * and t * , all position vectors r n m m = 1, · · · , M ; n = 1, · · · , N are transformed into the coordinate system x 1 -y 1 . All pairs of transformed points separated by a distance of less than a threshold distance (i.e., |r are regarded as associated pairs. 4) All associated pairs are processed using the Procrustes analysis, and the transformation parameters R * and t * are then updated. 5) For each human target, one of the displacements ) is selected that is most likely to reflect the target's respiratory motion. We proposed selection of the displacement with the waveform that is closest to a sinusoidal wave. Using the following fourth-order moment of the power spectrum, we then find the m that maximizes κ(n, m) for each n (1 ≤ n ≤ N ). We define and the optimum m is then found as which means that the m * (n)-th radar signal is used for the n-th human target. As a result, the displacement waveform that is most likely to be related to their respiration is selected for each person, thus leading to accurate measurement of their respiration.

IV. Experimental Evaluation of the Proposed Method A. Multiradar Experiment with Multiple People
We performed the experiments using both a pair of array radar systems and belt-type respirometers simultaneously to evaluate the accuracy of the multiradar-based respiratory measurements acquired using the proposed method. To evaluate the accuracy of the method in measuring the respiratory intervals, all participants wore a belt-type respirometer on their upper torso. Each participant was seated and was breathing normally, and the measurements were acquired using two radar systems that were installed with different orientations and at different positions. Note that the belt-type respirometers were only used to evaluate the accuracy of the radar-based measurements.
Both radar systems are frequency-modulated continuous wave (FMCW) radar systems with a center frequency of 79 GHz, a center wavelength of λ = 3.8 mm, and an occupied bandwidth of 3.9 GHz. The beamwidths of the individual radar array elements are ±4 • and ±35 • in the Eand H-planes, respectively. The radar array is composed of a multiple-input multiple-output (MIMO) array that contains three transmitting and four receiving elements,  with spacings of 7.6 mm (2λ) between the transmitting elements and spacings of 1.9 mm (λ/2) between the receiving elements. In the experimental setting, the MIMO array can be approximated using a virtual linear array. Specifically, the array used in this study can be approximated using a 12-element virtual linear array with element spacings of λ/2. The slow-time sampling intervals were 100 ms. We apply the proposed method to the radar data that were measured with the seven participants in two scenarios. We set different layouts for both the human targets and the radar systems in these scenarios. In scenario 1, as shown in Fig. 2, the seven participants were seated in a U-shaped arrangement with participant spacing of 1 m. We then used a pair of array radar systems to measure the participants as illustrated in Fig. 3. In scenario 2, the seven participants were seated in a different layout, as shown in Fig. 4, and the radar systems were also placed at different positions with different orientations, as illustrated in Fig.5. The measurement time was set at 120 s and the participants were instructed to remain still and breathe normally during the measurements. To evaluate the accuracy of the respiration measurements, belt-type contact respirometers were used simultaneously with the radar measurements.

B. Application of the Proposed Data Fusion Method to the Measured Radar Data
1) Application in Scenario 1: First, we apply the proposed method to the data acquired in scenario 1. The left and right panels of Fig. 6 show radar images |I 1 (t, r 1 )| 2 and |I 2 (t, r 2 )| 2 at t = 30 s. In the figure, the actual target positions are indicated by white circles with the target numbers. Next, a point cloud is generated from these images and the respiratory-space clustering algorithm [36] is then applied to generate the radar cluster images |C 1 (t, r 1 )| 2 and |C 2 (t, r 2 )| 2 shown in Fig. 7. In the figure, an alphabetical label is provided for each cluster. In image |I 1 (t, r 1 )| 2 in Fig. 6, the echo from target 3 is barely visible because of shadowing; as a result, in Fig. 7, target 3 is not classified as a cluster. The resulting number of clusters was thus estimated erroneously to be six rather than seven, which is equivalent to the estimated number of human targets. This type of shadowing problem can occur particularly when multiple people are densely spaced within the measurement scene. To mitigate the shadowing problem, this study uses multiple radar systems rather than a single system.
Next, the proposed method is used to calculate the correlation coefficient ρ C for each pair of displacement waveforms (see Table I). The proposed method finds two pairs of points with the largest and second largest values of ρ C . In Table I, we can find the largest correlation coefficient for the pairing of target E (radar 1) and target C (radar 2). We can also find the second largest correlation coefficient for the pairing of target C (radar 1) and target E (radar 2). These pairs (E, C), (C, E) are processed via the Procrustes analysis for N = 2 and the rigid transformation parameters x = −2.52 m, y = 1.90 m, and θ = −1.58 rad are obtained. Using these parameters, the radar image |I 2 (t, r 2 )| 2 is then transformed into |Î 2 (t, r 1 )| 2 . Using the transformation parameters, the proposed method associates the six cluster pairs (A, D), (B, B), (C, E), (D, F), (E, C), and (F, A) from the seven human targets, where the distance threshold was set at d th = 0.5 m. Note that cluster G from the second radar system was not associated in this process. The associated pairs are then processed via the Procrustes analysis and the transformation parameters x = −2.46 m, y = 1.86 m, and θ = −1.57 rad are updated, as illustrated in Fig. 8.
For the associated cluster pair (n 1 , n 2 ), the respiratory displacement waveforms ∠I 1 (t, r (n1) 1 ) and ∠I 2 (t, r (n2) 2 ) are then obtained. The detection rate for 120 s for each target is shown in Table II. The table shows that human target 3 was not detected by one of the radar systems because of shadowing. All the human targets, with the exception of target 3, were detected by both radar systems, which gives us selection options; we can thus select a more accurate radar system for each target. In the proposed method, the fourth-order statistical parameter κ is calculated for both displacement waveforms, from which the waveform with the larger κ value is selected every 30 s. Table V shows the errors in estimation of the respirations per minute (rpm) obtained when the two radar systems are combined. The error averaged over the seven participants is 0.33 rpm for the proposed process when combining the two radar systems. These results indicate that the proposed method using the multiradar system is effective in performing accurate respiration measurements of multiple people simultaneously.

2) Application in Scenario 2:
We also applied the proposed method to the radar data acquired in scenario 2. The layout of the target people in this case is shown in Fig. 5. Using radar systems 1 and 2, the radar images |I 1 (t, r 1    and |I 2 (t, r 1 )| 2 were generated as shown in Fig. 9. We can then generate the radar cluster images |C 1 (t, r 1 )| 2 and |C 2 (t, r 2 )| 2 shown in Fig. 10. In Fig. 10, no clusters are generated for targets 5 and 6 because the echoes from targets 5 and 6 are weak in image |I 1 (t, r 1 )| 2 , as shown in Fig. 9. Similarly, no cluster is generated for target 1 because the echo is weak in image |I 2 (t, r 1 )| 2 , as shown in Fig. 9. The proposed method is then used to calculate the correlation coefficient ρ C for each of the pairs of displacement waveforms (Table III). In Table III, we find that the largest correlation coefficient is that for the pairing of target C (radar 1) and target F (radar 2), and also find that the second largest correlation coefficient is that for the pairing of target A (radar 1) and target B (radar 2). These pairs, denoted by (C, F), (A, B), are processed via the Procrustes analysis for N = 2 and the rigid transformation parameters x = −0.17 m, y = 5.11 m, and θ = −3.07 rad are obtained.
Then, using these parameters, the radar image |I 2 (t, r 2 )| 2 is transformed into |Î 2 (t, r 1 )| 2 . Using the transformation parameters, the proposed method associates the four cluster pairs (A, B), (B, C), (C, F), and (E, D) from the clusters, where the distance threshold was set at d th = 0.5 m. Note that cluster D from the first radar system and clusters A and E from the second radar system were not associated in this process. Using the associated pairs, the transformation parameters x = 0.12 m, y = 5.09 m, and θ = 3.06 rad are updated, as illustrated in Fig. 11.
For the associated cluster pair (n 1 , n 2 ), the respiratory displacement waveforms ∠I 1 (t, r (n1) 1 ) and ∠I 2 (t, r (n2) 2 ) are obtained. The respiration detection rates for each target when using the proposed method are shown in Table  IV, which shows that human targets 1 and 5 were only detected by one of the two radar systems because of shadowing, whereas the other targets were detected by both radar systems. As in scenario 1, the fourth-order statistical parameter κ is calculated for each of the estimated displacement waveforms, from which the waveform with the larger value of κ is selected every 30 s. Table V shows the errors in respiration rate estimation when using the proposed data fusion technique for these radar systems. The average error for the seven participants was 1.24 rpm when using the proposed method to combine the radar data.

V. Conclusion
In this study, we have proposed a novel method for a multiradar system that can measure the positions and res- Fig. 9. Examples of radar images |I 1 (t, r 1 )| 2 (left panel) and |I 2 (t, r 2 )| 2 (right panel) at t = 30 s in scenario 2. piration characteristics of multiple people. By introducing multiple radar systems rather than a single system, the shadowing problem can be mitigated, which means that the respiration rates of multiple people can be measured even when these people are located closely together. In addition, when using multiple radar systems, we can measure each person from multiple directions, which improves the accuracy of the respiration measurements. The effective- Fig. 11. Radar image |Î 2 (t, r 1 )| 2 after the optimized rigid transformation is performed in scenario 2. ness of the proposed method was verified experimentally in two scenarios using a pair of radar systems and seven participants.
In both scenarios, the proposed method was demonstrated to be able to detect all participants by combining the data acquired from the two radar systems, whereas not all participants could be detected when using only one of the two radar systems because of shadowing. The average respiratory rate estimation error for the seven participants in the two scenarios was 0.79 rpm, which is sufficiently small when compared with the typical respiration rate range for adults (12-20 rpm). Use of the multiradar system with the proposed method has been demonstrated to be promising for use in accurate noncontact monitoring of the respiration of multiple people.