3D human posture segmentation by spectral clustering with surface normal constraint

doi:10.1016/j.sigpro.2011.04.003

Signal Processing

Volume 91, Issue 9, September 2011, Pages 2204-2212

https://doi.org/10.1016/j.sigpro.2011.04.003 Get rights and content

Abstract

In this paper, we propose a new algorithm for partitioning human posture represented by 3D point clouds sampled from the surface of human body. The algorithm is formed as a constrained extension of the recently developed segmentation method, spectral clustering (SC). Two folds of merits are offered by the algorithm: (1) as a nonlinear method, it is able to deal with the situation that data (point cloud) are sampled from a manifold (the surface of human body) rather than the embedded entire 3D space; (2) by using constraints, it facilitates the integration of multiple similarities for human posture partitioning, and it also helps to reduce the limitations of spectral clustering. We show that the constrained spectral clustering (CSC) still can be solved by generalized eigen-decomposition. Experimental results confirm the effectiveness of the proposed algorithm.

Highlights

► We introduced spectral clustering to 3D point cloud human posture segmentation. ► We integrated distance and surface normal similarities. ► We developed a constraint spectral clustering algorithm.

Introduction

With emergence of human body scanners in recent years, 3D human point cloud data is becoming more popular in various applications [25], [26]. Human behavior analysis is an important task in various applications [2], [6], [24] like human behavior recognition, real-time tracking system, human interaction system, human-machine control system, and so on. Considering significance of human behavior analysis and novelty of 3D human point cloud data, employing 3D point cloud data into human behavior analysis is an inevitable trend.

A fundamental and important issue in 3D point cloud based human behavior analysis is posture segmentation, i.e., partitioning the human body into semantic parts, such as, torso and limbs. The partitioning of human posture is valuable to motion estimation and recognition [19], [22]. By using the obtained clusters, which represent different parts of the human body, we can give a more compact and robust description of motion than using the raw data. Moreover, it also serves as a preprocessing and benefits the high-level tasks afterward, e.g., recognition and tracking of human behavior [29]. The partitioning of human posture into a small number of semantic parts helps reducing the complexity of the dynamic system of tracking, which is valuable to both improve the accuracy and reduce computation cost.

However, there are several difficulties in partitioning human posture represented by 3D point cloud. First, data from the point cloud are scattered on a 2D manifold of human surface embedded in the 3D space. Due to the utilization of Euclidean distance, classical sum of squared distance methods, e.g., k-means [13], or probability density based methods, e.g., Gaussian mixture models (GMM) [28], or other linearly separable algorithms [9], are unable to find suitable clusters on manifold. Thus, nonlinear approach should be used to integrate geodesic distance between points into the partitioning process so that the manifold structure can be properly addressed. Second, to obtain a partitioning result with reasonably semantic meanings, it is necessary to exploit multiple similarities between the data point on the manifold (human surface), e.g., distance similarity and surface normal similarity. However, the integration of multiple similarities into the partitioning of data point scattered on a manifold is more challenging.

In this paper, we propose a new algorithm, called constrained spectral clustering (CSC), for partitioning the point cloud data of human posture into semantic parts. CSC duly addresses all aforementioned difficulties. On one hand, CSC belongs to the graph-based partitioning algorithm, which only utilizes local similarities between data points, and thus is able to deal with the manifold structure of human surface. On the other hand, CSC formulates extra similarities (could be more than one although we use only one extra similarity in this paper) as constraints for optimization so as to obtain a partitioning result consistent with multiple similarities. We prove that, like spectral clustering (SC), CSC can also be effectively solved by using eigen-decomposition. Sufficient experiments are performed to evaluate the effectiveness of the proposed algorithm.

The rest of this paper is organized as follows. Section 2 gives a brief review of related works on human posture partitioning and clustering algorithms. Section 3 presents the multiple similarities, i.e., distance similarity and surface normal similarity, used in this paper for human posture partitioning. In Section 4, we derive the proposed algorithm, constrained spectral clustering (CSC). We report experimental results and time complexity in Section 5. Finally, Section 6 concludes this paper and gives discussions on future works.

Section snippets

Related works

In the past recent years, extensive researches have been done on 3D point cloud data and technology of segmenting human posture formed by 3D point cloud into semantic parts is one of them [3], [20], [25], [26]. But automatic segmentation of human body is a challenging problem, which is determined by several reasons. First, the body shape is both articulated and deformable. Second, the point cloud is unorganized.

The pioneer investigator in this area is Nurre [20]. He means to segment the human

Multiple similarities

In this section, we present the distance similarity and the surface normal similarity that are used in our partitioning algorithm.

Constrained spectral clustering

The proposed CSC is a constrained extension of SC. In this section, we first review SC briefly. Then, we propose CSC by utilizing the multiple similarities discussed previously, and derive an optimization algorithm for CSC based on eigen-decomposition.

Experimental results

In this section, we conduct a series of experiments to validate the proposed algorithm's effectiveness. Besides, at the end, we also analysis the time complexity of the surface normal similarity calculation and CSC algorithm.

The 3D human body point cloud used in our experiments is obtained from simulated sequences of human actions. To get a comprehensive evaluation of performance, we simulated data examples of different situations, e.g., full-body point cloud and semi-body point cloud, which

Conclusions

In this paper, we achieved the goal of automatically segmenting 3D human posture represented by point cloud by using the proposed CSC algorithm. We used local feature—distance similarity, encoded in the graph Laplacian, to keep the information about the manifold on which 3D data points scatter, meanwhile we additionally utilized human body surface normal similarity as constraint to enhance the partitioning result. The proposed CSC is an extension of SC by exploiting multiple similarities,

References (31)

P. Antonakaki et al.
Detecting abnormal human behaviour using multiple cameras
Signal Processing
(2009)
J.A. García et al.
A dynamic approach for clustering data
Signal Processing
(1995)
X. Li et al.
Optimum cut-based clustering
Signal Processing
(2007)
H. Woo et al.
A new segmentation method for point cloud data
International Journal of Machine Tools and Manufacture
(2002)
Y. Andrew et al.
On spectral clustering: analysis and an algorithm
J. Anupama, Segmentation and recognition of 3D point clouds within graph-theoretic and thermodynamic frameworks, Ph.D....
D.P. Bertsekas
Nonlinear Programming
(1999)
C.L.W. Charlie et al.
From laser-scanned data to feature human model: a system based on fuzzy logic concept
Computer-Aided Design
(2003)
C.-C. Chen et al.
Segmentation of human body parts using deformable triangulation
T. Coleman et al.
Spectral clustering with inconsistent advice

L. Dekker et al.

Building symbolic information for 3d human body modeling from range data

R. Grone et al.

The Laplacian spectrum of a graph

SIAM Journal on Matrix Analysis and Applications

(1990)

X. Ju et al.

Automatic segmentation of 3d human body scans

R. Kannan et al.

On clusterings: good, bad and spectral

Journal of the ACM

(2004)

T. Kanungo et al.

An efficient k-means clustering algorithm: analysis and implementation

IEEE Transactions on Pattern Analysis and Machine Intelligence

(2002)

Cited by (18)

Segmentation of 3D Point Cloud Data Representing Full Human Body Geometry: A Review
2023, Pattern Recognition
This article aims to present a review of the segmentation techniques of the 3D data representing human body in the form of point clouds. The techniques discussed are divided into three subgroups: 2D contour approaches, topological techniques, and machine learning heuristics. These subgroups are then reviewed regarding the following aspects: computation time, accuracy, reliability, dependency on human pose, and segment count. The authors emphasize an analysis of these algorithms with respect to their exploitation in the segmentation of 3D data varying in time, as well as further improvement of the applications in anthropometry. The conclusion reached is that machine learning approach tends to be the most suitable solution for future 4D applications. Another foreseeable direction of development in the field of segmentation algorithms is the classification of the points on the borders between segments and maintaining fluent and consistent edges of the segments between the subsequent frames.
Projection-preserving block-diagonal low-rank representation for subspace clustering
2023, Neurocomputing
In this paper, a novel model named projection-preserving block-diagonal low-rank representation (PBDIR) is proposed and can obtain a more distinguishable representation matrix for clustering. PBDIR acquires a more advantageous representation by extracting the essential features. Specifically, we introduce a projection matrix to our model to learn a new feature space that can capture more significant features. Therefore, our model learns a more robust representation, which can reduce noise interference. Meanwhile, we introduce a block-diagonal regularization to ensure that the obtained representation matrix involves a k-block diagonal, where k denotes the number of clusters. This term brings more benefits for clustering tasks. Experimental results on real datasets show that our model can significantly improve the clustering performance and the proposed approach is robust against Gaussian noise, Multiplicative noise, and Salt-and-Pepper noise.
3D anthropometric algorithms for the estimation of measurements required for specialized garment design
2017, Expert Systems with Applications
3D anthropometry is key to the automation and facilitation of tedious, costly, and time-consuming traditional anthropometric tasks such as the size designation of a specialized garment.
This article provides a short survey of 3D anthropometry in garment design. The results show that due to a lack of detailed, complex analyses of the entire anthropometric procedure from a raw body scan to a set of anthropometric measurements, there was a need to prepare and describe a comprehensive system from scratch. The aim of this article is, therefore, to present the developed 3D anthropometric solution, along with fully documented algorithms for automatic measurement extraction from a 3D body scan, results of tests, and its validation.
Based on a list of desired measurements, a general processing path was prepared. Its methods are described in detail, including algorithms for body segmentation, characteristic points localization (body landmarking), and final specific measurements of girth, arc, and linear lengths (widths and heights). Furthermore, pseudo-code for the algorithms and 3D description of the characteristic points is included in Appendices A and B. The scanning procedure and posture recommendations are explained. Finally, test results and analyses are presented.
The validation dataset consisted of 40 subjects (21 male and 19 female volunteers) aged between 25 and 55 years, weighing from 55 to 105 kg and 150 to 205 cm tall. Each of the subjects was measured once manually and three times automatically by both our system and the Human Solutions system to check the repeatability of the consecutive measurements. The performance of the 3D anthropometric system was evaluated by comparing its output with the manual measurements treated as the best possible ground truth. The output was also compared with the results from a leading commercial 3D anthropometric scanning solution provided by Human Solutions GmbH. The accuracy of both systems was measured using both the relative percent and millimeter difference error. Consistency and absolute agreement between systems and manual measurements were evaluated using the Intraclass Correlation Coefficient (ICC) method. Finally, comparability between the manual and automated measurements was assessed according to ISO 20685. It appeared that for our system, only 5 out of 26 measurement types passed the highly demanding test from ISO 20685 and for the Human Solutions system, 3 out of 26 types passed the test. However, most of the accuracy, consistency, and absolute agreement results appeared to be on a satisfactory level according to garment design experts taking part in the project. Girth measurements were characterized by the highest degree of consistency and absolute agreement, while arc length measurements were still acceptable, but least satisfactory. The degrees of general consistency and absolute agreement between our system and manual measurements fell into the excellent interval of the ICC practical significance measure. These tests also showed high inter-rater agreement between the proposed system and the commercial one (the Human Solutions product). It appeared that the current version of the system can provide results relatively close to those of the state-of-the-art solution. However the system still needs further development and tests on a larger population, as it is not completely free of errors, particularly those resulting from improper body landmarking and inconsistent posture.
Comparison of semi-supervised and supervised approaches for classification of e-nose datasets: Case studies of tomato juices
2015, Chemometrics and Intelligent Laboratory Systems
Citation Excerpt :
Recently, a state-of-the-art clustering method—spectral clustering—has become a topic of interest. By constructing an undirected weighted similarity graph, spectral clustering utilizes spectrum of the graph Laplacian to obtain a low dimensional representation of the data, and then does clustering using classical methods such as k-means [41]. In our previous research [42], spectral clustering was found better than six conventional clustering methods (ISODATA, FCM, k-means, single linkage clustering, CL and Ward's).
Supervised classification, which is a fundamental classification approach for e-nose data, requires sufficient labeled data for training. However, sufficient labeled data requires extensive money, materials, energy and time. In this paper, a semi-supervised approach—Cluster-then-Label—that simultaneously uses labeled and unlabeled data to build a better classifier with fewer training data was introduced to deal with e-nose data for the first time. A novel clustering algorithm—spectral clustering—was also introduced to improve this semi-supervised approach. Three experiments—discriminating storage shelf life (SL), identifying pretreatments and authenticating juices, respectively—were conducted on cherry tomato juices using a PEN 2 e-nose, generating three datasets of different data structures. For each dataset, only 20% of data were selected for training. Classifications of the datasets by this semi-supervised approach and four supervised approaches (linear discriminant analysis (LDA), quadratic discriminant analysis, multi-class support vector machine and back propagation neural network) were compared. The results indicate that this spectral clustering based semi-supervised approach outperforms the supervised approaches in all cases. By using this semi-supervised approach, it is now possible to build reliable classifiers with only a few labeled data. It is also worth mentioning that this new approach takes no remarkable superiority over LDA. Thus, our next plan is to use more e-nose datasets for test.
E-nose combined with chemometrics to trace tomato-juice quality
2015, Journal of Food Engineering
Citation Excerpt :
Thus, in this paper, a state-of-the-art clustering approach – spectral clustering – is also introduced. By constructing an undirected weighted similarity graph on the data, spectral clustering utilizes the spectrum of the graph Laplacian to obtain a low dimensional representation of the data, and then does clustering using classical methods, such as k-means (Chen et al., 2011). This graph-theoretic based clustering method can be solved efficiently by standard linear algebra software and very often outperforms conventional clustering algorithms (Von Luxburg, 2007).
An e-nose was presented to trace freshness of cherry tomatoes that were squeezed for juice consumption. Four supervised approaches (linear discriminant analysis, quadratic discriminant analysis, support vector machines and back propagation neural network) and one semi-supervised approach (Cluster-then-Label) were applied to classify the juices, and the semi-supervised classifier outperformed the supervised approaches. Meanwhile, quality indices of the tomatoes (storage time, pH, soluble solids content (SSC), Vitamin C (VC) and firmness) were predicted by partial least squares regression (PLSR). Two sizes of training sets (20% and 70% of the whole dataset, respectively) were considered, and R² > 0.737 for all quality indices in both cases, suggesting it is possible to trace fruit quality through detecting the squeezed juices. However, PLSR models trained by the small dataset were not very good. Thus, our next plan is to explore semi-supervised regression methods for regression cases where only a few experimental data are available.
Comparison of spectral clustering, K-clustering and hierarchical clustering on e-nose datasets: Application to the recognition of material freshness, adulteration levels and pretreatment approaches for tomato juices
2014, Chemometrics and Intelligent Laboratory Systems
Citation Excerpt :
Recently, spectral clustering has been researched as a popular topic. By constructing an undirected weighted similarity graph on the data, spectral clustering utilizes the spectrum of the graph Laplacian to obtain a low dimensional representation of the data, and then does clustering using classical methods, such as k-means [34]. This graph-theoretic based clustering method is simple to implement, and it can be solved efficiently by standard linear algebra software and very often outperforms conventional clustering algorithms [35].
Various clustering algorithms have been developed since conventional hierarchical cluster analysis (HCA) and partitioning clustering algorithms have their own limitations and scopes of applications. However, in the area of e-nose where clustering is applied, the conventional algorithms (mostly HCA) still play a dominant role. In addition, comparison among different clustering methods or validation of clustering results was seldom mentioned. In this paper, we present a state-of-the-art clustering method – spectral clustering – and compare it with six conventional clustering methods: K-clustering (ISODATA, FCM and k-means) and HCA (single linkage, complete linkage and Ward's). Three external validation criteria – mutual information criteria (MI), precision and rand index (RI) – were used to evaluate clustering performances on three independent e-nose datasets. The spectral clustering outperforms with statistical significance (alpha = 0.05) the performance of other methods, and the single linkage presents the worst (unacceptable) clustering result. In addition, the proposed approach – cluster validation criteria in combination with majority voting – in a way makes clustering a semi-supervised classification technique. Using this approach it is possible to compare clustering based semi-supervised methods with classification methods to find which method is better for discrimination of a certain e-nose dataset.

View all citing articles on Scopus

View full text

3D human posture segmentation by spectral clustering with surface normal constraint

Abstract

Highlights

Introduction

Section snippets

Related works

Multiple similarities

Constrained spectral clustering

Experimental results

Conclusions

Signal Processing

Signal Processing

Signal Processing

International Journal of Machine Tools and Manufacture

On spectral clustering: analysis and an algorithm

Nonlinear Programming

From laser-scanned data to feature human model: a system based on fuzzy logic concept

Computer-Aided Design

Segmentation of human body parts using deformable triangulation

Spectral clustering with inconsistent advice

Building symbolic information for 3d human body modeling from range data

The Laplacian spectrum of a graph

SIAM Journal on Matrix Analysis and Applications

Automatic segmentation of 3d human body scans

On clusterings: good, bad and spectral

Journal of the ACM

An efficient k-means clustering algorithm: analysis and implementation

IEEE Transactions on Pattern Analysis and Machine Intelligence