Pose-Based Gait Analysis for Diagnosis of Parkinson’s Disease

Abstract

Parkinson’s disease (PD) is a neurodegenerative disorder that is more common in elderly people and affects motor control, flexibility, and how easily patients adapt to their walking environments. PD is progressive in nature, and if undetected and untreated, the symptoms grow worse over time. Fortunately, PD can be detected early using gait features since the loss of motor control results in gait impairment. In general, techniques for capturing gait can be categorized as computer-vision-based or sensor-based. Sensor-based techniques are mostly used in clinical gait analysis and are regarded as the gold standard for PD detection. The main limitation of using sensor-based gait capture is the associated high cost and the technical expertise required for setup. In addition, the subjects’ consciousness of worn sensors and being actively monitored may further impact their motor function. Recent advances in computer vision have enabled the tracking of body parts in videos in a markerless motion capture scenario via human pose estimation (HPE). Although markerless motion capture has been studied in comparison with gold-standard motion-capture techniques, it is yet to be evaluated in the prediction of neurological conditions such as PD. Hence, in this study, we extract PD-discriminative gait features from raw videos of subjects and demonstrate the potential of markerless motion capture for PD prediction. First, we perform HPE on the subjects using AlphaPose. Then, we extract and analyse eight features, from which five features are systematically selected, achieving up to 93% accuracy, 96% precision, and 92% recall in arbitrary views.

1. Introduction

The motor skills required to walk without difficulty in different environments are vital for good quality of life. As with gait in general, these motor skills take many years to develop. Parkinson’s disease (PD) is a neurodegenerative disorder that is more common in elderly people and affects motor control, flexibility, and adaptability [1]. PD is caused by a decline in the production of dopamine, which negatively affects patients’ speech coherence and motor control. However, as is the case with most neurodegenerative disorders, the root cause of PD is unknown [2].

PD is progressive in nature, and if undetected and untreated, symptoms grow worse over time. PD has a stage in which the symptoms required for correct diagnosis are absent or difficult to detect [3]. The main visible signs of PD are reduced movement (hypokinesia) and loss of movement (akinesia) [1]. These symptoms can be observed as reduced range of motion, tremors in the upper limbs, body rigidity, difficulty in walking, reduced step length, and slower walking speed [4]. Early prediction could aid in treatment and therapy to slow down the degenerative process. This could prevent degradation to a clinical stage, thereby improving quality of life [3].

PD affects the central nervous system by reducing its control of motor neurons. This results in the condition referred to as dysphonia, which is the inability to form words coherently. Hence, PD can be detected using voice features. However, speech features obtained in unconstrained environments are often noisy [2]. Fortunately, PD can be detected early using gait features since the loss of motor control results in gait impairment. Early signs of PD shown in gait include reduced arm swing amplitude and symmetry, reduced gait speed, reduced step length, and increased time spent in the double-support phase [5].

Predicting PD from gait requires an appropriate gait capture technique. In general, techniques for capturing gait can be categorized as vision-based or sensor-based [6]. Sensor-based techniques are mostly used in clinical gait analysis and are regarded as the gold standard for PD detection. The sensors used can be wearable, such as inertial measurement units (IMUs) [7]; or non-wearable, such as floor sensors [5]. The main limitation of using sensor-based gait capture is the associated high cost and the technical expertise required for setup. In addition, subjects’ consciousness of worn sensors and being actively monitored may further impact their motor function.

The studies focusing on PD prediction can be grouped into two main categories based on the features used: studies using non-perceivable features, and those using perceivable features. Examples of non-perceivable features are those extracted from magnetic resonance imaging (MRI) scans of the brain. Such features could be analysed by experts, or prediction could be performed using deep neural networks [8,9]. PD prediction has also been performed using brain electroencephalogram (EEG) signals [10]. The limitations of these features include the requirement for expert knowledge and the level of interpretability of results obtained.

Perceivable features for PD prediction include voice, handwriting, face, and gait. Åström and Koker [2] proposed a neural network approach to predict PD based on voice features. However, speech features obtained in unconstrained environments are often noisy. Handwriting features have also been used in PD prediction [11,12]. In recent research [13,14], the face has been shown to contain PD-related features. Of all perceivable features, gait can be obtained unobtrusively in videos, and it has also been used for PD prediction [15,16,17], including a computer-vision approach to classifying slowness of movement based on arm motion captured by a smartphone camera [18].

Recent advances in computer vision have enabled the tracking of body parts in a markerless motion capture scenario via human pose estimation (HPE) algorithms such as AlphaPose [19] and OpenPose [20]. Although markerless motion capture has been studied in comparison with gold-standard motion capture techniques [21], it is yet to be used in the prediction of neurological conditions such as PD. Hence, in this study, we extract PD-discriminative gait features from raw videos of subjects and demonstrate the potential of markerless motion capture for PD prediction.

The main contributions of this study are as follows. First, we propose eight features for the diagnosis of Parkinson’s disease from videos. We explore pose estimation for markerless motion capture and perform feature extraction from the human pose estimation time series. Then, through systematic analysis and several experiments, we propose features that can aid in PD diagnosis from arbitrary views. We note that vision-based approaches can be based only on the detection of perceivable gait impairments, most of which can also be found due to ageing or other conditions. Hence, the proposed technique could serve the purpose of screening. If gait impairment is detected, further clinical investigations may be required to determine if it is due to PD or other reasons.

2. Methodology

In this section, we describe the approach taken for gait capture, feature extraction, and Parkinson’s disease prediction (Figure 1).

2.1. Video Capture and Preprocessing

Data are obtained online from public domain videos of people walking, and each video is manually labelled with the apparent PD status (Figure 2). Due to the unavailability of public video datasets for PD prediction, most of the PD videos are obtained from public domain recordings of PD therapy sessions and subjects walking in unconstrained environments. The data collected consist of 167 subjects: 93 healthy and 74 with PD. The videos are segmented to ensure there is only one subject in each video. Videos that feature multiple subjects walking too closely are discarded.

2.2. Human Pose Estimation

To track participants’ motor performance in videos without using physical markers, we perform human pose estimation (HPE) using AlphaPose [19]. For a walking sequence of T seconds in which we track K body keypoints in a video with frame rate f, HPE outputs a multivariate time series $\left\{(x_t^i,y_t^i,p_t^i)\right|i=1,\dots ,K;t=1,\dots ,fT\}$, where $(x_t^i,y_t^i)$ represent the 2D coordinates, and $p_t^i$ is the detection probability of the $i^{th}$ keypoint in frame t. Due to the noisy nature of HPE data, smoothing is performed using a Savgol filter with a window size of 5. The motion of two subjects is illustrated in Figure 3.

2.3. Feature Extraction

The early visible symptoms of PD include changes in posture, tremors, reduced arm swing, arm swing asymmetry, and reduction in step length and gait speed. Hence, we extract eight features from the body keypoint coordinates obtained as described in Section 2.2. If we let $u=\{x_t^i,y_t^i\}$ and $v=\{x_t^j,y_t^j\}$ be the coordinates of two adjacent keypoints i and j in frame t, the Euclidean distance $D_E$ between i and j can be found as:

$$D_E(i,j)={[{(x_t^i-x_t^j)}^2+{(y_t^i-y_t^j)}^2]}^{(1/2)}$$

(1)

Suppose j is a central keypoint, such as the elbow joint, with i and k as its adjacent nodes; we can form vectors from $ji$ and $jk$ in order to obtain the range of motion at j as:

$$\theta \left(j\right)=co{s}^{-1}\left(\frac{ji·jk}{\left|ji\right|\left|jk\right|}\right)$$

(2)

We obtain a measure of the steadiness of a body keypoint j as the standard deviation of the moving differences of j:

$$\mathcal{S}\left(j\right)=std\left[\{p_{t+1}^j-p_t^j|t=1,...,F\}\right],\mathcal{S}\left(j\right)\in {\mathbb{R}}^{F\times 2}$$

(3)

where $p_i^j\in {\mathbb{R}}^F$ is a vector representing the 2D positional coordinates of keypoint j in frame t, and F is the total number of frames. Hence, as shown in

Table 1. Description of features extracted from HPE.

Feature	Related PD Symptom	Visual Perception	Computation
posture	hunched posture	nose-to-foot distance	mean $D_E(nose,toe)$ *
arm steadiness	arm tremors	wrist steadiness	mean $\mathcal{S}\left(wrist\right)$ *
arm swing	reduced arm swing	nose-to-wrist distance	mean $D_E(nose,wrist)$ *
arm swing symmetry	arm swing asymmetry	uneven arm swings	mean $D_E(nose,LWrist)/D_E(nose,RWrist)$
step height	reduced step height	ankle-to-ground distance	$max\left(ankle\right)$ *
step	reduced step length	left-to-right ankle distance	mean $D_E(LAnkle,RAnkle)$
cadence	reduced cadence	lower step frequency	$n_{steps}/F$ (see Figure 4)
speed	reduced gait speed	slower movement	$step\times cadence$

Mean values are taken across F frames. * The side (left or right) closest to the camera is selected for each frame. All values are normalized with the total body height of the subject.

, subject posture, arm steadiness, arm swing, arm swing symmetry, maximum step height, mean step length, cadence (steps per minute), and gait speed (cycles per minute) are obtained from the keypoint coordinates obtained via pose estimation. As illustrated in Figure 4, the number of steps $n_{steps}$ taken by the subject in the video is estimated by counting the peaks of the step length. To compute the peaks, we take the step length values per frame as a time series. We then find all local maxima by a comparison of neighbouring values.

3. Experiments and Results

3.1. Experiment Setup

Data augmentation was performed in two ways. First, we flipped the original motion time series obtained via pose estimation to simulate the backward movement of the subjects. The flipped motion time series was then appended to the original, thereby doubling the length of the time series. Then, each sequence was divided into subsequences with a uniform length of 180 frames, which is equivalent to 6 s each (assuming a frame rate of 30 frames per second) (

Table 2. Subject distribution in the dataset.

	Total	Front View		Side View
		Healthy	Parkinsonian	Healthy	Parkinsonian
Number of subjects	167	67	48	26	26
After augmentation	974	314	360	92	208

). Classification was performed using a Random Forest classifier with 250 trees on the front view, side view, and all views. Each view was represented by a feature matrix $X\in {\mathbb{R}}^{N\times f}$ and a label vector $\mathbf{y}\in {\{0,1\}}^N$, where N is the number of samples and f is the number of features. Apart from accuracy, the precision, recall, and F1 score are also reported. Analysis of the features extracted in Table 1 is presented in Appendix A.

3.2. Feature Selection

For each view, experiments were performed to select the best features based on feature importance. We use a stratified 5-fold cross-validation. In each run, the least important feature was recursively removed until the optimum number of features were selected based on cross-validation scores (Figure 5). For each view, the number of features was selected from all eight features such that there was no significant reduction in overall accuracy, thereby determining the important features for each view.

Table 3. Feature selection.

Feature	Feature Ranks by View				Selected Features by View
	Front	Side	All	Average	Front	Side	All
posture	7	6	7	7	×	✓	×
arm steadiness	4	7	6	6	✓	✓	✓
arm swing	1	3	1	2	✓	✓	✓
arm swing symmetry	5	8	5	6	×	×	×
step height	3	4	3	3	✓	✓	✓
step length	2	2	2	2	✓	✓	✓
cadence	8	5	8	7	×	✓	×
gait speed	6	1	4	4	✓	✓	✓
number of selected features					5	7	5

shows the feature rankings and selection for each view. For the front view, a total of five features were selected, excluding the posture, arm swing symmetry, and cadence. This is expected, as these features are not easily obtained in the front view. On the other hand, all features except the arm swing symmetry were selected in the side view. The arm swing symmetry, which is the ratio of the swing of the left arm to the swing of the right arm, was not selected in any view. This could be because of its high correlation with the arm swing. Lastly, we found that the five features common to the front and side views were selected as important for all views, namely, arm steadiness, arm swing, step height, step length, and gait speed. Across all views, the step length and arm swing ranked highest, while posture and cadence ranked lowest among the features.

3.3. Experimental Results

This section shows the results of experiments performed to evaluate the selected features and the effect of smoothing on the pose estimation output. For each of the views, the dataset was split into a training set (75%) and a testing set (25%) and fitted on a Random Forest classifier with 250 trees. The summary of the results is shown in

Table 4. Results summary of PD prediction.

View	Train/Test	Features	Smooth	Acc	Healthy			Parkinsonian
					Precision	Recall	F1-Score	Precision	Recall	F1-Score
		8	×	95%	0.95	0.95	0.95	0.95	0.95	0.95
Front	505/169	8	✓	95%	0.95	0.95	0.95	0.95	0.95	0.95
		5	×	95%	0.95	0.95	0.95	0.95	0.95	0.95
		8	×	97%	1.00	0.92	0.96	0.96	1.00	0.98
Side	225/75	8	✓	97%	1.00	0.92	0.96	0.96	1.00	0.98
		7	×	97%	1.00	0.92	0.96	0.96	1.00	0.98
		8	×	92%	0.88	0.92	0.90	0.94	0.92	0.93
All	730/244	8	✓	93%	0.90	0.94	0.92	0.96	0.93	0.94
		5	×	93%	0.88	0.94	0.91	0.96	0.92	0.94

.

As shown in Table 4, smoothing shows no effect on the accuracy achieved in the front and side views. However, it results in a slight improvement in the accuracy and F1-scores achieved for all views. We observe that conventional filters such as the Savgol filter used in this study may not be suitable for smoothing human motion time series obtained via pose estimation, since important motion data may be distorted. In addition, using five features (arm steadiness, arm swing, step height, step length, and gait speed) on the front view yields the same results as using all features with smoothing. Similarly, excluding the arm swing symmetry feature on the side view still yields results similar to using all features with smoothing. Lastly, when all views are considered, the arm steadiness, arm swing, step height, step length, and gait speed prove sufficient for differentiating a healthy from a Parkinsonian gait.

4. Conclusions and Future Work

In this study, we proposed gait features extracted from videos using markerless motion capture techniques for PD detection. First, we performed HPE on the subjects using AlphaPose. The results of the experiments performed in this study suggest that PD can be predicted in videos using markerless motion capture with up to 93% accuracy from arbitrary views, 95% in the front view, and 97% accuracy in the side view. This is most likely because most PD-related gait features can be more easily observed in the side view of gait. Most of the extracted features show more discriminative ability in the side view, especially gait speed, step length, and arm swing. As expected, the five features common to the front and side views are selected as important for all views, namely, arm steadiness, arm swing, step height, step length, and gait speed. We propose that these five features can be used for PD prediction in arbitrary views.

Although this study has used a modest-sized self-collected dataset, the results show the potential of using markerless motion capture for the early detection of Parkinson’s disease. We note that vision-based approaches can be based only on the detection of perceivable gait impairments. Hence, further clinical investigations may be required to determine if a patient suffers from PD or gait impairment due to ageing or other conditions. Notwithstanding, the main advantage of this approach is its ease of use, potential reduction of cost, reduction of waiting times, and reduction of burden on trained clinicians. Future research will focus on investigating more features to detect different phases of Parkinson’s disease progression.