Biofeedback for Gait Retraining Based on Real-Time Estimation of Tibiofemoral Joint Contact Forces

Biofeedback assisted rehabilitation and intervention technologies have the potential to modify clinically relevant biomechanics. Gait retraining has been used to reduce the knee adduction moment, a surrogate of medial tibiofemoral joint loading often used in knee osteoarthritis research. In this paper, we present an electromyogram-driven neuromusculoskeletal model of the lower-limb to estimate, in real-time, the tibiofemoral joint loads. The model included 34 musculotendon units spanning the hip, knee, and ankle joints. Full-body inverse kinematics, inverse dynamics, and musculotendon kinematics were solved in real-time from motion capture and force plate data to estimate the knee medial tibiofemoral contact force (MTFF). We analyzed five healthy subjects while they were walking on an instrumented treadmill with visual biofeedback of their MTFF. Each subject was asked to modify their gait in order to vary the magnitude of their MTFF. All subjects were able to increase their MTFF, whereas only three subjects could decrease it, and only after receiving verbal suggestions about possible gait modification strategies. Results indicate the important role of knee muscle activation patterns in modulating the MTFF. While this paper focused on the knee, the technology can be extended to examine the musculoskeletal tissue loads at different sites of the human body.

There is an evolving understanding that MTFF underloading, in addition to overloading, may also be an important factor in the onset and progression of knee OA. Saxby and colleagues [16] have recently shown that at 2-year post anterior cruciate ligament reconstruction (ACLR) the MTFF are lower during walking, running, and side stepping compared to healthy controls. Additionally, healthy controls and ACLR without meniscal damage that have larger MTFF during walking have healthier cartilage and bone [17], while lower loading of MTFF at 2-year after ACLR may lead to greater risk of future onset of radiographic knee OA [18]. Currently it is unclear whether increasing or decreasing MTFF may offer therapeutic benefit; however, modifying gait to decrease MTFF has been proposed to slow the disease progression in those with established knee OA [19]- [23].
The KAM has been positively correlated with the shape of MTFF during walking [7], [39], but this is not always This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/ the case [3], [4], [37], [40]. Studies based on instrumented tibiofemoral prostheses [41] have shown that decreases in the KAM do not necessarily result in decreases to MTFF [37], and that only small changes occurred in the MTFF for gait patterns designed to reduce KAM [42]. The KAM-MTFF relationship probably breaks down due to large muscular contributions to MTFF that are similar in magnitude to the contributions from external loads [4], [6]. Thus, it is essential to consider both external joint moments and muscle forces in order to estimate MTFF accurately [4], [6].
OpenSim [43] is a popular musculoskeletal modelling software for the analysis of the human motion that can account for individual variations in anatomy and movement [44]. However, OpenSim estimates muscle forces via optimisationbased algorithms (e.g. static and dynamic optimization) [45] that cannot account for alterations in muscle excitation patterns evident in individuals with knee pathologies during gait [46]. Alternatively to optimization-based methods, electromyogram (EMG)-driven neuromusculoskeletal (NMS) models [47] use experimentally measured muscle excitations to estimate muscle forces, and have been able to correctly predict MTFF and lateral knee contact force (LTFF) measured directly from instrumented knee implants [41], [48], [49].
In this study, a calibrated EMG-informed NMS model, i.e. CEINMS [50], was used in combination with OpenSim to estimate MTFF in real-time. Subjects were asked to modify their gait pattern in order to modify the model estimate of MTFF which was provided as visual biofeedback. Because of the novel approach, it was unclear whether subjects using this technology would be able to modify their gait strategy in order to manipulate their MTFF on-demand or if they would require suggestions. Also, it was unclear how subjects would change their gait strategy to alter their MTFF and if a common preferred strategy would emerge across the different subjects.
The aims of this paper were (1) developing a software based on both OpenSim and CEINMS to estimate knee contact forces in real-time using musculoskeletal modelling techniques, (2) comparing real-time and offline estimates, and (3) exploring the use of musculoskeletal tissue loading as visual biofeedback for gait retraining by evaluating the response of 5 different subjects in distinct experimental conditions.

B. Software Description
We developed software (C++) based on CEINMS [50] and OpenSim [43], [51] that uses a marker-based Vicon motion capture system, force plates, and EMG to estimate MTFF in real-time (Fig. 1). The MTFF was displayed on a screen to provide augmented visual sensory feedback to the subject walking on the treadmill. The software has five different components: (1) anatomical model, (2) real-time estimation of joint angles and moments [51], (3) real-time estimation of tibiofemoral contact forces, (4) musculotendon parameter calibration, and (5) biofeedback visualization. These are briefly described below.

C. The Anatomical Model
The generic gait2392 OpenSim model [43] was first modified to allow calculation of external adduction/abduction moments about lateral and medial condyle contact points [40] and then scaled to fit the anthropometry of each participant. The hip joint centers were calculated using regression equations [54], while knee and ankle joint centers were calculated from markers on the femoral epicondyles and ankle malleoli, respectively. Hip, knee, and ankle joint centers were then used to scale the anatomical model to the individual's segments lengths using the OpenSim scale tool. The tibiofemoral intercondylar distances were scaled proportionally to femoral epicondyles distance [5], [6]. Further personalization of MTU insertion points and muscle paths was not performed. In the scaled anatomical model, tendon slack length and optimal fiber length of each MTU were estimated using the anthropometric scaling method developed by Modenese and colleagues [55].

D. Real-Time Estimation of Joint Angles and Moments
Vicon Nexus software automatically reconstructed and labelled markers in real-time. Joint angles and moments were estimated through OpenSim's inverse kinematics and inverse dynamics algorithms (Fig. 1). A state-space filter implementation of a 2 nd order low-pass Butterworth filter [56], with a cut-off frequency of 15 Hz [51], was used to filter both the GRFs and joint angles. An extended description is provided in [51].

E. Real-Time Estimation of Tibiofemoral Contact Forces
The tibiofemoral contact forces were estimated via CEINMS [50] in EMG-driven mode [47], [53]. This required, as inputs, MTU kinematics and muscle excitations. Multidimensional cubic B-splines, predefined using the OpenSim muscle analysis tool, calculated the MTU kinematics as a function of the joint angles [57] in real-time. Muscle excitations were calculated in real-time from raw EMGs that were high-pass filtered (30Hz), full-wave-rectified, low-pass filtered (6Hz) using a state-space 2 nd order Butterworth filter [47], [56], and amplitude-normalized using the maximal values extracted from MVC trials acquired offline previously. Medial (F MC ) and lateral (F LC ) tibiofemoral contact forces were calculated as follows [6], with,

F. Model Calibration
Prior to real-time capability, each participant's MTU parameters were calibrated using CEINMS [50]. The CEINMS calibration is an offline procedure whereby the MTU parameters are optimized to minimize the summed square error between predicted and experimental joint moments whilst minimizing the peak magnitude of both MTFF and LTFF [48]. The calibration was performed using a set of 3 walking stance phases recorded and processed offline [58] before starting the realtime protocol. The list of calibrating parameters and boundary conditions is the same used in [50]. The total time for initial data collection, offline data processing and calibration was approximately 1.5 hours. After calibration, the system was ready to estimate the contact forces in real-time.

G. Biofeedback Visualization
The real-time estimated MTFF was visualized as a continuous time series graph plotted on a screen placed in front of the subject (Fig. 2). Each new point was added to the rightmost part of the visualization window scrolling previous data points to the left, and permitted visualization of the last 10 seconds of data (blue line in Fig. 2). The target was calculated as percentage of the moving average of the previous 15 medial contact force peaks and was also plotted onto the graph (red line in Fig. 2). This feedback modality was chosen to guide and promote the subject to further decrease or increase their MTFF.

H. Experimental Protocol
Participants were given a period of familiarization with the treadmill and asked to select their preferred walking speed, which was maintained for the remainder of the session. The real-time protocol consisted of five different walking conditions. First, "baseline walking" was recorded for 1 minute and the mean of the peak values of MTFF was subsequently calculated. The subjects were then provided with the visual feedback of their MTFF and a target equal to 90% of the moving average of their MTFF peak. Subjects were then asked to decrease their MTFF driving the displayed value below the target ("decrease" condition). The subjects were then asked to try different strategies while trying to maintain a symmetric gait pattern; however in this instance specific instructions were not provided. The researchers verbally confirmed whether the strategy of the subjects was being effective or not when compared to their baseline level. When subjects found an effective gait strategy for reducing MTFF, they were asked to sustain it for 1 minute. Then a washout period of 5 minutes of normal walking was performed. Following washout, the same protocol was repeated asking the subjects to increase the medial force, using a target of 110% of MTFF peak displayed on screen ("increase" condition). After these self-guided gait trials, a washout period of 5 minutes of normal walking was performed, and then began a series of researcher-suggest gait patterns. The subjects were asked to decrease and then to increase their MTFF following the same protocol described above.
However, this time specific verbal suggestions were provided, which we refer to as "decrease with suggestions" and "increase with suggestions" conditions. The following gait strategies, previously found to modify external knee loading [31], [34], [59], were presented to the subjects in both the "decrease with suggestions" and "increase with suggestions" conditions: walking with toes pointed in [30], [31] or out [31], [32], increasing or decreasing side-to-side trunk sway [31], [33], using longer or shorter strides [31], [34], loading the inside or outside of the foot [34], changing the step width [35], [36], and changing the knee alignment [25], [37], [38], i.e. more medial or lateral knee positioning. Importantly, the suggested gait modifications were used as guidance to subject in order to find more effective gait modification strategies, but were not imposed.  Comparison between tibiofemoral contact forces predicted offline (solid blue) and in real-time (dashed red) for 5 different subjects. For each subject, medial and lateral tibiofemoral contact forces from 50 consecutive gait cycles of baseline trials were normalized to body weight (BW), time normalized, and averaged to create representative mean curves. The blue and red points represent the maximum mean value of offline and real-time tibiofemoral contact forces, respectively. Shaded area represents ± 1 standard deviation.
between calcaneus markers at subsequent heel strikes), hip internal rotation, knee flexion angle, knee flexion moment, knee adduction moment, the external load contribution to the medial force, the MTU contribution to the medial force, the mean activation of the MTUs surrounding the knee, and the directed co-contraction ratio (DCCR) calculated from knee muscle activations [6], [46]: where Act ext and Act flex are the mean activation of the knee extensors and flexors, respectively. For each variable, multiple paired t-tests were used to compare each condition to baseline.
Finally, examples of how subjects searched for an optimal strategy in order to increase or decrease their MTFF were reported.

III. RESULTS
The estimation of the MTFF was similar in real-time and offline, with an average (mean ± standard deviation) timealigned RMSE of 0.125 ± 0.069 BW and a time delay of 37.7 ± 1.1 ms (Table I and Fig. 3). Differences between realtime and offline estimation of LTFF were larger, resulting in an average time-aligned RMSE of 0.192 ± 0.60 BW and   (Table I and Fig. 3). The Bland-Altman analysis (Fig. 4) reported a bias of 2.30% and agreement limits of -3.36 and 7.95% for the MTFF, and a bias of 16.13% and agreement limits of -1.32 and 33.59% for the LTFF.
The total time delay introduced by the real-time computation was approximately 115 ms. This included 4 ms of data processing in Vicon Nexus, 57 ms to calculate contact forces (including inverse kinematics, inverse dynamics, MTU kinematics, filtering, and muscle force estimation), 37 ms of time delay caused by the filtering phase shift (Table I), and 17 ms of refresh time of the 60 Hz monitor used to provide the visual biofeedback.
No subjects were able to decrease their MTFF without suggestions, but three subjects significantly decreased it following suggestions (Table II). All of the subjects were able to increase the MTFF, where for subjects 1, 2, and 3, the greatest change occurred after being provided suggestions. Fig. 5 summarizes the results across subjects and conditions and Fig. 6 shows examples of strategy search to decrease and increase MTFF. For each of the final steady-state conditions, the MTFF peak was calculated for each gait cycle. The time points corresponding to the peaks of the MTFF were then used to analyse a variety of spatiotemporal, kinematic, kinetic, and muscular variables. The symbol ' * ' represents significant differences (p < 0.05 ÷ 4, i.e. with Bonferroni correction) from baseline. Error bars indicate 95% confidence intervals.

IV. DISCUSSION
We estimated MTFF and LTFF in real-time at 200 Hz, using a scaled OpenSim model and CEINMS in EMG-driven mode [50], [53]. EMG-driven NMS models have been previously used to estimate, in real-time, the MTU forces spanning knee and ankle [61], [62]. However, this is the first time they have been used to estimate, in real-time, articular loading from motion capture data. The real-time estimated MTFF was used as visual biofeedback to explore whether healthy individuals could modify their gait patterns to manipulate their MTFF. All subjects were able to increase their MTFF, whereas three subjects were able to decrease the MTFF after being provided with suggestions on how to change their gait patterns ( Fig. 5 and Table II).
Differences between real-time and offline tibiofemoral estimates were limited, although larger errors were present in the real-time LTFF when compared to offline estimation (Table I and Fig. 3). Errors were caused by the combined effect of signal distortions associated with the real-time filtering of force plates, EMG, and kinematics data [51], [63]. Also, muscle forces contribute more to LTFF than MTFF [6], thus explaining larger errors (Table I and Fig. 3) and the inter-subject variability evident in the Bland-Altman analysis (Fig. 4).
As measuring knee contact forces in-vivo is challenging, direct validation of real-time estimated MTFF on healthy subjects using direct experimental measures was not performed. Offline estimation of MTFF using CEINMS was previously validated in [48] using data from instrumented knee implants [41]. Despite the differences between instrumented knee implants and healthy knees, CEINMS accounts for personalized excitation patterns and the OpenSim anatomical model can be adjusted to include variations in knee geometry and alignment [44], [48]. Thus, we considered CEINMS estimations of internal forces to be valid also on healthy subjects, and the offline CEINMS estimates of MTFF were used as the criteria for real-time verification in this study (Table I, Fig. 3, and Fig. 4). Some subjects presented values of MTFF that were higher in the first peak than second peak for their baseline gait (Fig. 3). However, variability in MTFF across subjects has been shown using instrumented knee implants [39], so it is not surprising to observe such variability in healthy individuals as well.
It could be argued that multiple gait strategies could be first tested in laboratory and then analyzed offline to identify the best strategy to manipulate MTFF. While this is possible, subjects may not be able to replicate novel gait patterns across sessions, or may find a combination of strategies more successful than a single strategy or more suited to their personal walking style. Also, while our study focused only on increasing and decreasing MTFF, specific MTFF values could be used as biofeedback target, which could be extremely difficult, or impossible, to perform offline.
The association between action and biofeedback improves as the biofeedback delay decreases [64]. Subjects were able to visualize the MTFF biofeedback as a continuous time series graph with a total delay of 115 ms from the data measurement. While the time delay of our system may not be optimal (i.e. less than 75 ms [64]), it was considered an improvement from visualizing the MTFF peak only, which would require to complete the current gait cycle before visualizing the new value [34], thus introducing a larger delay.
While previous gait retraining studies have used kinematic [23], [38], [65] and kinetic [21], [23] variables as biofeedback, we used the MTFF. The MTFF arises due to the complex changes in and interactions between whole-body kinematics, kinetics, and muscle coordination. No subject achieved a MTFF reduction when provided with only verbal and visual biofeedback, and the three subjects that did reduce their walking MTFF only did so when suggestions were given on gait modification strategies. Studies on individuals with instrumented implants also showed that alternative gait A representative example of a strategy search to (a.) decrease MTFF (Subject 2, "decrease with suggestions" condition), and (b.) increase MTFF (Subject 4, "increase" condition). . For each MTFF peak percentage variations from baseline were plotted as bar charts. High values for changes in DCCR (b.) were caused by baseline values close to zero. The red shaded area represents ±1 standard deviation of MTFF peaks from 50 consecutive gait cycle of baseline trials. patterns did not lead to statistical significant reductions of MTFF [42], or that magnitude reductions were present in the second MTFF peak only [24], [37]. Interestingly, in the "decrease with suggestion" condition, subjects 1 and 5 were unable to decrease their MTFF lower than baseline (Table II and Fig. 5) and adopted altered gait that had the opposite effect, resulting in increased MTFF (Table II and Fig. 5). This may be a reflection of the study design and the subjects may have felt necessary to follow some of the provided suggestions even if they resulted in increased MTFF. Given the complex nature of the biofeedback, it is also plausible that subjects were unable to associate their action to the visual biofeedback. However, all subjects increased their MTFF, even without suggestions, indicating that a correct association between action and visual biofeedback was present. The inability of the subjects to reduce the MTFF without suggestions could be related to the visual biofeedback target magnitude, which was calculated from the peak of the MTFF of 15 previous gait cycles. Future studies investigating MTFF biofeedback should also consider evaluating what type of visual biofeedback (e.g. continuous time series, peaks, bars) works best, similarly to what has been done with KAM biofeedback [23], [29]. Finally, learning effects of the decrease strategy in the increase conditions cannot be excluded, as conditions were not randomized.
Gait strategies to modify the MTFF differed across subjects (Fig. 5). Previous gait retraining studies identified increased step width, medial knee thrust, increased hip internal rotation, and trunk sway as common strategies to decrease KAM [22], [66], even though large variations among subjects were reported [26], [27]. However, our model included joint kinematics, kinetics (e.g. KAM), and EMG-based MTU forces when estimating the MTFF. Since muscle activation patterns differ across tasks [67] and individuals even when the joint kinematics and kinetics are the same [68], [69], MTFF modulation strategies were expected to be different between different individuals. When using a large number of subjects, common kinematic patterns related to MTFF variations will likely emerge, but large variations would still be expected in the final strategies that individuals adopt.
A combination of kinematic, kinetic, and muscle activation changes were used by the subjects to manipulate their MTFF (Fig. 5). In the "decrease with suggestion" condition, subject 2 increased trunk sway and decreased hip internal rotation as their effective MTFF reduction strategy ( Fig. 5 and Fig. 6). Interestingly, in the "increase" condition, subject 2 increased their MTFF (Fig. 5) by increasing their total muscle cocontraction, but maintained consistent KAM and knee flexion moment. Conversely, subject 4 used a different gait strategy to successfully increase MTFF by combining changes in stride length, hip internal rotation, knee flexion angle, trunk sway, and muscle activations and co-contraction ( Fig. 5 and Fig. 6). Interestingly, subject 4 increased their MTFF by substantially increasing their total muscle activation, but reduced external loading with reduced KAM and knee flexion moment. Importantly, the use of the EMG-driven neural control solution in our NMS model was essential to identify changes in muscle forces due to variation in activation patterns and co-contraction, which static optimization based methods cannot correctly predict [70].
In this preliminary analysis, a reduction of MTFF did not always result in an increase of LTFF or vice-versa (Table II). However, LTFF results should be interpreted cautiously, as larger limits of agreement and bias are present in the LTFF estimates when compared to MTFF (Fig. 4). Also, variation in hip and ankle contact forces should be analyzed to better understand how variation in MTFF could affect other joints [71].
While this study aimed to evaluate whether the MTFF could be used for biofeedback, we recognize that subjects may struggle to adopt different kinematic patterns during gait, and that verbal suggestions may be essential to achieve MTFF reductions. Subjects were free to adopt any symmetrical gait pattern, but, as also observed by van den Noort and colleagues [23], this could result in extreme kinematics changes that are unsuitable for daily living activities. Furthermore, subjects were not allowed to walk with their new gait for a long period of time; thus, if they adopted a kinematic change, it is unlikely that they were walking with the most efficient coordination strategy [72]. Also, while multiple optimal solutions to decrease or increase MTFF may exist, subjects may have only explored a limited portion of the solution space. Computer simulations [25] and/or data-driven approaches [20] to suggest minimal gait kinematic changes could mitigate extreme gait patterns and address the limited time subject have to explore potential solutions.
This study has limitations that should be considered. Although scaled to the subjects' dimensions and mass, the anatomical model was based on a generic template. This potentially produces less accurate estimates compared to subject-specific models [48]. Also, the inter-condyle distance in the knee model was scaled proportionally to the femoral epicondyles, possibly resulting in MTFF magnitude errors as the contact model is sensitive to the contact geometry [44,49]. Further studies should include subject specific knee alignment [44], geometry [48], and kinematics [73] to improve the estimation of MTFF.
Prior to using CEINMS in real-time, it was necessary to scale the anatomical model and calibrate the MTU parameters. These operations are time consuming, but they were performed within the same session to avoid marker and EMG electrode repositioning errors. To reduce the time burden on the subjects, we limited the CEINMS calibration time to less than 1 hour using the stance phase, rather than the full gait cycle. This potentially resulted in sub-optimal MTU parameters, although due to the complex solution space of the NMS system it is not at all clear that more calibration time will necessarily improve MTU parameter estimates. Future improvements will focus on the development of faster calibration routines able to operate "in the background" to automatically calibrate and update MTU parameters during the real-time execution. Another limitation regards the use of the split-belt treadmill. The treadmill forced the subjects to maintain a minimum step width to avoid treading on the junction between the moving belts. Also, while the subjects were asked to maintain a symmetric gait, a measure of that symmetry was not provided as biofeedback. Thus, it is not possible to exclude compensatory changes in the contralateral limb, or in other joints. However, these should be considered in future clinical applications of this technology. Finally, while we acknowledge the listed limitations, we also stress the exploratory nature of this study and its main goal to assess whether people could use the MTFF as biofeedback to modify their gait.

V. CONCLUSIONS
This study presented the real-time estimation of MTFF and its use as a visual biofeedback variable for gait modification. The MTFF was estimated from joint angles and moments calculated through real-time inverse kinematics and inverse dynamics analyses, respectively, that were used within the CEINMS framework in EMG-driven mode [47], [50], [53].
The current study is not meant to be an extensive exploration of possible gait modification strategies, and further investigations into the use of MTFF for gait retraining involving a larger number of subjects is required before reaching general conclusions. Despite the modest sample size, these results show the importance of personalized NMS models that account for variations in movement, external joint loading, and muscle activation patterns across different individuals. Accounting for this variability is of particular importance when studying individuals with altered activation patterns [46], [74], joint pathology [75] or who have received specialized training [76]. Finally, the present research focused on the knee joint, yet the computational system is fully generic and could be applied to other musculoskeletal structures. While some limitations need to be addressed before translating this technology to clinical practice, merging real-time estimation of forces inside the human body with subject-specific anatomical models [77] can provide clinicians with accurate and relevant estimates of musculoskeletal loading, and has the potential to revolutionize the current use of post-hoc gait analysis for rehabilitation and training.