Innovative Multi Vibrotactile-Skin Stretch (MuViSS) haptic device for sensory motor feedback from a robotic prosthetic hand

In the field of upper limb prosthetics, the incorporation of sensory feedback is critical to cognitive processes and behavior. Studies have demonstrated that haptic feedback improves amputees’ control over their prostheses. This study presents the development of the MuViSS (Multi Vibrotactile-Skin Stretch) haptic device, which is worn on the wrist and forearm and provides sensory-motor feedback from a robotic prosthetic hand. An innovative feedback strategy is presented that has not been explored in the existing literature. By combining two already established strategies – namely, stretching the skin in conjunction with proprioception and incorporating cues on contact – the research offers an unexplored approach to sensory feedback. Adaptations were made to a commercially available Taska prosthetic hand to integrate sensors and capture data for haptic feedback. Two classes of tests performed on non-amputee subjects have shown promising efficacy and performance. A first class of tests, designed to assess the effectiveness of MuViSS feedback, was conducted with five participants, testing each feedback separately. In order to evaluate the effectiveness of the entire system, tests were also performed on nine subjects with MuViSS and the prosthetic hand being controlled. They allowed the comparison of the MuViSS feedback with the classical force feedback by vibration and with the condition without haptic feedback. The results showed that the new feedback solution was able to provide size and stiffness information in the absence of vision. In addition, the feedback improved the performance of a motor task, specifically grasping a marble, with vision. The study demonstrates that the system has the potential to improve control, enhance performance, and positively impact the user’s overall experience when operating prosthetic devices.


Introduction
The loss of a limb is a significant and drastic event, both personally and socially, especially if it is the upper limb.This is because the hand and arm are involved in fine and precise movements.Prostheses have been developed to compensate for this physical loss and to restore some of the lost functionality [1].Historically, body-powered prostheses were primarily operated using cable and harness systems, while the development of prostheses with electrically powered actuators began in the 1970s.A notable advancement in this field has emerged more recently, with the creation of prostheses featuring electrically powered actuators controlled by electromyographic signals derived from the muscles of the user's residual limb [2].These myoelectric prostheses, which are controlled by measuring the nerve activity in muscles, have become increasingly popular because they offer a more natural look, dexterity, and advanced functionality compared to body-powered designs and require less effort to perform movements [3].They rely on electronic control interfaces that eliminate the need for mechanical interactions to control prosthetic movement between the prosthesis and the amputee.By interpreting the intended movements from the myoelectric signal [4], they significantly enhance users' comfort [5].Nevertheless, this switch from mechanical to myoeletric prostheses did not solve the issue of the lack of information from the somatosensory system, and actually increased it.Upper limb prosthesis users have indeed a strong demand for sensory feedback [6].Cordella et al. [7] reported that sensory feedback https://doi.org/10.1016/j.mechatronics.2024.103161Received 4 October 2023; Received in revised form 28 January 2024; Accepted 16 February 2024 is widely considered a priority for prosthesis users because it plays a fundamental role in our cognitive processes, emotional state, and behavior.Furthermore, in a study by Raspopovic et al. [8], it was found that neural sensory feedback led to increased walking speed and self-reported confidence, along with reduced mental and physical fatigue, compared to trials without stimulation in lower limb amputated subjects.
Sensorimotor loops and sensory feedback refer to the concept of continuous exchange of information and feedback between sensory and motor systems in the brain during the execution of motor actions.However, conventional prostheses often lack the necessary sensory feedback to support effective sensorimotor looping.
Commercially available prosthetic hands such as the i-limb Ultra (Ottobock) and the BeBionic (Touch-Bionics/Ossur) prosthesis, lack tactile sensation capabilities when interacting with the environment and handling objects citeint17.The PSYONIC Ability Hand [9], conversely, is an example of a commercially available prosthetic hand that integrates sensorized digits and vibrotactile feedback.
In the absence of feedback the patient must learn compensatory strategies: use of visual feedback, unconscious use of acoustic feedback from interaction with the environment (sounds of motors, gears under load, or deformations of the environment).However, the slowness and attention required for this sensory integration make the use of the prosthesis complex and cognitively heavy [10].By providing sensory information, tactile feedback can help create a more natural and intuitive experience for the user and reduce the cognitive load required to control the prosthesis.Furthermore, there may no longer be a need to rely on diverted visual or auditory cues when using the prosthesis.Some studies have confirmed that haptic feedback can efficiently improve the control of grasping force during functional use of the prosthesis in amputees and provide useful information to the user [11,12], improve task performance [13,14], especially in complex tasks, reduce the error rate in movements and grasping tasks, improve training of grasping force, and reduce phantom pain.
In the field of prosthetics, a variety of physical parameters can be measured using different types of sensors, such as force, finger position (angle), slip, temperature.Typical information provided in studies with sensory feedback includes proprioception and interaction forces [15].Proprioception allows the users to perceive the position and movement of their body, allowing them to identify, without vision, the position and orientation of the prosthesis.Interaction force refers to the force exerted by the user and experienced through the prosthesis during interaction with objects or surfaces.Heidi et al. [16] combined both types of information and attempted to provide additional stiffness information.However, studies have also examined the impact of providing through haptic feedback information on the texture [17], stiffness [18], shape recognition [19], and contact cues [20].After this information is collected, it is processed and analyzed before being used to generate feedback.The tactile feedback generated is directed to a specific part of the user's body that is able to detect and interpret the tactile information.This tactile sensation is generated in response to the user's interaction with the device and provides a physical cue that allows the user to receive feedback beyond naturally generated visual or auditory cues.
Various modalities to provide the haptic stimulation have been explored over the years [21][22][23][24][25][26][27].Non-invasive methods use external devices that provide tactile feedback to the user on the skin or other body parts, eliminating the need for surgical intervention and reducing invasiveness and associated risks.Typical noninvasive methods include vibrotactile feedback, electrotactile feedback, spatial audio rendering (SAR), and mechanotactile feedback (e.g., skin stretch, pressure, and squeeze).Non-invasive stimulation systems with haptic feedback are often coupled with myoelectric hand prostheses [28] and the most extensively studied method is the use of an external haptic device worn on the arm.However, there has also been research on the use of haptic soft gloves on the controlateral hand to provide sensory feedback to amputees operating prosthetic hands [29].Although statistical analysis [30] showed that electrostimulation actuators are widely used because of their light weight, low cost, low power consumption, and low noise, it is important to note that electrotactile feedback can cause skin irritation, discomfort, distraction, and interference with electromyographic signals [10].

Restoring proprioception
An example of a mechanical-tactile device is the HapPro [31], which has been integrated into the SOftHand Pro, an anthropomorphic robotic hand [32], and uses proprioception information about the opening/closing position of the hand obtained from encoders in the motors.The haptic device allows a carriage to slide linearly over the skin of the forearm, giving the user of the prosthetic hand the sensation of grasped object size.Experiments conducted by the authors evaluated the device's performance in grasping balls and cylinders of various sizes and demonstrated its ability to provide accurate and reliable proprioceptive feedback.
Such concept of skin stretch has also been transferred to different applications, such as laparoscopic surgery to improve force feedback, with the primary goal of detecting tissue stiffness [33].Another notable haptic device is the Haptic Rocker [34,35] which uses the rotational stretch of the skin to convey valuable information about the size of objects to be grasped.It provides haptic feedback that correlates with the degree of hand opening.Proprioception has also been conveyed through vibration [36], but experiments by Bark et al. [37] indicate that skin stretch on hairy skin provides higher effective analog resolution compared to vibration amplitude.Skin stretch seems better suited for continuous feedback without sudden changes, such as when opening the hand.

Restoring force sensations
Vibrotactile feedback has been extensively explored for force feedback applications where vibromotors are brought into contact with the skin, as in the work of Nabeel et al. [38].In this case, the vibration motor used is an ERM (Eccentric Rotating Mass), while other solutions [39] involve an LRA (Linear Resonant Actuator); these motors have different vibration modes and directions and consequently different dynamics and sensations.A classic example is the VF-M system [40] (vibrotactile frequency modulation), which uses dual-frequency vibrotactile stimulation to restore the perception of strength and stiffness in individuals with sensory impairments.The system consists of five hardware capsules, with each capsule representing the sensory information of one finger.Through frequency-modulated sensory feedback, each capsule conveys information to the wearer that is proportional to the force applied to the corresponding fingertip.
A proportional force feedback problem is related to the position control of commercial hand prostheses.Due to the high velocity, it is challenging for users to control and modulate the applied force.To overcome this problem, Clemente et al. [20] developed a device which taps into established sensorimotor mechanisms to provide brief sensory cues instead of continuous feedback, using vibrotactile stimuli.This device provides feedback when a prosthetic hand makes and releases contact with objects, events that are important for normal grasping and lifting control.It is expected that the feedback will be naturally associated with the corresponding mechanical events.

Multimodal feedback
In recent years, more haptic devices have been developed that can reproduce multiple types of skin sensations.By using different tactile actuators, the amount of information provided can be increased compared to using a single type of actuator.This approach expands the range of stimuli that can be matched to the modality of stimulation of the prosthetic hand.However, it is critical to maintain the simplicity of the sensory feedback system to avoid overloading the cognitive process of interpreting the sensations perceived.An example of a wearable, multisensory haptic feedback system is the Missive (multisensory interface consisting of stretch, squeeze, and integrated vibration elements) [41].The Missive system is capable of providing three different cutaneous sensations on the upper arm: lateral rotary skin stretch, radial pressure, and vibration.It uses multiple sensory elements to provide a more comprehensive haptic experience.The device has not yet been tested for conveying tactile information to the user.
Pezent et al. presented another haptic device called Tasbi [39], which provides both vibrotactile and squeeze feedback functions.The results of their study show that the integration of pressure, vibrotactile, and visual stimuli can produce realistic effects and provide intuitive haptic sensations.An example of a multisensory device and feedback targeted to upper limb prostheses is presented by Clemente et al. [42].The designed device conveys information to the individual about the contact between the fingers and the environment through a vibrotactile stimulus and the grasping force exerted by the prosthesis on objects through force/pressure feedback.

The MuViSS
The development and application of various haptic devices have showed promising potential in restoring essential sensory feedback, such as proprioception and force sensations, to individuals using prosthetic limbs.Notably, extensive experimentation [43] has shown that subjects respond significantly better to haptic cues when the haptic display is placed around the wrist.This is why we propose here a new multisensory device called MuViSS (Multi Vibrotactile and Skin Stretch) designed for haptic feedback.This innovative device is specifically designed to be worn on the wrist and forearm.The device has two different types of tactile feedback: skin stretch and vibrotactile sensations.The development of the device emphasizes compatibility with the Taska hand [44], which has been modified to integrate sensors for measuring force and proprioception.
Compared to the devices described in the literature, the MuViSS is distinguished by its ability to reproduce a wide range of skin sensations and differs from devices such as the HaPPro or the VF-M system in its ability to reproduce a more extensive range of stimuli.Compared to devices such as MISSIVE and Tasbi, the MuViSS has a linear skin stretch module similar to the HaPPro, as well as different vibromotors that can produce different vibratory sensations.Specifically, the MuViSS is equipped with the vibration capabilities of an EMR and two LRAs Together with the device, this study introduces a novel feedback strategy that has not been investigated in the existing literature.By combining the two previously established strategies of Rossi et al. [31] and Clemente et al. [20], namely skin stretching associated to proprioception and the inclusion of cues on contact through vibrotactile stimuli, the feedback creates a unique and unexplored approach.
The remainder of the paper is organized as follows.Section 2 covers the conceptual design of the overall system, the MuViSS system, the integration of sensors into the Taska hand, and an in-depth investigation of our novel feedback strategy algorithm.Section 4 presents the experiments we conducted and the results we obtained.This is followed in Section 5 by a comprehensive discussion of our results and their implications.Finally, in Section 6 we summarize the results obtained and briefly describe future activities.

Concept of the multi-sensory based haptic device -robotic prosthetic hand system
The MuViSS haptic device in conjunction with the Taska Hand is designed to provide continuous, valuable tactile feedback to amputee users with simplicity and comfort in mind.The sensor-equipped Taska hand collects data about its movements and interactions forces and transmits this sensory information to the MuViSS haptic device, which translates it into tactile stimuli.This system combines proprioceptive data with force information to maintain a constant awareness of the hand's openness and alert the user when contact occurs without relying on other sensory modalities or strategies.
The main goal is to increase the utility of prostheses by providing tactile information, thus reducing the user's dependence on visual or auditory senses.Such a reduction in sensory dependence mitigates distractions and cognitive load, improving efficiency and overall performance on tasks related to recognition and motor skills.
The overall structure of the sensory feedback in combination with a prosthetic robotic hand is shown in Fig. 1.The Control Loop is dedicated to controlling the movement of the prosthesis based on EMG signals, while the Sensory Feedback provides the user with surface stimulation that correlates with the actions of the prosthesis.The MuViSS device can be worn either on the amputee's residual limb or on the opposite arm, depending on the degree of amputation.An example of wearing MuViSS and the hand prosthesis by an upper-arm amputee can be seen in Fig. 2. In this case, MuViSS is worn on the side of the non-amputated limb.MuViSS is a prototype, in future developments it can be made smaller and lighter, enabling its placement on the same arm of a transradial amputee.Following the conceptualization of the robotic hand haptic system, the MuViSS haptic system was realized.To develop the feedback strategy and control mechanism, it was necessary to integrate sensors into the Taska hand.Initially, tests were conducted to characterize the MuViSS and evaluate its effectiveness.Following the successful results of these tests, the entire system was subjected to comprehensive testing.

The MuViSS haptic device
The MuViSS haptic device is composed of three different types of stimuli: the stretching of the skin, the vibration of an ERM, and the vibration of two LRAs.
Fig. 3 provides an overview of the MuViSS-Taska Hand System and shows the haptic feedback of MuViSS and its correlation with the movement and force signals detected on the hand prosthesis.Two different types of feedback are generated: proprioceptive and contact feedback.A rotary potentiometer measures the degree of closure of the hand, which is correlated with haptic feedback of skin stretching.This is proprioceptive feedback.The signals detected by two force sensors on the fingertips of the thumb and index finger are converted into a sensory vibration stimulus.These are contact feedback.
All modules and the housing of the control board are connected with two elastic nylon straps that can be fastened with Velcro.The skin stretching module is placed on the upper part of the forearm.The linear motion of a suitably designed spherical tactor, made of silicone rubber (which is characterized by a high frictional coefficient) is generated by a small servomotor (SERVO DMS-MG90-A) and a gearbox.The tactor has a linear range of 3 cm, which corresponds to a spatial rotation of the motor of 180 degrees.The structure of this module is shown in Fig. 4.
The rotary movement of the servomotor is converted into an alternative translatory movement of the tactor by the rack and pinion system visible in Fig. 4. The tactor, which comes into contact with the skin, causes the skin to stretch by moving back and forth.Fig. 5 shows a view of the inside of the opened MuViSS.
The two LRA modules consist of a 3D-printed part that acts as a buckle and contains the motor (C12-003 from Microdrives) and its driver.The modules can slide along the elastic band and let the motor apply normal pressure to the arm.They are placed on the left and right side of the wrist or forearm.Linear resonant actuators produce vertical vibrations at a specific resonant frequency.An LRA motor functions as a voice coil motor resembling a mass-spring system, which operates as a resonance system driven by an alternating current voltage with a resonant frequency of around 200 Hz.Varying the input voltage amplitude allows for adjustments in vibration amplitude while maintaining a fixed vibration frequency.The current supplied activates the voice coil, creating a magnetic field that exerts a proportional mechanical force on a mass containing a permanent magnet.This mass moves up and down in a linear motion, aided by a spring.Contact information is detected using force sensors on the prosthesis.A contact is identified when the force exceeds a predetermined threshold.A second-order Butterworth low-pass filter has been designed to reduce the noise of the measure.To relay this information to the user, a 300 ms vibration impulse is generated by the LRA.If the contact duration is shorter, the vibration also lasts for the corresponding amount of time, e.g. a contact of 150 ms produces a vibration of 150 ms, while a contact of 2 s produces a vibration of 300 ms.Each LRA is associated with a specific fingertip and sensor, allowing for localized feedback.The two LRA are independent and can vibrate at the same time in case both sensors overcome the thresholds.If the sensor in the thumb perceives a contact, the corresponding LRA vibrates, if the index sensor perceives a contact, the other LRA vibrates, if both sensors perceive a contact, both LRAs vibrate, with the amount of time previously explained.
An ERM motor consists of a DC motor connected to a shaft with a mass.Altering the speed at which the offset mass spins changes the frequency and amplitude of the resulting vibrations.Typically, microcontrollers are employed to generate a pulse width modulation signal to control these motors.The motor's speed, determined by the input voltage, directly influences the vibration frequency but also amplitude.The ERM motor has a relatively slow acceleration and deceleration due to the mass's inertia.The ERM motor, a 10 mm diameter vibration motor, is attached directly to the bottom of the housing, which contains the control board, which is made of an elastic material.The ERM comes into contact with the skin on the lower part of the forearm or wrist.
LRAs exhibit faster acceleration, smaller physical size, higher efficiency, and longer lifespan, primarily because they lack brushes that can deteriorate over time.However, LRAs are constrained in terms of frequency due to their narrow resonance peak.Additionally, for optimal acceleration performance, an LRA needs to be operated close to its resonance frequency.LRAs are better suited for binary applications, whereas ERMs are more effective for encoding a greater amount of information and conveying complex signals.
Comparative tests were developed between LRAs and ERMs, which showed that LRAs have a faster response time compared to ERMs.ERMs had a noticeable rise and decay time, which does not help to create the desired sensation of contact information.However, the ERMs provide a higher amplitude, which is useful for generating different amplitudes for proportional force feedback.Based on these findings, the LRAs were selected for the generation of contact information, while the ERMs were retained for proportional force feedback.
The control board housing contains an Arduino MKR1010.Each vibromotor is driven by a DRV2506L haptic driver.Power is supplied to the Arduino board via a USB cable connected to a laptop.To achieve a more comfortable feeling when wearing the device, spongy material was added to the rigid plastic parts that attach the elastic straps.The assembly of the entire MuViSS is shown in Fig. 6, and the scheme for placing the stimuli and wearing the device are shown in Fig. 7.

Integration of MuViSS with the Taska hand
For the development of the setup, the commercially available Taska hand was used.It is an available myoelectric robotic hand with multiple  joints, and its wrist allows interchangeability with a wide range of in-hand controller terminals.
Additional sensors and an electronic acquisition unit were attached to the Taska hands to gather the information needed to feed the Mu-ViSS.Specifically, two force-resistant Flexiforce sensors and a rotary potentiometer were incorporated.New 3D-printed fingertip attachments were designed to accommodate the force sensors, as the fingertips of the Taska hand can be removed as separate components.The force sensors were placed in the thumb and index fingertips, respectively.Rectangular holes were drilled in the fingertips to insert the sensors, and a plastic pad was inserted through a central circular hole to attach and apply direct pressure to the sensitive surface of the sensor.The pad was positioned slightly above the surface of the fingertip so that it was the first point of contact when an external force was applied.
For the potentiometer, two 3D-printed parts were designed to attach the sensor to the hand and measure the openness of the hand.The first part, which serves as a fixed mount for the potentiometer, was screwed tightly onto the hand.The second part was inserted into a hexagonal recess on the tip of the index finger and has a shaft that rotates the rotary component of the potentiometer.Thus, the openness of the hand could be determined by measuring the angle of rotation of the index finger.
Finally, a custom 3D-printed enclosure was created to house an Arduino MKR1010, Flexiforce adapters for stable force sensor measurements, a LiPo battery, and a voltage step-up converter for operation at 5 V.The case is attached to the back of the hand with Velcro for easy removal.Fig. 8 illustrates the final setup of the modified Taska hand and shows the integrated components.

System control algorithm
Two circuits were developed, one for the hand prosthesis and one for the MuViSS.Their structure is shown in Appendix.WiFi communication is used to transmit the measurement data from the Arduino on the hand to the Arduino in the haptic device.
The presented new feedback strategy combines two stimuli, based on both proprioception and contact information, to improve the haptic feedback for the user.We called it PC feedback, where P stands for proprioception and C for contacts.A schematic representation of the implementation of the PC feedback can be seen in Fig. 9A.
Proprioception information is captured by the potentiometer and rendered to the user via the skin stretching device.This modality has already been investigated in projects such as Happro [31].A second order low pass filter with cutoff frequency equal to 2.5 Hz was developed to reduce the noise of the measurement.Contact information is detected using force sensors.A contact is detected when the force exceeds a predetermined threshold.A second order low pass filter with cutoff frequency equal to 13 Hz has been designed to reduce the noise of the measurement.To relay this information to the user, the LRA generates a vibration pulse of 300 ms.If the contact duration is shorter, the vibration also lasts a correspondingly shorter time.Each LRA is associated with a specific fingertip and sensor, allowing for localized feedback.Isolated studies [20] have shown that the implementation of contact information alone has led to performance improvements.
The choice of the LRA engine is based on its dynamic properties.Previous experiments [45] have shown that this engine is better suited for binary information and provides a faster response.LRA motor, unlike the ERM motor, has higher and more pronounced dynamics when responding to short pulses.The ERM is still integrated into the wristband, although it is not directly used in the feedback strategy of PC.The ERM produces a different sensory experience than the LRA, and experiments have shown that it is easier to detect different vibration amplitudes with this type of motor [45].
In addition to the presented multimodal PC strategy, a classical force feedback approach (FF) was developed for comparison purposes.Fig. 9B provides a schematic view of the implementation of the FF feedback.This method considers the maximum force detected between the two force sensors.The ERM can oscillate at four different amplitudes depending on the measured force.To prevent the sensation from becoming annoying to the user, the ERM is programmed to vibrate for a maximum of 3 s at a constant force range.Also in this case, the information about the hand is sent to the device via WiFi.Another reason for integrating the ERM into the wristband, although not used in the current feedback strategy, is to explore its potential as a new source of information in future studies.Given its ability to provide a particular sensory experience, the ERM holds promise for further investigation.
To ensure that delayed sensations are avoided, the time taken from recognizing the stimulus in the prosthetic hand to providing the sensation through the haptic device was carefully estimated.This was done to avoid confusion and create realistic sensations.The Flexiforce sensors have a declared response time of less than 5 μs.In addition, the time interval between measurement and actuation was measured.Several estimates were made for different events, e.g.touch, continuous force, proprioception, and their combinations.The average time obtained was about 8 ms.The time required to send the data was measured in a separate experiment and was less than 1 ms.
In both strategies, the time between the detection of the signal in the hand and the subsequent stimulus on the wristband was measured, and it was found to be less than 10 ms.Moreover, the onset time of the LRA is equal to 32 ms.Based on literature, papers, and experiments [46,47] it was determined that the time interval between a touch and the subsequent feedback should be more than 100 ms for subjects to perceive a delay and consider the two events as separate.Based on these results, it is safe to say that the total sensing, computation, and communication time is well below 100 ms.Therefore, the sensations provided can be reasonably associated with the touch of the prosthetic hand.

Assessment protocols
Table 1 contains a summary of the experiments performed, including the number of subjects, duration, data stored, specific parameters, and the objectives of each.The experiments were divided into two sessions: the first focused on characterizing the MuViSS device (assessing its performance), the second on evaluating users ability to integrate and benefit from different feedback sent by the MuViSS during two type of task with the hand prosthesis.

Protocol of the MuViSS assessment
These tests on the device aim to evaluate the effectiveness of each feedback independently.To achieve this, each modality was tested individually in short experiments to assess its clarity and effectiveness.Five participants took part in the experiments with the MuViSS device.This experiment is divided into three parts, each corresponding to a module of the device.These parts were presented to the participants  in a random order.In each part, participants wore the haptic MuViSS device on their wrist and noise-canceling headphones to eliminate all auditory cues (see Fig. 10).Final results were analyzed in Matlab.
In the LRA part, participants were presented with two buttons corresponding to the thumb and finger the training phase, they trials could a button and receive the associated stimuli.These stimuli mimicked the feedback experienced in real-life situations, consisting of a 300 ms pulse.In the test phase, participants received successive stimuli from the LRAs, which were randomly selected by the software.They had 6 s to press the correct key in response to the sensed stimulus.The time between the stimulus and the correct key press was recorded, and if they did not respond within 6 s, the trial was considered unsuccessful.The next stimulus was generated after a random time interval between 2 and 10 s.In total, the test included 30 stimuli randomly presented to a subject, with 15 discriminations for the two stimuli (thumb and index finger).
The ERM part borrows from Gathmann's work [40].First, participants are shown the minimum and maximum force they could experience.Then, each of the four force levels is provided in ascending order (habituation training).Participants are informed of the intensity level of force provided.Then, a round of 5 stimuli of a randomized reinforcement learning phase is performed.During the reinforcement phase, the participant was given correct when given a response.familiarization and reinforcement phase, the testing protocol is conducted to the participant's ability to classify 4 levels of force.A randomized sequence was created in which each strength level was repeated seven times.The participant is asked to recognize the force level, and the correct answer is not given.The recognized force levels by the user and the actual force levels are recorded.
The skin stretching phase was inspired by earlier experiments of Rossi [31] and Battaglia [35].In this phase, the constant stimuli method was used.Participants were presented with pairs of stimuli, each stimulus consisting of a shift of the tactor followed by its return to the initial position.Participants were then asked to indicate which stimulus they thought had a longer shift.Pairs consisted of a standard stimulus of 15 mm (90 • -degree space of the motors) and a comparison stimulus presented randomly.Five comparison stimuli were used, equally spaced between 7.5 mm (45 • ) and 22.5 mm (135 • ).The minimum and maximum deflections were chosen so that they were typically judged to be smaller or larger than the standard stimulus.A single trial included the presentation of the first stimulus, a 1-second interstimulus interval, the presentation of the second stimulus, and the participant's response.Participants were instructed to respond as quickly as possible by saying ''1'' or ''2'' to indicate whether the stimulus with the longer shift was the first or second of the pair.There was no time limit for the response to ensure that participants focused on the distance between the stimuli and not on their timing.Three different speeds were used to prevent reliance on timing.The skin-stretch phase consisted of 30 randomly presented stimulus pairs, including six discrimination pairs for each of the five stimulus levels.Randomization was performed independently for each stimulus sequence.In addition, the participant's arm was positioned so that he could not see the shift in the tactor.
The skin stretch phase of the experiment aimed to determine two important measures: the just noticeable difference (JND) and the point of subjective equality (PSE) of the skin stretch feedback.The JND refers to the minimum distance the wheel travels to produce a noticeable perceptual difference compared to a reference stimulus.The PSE is the point at which participants perceive two stimuli as equivalent.The percentage of subjects indicating that the respective comparison skin stretch is greater than the standard value for each subject is considered.

Protocol to measure MuViSS' impact on the user performing with prosthesis
The aim of the feedback experiments with the prosthetic hand is to test the effectiveness of feedback in typical activities of daily living.In this preliminary work, representative tricky tasks from the international Cybathlon competition [48], which challenges teams from all over the world to develop assistive technologies suitable for everyday use with and for people with disabilities, were considered.
They consist of two tasks: the first involves a recognition task, the second focuses on a motor task.In the recognition task, we compared the effect of PC and FF on the ability to recognize/discriminate size and stiffness of grasped objects.In the motor task, we performed tests with both feedback mechanisms and without haptic feedback (WF) to compare their effects on performance during a dynamic everyday activity.The order of the two tasks was randomized.Nine subjects were tested in both tasks.A forearm case was used for both experiments.It contains the control electronics and the battery.There are two buttons on the top that allow the user to control the opening and closing of the hand to simulate myoelectric control, as visible in Fig. 11.These are the only two movements performed during the experiment.By using this device, all problems related to reading the myoelectric signals and training the participants to control the hand through their muscle signals can be avoided.At the end of each condition, participants are asked to complete a questionnaire NASA TLX to evaluate the perceived workload.The MuViSS was worn on the wrist during both experiments.Final results were elaborated in Matlab and Jasp.

Recognition task
The experimental setup involved one participant sitting on a chair positioned toward a table.Using the previously mentioned prosthesis, the participant performed a haptic task.The task involved several spheres, three with different sizes (20, 35, 50 mm) and three with different stiffness levels (rigid wood ''approx.∞'', 1, 3 N∕m), attached to a corresponding base to fix them and facilitate the execution of the experiment.
To eliminate visual influences, the participant was blindfolded with a night mask during the experiment.The prosthesis was securely fixed in a specific position so that the participant could manipulate it by pressing buttons to open and close the hand.The size and stiffness recognition tasks were performed separately, with the participant performing size recognition in one session and stiffness recognition in another.The order of these two sessions was randomized.
In the training phase, each condition was tried once to observe the scene.In the reinforcement phase, the participant was blindfolded and given nine random sizes or stiffnesses, and when they gave an answer, they were given the correct answer.This was done by opening and closing the hand, using the available feedback to give the correct answer.This begins the testing portion.On each trial, a single ball was placed in the center of the prosthetic hand, and the participant's task was to determine the size or stiffness of the ball.The participant had 15 s to provide a response on each trial.Each session included fifteen trials, five for each sphere.The setup is shown in Fig. 11.
Throughout the experiment, the participant went through each of the previously described conditions, resulting in a total of four recognition phases: FF and PC for stiffness and size.

Motor task
In the motor experiment, the participant sat on a fixed chair in front of a table with a box divided into two compartments: one compartment contained a marble resting on a support, while the other compartment remained empty.
The participant used the prosthesis presented earlier.The task was to take the marbles one by one from the support and move them to the empty compartment on the table.To increase the difficulty and introduce randomness, two types of marbles were used: one made of wood and one made of hard plastic.These marbles were alternated on each trial.In total, the participant had to move 14 marbles in each condition, 7 of each type.A clear depiction of the setup can be found in Fig. 12.
Before each condition, participants were given one minute to familiarize themselves with the prosthesis and MuViSS.After releasing each marble, the participant had to fully open the hand again before attempting the next grasp.If a marble fell out of the holder during grasping, the participant had to restart the grasping procedure by fully opening the hand again.The order of execution of the feedback was randomly selected and distributed.The participant was informed that he was evaluated in terms of time and effort.During each trial, data from the force sensors and the encoder were recorded to collect relevant information about the grasping process and the movements involved.Force was assessed using the maximum average force between the two sensors while grasping the marble on each trial.Time was assessed by the interval between the onset of closure and the release of the marble.

Participants
The experimental study was carried out in accordance with the recommendations of Sorbonne Université ethics committee CER-SU, which approved the protocol.All asymptomatic participants, aged 18-30, volunteered for this experimental study.They all gave their informed consent, in accordance with the Declaration of Helsinki.

Statistics
Statistical tests were run on data.Normality was assessed with Shapiro-Wilk test; if normality was assessed, repeated measures ANOVA were run otherwise non parametric Friedman test and eventually Conover post-hoc comparisons were used.

Results of the characterization of MuViSS
The results of the experiments on the LRAs are shown in Table 2 and are presented as the percentage of successful or incorrect choices or elapsed time, and then the average time of each subject is calculated.In all subjects and trials, there was only one case in which a stimulus was not perceived by the participant, i.e., 1 out of 150 trials performed.In addition, there were no cases in which the thumb stimulus was mistaken for the index finger or vice versa.
Regarding the ERM, the cumulative results are shown in Fig. 13 as a confusion matrix with the sum of the responses of the individual subjects; the accuracy of the confusion matrix is 76%.
The skin stretch answers data were fitted to a general linear model with a logit link function to estimate the psychometric function for each subject.Fig. 14 shows the average data for all subjects and the fitted model.It can be observed that the two data sets for 7.5 mm   and 12.5 mm do not have an error bar as they are characterized by a standard deviation of zero, as all test participants correctly stated that the generated sensation was less strong than the reference sensation of 15 mm.The JND is defined as the difference between the 75% threshold and the 25% threshold divided by 2. The 50% is the point of subjective equality.In all subjects, the JND was 2.08 mm, whereas the PSE was 16.93 mm.

Experiments on the feedback with the prosthetic hand 4.2.1. Recognition task
The results are presented in the form of confusion matrices in Fig. 15.The accuracies achieved in the confusion matrices for size recognition using FF and PC are 92% and 88%, respectively, while for stiffness recognition, they are 81% and 79%.
Fig. 16 shows the average results of the surveys in recognizing and comparing the two types of feedback.Of all the questions in the survey, repeated measures ANOVA showed that only frustration in size recognition was found to have a statistically significant difference between the two feedbacks ( = 0.035).

Motor task
The average forces recorded by the sensors while the hand was grasping the marble are as follows: 0.896 ± 0.718  with the use of WF, 0.579±0.439 with FF, and 0.588±0.417 with PC.The Friedman test   revealed a significant main effect of type of feedback on force values ( 2 (2) = 17.206,  < 0.001, Kendall's  = 0.068).Connovers posthoc comparisons also revealed that PC produced lower forces than WF ( < 0.001), although it did not differ from FF ( = 0.9).WF and FF differed ( < 0.001).
The average time measured are as follows: 5222 ± 2316 ms using WF, 4481 ± 1089 ms using FF, and 5283 ± 2192 ms using PC.Fig. 17 shows the comparison of raincloud plots between the three feedbacks in terms of force and time.The Friedman test revealed a significant main effect of type of feedback on time values ( 2 (2) = 8.714,  < 0.013, Kendall's  = 0.035.Connovers post-hoc comparisons also revealed that PC generated less time than WF ( = 0.009) and FF ( = 0.015).WF and FF did not differ ( = 0.850).Further analyses were performed to understand the effects of the order of execution of the feedback and the response of each subject.For these studies, all force and time data were considered independent of the type of feedback.Kruskal-Wallis tests showed that time changed significantly with execution order ( 0.001), whereas it did significantly affect force ( = 0.332).In addition, subject significantly affected both time ( < 0.001) and force ( < 0.001).
Fig. 18 shows the average results of the surveys comparing the three types of feedback.Of all the questions in the survey, Friedman test showed that only physical demand had a statistically significant difference between the three feedbacks.

Assessment of the MuViSS device
The initial experiments aimed to evaluate the effectiveness of the MuViSS haptic device regardless of the feedback strategy chosen for the prosthesis.Overall, the results showed that the MuViSS effectively conveyed clear sensations.In particular, the LRA experiments showed that users could easily understand and distinguish feedback from the thumb and index finger.The fast response times indicated that users could understand the stimuli without much delay.Only one subject showed slightly slower reaction times, but overall performance was still considered acceptable.The binary nature and dynamic nature of the LRAs make the contact information a quick alert, allowing users to quickly absorb and understand the information.
The ERM portion of the experiments assessed participants' ability to detect the four different vibration/force levels.The accuracy achieved indicated that the subjects were able to effectively discriminate between the different levels.For the second and third stages, the recognition rates were slightly lower (60% and 66%), but all the errors were within adjacent levels.It is worth noting that after long and continuous vibration sensations, the skin remained able to perceive the stimulus at a lower intensity.In addition, a short training period was conducted to familiarize the participants with the experiment, giving them only a limited clue to the procedure.This allowed them to later test the intuitiveness of the areas.In view of the above, ERM sensation can be considered effective in identifying different force levels.
Regarding skin stretching, the values of JND and PSE gave an overview of the effectiveness of the skin stretching channel.The results (JND=2.08 mm with 3 cm range) were comparable to those obtained with the Happro device, which had a slightly different stretching range (4 cm) and obtained a JND of 3.10 mm.This difference could be due to the reduced number of tests performed.Nevertheless, the aim was here to evaluate the effectiveness of the skin stretching channel, and this part of the experiment confirmed that the users could understand the sensations.It is noteworthy that of each pair of stimuli with a stretch difference of 7.5 mm (both 7.5 mm and 22.5 mm stretch), only one out of thirty cases was not correctly guessed.
In summary, the initial experiments have demonstrated MuViSS's ability to deliver distinct and comprehensible haptic feedback, regardless of the chosen feedback strategy, whether it involves EMR, LRA, or skin stretch.Nevertheless, it is essential to address the bulkiness of the current prototype, as potential redesign opportunities exist to integrate it seamlessly into a prosthetic socket.Leveraging the internal sensor measurements available in modern polydigital hands could facilitate this transition, enhancing its suitability for real-world applications.

Assessment of MuViSS effect on user's ability to perform task with the prosthesis
Table 3 summarizes all the previous results and analyses.

Recognition task
The second part of the experiments aimed to test the effectiveness of the feedback.The results of the recognition experiments show that both the PC and FF feedback were able to help the participant recognize size and stiffness of grasped objects.The WF condition was not included in this task because the user had no information to rely on, and he/she would have guessed randomly because the audio and visual feedbacks were also blocked.Moreover, experiments in Happro have already shown that the absence of haptic feedback in a recognition task resulted in a well-distributed confusion matrix.
Starting with the size task, accuracies obtained with PC and FF were impressively high at 92% and 88% respectively, indicating that participants were able to successfully discriminate between different dimensions.Statistical tests also confirmed that there was no difference between the true positives for each feedback, although PC had a slightly higher value.
However, a crucial condition was present in the experiment, namely a fixed starting position of the hand.Thus, with FF, participants relied on the time between the onset of hand closure and vibration feedback to guess the size of a sphere.In contrast, with PC, participants were able to rely directly on the skin stretch channel to provide a signal when contact occurred.Subjects self-reported these observations after the experiment.In addition, previous studies have shown that skin stretch with proprioception is effective for size recognition, as seen with devices such as Happro or Haptic Rocker.This suggests that feedback from PC is more robust and intuitive, providing accurate cues.Furthermore, the skin stretch feedback from the PC imparts a sense of hand openness, directly correlating with dimensional information.Consequently, the feedback of PC provides clear cues about the size of objects, while FF can also provide information but is strongly influenced by the previous hand position and may more time and cognitive Analyzing the survey responses of the mean scores, PC performed better in terms of mental and time demands, perceived performance, and overall effort.A significant statistical difference was found in the frustration response, showing that FF was more frustrating than PC, which could be due to the reasons mentioned above.However, no significant statistical difference was found for the other survey responses.
In the stiffness task, accuracies obtained with PC and FF were slightly lower than in the size task, at 81% and 79%, respectively, indicating increased difficulty in recognition.However, the results were still considered acceptable given the complexity of the task.Similar to the size task, statistical test showed no difference in true-positive rates between the two feedback methods.In this case, FF is a more direct and intuitive type of feedback related to stiffness information.In addition, Gathmann et al. [40] previously demonstrated the effectiveness of FF in this task.However, PC showed the same performance even when participants had to rely on two different sensations.According to participants' feedback, FF provided clues to stiffness through maximum amplitude and slope to achieve it, especially for small levels of openness.On the other hand, PC provided cues through the stretch sensations after contact information.
When the survey responses were analyzed, no significant statistical difference was found, and the average values for each response were quite similar.
Considering these results, PC proves to be a valid alternative to FF in detecting stiffness and achieves comparable performance even though people tend to extract force from displacement information, and it is able to provide information about two different aspects simultaneously.

Motor task
The motor experiment was designed to investigate the influence of each feedback during a dynamic representative daily living task, during which complete visual feedback is available.
Statistical analysis revealed that both PC and FF decreased the applied force compared to WF.The lower force applied with FF was expected due to the nature of the feedback.However, the result of PC suggests that contact information can provide crucial cues for grasping an object that help users avoid excessive force application.Although some information is lost after contact, this cue remains useful for this purpose.
As for the temporal results, statistical analysis has shown that PC performs better in terms of speed than FF and WF.This suggests that the contact information not only regulated the force but also triggered a fast alarm, which improved task performance in terms of speed.Conversely, FF may have required more decoding and could have imposed a higher cognitive load, resulting in a slower response.This could also be due to the limited dynamics of the ERM.
In addition, it should be noted that PC also provided information on proprioception, although its usefulness was here reduced by the inclusion of vision.Nevertheless, it provided additional useful information about hand openness.Importantly, the inclusion of vision in this experiment may have influenced the results.If vision is partially eliminated, for example by visual obstacles which are common in everyday life, the results of the task could be more relevant and varied between feedback conditions in terms of applied force and, more importantly, completion time.
Overall, this experiment demonstrates that the introduction of feedback can improve performance on motor tasks even when vision is available.In addition, feedback from PC generally provided better results in terms of force regulation and completion time compared to WF and in terms of completion time compared to FF.
A key aspect is that the order of execution significantly affected the execution time.Participants tended to exercise and improve on each execution, even when there was a training phase.To mitigate this effect, the order of each condition was randomized for all participants to ensure that each condition was executed the same number of times in different positions of the order, resulting in valid and reliable results.Surprisingly, applied force was not statistically affected by the order of execution, suggesting that there was no strong training effect or improvement over time for strength-related aspects.Moreover, the different and individual behavior of each subject led to different results, underlining the subjective nature of feedback perception.This was confirmed by the statistical tests, which showed significant differences between subjects in terms of time and force levels.
The surveys show that PC provides better results on average, but no statistical differences were found between the different feedbacks.

Overall effect of the feedback
In general, the PC transmitted by the MuViSS represents a novel feedback modality which appears to be a promising approach for a multimodal feedback in prosthetic hands.The motivation for introducing a feedback modality other than the conventional FF is closely related to the control of existing commercial prosthetic hands.These devices are typically controlled in velocity and often operate at high speeds to achieve optimal performance.However, this velocity control makes it difficult for the user to effectively control and modulate the applied force.This often results in significant force fluctuations when attempting to perform delicate movements.
The inclusion of force feedback driven by abrupt and potentially overwhelming force changes could therefore potentially saturate the information provided to the user.While proprioception is more coherent and direct to the fingers position and movement control, it lacks certain important information.To address these limitations, additional contact cues were introduced.The introduction of contact cues has the advantage of providing important force-related information during interactions.When contact occurs, this system can easily and effectively convey force information to the user.While it is true that some information is lost compared to the force feedback approach, our analysis shows that force feedback is often underutilized due to the inherent limited hand control.Therefore, the proprioceptive and contact cues together provide a balanced and reliable feedback approach.
Furthermore, the combination of proprioceptive and contact modalities proved not to be cognitively burdensome for users.Contact information is presented in a binary fashion, whereas proprioception is continuous.This clear distinction between the two modalities allows users to identify different sensations without requiring much mental effort.
Another advantage of this solution is that movements and sensations are triggered only when needed, stretch sensation is activated when the user commands an opening or closing movement of the hand, The participants were able to recognize sizes.They relied on the skin stretch channel to receive an alert when a contact occurred.PC directly relates to the information from the potentiometer.It is more robust and provides accurate cues even from unknown positions.It was found to be less frustrating than FF.
The participants were able to recognize sizes.They relied on the time between the start of hand closure and the vibration feedback.The information is strongly influenced by the previous hand position and requires time and cognitive effort.It cannot provide reliable information from unknown positions.It was found to be more frustrating than PC.

Stiffness Recognition
The participants were able to recognize stiffness.It provides clues through the stretch sensations after contact information.The participants rely on two different sensations.Surveys and results showed equal performances to FF.
The participants were able to recognize stiffness.It provides clues through the maximum vibration amplitude and the gradient to reach it.FF is intuitive.Surveys and results showed equal performances to PC.
Force in picking and placing task The force applied was reduced compared to WF, but it did not differ statistically from FF.The contact information can provide crucial cues for grasping object, helping users avoid applying excessive force.
The force applied was reduced compared to WF, but it did not differ statistically from FF.The FF helps in the control of force.
Time in picking and placing task PC better performed with respect to FF and WF.
Contact information provides a rapid alert, improving the performance in terms of velocity.
FF performed worse compared to PC and it did not exhibit statistical differences from WF. FF requires decoding and might impose a higher cognitive load, leading to a slower response.
Relationship with the hand control actuation (position) PC is more coherent and direct due to its nature of conveying the finger's position and movement.
High speed control makes it challenging for users to control and modulate the applied force.This leads to significant force variations when attempting delicate movements, possibly saturating the information provided to the user.

Comfort
The solution lacks annoyance because movements and sensations are provided only when necessary.
It can provide prolonged vibrations related to force, which can be annoying and distracting.

Complexity
All the results suggest that PC is not cognitively heavy even if it combines two modalities.
All the results suggest that PC is not cognitively heavy even if it combines two modalities.
and vibrations are triggered only upon contact with an external object over a certain force level.Prolonged vibration feedback can be tiring and thus reduce the effectiveness of closed-loop control.The solution developed for contact feedback avoids this risk.The vibration feedback is only generated when contact occurs and only for a limited time.Therefore, the risk of excessive fatigue due to prolonged vibration feedback is very low.The feedback is designed to avoid continuous sensations, such as sustained vibrations associated with force, which can be both irritating and distracting.This design decision makes for a more comfortable and less mentally taxing experience for the user.

Conclusions
This study presents the MuViSS, a new wearable multimodal haptic device that can render three different types of sensory input, along with a new unique sensory feedback strategy (PC) tailored to upper limb prostheses and their users.The development of this innovative feedback approach represents a promising advance in hand prosthesis technology.
Experiments with the MuViSS demonstrated the effectiveness of the device in rendering various stimuli and laid the groundwork for later experiments on feedback with information coming from a prosthetic hand.The results demonstrated the ability of the device to provide valuable sensory feedback and improve both recognition and motor tasks for users.By effectively addressing the challenges associated with force feedback, the device provides a simple, effective, and comfortable user experience while reducing cognitive load, thereby improving the overall user experience.
Future developments include field tests with amputated, which will provide valuable insights and subjective feedback on the practical performance of the device.The presented version of MuViSS is a prototype for a proof of concept, that could easily be made smaller and lighter, thus increasing portability and user comfort.As for the hand side, a custom circuit board design may reduce the dimensions of the housing and the overall design.In addition, the introduction of more information can be considered, with the ERM already integrated.After having further improved the device, a campaign of experimental validation tests will be carried out, covering the main recognition and motor tasks involved in the activities of daily living, as well as tests on amputees.

Fig. 1 .
Fig. 1.Scheme of the Control Loop (in red) and Sensory Feedback (in blue) within the MuViSS -prosthetic hand system.

Fig. 2 .
Fig. 2. Example of wearing MuViSS and the hand prosthesis by an upper-arm amputee.

Fig. 4 .
Fig. 4. Scheme of the assemblage of the skin stretch module.

Fig. 7 .
Fig. 7. MuViSS worn by a person on the left.Scheme of stimuli placement on the right.

Fig. 9 .
Fig. 9. A. Scheme of the PC feedback.B. Scheme of the FF feedback.

Fig. 10 .
Fig. 10.Setup of the experiment for the characterization of the device.

Fig. 14 .
Fig. 14.Average result on the Skin Stretch feedback signal.

A
. Campanelli et al.

Fig. 15 .
Fig. 15.Confusion matrix results of the recognition experiment using Proprioception Contact Feedback (PC) and Force Feedback (FF).

Fig. 16 .
Fig. 16.Surveys results in recognition experiments.For all the categories, small values indicate better results.

Fig. 17 .
Fig. 17.Raincloud plot comparison of the three feedback, where WF is without feedback, FF is force feedback and PC is proprioception and contact feedback.

Fig. 18 .
Fig. 18.Surveys results in motor experiments.For all the categories, small values indicate better results.

Table 1
Resume of the performed experiments.

Table 2
Results of the experiments on the LRAs.
Fig. 13.Cumulative confusion matrix result on the ERM feedback signal.

Table 3
Comparison between PC and FF.