Data in Brief

We present a multi-sensor dataset of bimanual human-to-human object handovers. The dataset consists of 240 recordings obtained from 12 pairs of participants performing bi-manual object handovers with 10 objects, and 120 recordings obtained from the same 12 pairs of participants performing unimanual handovers with 5 of those objects. Each recording includes the giver and receiver’s 13 upper-body bone position and orientation trajectories, position trajectories for the 27 markers placed on their upper bodies, object position and orientation trajectories, and two RGB-D data streams. The motion trajectories are recorded at 120Hz and the RGB-D streams are recorded at 30Hz. The recordings are annotated with the three handover phases: reach, transfer, and retreat. The dataset also includes four anthropometric measurements of the participants: height, waistline height, arm span, and weight. Our dataset could help investigations of the bimanual reaching motions and grasps utilized by humans while performing handovers. Also, it can be used to train robots to perform bimanual object handovers with humans.


Subject
Engineering Specific subject area Human-Robot Interaction Type of data Tables and Videos How the data were acquired Motion Capture (Optitrack Flex 13); RGB-D camera (Microsoft Kinect v2.0); Measuring Tape; Weighing Scale Data format Raw and processed .csv,.mp4,.matDescription of data collection (400 characters) Pairs of participants performed the object handovers standing face-to-face, and the giver held the object before the start of the handover.A motion capture system and two RGB-D sensors were used to record these object handovers between each pair of participants.Data

Value of the Data
• This dataset is useful for studying various features of bimanual handovers between two persons, such as handover location, object orientation, grasp configuration, and reaching velocity profile.• This dataset can be used to build and evaluate models of human motion in bimanual handovers, a task that people frequently perform in their daily lives.• This dataset can be used to train a robot to perform the tasks of human-to-robot and robotto-human handovers which are essential for human-robot collaboration.• This dataset can be used to compare bimanual handovers and unimanual handovers, which can inspire new motion models and robot controllers for object handovers.• This dataset can benefit researchers studying human-robot interaction, robotic manipulation, and human motor skills.

Objective
There are numerous applications where bimanual handovers, i.e. handing over objects with two handed grasps, are useful or even necessary.Bimanual handovers are necessary when handing over large rigid objects, deformable objects, spherical objects, and delicate objects.Also, in some cultures, it is a rule of etiquette to hand over objects with two hands.The existing datasets of human-to-human handovers [1][2][3][4] only consider handovers of objects with single-handed grasps i.e. uni-manual handovers.To the best of our knowledge, there is no public dataset of human-to-human handovers of objects requiring bimanual grasps, even though such handovers are equally important and even more challenging.Our dataset is aimed at addressing this gap.

Data Description
This dataset contains 240 recordings collected from 24 volunteers performing bimanual handovers, i.e. handovers with two-handed grasps ( Figs. 1 and 2 ), with 10 objects ( Fig. 5 and Table 3 ).We also include 120 recordings from the same volunteers performing unimanual handovers, i.e. handovers with single-handed grasps, with 5 of those objects (object numbers 1 to 5 in Fig. 5 ).Each recording consists of:   The dataset also includes the height, weight, arm span, and waistline height of the participants.
The dataset contains one subfolder for each participant pair, named "P[ab]_P[cd]" where "ab" and "cd" are the two-digit participant numbers .Each of these subfolders contains four subfolders: 1. Kinect_1: This subfolder contains the data obtained from the first Kinect v2 sensor.There are three types of files inside this folder: "Pab_Pcd_Handover Type_Object.mp4","Pab_Pcd_HandoverType_Object_Depth.mat", and "Pab_Pcd_ HandoverType_Object_Depth_Timestamps.csv".In all three filenames, "Pab" corresponds   3 .Each object is fitted with a loop part of the hook-and-loop fastener to facilitate easy attachment and removal of the marker-set.to the participant number of the giver, "Pcd" corresponds to the participant number of the receiver, "HandoverType" is "double" for bimanual handovers and "single" for right-handed handovers, and "Object" is the name of the object being handed over.There are 10 objects: bag, basketball, cardboard, clothes, dishes, laptop, large cardboard box (lbox), plush toy, pot, and router.
The ".mp4" file is the RGB video recording of a handover with a resolution of 1920 × 1080 at 30fps.
The ".mat" file is the depth data of a handover with a resolution of 512 × 424.The timestamp of each depth frame with reference to the first video frame is provided in the ".csv" file.
2. Kinect_2: This subfolder contains the data obtained from the second Kinect v2 sensor.The file types inside this folder are similar to those in the Kinect_1 folder.3. OptiTrack_Global_Frame: This subfolder contains the data obtained from the Opti-Track Flex 13 Motion Capture System.Due to the occlusions of markers, a total of 0.65% values are missing in this data.Each file inside this folder is named in the format "Pab_Pcd_HandoverType_Object.csv",similar to the Kinect files.For example, "P07_P08_double_bag.csv" corresponds to the bimanual handover of the "bag" from "P07" to "P08".Each file contains the trajectories of 13 upper-body bones and 27 upper-body bone markers for both the participants and the trajectory of the object.All trajectories are in the reference frame of the motion tracking system and are recorded at 120fps.The trajectories are annotated with the handover phases (reach/transfer/retreat) of both the participants and the index and timestamp of each dataframe.Thus, each of these files is a 355-dimensional representation of a handover.The column headers in these files are described below: "Index" -Dataframe index, starts with 0 "Giver" -Handover phase of the giver, reach/transfer/retreat "Receiver" -Handover phase of the receiver, reach/transfer/retreat "Time" -Dataframe timestamp in seconds.milliseconds"Structure:Pab:Name:TrajectoryType:Axis" -Trajectory of a bone or a bone marker of the participant "Pab"."Structure" is either "Bone" or "Bone Marker".For bones, the "Name" is one of the 13 bone names listed in Table 1 and shown in Fig. 3 .For bone markers, the "Name" is one of the 27 marker names listed in Table 2 and shown in Fig. 4 ."TrajectoryType" is either "Position" or "Orientation".The positions are in meters ("Axis" = X or Y or Z) and the orientations are in quaternions ("Axis" = X or Y or Z or W).
"Rigid Body:object:TrajectoryType:Axis" -Trajectory of the object being handed over."Trajec-toryType" is either "Position" or "Orientation".The positions are in meters ("Axis" = X or Y or Z) and the orientations are in quaternions ("Axis" = X or Y or Z or W). 4. OptiTrack_Local_Frame: This subfolder contains the quaternion orientation trajectories of 13 upper-body bones for each participant with respect to their parental segment, generated by the OptiTrack Motive software, roughly representing the joint angles.Thus, each of these files is a 108-dimensional representation of a handover.Due to the occlusions of markers, a total of 0.59% values are missing in this data.The filename and column header format is the same as the files in "OptiTrack_Global_Frame" excluding the bone markers and the object.
The dataset also includes a "participant_anthropomorphic_measurements.csv"file which contains the height (in meters), weight (in kilograms), arm span (in meters), and waistline height (in meters) of the participants.

Experimental Design, Materials and Methods
We conducted a user study to create a multi-sensor dataset of bimanual human-to-human handovers [5] .This section describes the study protocol and data collection process.

• Study Setup
Our experiment setup, shown in Fig. 1 , consisted of an OptiTrack motion tracking system with 12 Flex-13 cameras, and two Kinect v2 sensors.The area was surrounded by green screens to make it easier for other researchers to do post-processing on the video recordings in the future.

• Motion Tracking System
The OptiTrack cameras were connected to a desktop computer (CPU: Intel Core i5-7600 3.5GHz, GPU: 2GB AMD Radeon T R5 430 and 4 GB Intel HD Graphics 630, RAM: 8GB, running Windows 10 Pro) which recorded the skeleton tracking data through OptiTrack's Motive software.To track the participants' motion in bimanual handovers, we used the "Conventional Upper Body" marker-set consisting of 27 reflective markers placed at the positions shown in Fig. 4 .Along with the 3D positions of these 27 markers, Motive also computes-from the marker positions-the 3D position and orientation of 13 upper-body bones shown in Fig. 3 .

• Kinect Sensor
The first Kinect v2 sensor (K1 in Fig. 1 ) was connected to a desktop computer (CPU: Intel Core i7-6700 3.4GHz, GPU: 4GB NVIDIA Quadro P1000, RAM: 16GB, running Windows 10 Pro) which recorded the data streams from the Kinect sensor in the Microsoft Kinect Studio software.The the Quaternion orientations of 13 upper-body bones for each participant with respect to their parental segment, roughly representing the joint angles.
We annotated each frame of the data recordings with the three phases of a handover: reach, transfer, and retreat.The "reach" phase corresponds to the reaching motion of the giver and receiver before the receiver makes contact with the object.The "transfer" phase starts when both the giver and receiver touch the object for the first time and ends when the giver releases the object.In the "retreat" phase, the giver and receiver retract their hands to the resting position.In our annotations, the "transfer" phase corresponds to the portion of the handover where the distance between the giver's and receiver's hands is within 20cm of the minimum distance of the handover, and their relative velocity is less than 15cm/s.These thresholds were obtained with trial-and-error.

Fig. 1 .
Fig.1.Schematic of the data collection setup of the dataset.We used an OptiTrack motion tracking system with 12 cameras (red squares), and two Kinect v2 sensors (black triangles).The green and blue circles represent the positions of the participants.The reference frame in the Fig.shows the reference frame of the motion tracking system whose origin was on the floor and behind the participant shown with the green circle.The participants were instructed to stand along the x-axis of this reference frame, at an interpersonal distance that is comfortable to them.

Fig. 2 .
Fig. 2. Example of a bimanual handover between two participants.The participants are wearing motion capture suits with 27 reflective markers per suit.A set of 6 reflective markers is attached to the object.

Fig. 5 .
Fig. 5. Objects used in the dataset.The object names and properties are listed in Table3.Each object is fitted with a loop part of the hook-and-loop fastener to facilitate easy attachment and removal of the marker-set.

Table 1
Names and acronyms of the 13 bones in the "Conventional Upper Body" skeleton.

Table 2
Names and acronyms of the 27 bone markers in the "Conventional Upper Body" skeleton.