Game controller modification for fMRI hyperscanning experiments in a cooperative virtual reality environment

Graphical abstract

The Magnetic Resonance Imaging (MRI) environment presents unique challenges to the introduction of any additional hardware into close proximity with the machine. If proper care and safety are not taken when bringing equipment into the bore, the strong magnetic field creates conditions that could harm subjects or the machine. In addition, inclusion of any active electronic device must be tested to make sure there is no interference with data collection or induction of artifacts into the data. Devices, like a computer keyboard, have previously been used to record complex motor responses within an MRI and have been determined to be both safe and to not interfere with data quality [1]. However, for natural, immersive control of a virtual avatar, a game controller with dual analog joysticks has been shown to be more effective and has become the commercial game industry standard. A modified game controller has been used within an MRI to control action of a single agent within a complex virtual representation (e.g. the path of a virtual taxi cab in London [2]), but details on the necessary modifications have not been previously published. Recording simultaneously from two separate brains has been shown to elucidate neural markers of subject-subject interaction, but has been mainly limited to social experiments [3]. Through the use of virtual avatar representations controlled simultaneously by the two subjects, the ambiguity of cooperative interaction is kept to a minimum leaving only the salient time series and events for analysis. The following is a detailed procedure for modifying a game controller to be MRI compatible, along with steps for creating hyperscanning virtual environments. As MRI-compatible controllers with the functionality required for hyperscanning in a virtual environment are not currently available on a commercial basis, the procedures described should prove to be useful for the development of experimental protocols in this area of research.

Method details
Step  [ ( F i g . _ 1 ) T D $ F I G ]  [ ( F i g . _ 2 ) T D $ F I G ] c. Disassemble the stock joystick assemblies, keeping the plastic components and the single (non-magnetic) metal axle. d. Cut the ultra-precision 302 stainless steel compression spring to the same size as the stock component e. Glue a plastic block, cut to the dimensions of the joystick button, to the base piece in lieu of the button (Fig. 5A). This supports one of the joystick axles. Note that the button is not needed to control the virtual environment. It is replaced in the present implementation due to the presence of magnetic metal within the button. f. Reassemble the joystick using the printed housing and base, along with ultra-precision 302 stainless steel compression spring and remaining stock pieces (Fig. 5A). g. Secure the assembly by gluing the housing and base together on the outside, making sure to not allow any glue to reach the inside (Fig. 5B f. On the other side of the cable, solder the wires to a D-sub 25 connector, using pins 1, 3, 5, and 7, for the wire leading to DND, D+, D-, and VOC, respectively (Fig. 6). Any connector with 4 pins or more can be used to the same effect (e.g. 2 BNC or serial port). g. Solder a further connection from the ground pin (1), to the metal housing of the D-sub 25 connector. This will effectively connect the room shielding to the controller shielding. No external filter is used. 8. Build secondary D-sub 25 to USB wire.

[ ( F i g . _ 3 ) T D $ F I G ]
a. Make sure to match the correct USB wire configuration with the correct pins used in step 7. This is to connect the MRI room wall pass-through to the experiment computer. 9. Apply shielding to controller housing.
a. Drill a hole in bottom half of the casing near the cable opening. b. Using painter's tape, tape off all holes and the sides of each housing piece on the outer side. c. Insert toothpicks into the screw holes to prevent blockage from paint application. [ ( F i g . _ 6 ) T D $ F I G ]  10. Insert a brass bolt into the drilled hole and secure tightly with a brass nut (circled in red). Attach a second brass nut without fully tightening it (Fig. 8). 11. Replace all of the button pieces into the controller housing. 12. Insert the circuit board back into the housing. 13. Secure the circuit board with the replacement 3/8 00 slotted brass round headed screws. 14. Attach the ''naked'' shielding wire from the cable to the brass bolt in controller housing and secure with the second nut. 15. Reconnect the two halves of the controller housing and secure with brass screws.
Step 2: Virtual environment construction Obtain objects and actors, along with a suitable environment for the scenario, using VBS2's development interface and object libraries (Fig. 9A). 2. Set situational logic and player/AI settings via VBS2's scripting language to streamline the development process and provide extensibility (Fig. 9B). a. Establish this by creating an initialization script that triggers when the scenario first starts along with other script files that contain particular functions. b. All scripts should all be placed in the same folder as the mission's files and have the file extension .sqf (See Supplementary Material below) 3. Bind the controller keys to basic in-game functionality by assigning appropriate actions to each of the keys through the options menu. 4. Create non-verbal communication interface a. Develop separate functions using VBS2's scripting language to display text locked to avatar position b. Assign the actions to the ''User Defined Actions'' option in the script. c. Bind the various ''User Defined Actions'' to the appropriate controller buttons using the options menu.
[ ( F i g . _ 8 ) T D $ F I G ] 5. Set-up additional data logging. a. Locate the correct scripting functions to interact with VBS2's native logger, After Action Review (AAR). b. Within the initialization script, add additional events to be logged into the AAR such as reaching a checkpoint, firing shots, and signaling to each other. 6. Initiate cooperative mission over Local Area Network (LAN) using VBS2's built-in networking protocol.
Step a. Only one Teensy is needed to trigger the two scanners from the master experiment computer. 5. Flash the Teensy 3.1 with custom code with: a. Serial interface to receive commands b. Ability to send a 10 ms square wave pulse. 6. A custom plugin for VBS2 allows it to connect to the Teensy 3.1 over a serial connection and send a pulse over BNC to both MRIs. 7. Initializing the two MRIs simultaneously allows two consecutive pulses (500 ms apart) to start both MRIs in sync from within the VBS2 environment.

Noise Tests
In addition to modifying the game controller to be safe for use within a MRI by removing all magnetic components, tests to determine whether or not the device will introduce excessive noise or artifacts into the MRI data were conducted. The [ ( F i g . _ 9 ) T D $ F I G ] scanners employed are the GE Discovery MR750 3T. Each scanner was used in conjunction with an in vivo 8 channel head coil. The tests were run with the controller plugged in, turned on, and placed within the bore at the approximate location it would be during an experiment. In addition, buttons were continuously pressed throughout the test by an experimenter located just outside the bore to simulate the hardware in use. The Center for Functional MRI at UC San Diego School of Medicine has established a signal to fluctuation noise ratio (SFNR) reading of 350 as the cut-off for an acceptable level of stability (see http://www.birncommunity.org/tools-catalog/function-birn-stability-phantom-qa-procedures/ for description). In addition, values for RMS stability and Weisskoff stability (RDC) [4] are obtained to give a more comprehensive analysis of scanner stability. A baseline reading without a controller was obtained, labeled phantom in Table 1, and had a SFNR of 388.3. Both controllers were run in the same noise test protocol in the conditions described above and resulted in SFNR readings of 375.0 and 377.7, respectively. Both of these values are well above the SFNR cut-off of 350, indicating a minimal level of noise induction by the modified game controller.
Along with testing the effect of the controller on fMRI data collection, the controllers were tested for induction of artifacts by the MRI on controller functionality. Both controllers displayed no artifacts in functionality for the entirety of the 10 min scans (data not shown). RDC > 4.5) used for the system quality assurance process at the UCSD Center for Functional MRI.