Simulating the perceptual effects of electrode–retina distance in prosthetic vision

In Argus II, the electrode array is attached to the retina by a single titanium tack located in the center of one narrow side (see figure 1(a)) [33, 57]. While the tacked side is attached to the retina, the other side is unattached and may tilt away from the retina (see figure 1(b)).

Zoom In Zoom Out Reset image size

Table 1 describes the terms regarding ERD. Given that the electrode array is implanted over the fovea [57], the largest ERD is likely to be around the center of the electrode array because of the macular concavity, as shown in different OCT studies associated with Argus II users [31, 35–38]. A heat map of the ERDs, which was used in all simulations (except those shown in figures 5(b) and (c)), is presented in figure 1(c).

The dependence of the threshold current amplitude on the ERD ranges from linear to an exponent of 2. One study has found that the threshold current amplitude increased with the ERD approximately linearly up to 0.5 mm [4]; other studies have found that the threshold current amplitude increased with the ERD squared in rabbits' retinas [43] and macaque's retinas [58]. This is reasonable as it fits the electric field decay with the square of the distance in an isotropic medium with distant boundaries. Other studies have found that the threshold current increased significantly for larger ERDs (ERD > 0.5 mm), and the relationship is approximately exponent of 2 [7, 39, 41]. In the simulation, a model that fits all these studies was adopted. Therefore, the threshold current dependence on the ERD is defined by:

$$\begin{align}{I_{{\text{th}}}}\left( {{\text{ERD}}} \right) = \left\{ \begin{array}{*{20}{l}} {a + b{\text{ERD}},}&{{\text{ERD}} \leqslant 0.5\,{\text{mm}},} \\ {a + b\exp \left( {2{\text{ERD}}} \right)/c,}&{{\text{ERD}} > 0.5\,{\text{mm}},} \end{array}\right.\end{align} \tag{ 1 }$$

where a is a parameter that represents the minimal current amplitude required to evoke phosphene at zero ERD, b describes the linear dependency between the threshold current amplitude and the ERD, and the factor c =5.437 was used to make the graph continuous at 0.5 mm. a = 130 µA and b = 0.5 A m⁻¹ were chosen to approximately fit the graph of the experimental data found with Argus II users, for 0.45 ms pulse at 20 Hz stimulation rate [4]. The maximum stimulation of 200 µm platinum electrodes (Argus II [4]) and 250 µm platinum electrodes (IRIS II [27], learning retinal implant system [28]) was 311 µA and 486 µA, respectively, with 0.45 ms pulse, within the safe charge density limit of 0.35 ${\text{mC}}\,{\text{c}}{{\text{m}}^{ - 2}}$ [59]. The graph of the threshold current as a function of the ERD is presented in figure 2(a).

Zoom In Zoom Out Reset image size

It has been found that electrodes, which are located farther from the retina, may not stimulate the retinal ganglion cells sufficiently, which affects the ability to elicit visual perception within safe charge density limits [4, 7, 10, 41]. The assumption that as the ERD increases, the phosphene's brightness decreases until it falls to the background brightness is valid because the electric field that reaches the ganglion receptive fields is lower as the ERD increases. This assumption is consistent with the literature because the stimulation amplitude decreases as the ERD increases, and a reduction in the stimulation amplitude is associated with the elicitation of darker phosphenes [48]. Therefore, electrodes cannot elicit phosphenes at a certain ERD in the simulation, referred to as ERD_crit. The graph of the phosphene brightness as a function of the ERD is presented in figure 2(b).

In cases where ERD > 0.36 mm, assuming a stimulation of the 200 µm electrodes with a pulse duration of 0.45 ms within the safe charge density limit, the Gaussian function is annihilated, i.e. there is no perception of phosphenes (black pixels). The same goes for 250 µm electrodes given the same pulse duration at ERD > 0.675 mm. The ERD affects the size of phosphenes. As the ERD increases, the stimulation amplitude reaches the retina decreases, and thus the phosphenes become smaller [48]. In the simulation, as the ERD increases, the phosphene brightness decreases. The decrease in brightness results in a perception of a much smaller phosphene because of the Gaussian profile. To control the amount that the size decreases, the function F_S (ERD) is used in equation (2). This function increases the phosphene size as the brightness falls in an ERD-affected phosphene to make the phosphene smaller but not miniature. The graph of the phosphene size as a function of the ERD is presented in figure 2(c). It was found in Argus I and II users that, as the ERD increases, the elicited phosphenes are less elongated or rounder [8]. In the simulation, phosphenes are perceived as round above a relatively low ERD (around 0.3 mm), as users consistently describe phosphenes as round [2, 7, 11, 19, 41, 60–62]. A graph presenting the modeled Gaussian dimensions that represent the phosphene shape as a function of the ERD is shown in figure 2(d).

The simulation was constructed in MATLAB R2020b (Mathworks Inc., Natick, MA, USA) based on the characteristics of the Argus II [33]. It includes an electrode array with 6 × 10 electrodes and is implemented on a desktop display at 144 × 240 pixels resolution. Based on an equation in the form of a 2D elliptical Gaussian function that describes the phosphene's brightness distribution and allows the simulation of size variations, shape variations, spatial shifts, and dropout [53], a more general equation that also considers the phosphene's brightness variations and the ERD effects was developed:

$$\begin{align}&f\left( {x,y,{\text{ERD}}} \right)\nonumber\\ &\; = AB{F_B}\left( {{\text{ERD}}} \right)\exp \{ - S{F_S}\left( {{\text{ERD}}} \right)[a{{\left( {x - i{x_0} - {x_1}} \right)}^2}\nonumber\\ & \quad + 2b\left( {x - i{x_0} - {x_1}} \right)\left( {y - j{y_0} - {y_1}} \right) + c{{\left( {y - j{y_0} - {y_1}} \right)}^2}]\} ,\end{align} \tag{ 2 }$$

where $f\left( {x,y,{\text{ERD}}} \right)$ is the pixel's value in location (x, y) at a specific ERD. A represents the phosphene brightness at the center; it is the mean value of a pixel block in the high-resolution camera's frame, and it is quantized into four normalized values (0, 1/3, 2/3, and 1). If a dropout is applied, a random 10% of the electrodes cannot elicit phosphenes by setting A to 0, regardless of the stimulus. B and S represent the phosphene brightness and size random variations and are applied to all phosphenes when considered; B is uniformly distributed between 0.7 (i.e. a 30% darker phosphene) and 1, i.e. no change in the brightness of a phosphene; S is uniformly distributed between 0.5 (i.e. a 50% bigger phosphene) and 1.5 (i.e. a 50% smaller phosphene). ${F_B}\left( {{\text{ERD}}} \right) = 1 - {I_{{\text{th}},{\text{norm}}}}\left( {{\text{ERD}}} \right)$, ${F_s}\left( {{\text{ERD}}} \right) = 1 - 0.5{I_{{\text{th}},{\text{norm}}}}\left( {{\text{ERD}}} \right)$; ${I_{{\text{th}}}}\left( {{\text{ERD}}} \right)$ is normalized for convenience, such that ${I_{{\text{th}},{\text{norm}}}}\left( 0 \right) = 0$ and ${I_{{\text{th}},{\text{norm}}}}\left( {{\text{ERD}} \geqslant {\text{ER}}{{\text{D}}_{{\text{crit}}}}} \right) = 1$, because the maximum allowed current is reached at ERD_crit. At ERD_crit, the phosphenes disappear (${F_B}$ (ERD_crit) = 0) and are twice as big (${F_S}$ (ERD_crit) = 0.5), because as the value ${F_S}$ (ERD) decreases, the phosphene becomes darker and bigger. a, b, and c, the shape parameters of the elliptic Gaussian, are defined as $a = \frac{{{{\cos }^2}\theta }}{{2\sigma _x^2}} + \frac{{{{\sin }^2}\theta }}{{2\sigma _y^2}}$, $b = - \frac{{\sin 2\theta }}{{4\sigma _x^2}} + \frac{{\sin 2\theta }}{{4\sigma _y^2}}$, and $c = \frac{{{{\sin }^2}\theta }}{{2\sigma _x^2}} + \frac{{{{\cos }^2}\theta }}{{2\sigma _y^2}}$, where ${\sigma _x}$ and ${\sigma _y}$ depend on the ERD and represent the spread of the phosphene in both axes. ${\sigma _x}$ ranges from 1 to 4/3 and ${\sigma _y}$ ranges from 3/4 to 1. $\theta$ is uniformly distributed between zero and 180. When shape variations were applied, ${\sigma _x}$ and ${\sigma _y}$ are randomly varied for 15% of the phosphenes within the same values range in a uniform distribution. The angle $\theta$ represents the clockwise rotation of the ellipsoid-shaped phosphene. The indices $i\,{\text{and }}j$ represent the phosphene's center point coordinates in the x and y axes, respectively. ${x_0}$ and ${y_0}$ are the phosphene's center position coordinates. ${x_1}$ and ${y_1}$ represent the spatial shifts; when applied, all the phosphenes are shifted in a uniformly distributed range from −6 to 6 pixels in the x-axis and the y-axis.

Groups of electrodes, such as quads, become necessary as the ERD increases. Phosphenes evoked by quads are usually bigger than phosphenes evoked by individual electrodes [12, 46]. The relation between the thresholds for evoking a phosphene by a single electrode (${T_{SE}}$) and by a quad (${T_Q}$) in the Argus II is T_SE = 2 × T_Q [32]. Figure 3 demonstrates the effect of quads on the threshold current amplitude for both 200 µm and 250 µm platinum electrodes stimulated by a pulse duration of 0.45 ms. Using quad stimulation instead of single-electrode stimulation for 200 µm electrodes increases ERD_crit from 0.36 mm to 0.685 mm.

Zoom In Zoom Out Reset image size

Another approach proposed for reducing the effects of ERD is by changing the input signal sent to the prosthesis. The first step is to measure the ERDs of all the electrodes in the array. This can be done by OCT measurements [32]. An ERD matrix should be created accordingly (like figure 1(c)). Subsequently, an ERD correction threshold should be defined, such that the pixels' value in the input image that corresponds to electrodes with ERDs above the ERD correction threshold is increased, and the pixels' value in the input image that corresponds to electrodes with ERDs below the ERD correction threshold is left untouched. Simple point-by-point matrix multiplication between the low-resolution input image (i.e. that fits the size of the electrode array, 10 × 6 for the Argus II) and the correction matrix is then performed. Figure 4 presents the ERD matrix from figure 1(c) and two examples for corresponding correction matrices. Different correction matrices could be helpful in different cases (elaborated in the discussion). In figure 4(b), the ERD correction threshold was defined as the minimum ERD (350 µm) in the ERD matrix. In figure 4(c), the ERD correction threshold was defined as an arbitrary higher value in the ERD matrix—420 µm. Any value can be taken for the ERD correction threshold; this value can be smaller or larger than the minimum ERD.

Zoom In Zoom Out Reset image size

An experiment with subjects and full-reference image similarity metrics were carried out to quantitatively assess the effects of quads and image enhancement on prosthetic image quality. Twenty normally sighted subjects (10 females), aged 20–62 (average: 32.5), were recruited to verify whether quads and image enhancement improve the quality of prosthetic images. The experiment was approved by the Ben-Gurion University Human Research Ethics Committee, and all subjects provided their informed consent. Ten original binary images (figure 5) were simulated using four different methods to illustrate four versions of prosthetic vision: ERD-affected image, ERD-affected image with a quad, ERD-affected image with input-image enhancement, and ERD-affected image with quad and input-image enhancement. The order of the prosthetic views was different for each binary image.

Zoom In Zoom Out Reset image size

The subjects were asked to rank the four prosthetic views according to their similarity to the original image in one session that lasted approximately 10 min (one minute for each image set ranking). The ranking was ordinal between 1 and 4; 1 for the least similar image and 4 for the most similar image. The significance of the experiment (for all the algorithms together) was estimated using the Friedman test, and the multiple comparisons significance between every two algorithms was estimated using the Wilcoxon rank test in SPSS 26.0 software (SPSS Inc., Chicago, IL).

In addition, three full-reference image similarity metrics based on the human visual system (HVS) were used to produce objective quantitative assessments of the similarity between each prosthetic view and its corresponding original image. PSNR-HVS-M (peak signal-to-noise ratio—human vision system—modified) metric is based on the peak signal-to-noise ratio and considers the contrast sensitivity function and between-coefficient contrast masking of discrete cosine transform basis functions [63]. MS-SSIM (multiscale structural similarity) metric is based on the high adaptation of the HVS in extracting structural information from a scene, and it is more robust than the traditional SSIM metric because it combines the SSIM index of several versions of the image at various scales [64]. CW-SSIM (complex-wavelet structural similarity) metric is a unique structural similarity metric based on the idea that distortions lead to consistent phase changes in the local wavelet coefficients, thus not changing the image's structural content [65].

Scanning an object with slow, patterned head movements can also reduce the perceptual loss of information in the prosthetic FOV caused by the ERD. The head movements allow different electrodes with smaller ERD to convey the information about the scene that the electrodes with the large ERD failed to convey. During object scanning, the phosphenes persist. The phosphene persistence period (PER) is defined as the time it takes a phosphene to fall to the value of the background following stimulation [17]. Generally, users who experience short persistence should scan objects faster than users who experience long persistence [17]. To define the scanning speed in deg s⁻¹, it is necessary to define the simulation's FOV. The FOV is set according to the Argus II FOV—19° × 11° [26] with approximately 13 pixels deg⁻¹. The results demonstrate object scanning at approximately 4.5 deg s⁻¹ without persistence and 9 deg s⁻¹ with and without persistence.

The ERD affects three spatial aspects that change the appearance of phosphenes: size, brightness, and shape. Phosphenes that correspond to electrodes located farther from the retinal surface (large ERDs) appear smaller, darker, and less elongated. The simulated impact of various ERDs_max (0.1, 0.3, 0.5, and 0.7 mm) in an electrode array that contains 60 electrodes is shown in figure 6. The largest ERD_max that was taken is 0.7 mm (figure 6(d)), as found in the literature for the Argus II [32]. For 250 µm electrodes, the ERD_crit is 0.675 mm, and thus there is no phosphene perception in the middle of figure 6(d), where ERD_max is 0.7 mm. Note that for 200 µm electrodes (as in Argus II), these results would be repeated for much smaller ERDs_max (0.36 mm). The simulations presented in figure 6 include spatial effects that the ERD causes, without random spatial variations.

Zoom In Zoom Out Reset image size

The images presented in figure 7 demonstrate how electrode grouping may improve perception mediated by electrodes located far from the retina (i.e. around ERD_crit or above). For 250 µm electrodes, the ERD_crit is 0.675 mm, and thus quads would be beneficial for ERD > 0.675 mm, but even for smaller ERDs, such as 0.6 mm, quads can help in eliciting clearer phosphenes. Note that 200 µm electrodes (as in Argus II) would require quads for ERD > 0.36 mm. The use of 15 quads, as presented in figure 7(b), is unnecessary because it decreases the resolution from 60 to 15, while most of the electrodes can elicit phosphenes without grouping; only a small number of electrodes in the center of the FOV cannot elicit phosphenes because of their large ERDs. Using quads only for specific electrodes with large ERDs could help elicit phosphenes where single electrodes cannot (or barely) do so without decreasing the resolution. The simulations presented in figure 7 do not include the random spatial variations of the phosphenes.

Zoom In Zoom Out Reset image size

Figure 8 compares the perceptual effects of various random spatial variations, such as size, brightness, dropout, and spatial shifts, that were previously simulated [53–56] with the perceptual effects caused by the ERD. Figure 8(b) presents a naïve prosthetic vision simulation where no spatial aspects are considered. Figure 8(c) illustrates how adding the spatial variations to the prosthetic view decreases the image quality. The perceptual effects caused by the ERD may have a more significant influence on the quality of the prosthetic image than the effects of the random spatial variations, as demonstrated in figure 8(d) in comparison to figure 8(c). In figure 8(e), the use of quads in restoring vision loss is demonstrated again.

Zoom In Zoom Out Reset image size

A reduction in the adverse spatial effects caused by ERD, especially the incapability to evoke phosphenes above ERD_max, is demonstrated by using the correction matrix from figures 4(b) and (c) by considering the ERD matrix presented in figure 1(c). Figure 9 presents an image of a key and its corresponding ERD-affected prosthetic view, with an ERD_max of 0.7 mm and 60 electrodes, each with a diameter of 250 µm.

Zoom In Zoom Out Reset image size

Figure 10 presents an example of a binary image and its four prosthetic views. 60 electrodes with an electrode diameter of 250 µm, pulse duration of 0.45 ms, and a constant ERD_max of 0.6 mm (just below ERD_crit = 0.675 mm) were chosen for all the prosthetic views of each binary image. Figure 11 presents a box plot of the central and spread tendencies of the four prosthetic views as found in the experiment. The experiment found significant differences across all the four prosthetic views (Friedman test: χ² = 397.16, P < 0.0001) and between every two algorithms (Wilcoxon rank test: P < 0.0001 for all comparisons). According to figure 11, the subjects' significant preference was for the ERD-affected image with quad and input-image enhancement (median: 4, 25th percentile: 3) and ERD-affected image with input-image enhancement (median: 3, 75th percentile: 4) over the ERD-affected image with a quad (median: 2, 75th percentile: 2.5) and ERD-affected image without improvements (median: 1, 75th percentile: 2). The ERD-affected image with a quad score was sufficiently higher than the ERD-affected image without improvements to be considered helpful in prosthetic vision.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Table 2 presents the mean scores of the computational image similarity metrics: PSNR-HVS-M, MS-SSIM, and CW-SSIM. The prosthetic view with both the quad and image enhancement obtained the highest score according to the PSNR-HVS-M and MS-SSIM metrics, and the prosthetic view with only the image enhancement obtained the highest score according to the CW-SSIM metric. These results are in line with the results from the experiment with subjects.

Figure 12 illustrates the perception of the key during scanning at a fast speed of 9 deg s⁻¹ performed by the user with and without enhancement of the input image and with and without the persistence effect (the videos are provided in the supplementary data (available online at stacks.iop.org/JNE/19/035001/mmedia)). When the user performs head movements to scan an object, the object is observed sequentially in different parts of the FOV, for some of which the electrodes can evoke phosphenes, thus allowing the perception of more parts of the object (figures 12(2(b) and (c))), including those that were missing in the first static image (the narrow part of the key). During the head movements, the object may be perceived with a smearing caused by the persistence effect (figures 12(3(b), (c), (e) and (f))). In the last part of the video, the scanning focused only on the narrow part of the key, helping the user perceive it as it has initially been affected by the ERD, without the smearing caused by the persistence of the wide part of the key. Enhancing the input image using the correction matrix improves the perception of the object by eliciting brighter phosphenes at large ERD locations, without scanning and during the process of scanning (figures 12(d)–(f)). However, as the phosphenes become brighter, the smearing caused by the persistence becomes more visible. In this case, it can be helpful to decrease the scanning speed [17]. Figure 13 illustrates the scanning of the key with 2 s of persistence at two different speeds, 4.5 deg s⁻¹ and 9 deg s⁻¹. It is demonstrated that a slower scanning speed preserves the advantages of improving the object's visibility but also reduces the adverse effect of smearing. In the videos presented in the links provided in the captions of figures 12 and 13, the electrode activation rate was set to 6 frames per second (FPS), according to the Argus II default stimulation rate, which is 6 Hz [45], the display frame rate of the simulation was 30 FPS, and the PER was either 0 or 2 s.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Currently, there is not enough data on the effect of the ERD on phosphene size. In the simulation, we assumed that the size decreases as the ERD increases, but with a limited reduction controlled by ${F_S}\left( {{\text{ERD}}} \right)$. It has been found that decreasing the stimulation amplitude results in the elicitation of smaller phosphenes [48], and, because the stimulation amplitude decreases as the ERD increases, it could be associated with decreasing the phosphene size. However, distant electrodes may stimulate more receptive fields, resulting in a bigger combined phosphene.

The ERD may increase in time following implantation, as was found in users implanted with suprachoroidal prostheses [39], which means that the ERD effects may get worse after implantation. However, this effect was not demonstrated in this study.