Opportunities for integrated photonic neural networks

2.1 Integrated photonic RC systems

RC is a computation concept well suited for sequential data processing (Figure 4) [27], [28]. A stream of input data is coupled into a reservoir, which consists of recurrently connected neurons. The synaptic interconnects between the input and the reservoir, as well as within the reservoir, are assigned randomly and kept fixed. RC systems are therefore a special type of recurrent neural networks (RNNs). The connections in the reservoir are typically sparse (<20%). To avoid exponential growth of the signals in the reservoir, the weights in the reservoir are scaled such that the system fulfills the echo state property [27]. During training, only the weights at the output layer are learned. RC has been of great interest as it massively simplifies the training compared to general RNNs. In an echo state network with a linear output layer, the weights can be learned by a simple ridge regression. While the simple training method is still beneficial, deep learning methods have made great progress over the last years and allow for very effective application of RNNs on complex tasks that could hardly be solved by RC systems. Nevertheless, RC remains an interesting concept for neuromorphic systems as the fixed reservoir weights map very well to a variety of non–von Neumann hardware implementations. Tanaka et al. [29] review various physical RC implementations ranging from electronic to optical and mechanical as well as biological implementations. Bulk, fiber and integrated photonic RC systems are reviewed in detail in the study by Van Der Sande et al. [30]. Here, we will give an overview of the integrated systems.

Figure 4:

Illustration of the reservoir computing approach. The weights at the input and in the reservoir are randomly selected and kept fixed. The connectivity in the reservoir is sparse, and the connections are recurrent. The weights at the output layer can therefore be trained by a simple ridge regression.

Some of the early concepts for integrated photonic reservoir systems evolved around networks of semiconductor optical amplifiers (SOAs). Each SOA provides an optical nonlinearity owing to its power saturation behavior and has a rich internal dynamic behavior. In the study by Vandoorne et al. [31], a waterfall network architecture with feedback connections and SOA nodes was suggested. Its performance was compared to an echo state network with tanh activation function (classical software implementation) on a simple pattern recognition task, indicating a slightly better performance for the SOA network, despite the simple network architecture. Improved architectures were proposed, and the performance benefit over traditional software implementations has been demonstrated in numerical simulations for various tasks, such as, for example, spoken digit recognition [32]. However, owing to the large power consumption of the SOAs and, hence, the limited power efficiency of these networks, the first hardware realization was based on a different concept, a passive silicon photonic implementation.

The coherent silicon photonic RC implementation presented in the study by Vandoorne et al. [33] is based on a passive, linear photonic circuit. The linear reservoir nodes are arranged on a grid and connected through waveguides (delay lines), splitters and combiners to their neighbors (Figure 5a). The state at each reservoir node is read out through a detector, introducing a square nonlinearity at the output of the system; the reservoir itself remains linear. The operation speed of the reservoir is determined by the length of the waveguide delay lines between the neighboring nodes, which in this case was set to 2 cm (280 ps), to match the speed of the available measurement equipment. Using shorter delay lines, the operation speed could easily be increased to 100’s of Gb/s. Operation up to 12.5 Gb/s was demonstrated for a 16-node system and based on various tasks. In the experiments, the input signal was fed into a single node, and the output signals at 11 nodes were recorded. The output signals were digitized and weighted and combined in software. The training was performed offline using ridge regression. Excellent performance was reported for timewise Boolean operations like exclusive or (XOR), bit header recognition or spoken digit recognition. Various improvements on the architecture have been applied over the years. The input scheme was optimized, by injecting the input signal to multiple nodes for a better power distribution in the network [34]. The use of multimode waveguides was suggested in the study by Katumba et al. [35] to minimize the combining loss of Y-junctions, and a novel architecture based on four-port devices for minimal loss and improved state-mixing behavior was introduced in the study by Sackesyn et al. [36] (Figure 5b). However, the missing nonlinearity inside the reservoir, the bandwidth limitations and latency imposed by detecting and weighting the output signals in the electronic domain, as well as the large number of required photodetectors for parallel operation (one detector per node), will significantly limit the practical applicability of these systems.

Figure 5:

Two integrated photonic reservoir computing architectures were suggested in the study by Sackesyn et al. [36]. The swirl architecture (a) and the four-port architecture (b), wherein each node (black box) is connected to four other nodes. The four-port architecture offers improved power efficiency as it does not use Y-junctions but four-port devices to mix and redistribute the signals.

To overcome the later, all-optical photonic RC concepts have been suggested in the studies by Freiberger et al. [37] and Stark et al. [38] (Figure 6). The amplitude and phase of each node’s output signal are weighted in the optical domain using ring resonators [39] and phase shifters or Mach-Zehnder interferometers [40], and subsequentially, the weighted signals are summed together using a coherent photonic combiner tree. The optical output signal can then either be kept in the optical domain, e.g., for nonlinear dispersion compensation in an optical link [41], or be converted into the electronic domain for further processing using a single fast photodetector. However, these systems cannot make use of the nonlinearity through the detection as the detection occurs only after the node signals have been coherently combined. It is nevertheless appealing to work with such coherent, linear photonic systems as some problems like the well-known XOR function are linearly separable in the complex domain (ℂ), but not in the real domain (ℝ) [42]. An additional challenge for all-optical systems is that the individual complex node states cannot be read out; hence, training these all-optical networks requires iterative optimization techniques to optimize both phase and amplitude of each weight. A method to reconstruct the complex states at each node is discussed in the study by Freiberger et al. [43], showing promising performance in numerical simulations. To reconstruct the amplitude of the states, all weights are set to zero, and sequentially, a single weight is enabled, and the output is recorded. In a second step, the relative phase difference between the states and a selected reference state is obtained, by turning on this pair of weights. Blackbox optimization techniques like the covariance matrix adaptation evolution strategy (CMA-ES) work as well but require a much longer training time.

Figure 6:

Concept for an all-optical integrated reservoir computing system. The reservoir consists of a network of nodes (blue boxes) based on semiconductor optical amplifiers (SOAs) and multimode interferometers (MMIs), which are connected by delay line waveguides (blue dots). For the node connection, a four-port architecture was used. Additionally, a fraction of the light is coupled out at each node and transmitted to a photonic weight. We suggest using electro-optic switches based on barium titanate to implement the signal weights. After the weighting, the signals are combined through a coherent combiner tree, and the output signal is either converted into the electrical domain using a photodetector or kept in the optical domain.

One way to implement the missing nonlinearity in these systems is to embed SOAs based on III–V materials on top of the silicon photonic stack. In the study by Stark et al. [38], we demonstrated a concept for such a system (Figure 6), which is based on a four-port architecture, and besides the SOAs, uses nonvolatile optical weights based on electro-optic barium titanate technology. The main drawback of using SOAs is their large energy consumption and the rather complex fabrication process. An alternative concept to bring nonlinearity into the silicon photonic reservoir is the use of nonlinear microring resonators [44]. The nonlinearity occurs through two-photon absorption, free carrier absorption and dispersion in the ring resonator. By carefully optimizing the operation point and delay line lengths, promising performance and excellent energy efficiency was obtained in numerical simulations for a timewise XOR task.

In the following sections, we will discuss two specific examples where we see potential for integrated photonic implementations of analog MAC accelerators. For comparison, we reference some bulk optical systems as well. In the first example, we focus on neural network inference, whereas the second option also opens a path toward efficient neural network training.

2.2 Integrated photonic devices for neural network inference

For neural network inference, there is a need for real-time power-efficient vector-matrix multiplications of high-speed signals. In literature, several examples were demonstrated of cascaded Mach-Zehnder interferometer structures performing a unitary matrix calculation. The matrix elements, representing the synaptic weights, are externally controlled by setting the phase of electro-optic tuning elements in the Mach-Zehnder arms [45]. Alternative architectures are, for example, based on ring resonator filters [46]. Barium titanate on silicon photonics electro-optic devices offer ultrahigh-speed phase modulation [22] and ultralow power tuning in the nanowatt range [47]. The maturity of state-of-the-art silicon photonics and silicon nitride and indium phosphide platforms is well suited for the implementation of this type of devices. Most demonstrations are limited to 4 × 4 matrix sizes, and indeed, challenges arise in scaling to larger matrices. Phase errors limit the performance, and as full control of an N × N matrix requires 2N² phase shifters, setting all elements requires many electrical signals to be controlled. This makes this option well suited for small-size matrix operations (N < 32) as, for example, in convolutional signal processing [48]. The inference calculation is performed in a fully parallel manner, and execution time and effort do not depend on the matrix size. An important aspect is the option to adjust the weight values very fast out of a prestored set of values. This opens the opportunity to adapt the matrix at the same speed as data is coming in, for example, to perform different types of convolutional operations. Another promising approach for neural network inference is based on the integration of phase-change materials as adjustable absorbers with integrated optic circuits [49], [50].

2.3 Integrated photonic devices for neural network training

The second exciting opportunity for integrated optic technology is related to artificial neural network training. Establishing an enhanced technology platform for neural network training is of utmost interest. Recent publications show the large environmental footprint of today’s technology in neural network training, as described in the introduction [7]. There are two basic approaches to optimize the training of photonic neural networks. Either the training methods are adapted to match the system capabilities, or the operations used in a general training method like stochastic gradient with backpropagation [15] are accelerated through photonic hardware.

An example for the former concept is given in the study by Bueno et al. [51], wherein Bueno et al. implemented an iterative photonic learning method based on a greedy learning algorithm for a 4f free-space RC system with 900 nodes. A digital micromirror device is used to set the output weights, which are therefore binary. The greedy learning algorithm randomly selects an output weight and switches its state. The system performance is evaluated, and if it improved, the new output weight configuration is kept; otherwise, the previous configuration will be restored. These steps are applied iteratively, until the required performance is achieved, or the error rate converges. The training method was able to optimize the weights in about 900 learning iterations for one-step-ahead prediction of the chaotic Mackey-Glass time series, with good generalization and performance.

For the latter, we present a concept, which extends the inference calculation of the synaptic connection between two neural layers to a technology platform in which also the backpropagation and weight update steps are performed in a fully parallel manner by optical signal processing. In the Mach-Zehnder interferometer-based vector-matrix multiplication concept, the matrix element values are set by an external subsystem. Hence, changing these values in an optimization procedure would require signals to flow from the neural network output to the control system. An in situ training algorithm for this type of structures was proposed [52] supporting the backpropagation algorithm [53]. It is based on performing intensity measurements in the device and storing the obtained values for processing in the subsequent steps. This communication path would still introduce an information flow bottleneck and therefore limit the performance and power efficiency of the training algorithm. A local weight update mechanism is required, directly fetching the signals in the network itself. Here, we first summarize the backward propagation algorithm as this helps to understand the merits of the optical signal processor presented thereafter. To train a feedforward DNN, we can use stochastic gradient descent together with backpropagation as follows (Figure 7) [54]:

1. Forward propagate training input samples x_k with target response t_k and store the corresponding outputs y.

2. For each training sample, compute the loss between the target output and the obtained outputs using a loss function. Often, the squared error is used as a loss function.

3. For each training sample, find the error signal δi=δLδzi, which indicates how large the influence of the input at a neuron on the total loss is. This error signals can be obtained by propagating the loss backward through the network with transposed weight matrices and using derivative of the activation functions [54]. http://neuralnetworksanddeeplearning.com/chap2.html

4. Using the error signals obtained in step 3, update the weights as follows, to minimize the loss

where α is the learning rate and xi⊗δi=δLδWi is the partial derivative of the loss with respect to the weights, which is averaged over all training samples.

1. Iteratively repeat steps 1–4 until the loss reaches a minimum.

Figure 7:

Illustration of forward- and backpropagation through a feedforward neural network with two hidden layers (similar to Fig. 3b) for training of the network weights. To train the network many training samples x with target output t are forward propagated through the network and the resulting output y are stored. Next, the loss which describes the error between the output y and the desired output is computed. Using backpropagation, we compute the error signals δi for each layer for all training samples. The weight updates are averaged over all training samples. This procedure is repeated iteratively until the loss is minimized.

Already in the 1990s, a photonic system was demonstrated in which the weighting elements are stored in a bulk crystal of a photorefractive material [55]. The MAC operation is obtained through the diffraction efficiency of a refractive index grating formed in the photorefractive crystal through the interference of two beams. A detailed description of signal processing in photorefractive materials is given in chapter 3 of the study by Denz [20]. In Figure 8, we depict the formation and operation principle of first a single weight and then two synaptic weights.

Figure 8:

(a) A synaptic weight is formed in the photorefractive crystal through the interference of two light beams. Charge carriers are optically excited and diffuse to dark regions of the interference pattern. The charge separation induces an electrical field, and through the Pockels effect, a modulation of the refractive index is induced. (b) An optical signal (S1) impinging from the direction of source 1 will now be diffracted toward destination 1, resembling the operation of a single synapse. (c) A second grating is formed by applying a second source (source 2), while the destination direction is kept the same. (d) By subsequently applying input signals from the directions of source 1 and source 2, an analog multiply and accumulate operation is performed toward destination 1.

The diffraction efficiency of an input signal to one of the outputs represents the respective synaptic weight. Also, a part of the other input signals is diffracted toward the same output where they are coherently combined, representing the accumulation function. Because the full input vector can be applied in one inference cycle, the vector-matrix calculation is of O(1), independent of the size of the matrix. The inference calculation is now visualized for multiple beams in Figure 9a.

Figure 9:

All critical vector-matrix calculations for neural network inference and training are efficiently performed as O(1) operations. (a) The input light is diffracted by the refractive index grating (green), which is stored in the photorefractive material. The weight matrix is given by the diffraction efficiency of the input signals to the different outputs. (b) The same refractive index grating can be used to compute the product of the transpose of the weight matrix by using alternate inputs and outputs. (c) A weight update is performed by writing a new refractive index grating by applying the input and error vector at the same time.

The backpropagation function is performed on the transpose of the matrix (W^T). The alternate input and output sections enable a straightforward calculation of the transpose operation on the same photorefractively imprinted grating, as indicated in Figure 9b. Finally, the weight update is induced by applying both the input and the error vectors at each network layer. All operations are of O(1), imposing a massive performance improvement compared to digital signal processing in von Neumann systems. Compared to the method proposed in the study by Hughes et al. [52], only intensity measurements at the device periphery are required and need to be transferred between layers of the network.

The availability of silicon photonics and the cointegration of materials like barium titanate or thin III–V layers [56] open opportunities for chip-level implementation of an analog photonic synaptic processing unit. In Figure 10, we show a device layout to implement the neural network operations based on the photorefractive effect, as depicted in Figure 9.

Figure 10:

Schematic representation of an integrated photonic synaptic processor for inference and training. The device has one coherent optical input. The electrical input signals drive electro-optic modulators in the transmitter array sections to set the optical input vector. Collimating mirrors convert the diverging optical waves in the planar waveguide sections to collimated beams entering the photorefractive region under slightly different angles (compare with Figure 8). Both transmitter arrays are operated simultaneously to update the gratings stored in the photorefractive material. For the inference and backpropagation steps, only one transmitter array is used. The resulting optical output signal is detected by the receiver array and converted back to the electrical domain.

A thin layer of a photorefractive material is bonded to the silicon photonics wafer in which the periphery to operate the processing section is integrated. Electro-optic modulators convert the electrical input vector to the required power and phase of the optical beams. Detector arrays convert the vector-matrix output signals back to the electrical domain. To theoretically estimate the viability of this concept, we take gallium arsenide (GaAs) as the photorefractive material. The retention time of GaAs as a weight storage medium is approximately 300 ms. With a projected cycle time for the inference and backpropagation steps of 20 ns and an update cycle of 100 ns, approximately 10⁴–10⁶ operations can be performed before the weight values must be refreshed. Therefore, this does not represent a hurdle for the applicability of this technology for neither neural network inference nor training. The theoretical storage density is large; we evaluated that the storage of 10⁶ weights in a thin layer with dimensions of 5 × 5 mm² is possible. Note that the peripheral devices must be added to the required area, resulting in a total size of 25 × 25 mm², which is still feasible. The estimated total operating power is less than 5 W. In addition to the parallelization, this in-memory computing concept avoids communication to and from memory, which is an essential aspect in meeting the power efficiency advancement. Calculations for electrical systems predict a power efficiency and performance advancement of factors larger than 100 [13]. For the optical case as presented here, we anticipate similar values as the power requirements for operating the devices are on the same order of magnitude as for the electrical memristive structure. Whether an optical implementation will be a viable solution compared to the electrical memristive structure will depend on the performance, form factor and application. Clearly, the electrical solution will have an overall larger areal density of about a factor of 20–50. Inherently, the photorefractive effect provides well-controlled setting and trimming of the weight values. This is important for efficient training and opens opportunities for analog vector-matrix multiplications with regularly updated matrix elements.