Computational Phase Transitions: Benchmarking Ising Machines and Quantum Optimisers

While there are various approaches to benchmark physical processors, recent findings have focused on computational phase transitions. This is due to several factors. Importantly, the hardest instances appear to be well-concentrated in a narrow region, with a control parameter allowing uniform random distributions of problem instances with similar computational challenge. It has been established that one could observe a computational phase transition in a distribution produced from coherent Ising machine(s). In terms of quantum approximate optimisation, the ability for the quantum algorithm to function depends critically on the ratio of a problems constraint to variable ratio (called density). The critical density dependence on performance resulted in what was called, reachability deficits. In this perspective we recall the background needed to understand how to apply computational phase transitions in various bench-marking tasks and we survey several such contemporary findings.


Physics of computation and hardware benchmarking
Physical processes have inspired many computational algorithms such as simulated annealing, Monte Carlo methods, random walks and more. These algorithms function by simulating a physical process-such as cooling-for the purpose of finding extrema (i.e. minimisation, a.k.a. optimisation). Recently such physics inspired algorithms have become increasingly replaced by an actual physical device. In other words, instead of simulating the physics inspiring a computer algorithm, we can build a physical computing device that actually implements e.g. the cooling process. Quantum effects are sought to accelerate the physical optimisation processes.
Recent experimental progress has culminated to produce a range of physical optimisers. These optimisers essentially allow us to create energy minimisation of what physicists call (tunable) Ising models. This is equivalent to what computer scientists call, quadratic binary optimisation. This approach is realized, for example, using Ising machines based on optical parametric oscillators [1,2], annealers based on superconducting bistate systems [3,4,5,6], and gate based superconducting electronics qubits executing quantum approximate optimisation (QAOA) [7,8]. We can also find quantum simulators and processors based on trapped ions [9] as well as optical approaches [10,11] and even few-monolayer spin ice films [12]. Approaches based on silicon or diamond defect can also be found [13].
As each physical optimiser has been built and developed based on different technologies, these physical processors have their own advantages and disadvantages. D-Wave processors-based on superconducting circuits-utilise some quantum effects to find a minimum of a given objective function. However, connectivity between binary units in this type of hardware is sparse and thus restricts the forms of objective functions to be implemented. On the other hand, coherent Ising machines [1,2] provide all-to-all connectivity. Such approaches [1,2] lack quantum effects.
We do not know when these physical processors will outperform standard supercomputers. The average or general performance is not the only concern: indeed, what types of problem(s) might these physical processors perform well on? And will they perform well when standard numerical approaches fail?
Although there are several avenues to benchmark physical processors, we have advocated heavily that one will consider computational phase transitions [14,15,16]. This is due to several factors. Importantly the hardest instances appear to be wellconcentrated in a narrow region, with a control parameter allowing uniform random distributions of problem instances which appear to pose similar computational challenge.
In a recent theoretical proposal, several of the authors have sought to physically observe a computational phase transition in a distributions produced from coherent Ising machine(s) [14]. In addition, several of us have also applied these same tools to quantify the performance of QAOA [15,16]. Regarding QAOA, we found that the easy-to-hard transition plays an important role. Namely, the ability for the quantum algorithm to function depends critically on the ratio of a problems constraint to variable ratio (called density). The critical density dependence on performance resulted in what was called, reachability deficits [15,16].

Complexity, Physics and Phase Transitions
In statistical physics, the concept of phase transitions has been used to describe an abrupt change in certain properties of a system as a result of the change in external conditions. The early concept arose to explain the phases of matter with transitions between e.g. solid, liquid, and gas (or even plasma in rare cases) due to the effects of changes in parameters such as temperature and pressure. In so called, complex systems, phase transitions can be related to the self-organized processes-which are emergent patterns-as well as order in a system arising from internal processes involving local interactions. Thus a system might transform from disorder to order. Mathematically, a system undergoing a phase transition exhibits discontinuities (at the transition point) in the large system size limit.
In computer science, the phenomena of phase transitions, called computational (or algorithmic) phase transitions, has been studied extensively. The phenomena was first reported in variants of the satisfiability (SAT) problem [17,18,19,20,21,22], which are NP-complete decision problems [23] and hence represent NP-hard optimisation problems. The problem consists of determining whether a Boolean formula of binary variables can evaluate to TRUE, and hence be satisfied. If k is the number of variables in each clause, the problem is called k-SAT.
The problem undergoes drastic structural changes (phase transitions) at critical values of an order parameter. It has been observed that the rapid change from satisfiable instances to unsatisfiable instances takes place at a certain clause to variable ratio (clause density), and moreover we empirically observe that computational algorithms seem to slow down at that critical point (see Fig. 1). The requirement of increasing computational resources to solve instances near this transition point suggests that difficult instances for the k-SAT problem concentrate around this critical density (an easy-hard-easy transition) [17,18,19,20,21,22]. These two features indeed show the signature of the so called computational (or algorithmic) phase transition.
These phase transitions seem to be a priori different; in physical sciences, one is the onset of non-trivial macroscopic collective behavior in a system composed of a large number of elements that follow simple microscopic laws [41], and in computer science, one is computational difficulty existing in a solution search process of difficult computational problems. Perhaps this connection stems from the Church-Turing-Deutsch (CTD) principle [42] which states that a universal quantum computing device can simulate every physical process (and conversely). By this concept, clearly every  [14] and based on works appearing in [17,18,19,20,21,22].
possible computing machine must be governed by the laws of physics. Meanwhile certain physical processes can be made to solve computational problem instances. This principle implicitly interrelates physical processes to computational complexity. The connection between computational complexity and phase transitions have been studied [43,44,45,46,47,48,37,49,50,51]. By this connection, randomly generated k-SAT instances can be interpreted as random realizations of a spin glass written in the form of the generalized Ising Hamiltonian with at most k-body interactions. This correspondence has allowed physicists and computer scientists to analyze the critical behavior of k-SAT in the language of statistical mechanics. It also has provided some logical basis for considering k-SAT in a physical spin system capable of exhibiting phase transitions.
However, as reported in [14], numerical prediction shows that the algorithmic phase transition signature is possible to be observed in such contemporary physical computing devices via Gibbs sampling. Considering the thermal Gibbs states of physical systems, the probability of ground states occupancy decreases around the phase transition point. This finding indicates that the difficulties in sampling solutions of the SAT problem concentrate around the critical point. Indeed we observe that, for any fixed temperature, there exist problem instances that would require significant sampling time to retrieve the ground states, especially at the phase transition point.
This prediction connects the computational phase transition signature with physical processes that feature in observable quantities. Since recent Ising devices [1,2] and quantum annealers [3,4,5] have been constructed with increasing programability, and SAT instances can be directly embedded into the generalize Ising Hamiltonian, the computational phase transition signature has the potential to be physically observed.
Such an experiment provides a bench-marking tool for devices to locate difficult problem instances. However, simulations of thermal Gibbs states in such devices have been practically challenging as a required exact numerical solution.

Benchmarking quantum processors via QAOA
Variational hybrid quantum/classical algorithms have been developed to utilize noisy intermediate-scale quantum (NISQ) devices to find approximate solutions to combinatorial optimization problems. These variational algorithms such as the variational quantum eigensolver (VQE) and the quantum approximate optimization algorithm (QAOA) train parameterized quantum circuits by using measurementfeedback loops to optimize a given objective function.
The QAOA is the most studied gate based model for optimization on NISQ architecture. Due to its relative ease in realization on existing processors and also due to recent universality results, QAOA shows increasing potential [58,59,7,11,53,8,60].
MAX-SAT or maximum satisfiability, which is the optimization version of decision SAT, also features the computational phase transition and therefore fits our purpose of benchmarking quantum processors via QAOA. Unlike decision SAT, MAX-SAT focuses on finding variable assignments that maximize the number of satisfied clauses in a given instance. This problem exhibits a phase transition (an easy-hard transition) at a critical clause density α c = 1.
A so called backbone corresponds to critically constrained variables that have fixed values in all optimal solutions. At the critical point, MAX-SAT undergoes a sharp transition from nonexistence of backbones when under-constrained to large backbones when over-constrained [61]. In random graph theory, the transition is generally related to the birth of a giant component in a random graph [62].
Interestingly, although 2-SAT and 3-SAT differ considerably in the exhibited point of their phase transition, the optimization versions exhibit the criticality at the same α c = 1. Reported in [15], the authors numerical studied QAOA performances on the MAX-SAT problem and found that the average error in QAOA approximation (error) starts to increase around the critical clause density. This suggests that the QAOA is sensitive to the most essential feature of problem hardness (see Fig. 2). The approximation inefficiency for fixed depth QAOA brought about by reachability deficits, strongly correlates with the density, which acts as an order parameter for QAOA performance. Moreover, the effects are persistent regardless of increasing depth.  Squares illustrate the average value obtained over 100 randomly generated instances for 6 qubits with error bars indicating the standard mean error . Plots also illustrate improved performance for higher depths, but the phase transition always occur at α c = 1. Figure reproduced from [15].

Conclusions
Computational problems such as Boolean satisfiability feature abrupt changes similar to the phase transitions that are exhibited in condensed matter systems. From a computational standpoint, it is at these critical regions that a worst-case scaling in computational resources is observed, making them more difficult to solve.
Recently-with the advancements made in physical computing devices-where the aim is to allow physical devices to naturally solve computational problem instances, an apparent connection first proposed by the Church-Turing-Deutsch principle appears important. If instances at critical regions are difficult to solve computationally, so should the hardness be reflected in physical computing devices. It is under this notion we propose to use computational phase transitions as a tool to benchmark different physical computing devices [14,15,16].