Automatic discovery and optimization of chemical processes

This paper presents the ﬁrst overview of recent developments in techniques and methods that enable closed-loop optimization, also sometimes called ‘self optimization’, as well as discovery in different areas of molecular sciences. The closed-loop experimental platforms offer tremendous new opportunities by signiﬁcantly increasing productivity, as well as enabling completely new types of experiments to be performed. Such experiments involve three main enabling technology areas: automated experimental systems, analytical instruments connected to automated chemoinformatics software and optimization or decision-making algorithms. We review the most exciting developments concerning robotic experiments, 3D printed lab-ware, experimental systems with multiple analytical instruments and advanced optimization algorithms based on machine learning approaches. A range of different chemical problems is described, which show the breadth of potential applications of this emerging experimental approach


Introduction
Recent advances in laboratory equipment automation and in new analytical methods rapidly transform the way in which experiments in molecular sciences are performed. These changes indicate a remarkable transition, which profoundly affects the research and development processes, and scientific methodologies across many areas of molecular sciences, as well as creating new business opportunities and new business models. Several pioneering studies have recently demonstrated a range of possibilities in automatic optimization of process conditions [1,2,3 ,4-6], exploring new chemical structures [7 ] and in the search for new bio-active compounds [8].
Technical underpinnings of this transition could be traced to the emergence in recent years of several new capabilities: rapid in situ or in line analysis of the outcomes of chemical transformations, chemoinformatics tools for treating large amounts of analytical data from data-rich experiments, miniaturization of experiments to perform reactions in microreactors and in microdroplets, automation of experiments and ubiquity of computing, which enable the development of closed loop systems with advanced design-of-experiments algorithms, and, finally, the emergence of additive manufacturing as a method of producing reaction ware.
Thus, the transition to the new experimental paradigm is a nexus of several scientific and technical developments that came to maturity at a more or less the same time.
Here we give a concise overview of the techniques and methods that underpin the new experimental platforms, and focus on the emerging trends and gaps in our technical capabilities and knowledge. The focus of this paper is on new experimental and software platforms that open completely new opportunities in optimization and discovery. The highly active and exciting area of automation and control of large-scale chemical processes is out of scope of this paper.

Hardware for self-optimization of closed-loop systems Labware
The new experimental platforms for optimization and discovery require very different lab-ware from the conventional round bottom flasks or even simple flow microreactors. Such experimental systems combine automation of reaction ware and of analytical instruments to create a closed-loop control system, schematically shown in Figure 1. The control algorithm in this case is implementing a specific strategy of experimental design (DoE).
An example of a highly automated and instrument-rich reaction ware is the Automatic Continuous Online Monitoring of Polymerization Reactions (ACOMP) system, which included a recirculation loop with automated sampling and sample preparation (dilution) to allow measurements of refractive index, UV-vis, viscosity, multi-angle light scattering and gel-permeation chromatography (GPC), all connected in series [9 ]. This reactor system was made for close to real-time monitoring of polymerization reactions without human interaction and was able to operate in batch, semi-batch and continuous experiments [10-12].
The important feature of such a system is a combination of analytical techniques that provide complementary information for a complex chemical system and largely include fast analytical methods based on optical and spectroscopic measurements, but also include more slow methods, such as GPC. ACOMP represents an example of an automated reaction system with several in-line analytical instruments, but it lacks two other necessary components for a closed-loop optimization: chemoinformatics and design of experiments.
The ACOMP system was built using a conventional approach to laboratory automation, using signal conditioning on data acquisition National Instruments platform and LabVIEW software interface. A similar approach has been followed in most systems reported to date, in which self-optimization has been implemented. These systems cover different areas of chemistry: polymerization reaction [13 ,14 ], heterogeneous catalytic processes [1,2,5], droplet formation [15,16] assembly of nanoparticles [17] and even multi-step syntheses [4,[18][19][20][21]. As we shall show below, this approach to automation is readily extendable to include chemoinformatics and design of experiments, using data exchange standards, such as OPC. This brings the developed automated optimization and discovery systems very close to industrial production systems with respect to control hardware architectures. A significant departure from this approach is the use of dedicated robotic experimental systems, which could either involve parallel batch or flow experiments. Such systems are developed for high-throughput experiments and enable rapid discovery of new reactions [22,23,24 ], or optimization of process conditions of known reactions [25,26]. A flow array system has been used to discover a new inorganic cluster [7 ]. A recent example from Merck & Co. Inc. shows that high throughput robotic experiments could be performed with very small quantities of reagents, making this technology highly appealing as a platform for chemical discovery [27].
Another significant recent development is the ability to reproduce labware through additive manufacturing, such as 3D printing [28 ]. This enables not only rapid development of ideas for new reactor types, but also sharing of ideas across many laboratories through universal access to identical labware. The uptake of this technology strongly depends on the continuing decrease in the cost of 3D printing equipment, the ability to print chemically resistant and thermally stable devices, and on the development of user resources and user communities with an open innovation philosophy.  Target product or process Discovery of new product or process

Initial input into DoE
On-line, in situ and in-line analytical methods Spectroscopic methods are providing fast and non-destructive chemical and process information about reacting systems, which is critical for the implementation of selfoptimization experiments. The use of in-line optical spectroscopy methods for real time sensing and control has been studied intensively over the last decade [2,29 ,30 ,31 ,32]. Major drawbacks of most spectroscopic analytical methods include relatively low sensitivity (excluding fluorescence spectroscopy) and the need for calibration models for multi-component reaction systems.
The problem of low sensitivity is particularly acute for tasks requiring quantification of minor components, such as optimization of medicinal syntheses. It is less of an issue when large variations in the measured variables are monitored, such as moisture (e.g. to enable real-time optimization of drying), pH, conductivity, etc. The latter methods carry less 'chemical' information compared to spectroscopic methods that provide evidence of molecular structure and concentration. The significant advantage of in-line or in situ spectroscopic methods over off-line methods is revealed in the case of unstable samples, when degradation of the substance of interest makes off-line analysis highly unreliable [33]. This highlights the importance of sampling time in the range of applications of interest for the new reaction platforms, and allows to further differentiate from the conventional applications of sensors in real-time control.
Here the aim is to either determine if a reaction outcome satisfies the criteria set out for a new process (in the process optimization/discovery scenario) or for a new product function (in the product discovery scenario). In this case, one can imagine that experimental system may be kept in steady state or idle till analysis results are available, prior to a new set of input conditions is established. This is rather different from real-time process control, when sampling time must be commensurate with the system's response time to allow predictive control.  input parameters that must be tested is large.
Self-optimization instrumentation and experimental philosophy differs from such classical DoE in the sense that there is an opportunity to learn from each experiment and update a DoE model in a sequential fashion. In this case a DoE algorithm may take the form of an optimization algorithm. This strategy has been implemented in several studies. Thus, simple linear algorithms have been used for optimization of flow and temperature in catalytic reactions [3 ,4,5]. In the case of the more complex problems with a large number of independent input variables such linear algorithms, for example a simplex algorithm [40,41], would lead to an unreasonably large number of experiments.
A more advanced approach to sequential design of experiments is to use machine learning optimization algorithms that allow taking advantage of the results of the previous experiments. The class of algorithms that are suitable for this task are either global or target optimization algorithms. One of the most prominent algorithms for such sequential optimization is the efficient global optimization (EGO) algorithm [42]. Since its introduction, the algorithm has been adapted for different types of optimization problems, including target optimization. One idea is to make use of the concepts of desirability [43] and virtual observations [44] to construct an algorithm capable of identifying and, with each iteration, improving on a cluster of solutions that best associate with target values [45]. Even though the algorithm undoubtedly explores globally throughout the search, it is not designed to actively search for solutions that would allow one to gain the most information about the underlying process (i.e. solutions optimal in terms of experimental design).
Machine learning algorithms require construction of a statistical surrogate model, which is then used to predict the outcomes of the future experiments, the decision making process. Gaussian Process is a popular surrogate model type as it provides a principled way of assessing uncertainty of the model and has successfully been used in many optimization problems, including chemistryrelated problems [46][47][48]. The advantage of using Gaussian Processes is in the ability to deal efficiently with both demands on the sampling criterion: exploration and exploitation. A similar approach is to use a surrogate based online evolutionary algorithm [39 ,42,49 ,50 ,51-54]. The attractive features of this algorithm are that (i) it has an evolutionary algorithm at the core, capable of solving multi-dimensional multimodal problems, and (ii) attempts to strike a balance between the need to reduce the amount of expensive evaluations and the need to improve on the quality of the surrogate model. Although unfamiliar to most practicing chemists, such advanced optimization methods have already been applied in several chemical processes. Thus, Gaussian Processes has been used in prediction of quality of polypropylene [55 ], in simulation of catalytic batch etherification reaction [56], in real-time prediction of properties for industrial rubber mixing processes [57] and in screening of new additives for a Friedel-Crafts catalyst [58 ].
Recently a new combination of Gaussian Processes, mutual information and a genetic algorithm for multi-target optimization has been published -a multi-objective active learner (MOAL) algorithm [39 ]. This algorithm outperformed another published algorithm, the surrogate 4 Reaction engineering and catalysis based on-line evolutionary algorithm (SOEA) [59,60 , 61,62], compared on the basis of in silico tests with different mathematical functions. Due to its objective of multi-target optimization, the MOAL algorithm can be used as a decision making software in discovery of products with specified properties or new chemical reaction routes with a specified target.
MOAL algorithm was recently applied as a decision making element of a closed-loop system in a process of experimental discovery of recipes for semi-batch emulsion co-polymerization [63]. For this the algorithm was extended to incorporate a Gaussian Process (GP) binary classification model [64]. The task of the classification model is to learn the regions of feasible experiments, assuming that the experiments are done for a problem with unknown bounds of experimental input variables.
The classification was performed in reduced space via application of a dimensionality reduction technique. Normally this can be done by principal component analysis (PCA) [65] or multidimensional sealing (MDS) [66] techniques. These techniques are limited by their global linearity. To use classification model in a highly dimensional nonlinear case, manifold learning techniques, such as locally linear embedding (LLE) [67] and Isomap [68 ] can be used. In the MOAL algorithm the SIsomap [69] was incorporated as a classification model to speed up the process of identification of the feasible experimental range. This system was able to find a feasible recipe for a two-monomer semi-batch co-polymerization within 20 experiments. The result is important in the context of the large number of input variables -in this case 14 input variables were treated as independent and used in the optimization.

Outlook
Closed-loop experimental systems have recently emerged as a powerful tool for optimization of process conditions and as a potential discovery platform. This platform thus far was only developed for sequential experiments, where advantage is taken of the knowledge obtained in previous experiments. Here, the more sophisticated statistical algorithms are capable to design experiments and minimize the number of experiments required for optimization or discovery. The area of automated experimental discovery is still in its nascent state. However, the range of the already demonstrated applications shows the potential significance of this approach: optimization of heterogeneous and homogeneous catalytic reactions, synthesis of new materials, synthesis of new bioactives, discovery of new process recipes for semi-batch reactions.
Current state of technology is characterized by several limitations, mainly in its analytical and decision making components. At present a narrow range of analytical techniques can be used in automated experiments.
Development of fast and cheap in-line LC-MS and NMR techniques, as well as methods of observing product properties (particulates properties, for example) in real time and under reaction conditions, would significantly broaden the range of applications. The robotic experimental platforms available on the market are already better than what can at present be realistically used in closed-loop systems due to limitations of both the analytical instrumentation and in the design of experiments algorithms. A serious current limitation of the decision-making algorithms is the lack of use of a priori knowledge of the chemical systems in design of experiments. These current limitations are all being addressed simultaneously and, undoubtedly, the technology of automated optimization and discovery in molecular systems will see rapid development over the next few years.

9.
Alb AM, Drenski MF, Reed WF: Automatic continuous online monitoring of polymerization reactions (ACOMP). Polymer Int 2008, 57:390-396. This is the most instrument-rich automated experimental system with inline monitoring which provides many sensor outputs for development of process understanding in a highly complex chemical system.

28.
Kitson PJ, Marshall RJ, Long D, Forgan RS, Cronin L: 3D printed high-throughput hydrothermal reactionware for discovery, optimization, and scale-up. Angew Chem Int Ed Engl 2014, 53:12723-12728. This is the first paper on 3D printing of labware. Most notably the group of Cronin developed polypropylene printed devices for improved chemical resistance.

29.
Mozharov S, Nordon A, Littlejohn D, Wiles C, Watts P, Dallin P, Girkin JM: Improved method for kinetic studies in microreactors using flow manipulation and noninvasive raman spectrometry. J Am Chem Soc 2011, 133:3601-3608. The 'push-out' method of collecting kinetic information allows to generate large amounts of data in a flow experiment, making it closer to conventional batch kinetic experiment. This contributes to the debate whether flow or batch experiments are better for collecting the intrinsic kinetic information. The difficulty of the 'push-out' method is in the impact of specific residence time distribution of the device used on observed reaction rates. The assumption of plug-flow is very difficult to attain in conventional 'flow' reactors used by most groups. Hence, the obtained kinetic data is not the 'true' kinetics, but must be treated mathematically having the experimental residence time distribution.

34.
Danieli E, Perlo J, Duchateau LL, Verzijl GKM, Litvinov VM, Blü mich B, Casanova F: On-line monitoring of chemical reactions by using bench-top nuclear magnetic resonance spectroscopy. ChemPhysChem 2014, 15:3060-3066. This is the first paper in which on-line monitoring of chemical reactions by bench-top NMR was presented. The following paper (34) also shows how to include flow NMR measurements into a closed-loop experimental system.