A linear optimal feedback control for producing 1,3-propanediol via microbial fermentation

In this paper, we consider a multistage feedback control strategy for the production of 1,3-propanediol(1,3-PD) in microbial fermentation. The feedback control strategy is widely used in industry, and to the best of our knowledge, this is the first time it is applied to 1,3-PD. The feedback control law is assumed to be linear of the concentrations of biomass and glycerol, and the coefficients in the controller are continuous. A multistage feedback control law is obtained by using the control parameterization method on the coefficient functions. Then, the optimal control problem can be transformed into an optimal parameter selection problem. The time horizon is partitioned adaptively. The corresponding gradients are derived, and finally, our numerical results indicate that the strategy is flexible and efficient.

1. Introduction. 1,3-Propanediol (1,3-PD) has a wide range of applications in cosmetics, polymers, adhesives, lubricants and medicines [12]. At present, there are two methods for producing 1,3-PD: chemical synthesis and microbial conversion. The second method is preferred because it is relatively easy to implement and does not generate toxic byproducts. However, microbial conversion usually yields a lower 1,3-PD concentration when compared with traditional chemical synthesis methods.
There are two ways to solve this problem: one is to improve the conversion rate of the substrate through improvement of the bio-reaction technology, and the other is to improve the conversion rate by optimizing the biochemical processing technology [21]. Improving the productivity of the microbial conversion process by optimization techniques is urgently necessary.
There are three conventional methods of microbial fermentation to produce 1,3-PD: batch culture, continuous culture and fed-batch culture. Although batch culture and fed-batch culture are preferred in large-scale industrial fermentation, the continuous culture is more appealing because of the higher production as well as the increased stability and automation. This paper focuses on studying continuous culture in which fresh medium is added while the old medium is removed during the reaction [22,3,7,25].
There are significant improvements in continuous fermentation. Zhang et al. [30] presented a nonlinear hybrid system, which described the intracellular reductive pathway according to the possible transport mechanisms of 1, 3-propanediol across a cell membrane. Gao et al. [4] discussed three different ways for glycerol and one way for 1,3-PD (passive diffusion and active transport) to pass the cell membrane. They also established a modified fourteen-dimensional nonlinear hybrid dynamic system with genetic regulation to describe the continuous fermentation. Lv [19] considered a nonlinear non-differentiable dynamic system in continuous cultures involving all possible metabolic pathways of the inhibition mechanisms of 3hydroxypropionaldehyde onto the cell growth and the transport systems of glycerol and 1,3-PD across the cell membrane.
The study of the microbial conversion process for synthesizing 1,3-PD started in the 1980s [29]. There is more focus on the research of the fermentation process of 1,3-PD with different methods now. The optimal control problems with state constraints [13,16] arise in a wide range of practical applications, such as continuous culture in the production of 1,3-PD [18]. Also, researchers proposed several new computational methods [20] to solve this problem with continuous inequality constraints. Although these methods were proven to be effective, they were only capable of producing an open loop control, which may not be robust in practice [8]. An optimal feedback control, expressed as a function of the current system state, is usually more effective. In this paper, we will consider a multistage feedback control strategy to maximize the yield of 1,3-PD.
The traditional approach to determine an optimal feedback control involves solving the well-known HJB partial differential equation. Other relevant approaches to solve the feedback control problem include the sensitivity penalization approach for computing robust suboptimal controllers [17], and the neighboring extremal approach [2,6]. Loxton [8] considered a general optimal control problem in which a feedback controller of a given structure was optimized by altering certain adjustable parameters. However, none of the previous studies considered the feedback control parameters function with time dependence. We propose to generate an adaptive grid for time to incorporate the variations of parameters in different time intervals. Therefore, the strategy has more flexibility comparing to those methods with fixed parameters throughout the whole interval of interests. To solve the optimal feedback problem, we use the control parameterization method. Control parameterization is a powerful numerical technique to solve optimal control problems with general nonlinear constraints [15]. The main idea is to discretize the control space by approximation with a piecewise-constant or piecewise-linear function, thereby yielding an approximate nonlinear programming problem [9]. Reference [31] studied a control vector parameterization (CVP) based hybrid algorithm, HAPSODSA-CVP, which solves the nonlinear chemical dynamic optimization problems. In this algorithm, the adaptive particle swarm optimization (APSO) is applied to enhance the global search ability, while the differential search algorithm (DSA) is used to improve the local exploitation ability.
In this paper, we present an innovative strategy to study the optimal control problem in the continuous culture of microbial fermentation. This method has four significant contributions. Firstly, the optimal control law is given as a generalized linear feedback control. Secondly, the parameters in the linear form are considered as a continuous function of time t, since the continuous culture is a long-term process control problem. Thirdly, the time-based process control is transformed into a multistage linear feedback control based on the control parameterization technique. Last but not least, both the number of stages and the length of each stage's interval are self-adapted in this scheme.
The structure of the paper is as follows. A nonlinear dynamic model is introduced to describe the microbial continuous culture process in Sec. 2. Then, the corresponding optimal control problem and the strategy of the multistage feedback control are discussed in Sec. 3 and Sec. 4, respectively. In the following sections, we use techniques in control parameterization method, and the optimal feedback control problem is transformed into a multi-stage linear feedback control form. Finally, an adaptive gradient-based optimization algorithm is developed to solve the optimal feedback control problem is given in Sec. 7, and the numerical results are explained in Sec. 8.
2. Problem statement. Following the continuous fermentation process and the hypothesis used in [5,26], we propose two assumptions for our dynamical model. Assumption 2.1. The material composition in the fermentation tank is homogeneous. Namely, it does not change in space, and the solution in the reactor is sufficiently well-mixed such that the concentrations of reactants are uniform.
Assumption 2.2. The continuously added medium only contains glycerin, and the substance in the reactor is exported at a dilution rate, denoted by D.
Under these assumptions, the mass balance relationships of biomass, substrate, and products in the microbial continuous fermentation can be described by the following nonlinear dynamic system: and where x i (t), i = 1, 2, 3, 4, 5, represents the concentration (in mmolL −1 ) of biomass, extracellular glycerol, extracellular 1,3-PD, acetate, and ethanol at time t, respectively; x 0i , i = 1, 2, 3, 4, 5, are the corresponding initial concentrations; t ∈ [0, t f ], where t f is the terminal time; the constant D denotes the dilution rate, and C s0 is the control variable denoting the concentration of glycerol in the input. Also, µ is the specific growth rate of cells (in h −1 ); q 2 is the specific consumption rate of substrate (in h −1 ); q i , i = 3, 4, 5, are the specific formation rate of 1,3-PD, acetate and ethanol (in h −1 ) respectively; and their formulations are listed below: Here, µ m = 0.67(h −1 ) is the maximum specific growth rate and k s = 0.28 (mmolL −1 ) is the Monod saturation constant for substrate. Under anaerobic conditions at 37 • C, pH=7.0, other parameters used in Eqs. (1) -(4) are available in literature [28] and are listed in Table (1) below: Table 1. The values of some parameters used in Eqs. (1) -(4).
The nonlinear control system can be formulated in the following way: For the actual bioprocess, it should be noted that there exist critical concentrations for the state vector x. Therefore, it is natural to restrict the concentrations of biomass, glycerol and products in a given set W defined as where x * i and x i * , respectively, denote the upper and lower bound of the corresponding state variables.
Equation (6) can be equivalently transformed as a continuous state inequality constraints by introducing the functions as follows: where In this paper, the concentration of glycerol in the input feed is chosen as the control variable. It is obvious that the control variable is also constrained: where u * and u * are the lower and upper bound of u(t).
Let x(· | u(t)) be the solution of (5) under the control of u(t), then x 3 (· | u(t)) is the concentration of 1,3-PD. We can describe the optimal problem as follows: Problem P 0 . Find u(t) to minimize the cost function with constraints.
4. Feedback control. The linear state feedback control law is one of the most common feedback control structures [1]. In microbial fermentation, the most important factors to influence the final concentration of 1,3-PD are the concentrations of biomass and glycerol. Thus, the feedback controller is set to be of a linear form with respect to the two concentrations: where ξ(t) = (ξ 1 (t), ξ 2 (t)) T is a vector function of the feedback control parameters.
As showed in (8), ϕ is a given continuously differentiable functional, and the control parameter function, ξ(t), is a decision function to be chosen optimally.
The following constraints are imposed on the feedback control parameters: Substituting (8) into (5) giveṡ Let x(· | ξ(t)) denote the solution of system (9) withf defined in (10). Then, the constraint conditions (7) become Our goal is to present a state feedback control strategy to maximize the final concentration of 1,3-PD. We now consider the problem of choosing the feedback control parameters ξ 1 (t) and ξ 2 (t) to minimize the total system cost subject to constraints (11). Problem P. Choose ξ(t) ∈ U ad to minimize the cost function J 1 (ξ).

5.
A penalty approach and approximate problem. The Problem P is a nonlinear optimization problem in which a finite number of decision variables (the feedback control parameters) need to be optimized subject to a set of constraints. This is a challenging optimization problem to solve because each continuous inequality constraint in (11) constitutes of infinite constraints-one for each point in [0, t f ]. Hence, the Problem P can be viewed as a semi-infinite optimization problem. Then, we will use a penalty approach to approximate the Problem P. [23,10] The condition x(t) ∈ W is equivalently transcribed into Clearly, G(ξ(t)) = 0 if and only if x(t) ∈ W . However, the equality constraint (12) is not smooth at the points when h i = 0. Consequently, standard optimization routines would have difficulties in dealing with this type of equality constraints. Let be a smooth function in ξ, where the smoothing parameter is a tiny positive number, and ϕ : R → R is defined by Then, the objective function of the Problem P can be reformulated as where ρ > 0 is the given penalty parameter. Hence, P can be approximated by the following problem: Problem Q. Choose ξ ∈ U ad to minimize the penalty function J(ξ).
Theorem 5.1. Let ξ * be the optimal solution of the approximate the Problem Q. Suppose that there exists an optimal solution ξ * of the original Problem P. Then Theorem 5.1 guarantees that any local solution of the approximate problem can be used for generating a corresponding local solution of the original problem when the smoothing penalty parameter is sufficiently small. 6. Control vector parametrization technique. To solve the Problem Q numerically, the control vector parametrization approach is applied [24,11], in which the feedback control variable ξ(t) = [ξ 1 (t), ξ 2 (t)] are discretized. Partition the time hori- Using the piecewise-constant policy, the feedback control variable ξ i (t) is approximated by and χ k is defined as With the ξ ∈ U ad , the differential equation (9) is of form: wheref The initial condition remains the same as x(0) = x 0 . Let x(·|σ) be the solution of the system (13) corresponding to the control parameter vector σ. Then, the Problem Q can be reformulated as a NLP problem, in which σ is the decision vector. We may now specify the approximating Problem Q(p) in the following way: Problem Q(p). Find a control parameter vector σ ∈ U ad to minimize the cost function J(σ).
Theorem 6.1. Letξ * be the optimal control of the approximating Problem Q(p). Suppose that the original Problem Q has an optimal control ξ * . Then, To solve the Problem P as a mathematical programming problem, we need to know the gradient for the function J(σ). We derive the formulae as follows [23]: Let the corresponding Hamiltonian function for the cost function be defined by and is the solution of the costate systeṁ with the boundary continuous λ(t f ) = (0, 0, 0, 0, 0) T . And the gradient of J is computed from Very often the control parametrization is carried out on an partition of the interval [0, t f ] in simulations. Each component of (14) can be written in a more specific form as shown below: 7. Gradient-based adaptive algorithm. Based on the control vector parametrization approach above, we adopt a gradient-based adaptive refinement method [27] to solve the Problem P. The algorithm is adaptive to obtain time-efficient and cost-effective discretization grids. In this way, a high-quality solution can be obtained with low computational cost. Define J * l as the optimal objective function value; defineξ * l i = [σ * l i,1 , ..., σ * l i,p ], (i = 1, 2) as the optimal solution, and define ∆ l = [t l 0 , ..., t l p ] T as the corresponding discretization time grid found in iteration l. ∆ l := [t l 0 , ..., t l 2p ] T is obtained by bisecting each subinterval in ∆ l with two initial control variables, ξ l 1 = [σ * l 1,1 , σ * l 1,1 , .., σ * l 1,p , σ * l 1,p ] T and ξ l 2 = [σ * l 2,1 σ * l 2,1 , .., σ * l 2,p , σ * l 2,p ] T . Suppose J * l , ξ * l 1 and ξ * l 2 are the optimal objective function value and the optimal solution in iteration l , respectively. We hope to find a new discretization grid to make it better adapted to the solution.
The main steps of this algorithm are as follows:
Step 2. Useξ l as the starting point and ∆ l as the starting time grids. By using a nonlinear programming algorithm to solve the NLP to obtain the optimal objective function value J * l and the optimal solutionû * l .
With those parameters, the algorithm meets the stopping criteria and terminates in six iterations. The evolution of time grids is illustrated in Fig 1. The concentration of 1,3-PD at the terminal time is 686.5683 mmol/L, which agrees with experimental data [5]. Hence, the control found at this time grid point is optimal. The feedback control parameters are shown in Fig 2. The changes of concentrations of biomass, glycerol, 1,3-PD, acetate and ethanol under the optimal feedback control over 100 hours are shown in Fig 3. The computational results verify the effectiveness of our proposed method. 9. Conclusions. To the best of our knowledge, a robust multistage feedback control strategy with a closed-loop is proposed, for the first time, for the production of 1,3-propanediol in microbial fermentation. In this paper, we also develop an adaptive gradient-based optimization algorithm to obtain the global solution. In Sec. 8, the numerical results show that the method is successful at producing high-quality control strategies. In the future, we plan to set D as another controller to maximize the productivity of the process.