Assessment of learning curves in complex surgical interventions: a consecutive case-series study

Background Surgical interventions are complex, which complicates their rigorous assessment through randomised clinical trials. An important component of complexity relates to surgeon experience and the rate at which the required level of skill is achieved, known as the learning curve. There is considerable evidence that operator performance for surgical innovations will change with increasing experience. Such learning effects complicate evaluations; the start of the trial might be delayed, resulting in loss of surgeon equipoise or, if an assessment is undertaken before performance has stabilised, the true impact of the intervention may be distorted. Methods Formal estimation of learning parameters is necessary to characterise the learning curve, model its evolution and adjust for its presence during assessment. Current methods are either descriptive or model the learning curve through three main features: the initial skill level, the learning rate and the final skill level achieved. We introduce a fourth characterising feature, the duration of the learning period, which provides an estimate of the point at which learning has stabilised. We propose a two-phase model to estimate formally all four learning curve features. Results We demonstrate that the two-phase model can be used to estimate the end of the learning period by incorporating a parameter for estimating the duration of learning. This is achieved by breaking down the model into a phase describing the learning period and one describing cases after the final skill level is reached, with the break point representing the length of learning. We illustrate the method using cardiac surgery data. Conclusions This modelling extension is useful as it provides a measure of the potential cost of learning an intervention and enables statisticians to accommodate cases undertaken during the learning phase and assess the intervention after the optimal skill level is reached. The limitations of the method and implications for the optimal timing of a definitive randomised controlled trial are also discussed. Electronic supplementary material The online version of this article (doi:10.1186/s13063-016-1383-4) contains supplementary material, which is available to authorized users.


Two-phase model formulation
In what follows, we give the details of the model formulation for a binary and a continuous outcome respectively.
Generalised non-linear model for a binary outcome The binary outcome is denoted by y ij , with indices i and j corresponding to surgeon number and operation order respectively. Operation order is determined by ordering each procedure within a surgeon's series by the date conducted. Since our outcome is dichotomous, y ij takes the values 0 for no event and 1 for an event/failure. The generalised and two-phase models are specified as follows: Generalised non-linear model : where the event is assumed to follow a Bernoulli distribution such that y ij ∼ Bernoulli(p ij ), i = 1, 2, 3, j = 1, ..., n i , p ij is the probability of event for the j th operation performed by the i th surgeon and n i the total number of operations performed by the i th surgeon. Experience, denoted by x ij , is the order of the j th operation performed by the i th surgeon i.e. x ij = j from the i th surgeon's series; z kij represents the k th covariate for the j th operation performed by the i th surgeon and w is the total number of covariates. φ(.) is the link function and since the response variable is binary, the respective canonical link function i.e. the logistic logit(p ij ) = log( pij 1−pij ) is preferred. g(.) is a candidate (linear or non-linear) function considered best for describing the learning data. Finally, θ is a scalar or vector of parameters related to the experience covariate and τ is the duration of learning parameter. Models 1 and 2 are fitted via ML estimation. The detailed procedure is as follows.
Recall a Bernoulli likelihood for the j th operation of the i th surgeon is: L(y ij ) = p yij ij (1 − p ij ) 1−yij and, from model 1 using the logit link function we get We denote the full vector of parameters as θ = (θ, δ 1 , ..., δ w ); the respective log- By independence, the log-likelihood for the full series of the i th surgeon is: Thus, the log-likelihood to be maximised is: For the two-phase model this becomes: The score vector for the Generalised Non-Linear Model is equivalent to the gra- for all k and all elements in θ.
Setting the above partial derivatives to zero and solving them simultaneously gives the ML estimators of the parameters.
This estimation process was implemented in the statistical programming language R 3.0.1 using the optimisation command optim() , selecting a quasi-Newton method also known as a variable metric algorithm.

Generalised non-linear model for a continuous outcome
A continuous outcome variable is denoted by y ij , with indices i and j corresponding to surgeon number and operation order respectively. The generalised and two-phase models are specified as follows.
Generalised non-linear model : Two-phase model: with the errors following a Normal distribution e ij ∼ N ormal(0, σ 2 ), i = 1, 2, 3, j = 1, ..., n i . Hence, the continuous variable is assumed to follow a Normal distribution such that y ij ∼ Normal(µ ij , σ 2 ), i = 1, 2, 3, j = 1, ..., n i and µ ij = α + g(x ij ; θ) + w k=1 δ k z kij . In the two-phase model, at the first phase µ ij = g(x ij ; θ) + w k=1 δ k z kij and at the second phase, µ ij = α + w k=1 δ k z kij . Experience is represented by x ij , the order of the j th operation performed by the i th surgeon; z kij represents the k th covariate for the j th operation performed by the i th surgeon and w is the total number of covariates. g(.) is a candidate (linear, or non-linear) function considered for describing the learning data best. Finally, θ is a scalar or vector of parameters related to the experience covariate and τ is the duration of learning parameter. Models 3 and 4 are fitted via ML estimation. The detailed procedure is as follows.
Recall a Normal likelihood for the j th operation in the i th surgeon's series is: We denote the full vector of parameters as θ = (θ, δ 1 , ..., δ w , σ 2 ); the respective log-likelihood is ij (θ ) = yij µij −µ 2 ij /2 σ 2 − 1 2 log 2π − log σ − y 2 ij 2σ 2 . By independence, the joint log-likelihood for the series of the i th surgeon is: For the two-phase model, the likelihood becomes: This can be split in three parts: The score vector for the Generalised Non-Linear Model (GNLM) is equivalent to the gradient vector U = ( ∂σ 2 ) for all k and all elements in θ. δ k z kij