1 Introduction

Many institutes evaluate proposals from collaborating professionals. Such proposals may be research proposals by collaborating researchers, tenders by consortia of firms, proposals for new films by collaborating producers and filmmakers, and so on. These professionals collaborate on a project basis. The evaluations of the proposals are often used to allocate funding to the best proposals before these proposals start. Therefore, it is of huge importance to recognise the proposals that are likely to be successful in a very early stage. We refer to this as the potential of the proposal.

The evaluation process usually judges the contents of the proposal only: whether or not the proposal is new, exciting, promising, successful, and so on. Such a judgement is subjective; the outcome depends on the interests of the reviewers. Also, the earlier results and experiences of the professionals are neglected. These earlier results are an expression of the talents and capacities of the professionals that contribute to the success of the proposal. What is needed is a good and more objective estimate of the expected potential of a proposal at a very early stage.

In this paper, we introduce a new method to estimate the potential of the proposal based on numerical data instead of on reviewers expertise. Our method first estimates the expected potential of each of the collaborating professionals. These are combined to estimate the potential of the proposal by the collaboration. We apply our method to proposals for new films in the Netherlands.

Several research fields study the performance of collaborations. The field of citation analysis studies the scientific performance of groups of scholars using citation counts (Garfield 1979). Performance measurement evaluates the efficiency of individual and organisational performance (Lampe and Hilgers 2015). Public procurement evaluates tenders using a scoring rule, with the goal of achieving high-quality goods at a low price (Bergman and Lundberg 2013). Sports science investigates team sports efficiency, usually with econometric methods (Fizel 2006). Social network analysis studies team performance using the relations among team members, and of the team with other people (Guimerà et al. 2005).

From an economic point of view, collaboration among professionals is considered as team work. During the collaboration, problems like free riding and moral hazard could arise. Relative performance evaluation can be helpful in reducing moral hazard costs (Holmstrom 1982). In this paper, we focus on absolute performance evaluation, using the past performances of the professional.

Our work is related to Shugan (1999). The author uses a team-member evaluation approach for very early predictions for new products, or projects. The expected contribution of team members is used to predict the outcome. First, an individual team members potential is estimated by the individual’s best past outcome. Then, the expected team outcome is a weighted sum of the team members potentials. Applying these results to the US motion picture industry, this approach explains 27.8% of the outcome variance of the box-office outcomes of films in an empirical study. Hence, the people that make the film, screenwriter, cast, director and producer, are important for its success. Our study differs since we consider the screenwriter, director, and producer, and not the cast, which is not yet known, and we use all past outcomes instead of the single best past outcome to estimate the potential of an individual filmmaker.

A movie is the result of collaboration among filmmakers. A large part of the literature on movies is devoted to the US motion picture industry (Eliashberg et al. 2006). In that industry, the form of organisation of movie production has changed during the years from an environment with a large number of studios, to a more flexible collaboration format (Lampel and Shamsie 2003). Movies with higher risks are more likely to be created by an alliance (Palia et al. 2008). Few papers consider applications of Operations Research. For example, Bomsdorf and Derigs (2008) investigates the creation of movie shoot schedules as resource constrained project schedules.

Two main performance measures for movie success are cumulative box-office performance and artistic performance. Box-office revenues are widely studied (Basuroy et al. 2003; Hadida 2009). These indicate the financial success of the movie after release in the theatres. Most papers consider forecasting the box-office revenue after the release of the movie (e.g. Vany and Walls 2002; Ravid 1999; Walls 2005), or before the release of the movie but after production (e.g. Eliashberg et al. 2000; Eliashberg and Shugan 1997; Foutz and Jank 2010; Mestyán et al. 2013; Shugan and Swait 2000).

Before production takes place, the movie should be financed. The main sources of the financing of movies are industry sources, lenders and investors (Vogel 2004). These sources need tools to decide on the best movies to invest in. Since large amounts of money are involved, there is a need for very early and good forecasts of revenues. Forecasting may be done by using artificial neural networks (Ghiassi et al. 2015; Sharda and Delen 2006). These models use input variables like MPAA rating, competition, star value, and genre. The goal is to correctly classify the success of a movie in one of several categories. As mentioned before, Shugan (1999) uses a team-member evaluation approach to predict the box-office results. More recently, Eliashberg et al. (2007) evaluates movie scripts using a forecast on a movie’s return on investment.

The commercial track record of a director is shown to have a positive impact on the commercial success of a movie (Hadida 2010). Further, past artistic success turns out to be a good predictor of artistic performance. Also for Dutch films, track records, or reputations, are important in the search for investment capital (Ebbers and Wijnberg 2012b). In that paper, the authors study the impact of different types of reputations of producers and directors on the investment decisions of distributors, television broadcasters and the Netherlands Film Fund. Although past commercial successes of directors and producers are evaluated differently by distributors and television broadcasters, no support was found for differences in evaluations by the Netherlands Film Fund. The commercial and the artistic reputation of producers and directors are investigated in Ebbers and Wijnberg (2012a).

The contribution of our work is as follows. We introduce a new numerical method to evaluate proposals from collaborating professionals at a very early stage. Our approach is new and contributes to the line of objective evaluation tools. Besides, our method also evaluates the team members using their track records. This differs from Shugan (1999), who only takes the best past result for each individual into account.

We apply our method to the Dutch film industry and estimate the potential of (proposals for) new movies before production is started. The potential represents commercial and artistic success of the film, as measured by the number of visitors and the awards won, respectively. It is represented by a (numerical) value, instead of a category. The Theil U statistic indicates that our estimates are good.

The outline of this paper is as follows. Section 2 introduces our model of cooperating professionals that are involved in projects. It includes the evaluation of proposals given the potentials of the professionals. We estimate the expected potential of a professional in Sect. 3. Thereafter, we apply our model to the evaluation of film proposals by the Netherlands Film Fund in Sect. 4. Section 5 concludes and provides managerial insights.

2 Model

To be able to evaluate proposals from collaborating professionals, we first model how professionals contribute to projects, and how these contributions determine the value or potential of the project.

Let \(\mathcal{P}=\{p_1,\ldots ,p_N\}\) be a set of N professionals, or players, and \(C\subset \mathcal{P}\) a group of collaborating professionals, or a coalition of players. The set \(\mathcal{C}\) of all coalitions is the power set of \(\mathcal{P}\). Let \(\mathcal{F}=\{f_1,\ldots ,f_M\}\) denote a set of projects, and \(C(f)\in \mathcal{C}\) the coalition that carries out project f, \(f \in \mathcal F\). A coalition may carry out multiple projects, and a player may be a member of multiple coalitions simultaneously, but each project is carried out by a unique coalition. The set of projects involving player p is \(\{f:p\in C(f)\}\). We assume projects are completed in periods \(t=-1,-2,\ldots \), that is, 1, 2 or more periods ago, and that each project \(f\in \mathcal{F}\) has a unique period \(t_f\) of completion. A project f carried out by coalition C(f) is influenced by a random environment. We assume that this influence is common for all projects.

Each individual has its talents and capacities, or potential, which determines his contribution to a project. The potential refers to the ability that a person has, which can be developed to make the person better or more successful. Depending on circumstances beyond this person’s control, we assume this contribution to fluctuate around a mean value. Said otherwise, the potential \(x_{p,f}\) of player p in project f is influenced by a random environment. This potential \(x_{p,f}\) is centred around its mean value at time \(t_f\) disturbed by noise,

$$\begin{aligned} x_{p,f}=\mu _p(t_f)+u_{p,f},\quad p \in C(f),\quad f\in \mathcal{F}. \end{aligned}$$
(1)

Here \(\mu _p(t_f)\) is the expected potential of player p in period \(t_f\), which represents the added value (e.g. skills, talents, capacities) that player p contributes to a project completed in period \(t_f\), and \(u_{p,f}\) is the noise, i.e. the influence of the random environment. Both \(x_{p,f}\) and \(u_{p,f}\) are random variables. The assumption that the influence of the random environment is common for all projects implies that the \(u_{p,f}\) are i.i.d. random variables. The noise is assumed to be zero on average,

$$\begin{aligned} {\mathbb E}[u_{p,f}]=0. \end{aligned}$$

Then the expected potential of player p in project f equals the mean value of this player in period \(t_f\),

$$\begin{aligned} {\mathbb E} [ x_{p,f}]=\mu _p(t_f), \end{aligned}$$

and its variance equals the variance of the noise,

$$\begin{aligned} {\mathbb V}\mathrm{ar}(x_{p,f})={\mathbb V}\mathrm{ar}(u_{p,f}), \end{aligned}$$

which we denote by \(\sigma ^2:={\mathbb V}\mathrm{ar}(u_{p,f})\).

We are interested in evaluating proposals of collaborating professionals, that is, estimating the potential of a proposed project f to be completed in period 0, \(t_f=0\). For this, we use the past performances of the players, which are the realizations of previous projects completed in periods \(t=-1,-2,\ldots \). Discounting the potential (1) of a player to time 0 results in \(X_{p,f}\), the potential in period 0:

$$\begin{aligned} X_{p,f} = \mu _p + U_{p,f},\quad p \in C(f),\quad f\in \mathcal{F}. \end{aligned}$$
(2)

Here \(\mu _p:=\mu _p(0)\) is the expected potential of player p at time 0, and \(U_{p,f}\) denotes the randomness discounted from period \(t_f\) to period 0.

The example below illustrates the influence of the random environment.

2.1 Example: a model for the influence of the random environment

To evaluate the potential of a player in a project at time 0, we assume that the randomness, or noise, \(U_{p,f}\) is characterised by the current experience \(w_p\) of player p, and the current influence \(v_{p,f}\) of project f. Experience is gained in the projects in which a player participated. The more experience a player has, the less noise there is; the noise \(U_{p,f}\) decreases in the experience \(w_p\). Further, when more time has elapsed since the project was completed, the value of \(v_{p,f}\) increases. Because the project result was established a longer time ago, the influence of the project decreases, resulting in more noise. Said otherwise, a more recent project has more impact on the current potential of a player, and as such results in less noise.

$$\begin{aligned} U_{p,f} = u_{p,f}v_{p,f} {/} w_p,\quad p \in C(f),\quad f\in \mathcal{F}. \end{aligned}$$
(3)

Because the noise variables \(u_{p,f}\) are i.i.d. with expectation \({\mathbb E}[u_{p,f}]=0\) and variance \({\mathbb V}\mathrm{ar}(u_{p,f})=\sigma ^2\), the noise variables \(U_{p,f}\) at time 0 are also independent, and on average they are zero,

$$\begin{aligned} {\mathbb E}[ U_{p,f}]=0. \end{aligned}$$
(4)

The variance depends on the current influence of noise and the experience and follows from (3), namely

$$\begin{aligned} {\mathbb V}\mathrm{ar}(U_{p,f}) = \sigma ^2 v_{p,f}^2 / w^2_p. \end{aligned}$$
(5)

Further, by (4) the expectation of the potential of player p in project f at time 0 equals \({\mathbb E}[X_{p,f}]=\mu _p\), and the corresponding variance is \({\mathbb V}\mathrm{ar}(X_{p,f})={\mathbb V}\mathrm{ar}(U_{p,f})\). \(\Box \)

In a project, we assume that the contributions of the players are independent; interactions between the contributions of different players are not taken into account. This assumption suits our application. Then the potential \(V_f\) of a project f equals the sum of the potentials of the players involved in the project,

$$\begin{aligned} V_f= \sum _{p\in C(f)} X_{p,f}. \end{aligned}$$

Consequently, this potential is a random variable. Thus, the contributions of the players add to the potential of the project, and may strengthen each other. In Sect. 3 we describe how to estimate the expected potentials of the players. We use these to estimate the potential of a proposal or project.

Let \(\mathcal{F}_0 \subset \mathcal{F}\) be the set of proposed projects to be completed in period 0. We could estimate the potential of project \(f\in \mathcal{F}_0\) by its expected potential, which is the sum of the potentials at time 0 of the involved players, \( {\mathbb E}[ V_f]= {\mathbb E}[\sum _{p\in C(f)} X_{p,f}]= \sum _{p\in C(f)} \mu _{p}. \) However, such an estimate does not take into account the uncertainty and variance in the potentials of the players. Therefore, we estimate the potential of a project with its probability of success

$$\begin{aligned} {\mathbb P}(V_f>c), \end{aligned}$$

which is the probability that the potential of the project exceeds a certain threshold c.

Besides, in certain projects, players in a coalition may have different weights. This may happen, for example, if one player has a smaller contribution than another player. To this end, let \(\delta _{p,f}\) denote the weight of player p in project f. The potential of project f is then a weighted sum of the potentials of the players:

$$\begin{aligned} V^{\delta }_f = \sum _{p\in C(f)} \delta _{p,f}X_{p,f}. \end{aligned}$$

Again, we estimate the potential of the project with its probability of success,

$$\begin{aligned} {\mathbb P}\left( V^{\delta }_f>c\right) . \end{aligned}$$

The example below illustrates this.

2.2 Example continued: ranking under normal randomness

We assume that the noise \(u_{p,f}\) has a normal distribution,

$$\begin{aligned} u_{p,f}\sim \mathcal{N}(0,\sigma ^2), \end{aligned}$$

with expectation 0 and variance \(\sigma ^2\). This implies that the noise at time 0 also has a normal distribution with an expectation of zero (3), and variance as described in (5),

$$\begin{aligned} U_{p,f}\sim \mathcal{N}\left( 0,\sigma ^2 v^2_{p,f} / w^2_p\right) . \end{aligned}$$

Using (2), the potential of player p in project f at time 0 follows a normal distribution with expectation \(\mu _p\),

$$\begin{aligned} X_{p,f}\sim \mathcal{N}\left( \mu _p,\sigma ^2 v^2_{p,f} / w^2_p\right) , \end{aligned}$$

and the potentials of proposed projects \(f\in \mathcal{F}_0\) are also normally distributed

$$\begin{aligned} V^{\delta }_f \sim \mathcal{N} \left( {\textstyle \sum \limits _{p\in C(f)}\delta _{p,f} \mu _p, \sum \limits _{p\in C(f)} \sigma ^2\delta ^2_{p,f}v^2_{p,f}/w^2_p}\right) . \end{aligned}$$

The potential of proposal \(f\in \mathcal{F}_0\) is estimated according to the success probability

$$\begin{aligned} {\mathbb P}(V^{\delta }_f>c)=1-\Phi \left( \left( c-\sum _{p\in C(f)} \delta _{p,f} \mu _p\right) \left( \sqrt{\sum \nolimits _{p\in C(f)} \sigma ^2\delta ^2_{p,f} v^2_{p,f}/w^2_p}\right) ^{-1}\right) , \end{aligned}$$

that follows immediately from the normal distribution, where \(\Phi (x)\) denotes the standard normal distribution function. \(\Box \)

3 Estimation of the expected potential of a player

The potential of a project depends on the expected potentials of the players involved in the project. Since these are unknown, we use estimations instead. In particular, we use the best linear unbiased estimator (BLUE) for the potential of a player p.

The BLUE is chosen as an estimator because it has three interesting properties. First, it has a simple form. Namely, it is linear in the potentials \(X_{p,f}\) of the projects f involving player p. Second, it is unbiased, meaning that the expected value of the estimator equals the mean value \(\mu _p\). Third, it has the smallest spread; that is, it has minimal variance among all linear and unbiased estimators of the potential. The BLUE may be written as \(\widehat{\mu }_p=\sum _{f: p\in C(f)} d^*_{p,f} X_{p,f}\). In the application, we use the realisations of the values \(X_{p,f}\) to obtain the BLUE. (In Appendix A, we show how to derive the coefficients \(d^*_{p,f}\). Further, we discuss how to estimate the variance \(\sigma ^2\)).

3.1 Example continued: BLUE of the expected potential of a player

In our example, the BLUE of the potential \(\mu _p\) of player p is

$$\begin{aligned} \widehat{\mu }_p =\sum _{ \{f:p\in C(f)\}} \frac{z}{v_{p,f}^2} X_{p,f}, \end{aligned}$$
(6)

with normalising constant \(z=1/\left( \sum _{\{f:p\in C(f)\}} 1/v_{p,f}^2\right) \). The derivation of this BLUE is shown in Appendix A. The BLUE is a weighted average of the potentials \(X_{p,f}\). The weights depend on the current influence of noise \(v_{p,f}\). Projects that finished some time ago have large values of \(v_{p,f}\), and thus small coefficients \(z/v^2_{p,f}\). These projects have less influence on the estimated potential than more recent projects, which have smaller values of \(v_{p,f}\), and larger coefficients \(z/v^2_{p,f}\). \(\Box \)

4 A tool for evaluating proposals of films by the Netherlands Film Fund

In this section, we apply our method to the Dutch film industry. We evaluate proposals of new films by collaborating filmmakers. As a by-product, rankings of Dutch filmmakers by type are obtained.

The Netherlands Film Fund is responsible for distribution of funds to support the production of Dutch films.Footnote 1 To this end, a large share of the proposals by consortia of filmmakers are judged via peer review by consultants of the Netherlands Film Fund. Films are classified in various categories. For feature films, the Netherlands Film Fund distinguishes films targeted towards film festivals, and commercial films targeted towards a broad audience.

To avoid subjective judgement of the proposals of new films, we apply our method to estimate the potentials of proposals for new films. This estimation is based on the past performance of the film team (a producer, a director and a screenwriter) that submits a proposal for funding of the production of a film. Our method takes into account and balances the artistic and box-office achievements of the members of the production team. We tested it with data of Dutch films, and parameters according to the policy of the Netherlands Film Fund. The results show that our method provides good estimations of the success of proposals for new films.

4.1 Value of a film

The method is based on the value of a film, which is the historical realisation of the potential of that film. This value represents box-office revenues and awards won at film festivals. To this end, in cooperation with the Netherlands Film Fund we developed a value function for Dutch films. This value function takes into account the total number of visitors to the film in the cinemas and the artistic value via awards won at film festivals, where more points are obtained for an award at a more prestigious film festival. Table 1 gives an overview of film festivals, their awards and corresponding points.

Table 1 An overview of film festivals and the points for their awards

The value function has several properties. First, the larger the number of visitors, the larger the value of the film. Also, the larger the number of visitors, the smaller the added value of a visitor to the film value. Thus, the value function increases with the number of visitors, and it shows a decreasing marginal value. Second, the value of the film increases with the prestige of the awards won. Also, the more awards won, the smaller the marginal value of an award to the value of the film. This means that the value function must be concave in both number of visitors and number of points for awards. Finally, we modified the value function to avoid disproportional effects of a film that receives a very low number of visitors or a very low number of award points (this would have a disproportionally large effect on the expected potential of a filmmaker). For this, the minimal value of a film is set to 2. Fitting to target values indicated by the Netherlands Film Fund, we arrived at the following formula for the value \(y_f\) of film f:

$$\begin{aligned} y_f = 10\left( 1-\frac{2}{10}^{({c_{1f}}/500{,}000 + {c_{2f}}/4 + 0.231)} \right) , \end{aligned}$$
(7)

with \(c_{1f}\) the number of visitors/viewers, and \(c_{2f}\) the artistic score from awards won by film f.

Notice that 500,000 visitors or an artistic score of 4 points yield the same value: 8.6. For 1,000,000 visitors, this value increases to 9.7, which is also obtained for 500,000 visitors and 4 artistic points. Further, the policy of the Netherlands Film Fund determined three parameter values: (1) the rate of increase of the value \(y_f\), determined by the factor 2/10, (2) the weight of the number of visitors compared to the artistic points, determined by the numbers 500,000 and 4, and (3) the minimal grade, determined by the start value 0.231. The multiplicative factor 10 is included to allow the value to be interpreted as a grade as used in the Dutch educational system.

Table 2 An overview of the values of some Dutch films in 2010

As an illustration, Table 2 gives the values of a number of films completed in 2010. We are not able to provide the most recent values (because of inavailability of information, and sensitivity of information with regard to subsidies). Our results in Sect. 4.5 clearly show that these films are most successful artistically (w.r.t. awards won) or commercially (w.r.t. numbers of visitors) in 2010, as in agreement with the expert judgement of the Netherlands Film Fund. Hence, the formula for \(y_f\) in Eq. (7) adequately captures the value of a film.

Together with the Netherlands Film Fund, we chose to combine the number of visitors and the artistic score of a film into the value function (7). Depending on the interests, other choices are also possible like considering only the visitors, or the artistic score, or some other measure(s). Table 2 seems to suggest that films either have many visitors or a large artistic score. However, that does not hold in general. Therefore, our value function combines the number of visitors and the artistic score into a single numerical value.

4.2 Potential of filmmakers

We model the potential of filmmakers as in the model of the Example with normal noise. In Appendix B we motivate this. The player set \(\mathcal P\) is the set of filmmakers (including producers). The value \(x_{p,f}\) of player p in film f is determined by the value of the film, his profit share \(\beta _{p,f}\) in this film, and noise: \(x_{p,f} = \beta _{p,f}y_f+u_{p,f}\).

We assume that the potential \(X_{p,f}\) of filmmaker (player) p in film f discounted to time 0 is subject to less noise when filmmaker p is more experienced. Experience is gained through participation in projects. Experience obtained more periods ago is of less predictive value than recent experience. To represent this, we let the influence of experience on the potential \(X_{p,f}\) decay over time with a decay factor \(\gamma _w\) per period. The decay rates determine, e.g. the half-life time of the influence of experience. If the half-life time is T years, then the corresponding decay rate is \(\gamma _w=\root T \of {1/2}\). The value for T is set by the Netherlands Film Fund.

Further, we assume that a filmmaker gathers more experience when his profit share \(\beta _{p,f}\) in the film is larger. The experience \(w_p\) of player p in period 0 is defined as

$$\begin{aligned} w_p^{2}= \sum _{\{f:p\in C(f),~t_f < 0\}} \beta _{p,f}\gamma _w^{-t_f}. \end{aligned}$$
(8)

Also, values of recent films are subject to less noise. Let noise decay over time with a factor \(\gamma _v\) per period. We define the current influence of the noise \(v_{p,f}\) of film f completed in period \(t_f\) in the variance of \(X_{p,f}\) by

$$\begin{aligned} v_{p,f}^{2} = \gamma _v^{t_f}. \end{aligned}$$
(9)

Using (6), the estimator

$$\begin{aligned} \widehat{\mu }_p=\sum _{ \{f:p\in C(f)\}} \frac{z}{\gamma _v^{t_f}} X_{p,f}, \end{aligned}$$

with normalising constant \(z=1/\left( \sum _{\{f:p\in C(f)\}} 1/\gamma _v^{t_f}\right) \), is the BLUE of the expected potential \(\mu _p\) of filmmaker p. It is a weighted average of the past performance of the filmmaker. We use the same type of evaluation for any type of filmmaker, since the Netherlands Film Fund does not distinguish between types (Ebbers and Wijnberg 2012b).

4.3 Evaluation of film proposals

A film proposal is usually made by a film team (coalition), consisting of three types of filmmakers: a producer, a director and a screenwriter. The contributions of these types to the film are independent. Sometimes, several filmmakers of the same type cooperate. For example, a film team may have two cooperating producers. Let the potential \(X_P\) resemble the joint potential of the cooperating producers, and let \(C_P(f)\) denote the set of producers in the film team of film f. Since production is a team effort, we consider the production team to be a (fictive) producer. We consider all films made by all producers in the production team, and let \(X_P\) be the potential as if all those films were made by the fictive producer.

Further, we may define the sets \(C_D(f)\), and \(C_S(f)\) of directors and screenwriters of film f, respectively. Since directors and screenwriters perform a large part of their task independently, their joint potentials \(X_D\) and \(X_S\) are determined as follows. Let the fraction \(\delta _{p,f}\) denote the weight of director \(p\in C_D(f)\), \(\sum _{p\in C_D(f)}\delta _{p,f}=1\). For example, if two directors cooperate, and one has no experience, we may set the weight of the unexperienced director to 0. Thus, the joint potentials \(X_D\) and \(X_S\) of directors and screenwriters are

$$\begin{aligned} X_D = \sum _{p\in C_D(f)} \delta _{p,f} X_{p,f}, \qquad X_S=\sum _{p\in C_S(f)} \delta _{p,f} X_{p,f}. \end{aligned}$$

Given the potentials of producers, directors and screenwriters, the potential of the film \(V_f\) is the weighted average of the potentials of the film team,

$$\begin{aligned} V_f = \frac{\alpha _PX_P + \alpha _DX_D + \alpha _SX_S}{\alpha _P+\alpha _D+\alpha _S} \end{aligned}$$
(10)

The weights \(\alpha _P\), \(\alpha _D\), and \(\alpha _S\) are determined by the Netherlands Film Fund.

The potential of a movie is a stochastic variable. As before, we estimate this potential using the success probability \({\mathbb P}(V_f > c)\) where the constant c is determined by the Netherlands Film Fund. Since the value of a film ranges from 1 to 10, we use

$$\begin{aligned} \hat{y}_f = 10{\mathbb P}(V_f > c) \end{aligned}$$
(11)

as the estimate of the potential of a proposal for film f. The expression on the right-hand side combines both the expected potential of the film \({\mathbb E}[V_f]\) and its variance \({\mathbb V}\mathrm{ar}(V_f)\) to measure the success of the film team. As such, it mimics the value function that also measures the success of a film.

4.4 Evaluation of individual filmmakers

Besides evaluating film proposals, we can also evaluate individual filmmakers. In Sect. 2, we have seen that the potential of a filmmaker p in film f at time 0 follows the normal distribution \(X_{p,f}\sim \mathcal{N}(\mu _p,\sigma ^2 v^2_{p,f} / w^2_p)\). Now we consider a fictitious film f solely made by this filmmaker at the current time period \(t=0\). The current influence of this fictitious film equals \(v_{p,f}^2=1\). This implies that the current potential of filmmaker p has mean \(\mu _p\) and variance \(\sigma ^2/w_p^2\). The experience \(w_p^2\) is obtained using time discounting (8). The filmmaker is now evaluated according to the success probability

$$\begin{aligned} {\mathbb P}(X_p > c), \end{aligned}$$

where the constant c is determined by the Netherlands Film Fund.

4.5 Implementation and results

In this section, we use our method to evaluate proposals of Dutch films released in 2010. Besides, we rank the individual filmmakers by type.

The data of Dutch films till 2011 were gathered from publicly available sources.Footnote 2 \(^{,}\) Footnote 3 \({^{,}}\) Footnote 4 For each filmmaker, we registered the films made by him or her. For each of these films we collected the year of release of the film, the number of visitors, the awards won and the corresponding artistic score, and the profit share. These shares are determined per type of filmmaker, and all filmmakers of the same type are assumed to have an equal share. For example, a single producer has a share of 100%, and in case of two producers each has a share of 50%. The parameter values used are according to the policy of the Netherlands Film Fund: \(c= 5\), \(T=20\), \(\gamma _v=\root T \of {1/2}\), \(\alpha _P=3\), \(\alpha _D=2\), and \(\alpha _S=1\).

Following the procedure in Sect. 4.3, for each film we first derive the expected potential for each type of filmmaker. Thereafter, these are combined to obtain the expected potential of the film team for film f, \({\mathbb E}[V_f]\), and its variance, \({\mathbb V}\mathrm{ar}(V_f)\), using (10) and the independence of the types of filmmakers. Finally, by (11), this results in the estimated film value \(\hat{y}_f\). The estimated potentials of Dutch films released in 2010, and the characteristics of the corresponding film teams are shown in Table 8 in Appendix B. The realised film values are shown in Table 9 in Appendix B.

Table 3 Estimated and realised film values for Dutch films in 2010. The Theil U statistic is 0.40, indicating that our estimates are very good
Table 4 Description of results of individual filmmakers, as used in Tables 5, 6 and 7

Table 3 lists the estimated potentials (film values) \(\hat{y}_f=10{\mathbb P}(V_f>c)\) and the realised film values \(y_f\) of Dutch films released in 2010. Overall, the estimated values are rather close to the realised values. Some films have an estimated film value more than two points below the realised film value; their performances are better than estimated. These differences are caused by debuting filmmakers in the film team. This is the case for the films New Kids Turbo (debuting director and screenwriter), Gelukkige Huisvrouw (debuting director and screenwriter), Dik Trom (debuting director), Loft (debuting director and screenwriter), and Ernst, Bobbie en het geheim van de Monta Rossa (debuting director and screenwriter). Our method estimates the film values based on past realisations of the filmmakers. Debuting filmmakers have no results yet, making it hard to estimate their results.

Further, two films have an estimated film value more than two points above the realised film value; their performances are worse than estimated. First, the film Het Geheim is a movie for children that did not attract as many visitors as expected. The film is based on an original story and was not based on a bestselling book. Hence, a good promotion was needed. Further, this is the third film of the scenarist, making him a beginning scenarist with limited experience. His potential is not easy to estimate with our method. Second, the film Vliegenierster van Kazbeck is a movie that was expected to win awards. Unfortunately that did not happen. Furthermore, this is the second movie of the director, making her a starting filmmaker. Therefore, it is not easy to estimate her potential.

To evaluate our estimations, we use the Theil U statistic (Theil 1961). This statistic is widely used to measure the accuracy of estimates. Since \(\hat{y}_f\) denotes the estimated value of film f, and \(y_f\) the realised value, the Theil U statistic equals \(U=\sqrt{\sum _f (y_f-\hat{y}_f)^2/ \sum _f y_f^2}\). The value U has the following meaning. If \(U>1\), then the estimate is not good. If \(U<1\), then the estimate is good, and the closer it is to 0, the better the estimate. In general, values of 0.55 or less are considered very good. For our estimated and realised film values, the Theil U statistic has the value \(U=0.40\), indicating that our estimates are very good. Hence, our method is a useful tool for more objective judgement of proposals for new films.

Table 5 Ranking of top 10 producers of Dutch films per 1/1/2011
Table 6 Ranking of top 10 directors of Dutch films per 1/1/2011
Table 7 Ranking of top 10 screenwriters of Dutch films per 1/1/2011

Besides, we use our model to evaluate the individual filmmakers. If, e.g. a producer was a director in the past, then this directing experience is not taken into account in the evaluation of the producer. We only consider the experience of a filmmaker per role since the experience gained in a film depends on the specific tasks and responsibilities related to that role. The resulting evaluations of the filmmakers are confidential. Therefore, we do not mention the names of the filmmakers, but we describe their results based on their past performances as indicated in Table 4. The ranked list of top 10 producers with largest evaluations is shown in Table 5. In the table, we list the evaluation values \(10{\mathbb P}(X_p > c)\) for each filmmaker p; these values may be interpreted as grades. The rankings of top 10 directors and screenwriters follow in Tables 6 and 7. These tables show that experienced filmmakers have large values. These rankings are concluded to be representative.

5 Conclusions

In this paper, we introduced a new numerical method to estimate the potential of proposals from collaborating professionals. Our method uses the past performances of the professionals to indicate their current potentials. These are combined to obtain an estimate of the potential of the proposed project by the collaboration.

We applied our method to estimate the potentials of proposals for Dutch films released in 2010. Our method is shown to obtain good results. Therefore, it is a useful tool for more objective judgement of proposals for new films. Besides, we rank producers, directors and screenwriters of Dutch films. These rankings are concluded to be representative. This application also shows that experienced filmmakers are highly valued, and that cooperation with new talented filmmakers is encouraged.

In general, our method may be used as a selection method for proposals that is more objective than reviewers expertise. It provides a tool for managers to estimate the potential of a proposal from collaborating professionals based on numerical data. Our model provides clear directives on which the estimate, and consequently the selection, is based.