Abstract
Professionals often collaborate in projects. Some of these projects require funding, so before the collaboration can start a proposal for the project is submitted. This proposal will then be evaluated by a committee. The goal of the committee is to recognise proposals that are likely to be very successful. In this paper, we introduce a new numerical method to estimate the expected potential of a proposal. This method helps in identifying proposals that may turn out to be the most successful. The estimation is derived from the past performances of the professionals involved and takes into account the uncertainty of a contribution of a professional to a proposal. We apply our method to the Dutch film industry. We estimate the potential of proposals for new films released in 2010. The value of a film depends on the number of visitors in cinemas and the artistic prizes won. Our estimates are very good, indicating that past performances of filmmakers provide a very good indication of the potential of their new film. As a by-product of our method, rankings of producers, directors, and screenwriters of Dutch films up to 2011 are obtained.
Similar content being viewed by others
1 Introduction
Many institutes evaluate proposals from collaborating professionals. Such proposals may be research proposals by collaborating researchers, tenders by consortia of firms, proposals for new films by collaborating producers and filmmakers, and so on. These professionals collaborate on a project basis. The evaluations of the proposals are often used to allocate funding to the best proposals before these proposals start. Therefore, it is of huge importance to recognise the proposals that are likely to be successful in a very early stage. We refer to this as the potential of the proposal.
The evaluation process usually judges the contents of the proposal only: whether or not the proposal is new, exciting, promising, successful, and so on. Such a judgement is subjective; the outcome depends on the interests of the reviewers. Also, the earlier results and experiences of the professionals are neglected. These earlier results are an expression of the talents and capacities of the professionals that contribute to the success of the proposal. What is needed is a good and more objective estimate of the expected potential of a proposal at a very early stage.
In this paper, we introduce a new method to estimate the potential of the proposal based on numerical data instead of on reviewers expertise. Our method first estimates the expected potential of each of the collaborating professionals. These are combined to estimate the potential of the proposal by the collaboration. We apply our method to proposals for new films in the Netherlands.
Several research fields study the performance of collaborations. The field of citation analysis studies the scientific performance of groups of scholars using citation counts (Garfield 1979). Performance measurement evaluates the efficiency of individual and organisational performance (Lampe and Hilgers 2015). Public procurement evaluates tenders using a scoring rule, with the goal of achieving high-quality goods at a low price (Bergman and Lundberg 2013). Sports science investigates team sports efficiency, usually with econometric methods (Fizel 2006). Social network analysis studies team performance using the relations among team members, and of the team with other people (Guimerà et al. 2005).
From an economic point of view, collaboration among professionals is considered as team work. During the collaboration, problems like free riding and moral hazard could arise. Relative performance evaluation can be helpful in reducing moral hazard costs (Holmstrom 1982). In this paper, we focus on absolute performance evaluation, using the past performances of the professional.
Our work is related to Shugan (1999). The author uses a team-member evaluation approach for very early predictions for new products, or projects. The expected contribution of team members is used to predict the outcome. First, an individual team members potential is estimated by the individual’s best past outcome. Then, the expected team outcome is a weighted sum of the team members potentials. Applying these results to the US motion picture industry, this approach explains 27.8% of the outcome variance of the box-office outcomes of films in an empirical study. Hence, the people that make the film, screenwriter, cast, director and producer, are important for its success. Our study differs since we consider the screenwriter, director, and producer, and not the cast, which is not yet known, and we use all past outcomes instead of the single best past outcome to estimate the potential of an individual filmmaker.
A movie is the result of collaboration among filmmakers. A large part of the literature on movies is devoted to the US motion picture industry (Eliashberg et al. 2006). In that industry, the form of organisation of movie production has changed during the years from an environment with a large number of studios, to a more flexible collaboration format (Lampel and Shamsie 2003). Movies with higher risks are more likely to be created by an alliance (Palia et al. 2008). Few papers consider applications of Operations Research. For example, Bomsdorf and Derigs (2008) investigates the creation of movie shoot schedules as resource constrained project schedules.
Two main performance measures for movie success are cumulative box-office performance and artistic performance. Box-office revenues are widely studied (Basuroy et al. 2003; Hadida 2009). These indicate the financial success of the movie after release in the theatres. Most papers consider forecasting the box-office revenue after the release of the movie (e.g. Vany and Walls 2002; Ravid 1999; Walls 2005), or before the release of the movie but after production (e.g. Eliashberg et al. 2000; Eliashberg and Shugan 1997; Foutz and Jank 2010; Mestyán et al. 2013; Shugan and Swait 2000).
Before production takes place, the movie should be financed. The main sources of the financing of movies are industry sources, lenders and investors (Vogel 2004). These sources need tools to decide on the best movies to invest in. Since large amounts of money are involved, there is a need for very early and good forecasts of revenues. Forecasting may be done by using artificial neural networks (Ghiassi et al. 2015; Sharda and Delen 2006). These models use input variables like MPAA rating, competition, star value, and genre. The goal is to correctly classify the success of a movie in one of several categories. As mentioned before, Shugan (1999) uses a team-member evaluation approach to predict the box-office results. More recently, Eliashberg et al. (2007) evaluates movie scripts using a forecast on a movie’s return on investment.
The commercial track record of a director is shown to have a positive impact on the commercial success of a movie (Hadida 2010). Further, past artistic success turns out to be a good predictor of artistic performance. Also for Dutch films, track records, or reputations, are important in the search for investment capital (Ebbers and Wijnberg 2012b). In that paper, the authors study the impact of different types of reputations of producers and directors on the investment decisions of distributors, television broadcasters and the Netherlands Film Fund. Although past commercial successes of directors and producers are evaluated differently by distributors and television broadcasters, no support was found for differences in evaluations by the Netherlands Film Fund. The commercial and the artistic reputation of producers and directors are investigated in Ebbers and Wijnberg (2012a).
The contribution of our work is as follows. We introduce a new numerical method to evaluate proposals from collaborating professionals at a very early stage. Our approach is new and contributes to the line of objective evaluation tools. Besides, our method also evaluates the team members using their track records. This differs from Shugan (1999), who only takes the best past result for each individual into account.
We apply our method to the Dutch film industry and estimate the potential of (proposals for) new movies before production is started. The potential represents commercial and artistic success of the film, as measured by the number of visitors and the awards won, respectively. It is represented by a (numerical) value, instead of a category. The Theil U statistic indicates that our estimates are good.
The outline of this paper is as follows. Section 2 introduces our model of cooperating professionals that are involved in projects. It includes the evaluation of proposals given the potentials of the professionals. We estimate the expected potential of a professional in Sect. 3. Thereafter, we apply our model to the evaluation of film proposals by the Netherlands Film Fund in Sect. 4. Section 5 concludes and provides managerial insights.
2 Model
To be able to evaluate proposals from collaborating professionals, we first model how professionals contribute to projects, and how these contributions determine the value or potential of the project.
Let \(\mathcal{P}=\{p_1,\ldots ,p_N\}\) be a set of N professionals, or players, and \(C\subset \mathcal{P}\) a group of collaborating professionals, or a coalition of players. The set \(\mathcal{C}\) of all coalitions is the power set of \(\mathcal{P}\). Let \(\mathcal{F}=\{f_1,\ldots ,f_M\}\) denote a set of projects, and \(C(f)\in \mathcal{C}\) the coalition that carries out project f, \(f \in \mathcal F\). A coalition may carry out multiple projects, and a player may be a member of multiple coalitions simultaneously, but each project is carried out by a unique coalition. The set of projects involving player p is \(\{f:p\in C(f)\}\). We assume projects are completed in periods \(t=-1,-2,\ldots \), that is, 1, 2 or more periods ago, and that each project \(f\in \mathcal{F}\) has a unique period \(t_f\) of completion. A project f carried out by coalition C(f) is influenced by a random environment. We assume that this influence is common for all projects.
Each individual has its talents and capacities, or potential, which determines his contribution to a project. The potential refers to the ability that a person has, which can be developed to make the person better or more successful. Depending on circumstances beyond this person’s control, we assume this contribution to fluctuate around a mean value. Said otherwise, the potential \(x_{p,f}\) of player p in project f is influenced by a random environment. This potential \(x_{p,f}\) is centred around its mean value at time \(t_f\) disturbed by noise,
Here \(\mu _p(t_f)\) is the expected potential of player p in period \(t_f\), which represents the added value (e.g. skills, talents, capacities) that player p contributes to a project completed in period \(t_f\), and \(u_{p,f}\) is the noise, i.e. the influence of the random environment. Both \(x_{p,f}\) and \(u_{p,f}\) are random variables. The assumption that the influence of the random environment is common for all projects implies that the \(u_{p,f}\) are i.i.d. random variables. The noise is assumed to be zero on average,
Then the expected potential of player p in project f equals the mean value of this player in period \(t_f\),
and its variance equals the variance of the noise,
which we denote by \(\sigma ^2:={\mathbb V}\mathrm{ar}(u_{p,f})\).
We are interested in evaluating proposals of collaborating professionals, that is, estimating the potential of a proposed project f to be completed in period 0, \(t_f=0\). For this, we use the past performances of the players, which are the realizations of previous projects completed in periods \(t=-1,-2,\ldots \). Discounting the potential (1) of a player to time 0 results in \(X_{p,f}\), the potential in period 0:
Here \(\mu _p:=\mu _p(0)\) is the expected potential of player p at time 0, and \(U_{p,f}\) denotes the randomness discounted from period \(t_f\) to period 0.
The example below illustrates the influence of the random environment.
2.1 Example: a model for the influence of the random environment
To evaluate the potential of a player in a project at time 0, we assume that the randomness, or noise, \(U_{p,f}\) is characterised by the current experience \(w_p\) of player p, and the current influence \(v_{p,f}\) of project f. Experience is gained in the projects in which a player participated. The more experience a player has, the less noise there is; the noise \(U_{p,f}\) decreases in the experience \(w_p\). Further, when more time has elapsed since the project was completed, the value of \(v_{p,f}\) increases. Because the project result was established a longer time ago, the influence of the project decreases, resulting in more noise. Said otherwise, a more recent project has more impact on the current potential of a player, and as such results in less noise.
Because the noise variables \(u_{p,f}\) are i.i.d. with expectation \({\mathbb E}[u_{p,f}]=0\) and variance \({\mathbb V}\mathrm{ar}(u_{p,f})=\sigma ^2\), the noise variables \(U_{p,f}\) at time 0 are also independent, and on average they are zero,
The variance depends on the current influence of noise and the experience and follows from (3), namely
Further, by (4) the expectation of the potential of player p in project f at time 0 equals \({\mathbb E}[X_{p,f}]=\mu _p\), and the corresponding variance is \({\mathbb V}\mathrm{ar}(X_{p,f})={\mathbb V}\mathrm{ar}(U_{p,f})\). \(\Box \)
In a project, we assume that the contributions of the players are independent; interactions between the contributions of different players are not taken into account. This assumption suits our application. Then the potential \(V_f\) of a project f equals the sum of the potentials of the players involved in the project,
Consequently, this potential is a random variable. Thus, the contributions of the players add to the potential of the project, and may strengthen each other. In Sect. 3 we describe how to estimate the expected potentials of the players. We use these to estimate the potential of a proposal or project.
Let \(\mathcal{F}_0 \subset \mathcal{F}\) be the set of proposed projects to be completed in period 0. We could estimate the potential of project \(f\in \mathcal{F}_0\) by its expected potential, which is the sum of the potentials at time 0 of the involved players, \( {\mathbb E}[ V_f]= {\mathbb E}[\sum _{p\in C(f)} X_{p,f}]= \sum _{p\in C(f)} \mu _{p}. \) However, such an estimate does not take into account the uncertainty and variance in the potentials of the players. Therefore, we estimate the potential of a project with its probability of success
which is the probability that the potential of the project exceeds a certain threshold c.
Besides, in certain projects, players in a coalition may have different weights. This may happen, for example, if one player has a smaller contribution than another player. To this end, let \(\delta _{p,f}\) denote the weight of player p in project f. The potential of project f is then a weighted sum of the potentials of the players:
Again, we estimate the potential of the project with its probability of success,
The example below illustrates this.
2.2 Example continued: ranking under normal randomness
We assume that the noise \(u_{p,f}\) has a normal distribution,
with expectation 0 and variance \(\sigma ^2\). This implies that the noise at time 0 also has a normal distribution with an expectation of zero (3), and variance as described in (5),
Using (2), the potential of player p in project f at time 0 follows a normal distribution with expectation \(\mu _p\),
and the potentials of proposed projects \(f\in \mathcal{F}_0\) are also normally distributed
The potential of proposal \(f\in \mathcal{F}_0\) is estimated according to the success probability
that follows immediately from the normal distribution, where \(\Phi (x)\) denotes the standard normal distribution function. \(\Box \)
3 Estimation of the expected potential of a player
The potential of a project depends on the expected potentials of the players involved in the project. Since these are unknown, we use estimations instead. In particular, we use the best linear unbiased estimator (BLUE) for the potential of a player p.
The BLUE is chosen as an estimator because it has three interesting properties. First, it has a simple form. Namely, it is linear in the potentials \(X_{p,f}\) of the projects f involving player p. Second, it is unbiased, meaning that the expected value of the estimator equals the mean value \(\mu _p\). Third, it has the smallest spread; that is, it has minimal variance among all linear and unbiased estimators of the potential. The BLUE may be written as \(\widehat{\mu }_p=\sum _{f: p\in C(f)} d^*_{p,f} X_{p,f}\). In the application, we use the realisations of the values \(X_{p,f}\) to obtain the BLUE. (In Appendix A, we show how to derive the coefficients \(d^*_{p,f}\). Further, we discuss how to estimate the variance \(\sigma ^2\)).
3.1 Example continued: BLUE of the expected potential of a player
In our example, the BLUE of the potential \(\mu _p\) of player p is
with normalising constant \(z=1/\left( \sum _{\{f:p\in C(f)\}} 1/v_{p,f}^2\right) \). The derivation of this BLUE is shown in Appendix A. The BLUE is a weighted average of the potentials \(X_{p,f}\). The weights depend on the current influence of noise \(v_{p,f}\). Projects that finished some time ago have large values of \(v_{p,f}\), and thus small coefficients \(z/v^2_{p,f}\). These projects have less influence on the estimated potential than more recent projects, which have smaller values of \(v_{p,f}\), and larger coefficients \(z/v^2_{p,f}\). \(\Box \)
4 A tool for evaluating proposals of films by the Netherlands Film Fund
In this section, we apply our method to the Dutch film industry. We evaluate proposals of new films by collaborating filmmakers. As a by-product, rankings of Dutch filmmakers by type are obtained.
The Netherlands Film Fund is responsible for distribution of funds to support the production of Dutch films.Footnote 1 To this end, a large share of the proposals by consortia of filmmakers are judged via peer review by consultants of the Netherlands Film Fund. Films are classified in various categories. For feature films, the Netherlands Film Fund distinguishes films targeted towards film festivals, and commercial films targeted towards a broad audience.
To avoid subjective judgement of the proposals of new films, we apply our method to estimate the potentials of proposals for new films. This estimation is based on the past performance of the film team (a producer, a director and a screenwriter) that submits a proposal for funding of the production of a film. Our method takes into account and balances the artistic and box-office achievements of the members of the production team. We tested it with data of Dutch films, and parameters according to the policy of the Netherlands Film Fund. The results show that our method provides good estimations of the success of proposals for new films.
4.1 Value of a film
The method is based on the value of a film, which is the historical realisation of the potential of that film. This value represents box-office revenues and awards won at film festivals. To this end, in cooperation with the Netherlands Film Fund we developed a value function for Dutch films. This value function takes into account the total number of visitors to the film in the cinemas and the artistic value via awards won at film festivals, where more points are obtained for an award at a more prestigious film festival. Table 1 gives an overview of film festivals, their awards and corresponding points.
The value function has several properties. First, the larger the number of visitors, the larger the value of the film. Also, the larger the number of visitors, the smaller the added value of a visitor to the film value. Thus, the value function increases with the number of visitors, and it shows a decreasing marginal value. Second, the value of the film increases with the prestige of the awards won. Also, the more awards won, the smaller the marginal value of an award to the value of the film. This means that the value function must be concave in both number of visitors and number of points for awards. Finally, we modified the value function to avoid disproportional effects of a film that receives a very low number of visitors or a very low number of award points (this would have a disproportionally large effect on the expected potential of a filmmaker). For this, the minimal value of a film is set to 2. Fitting to target values indicated by the Netherlands Film Fund, we arrived at the following formula for the value \(y_f\) of film f:
with \(c_{1f}\) the number of visitors/viewers, and \(c_{2f}\) the artistic score from awards won by film f.
Notice that 500,000 visitors or an artistic score of 4 points yield the same value: 8.6. For 1,000,000 visitors, this value increases to 9.7, which is also obtained for 500,000 visitors and 4 artistic points. Further, the policy of the Netherlands Film Fund determined three parameter values: (1) the rate of increase of the value \(y_f\), determined by the factor 2/10, (2) the weight of the number of visitors compared to the artistic points, determined by the numbers 500,000 and 4, and (3) the minimal grade, determined by the start value 0.231. The multiplicative factor 10 is included to allow the value to be interpreted as a grade as used in the Dutch educational system.
As an illustration, Table 2 gives the values of a number of films completed in 2010. We are not able to provide the most recent values (because of inavailability of information, and sensitivity of information with regard to subsidies). Our results in Sect. 4.5 clearly show that these films are most successful artistically (w.r.t. awards won) or commercially (w.r.t. numbers of visitors) in 2010, as in agreement with the expert judgement of the Netherlands Film Fund. Hence, the formula for \(y_f\) in Eq. (7) adequately captures the value of a film.
Together with the Netherlands Film Fund, we chose to combine the number of visitors and the artistic score of a film into the value function (7). Depending on the interests, other choices are also possible like considering only the visitors, or the artistic score, or some other measure(s). Table 2 seems to suggest that films either have many visitors or a large artistic score. However, that does not hold in general. Therefore, our value function combines the number of visitors and the artistic score into a single numerical value.
4.2 Potential of filmmakers
We model the potential of filmmakers as in the model of the Example with normal noise. In Appendix B we motivate this. The player set \(\mathcal P\) is the set of filmmakers (including producers). The value \(x_{p,f}\) of player p in film f is determined by the value of the film, his profit share \(\beta _{p,f}\) in this film, and noise: \(x_{p,f} = \beta _{p,f}y_f+u_{p,f}\).
We assume that the potential \(X_{p,f}\) of filmmaker (player) p in film f discounted to time 0 is subject to less noise when filmmaker p is more experienced. Experience is gained through participation in projects. Experience obtained more periods ago is of less predictive value than recent experience. To represent this, we let the influence of experience on the potential \(X_{p,f}\) decay over time with a decay factor \(\gamma _w\) per period. The decay rates determine, e.g. the half-life time of the influence of experience. If the half-life time is T years, then the corresponding decay rate is \(\gamma _w=\root T \of {1/2}\). The value for T is set by the Netherlands Film Fund.
Further, we assume that a filmmaker gathers more experience when his profit share \(\beta _{p,f}\) in the film is larger. The experience \(w_p\) of player p in period 0 is defined as
Also, values of recent films are subject to less noise. Let noise decay over time with a factor \(\gamma _v\) per period. We define the current influence of the noise \(v_{p,f}\) of film f completed in period \(t_f\) in the variance of \(X_{p,f}\) by
Using (6), the estimator
with normalising constant \(z=1/\left( \sum _{\{f:p\in C(f)\}} 1/\gamma _v^{t_f}\right) \), is the BLUE of the expected potential \(\mu _p\) of filmmaker p. It is a weighted average of the past performance of the filmmaker. We use the same type of evaluation for any type of filmmaker, since the Netherlands Film Fund does not distinguish between types (Ebbers and Wijnberg 2012b).
4.3 Evaluation of film proposals
A film proposal is usually made by a film team (coalition), consisting of three types of filmmakers: a producer, a director and a screenwriter. The contributions of these types to the film are independent. Sometimes, several filmmakers of the same type cooperate. For example, a film team may have two cooperating producers. Let the potential \(X_P\) resemble the joint potential of the cooperating producers, and let \(C_P(f)\) denote the set of producers in the film team of film f. Since production is a team effort, we consider the production team to be a (fictive) producer. We consider all films made by all producers in the production team, and let \(X_P\) be the potential as if all those films were made by the fictive producer.
Further, we may define the sets \(C_D(f)\), and \(C_S(f)\) of directors and screenwriters of film f, respectively. Since directors and screenwriters perform a large part of their task independently, their joint potentials \(X_D\) and \(X_S\) are determined as follows. Let the fraction \(\delta _{p,f}\) denote the weight of director \(p\in C_D(f)\), \(\sum _{p\in C_D(f)}\delta _{p,f}=1\). For example, if two directors cooperate, and one has no experience, we may set the weight of the unexperienced director to 0. Thus, the joint potentials \(X_D\) and \(X_S\) of directors and screenwriters are
Given the potentials of producers, directors and screenwriters, the potential of the film \(V_f\) is the weighted average of the potentials of the film team,
The weights \(\alpha _P\), \(\alpha _D\), and \(\alpha _S\) are determined by the Netherlands Film Fund.
The potential of a movie is a stochastic variable. As before, we estimate this potential using the success probability \({\mathbb P}(V_f > c)\) where the constant c is determined by the Netherlands Film Fund. Since the value of a film ranges from 1 to 10, we use
as the estimate of the potential of a proposal for film f. The expression on the right-hand side combines both the expected potential of the film \({\mathbb E}[V_f]\) and its variance \({\mathbb V}\mathrm{ar}(V_f)\) to measure the success of the film team. As such, it mimics the value function that also measures the success of a film.
4.4 Evaluation of individual filmmakers
Besides evaluating film proposals, we can also evaluate individual filmmakers. In Sect. 2, we have seen that the potential of a filmmaker p in film f at time 0 follows the normal distribution \(X_{p,f}\sim \mathcal{N}(\mu _p,\sigma ^2 v^2_{p,f} / w^2_p)\). Now we consider a fictitious film f solely made by this filmmaker at the current time period \(t=0\). The current influence of this fictitious film equals \(v_{p,f}^2=1\). This implies that the current potential of filmmaker p has mean \(\mu _p\) and variance \(\sigma ^2/w_p^2\). The experience \(w_p^2\) is obtained using time discounting (8). The filmmaker is now evaluated according to the success probability
where the constant c is determined by the Netherlands Film Fund.
4.5 Implementation and results
In this section, we use our method to evaluate proposals of Dutch films released in 2010. Besides, we rank the individual filmmakers by type.
The data of Dutch films till 2011 were gathered from publicly available sources.Footnote 2 \(^{,}\) Footnote 3 \({^{,}}\) Footnote 4 For each filmmaker, we registered the films made by him or her. For each of these films we collected the year of release of the film, the number of visitors, the awards won and the corresponding artistic score, and the profit share. These shares are determined per type of filmmaker, and all filmmakers of the same type are assumed to have an equal share. For example, a single producer has a share of 100%, and in case of two producers each has a share of 50%. The parameter values used are according to the policy of the Netherlands Film Fund: \(c= 5\), \(T=20\), \(\gamma _v=\root T \of {1/2}\), \(\alpha _P=3\), \(\alpha _D=2\), and \(\alpha _S=1\).
Following the procedure in Sect. 4.3, for each film we first derive the expected potential for each type of filmmaker. Thereafter, these are combined to obtain the expected potential of the film team for film f, \({\mathbb E}[V_f]\), and its variance, \({\mathbb V}\mathrm{ar}(V_f)\), using (10) and the independence of the types of filmmakers. Finally, by (11), this results in the estimated film value \(\hat{y}_f\). The estimated potentials of Dutch films released in 2010, and the characteristics of the corresponding film teams are shown in Table 8 in Appendix B. The realised film values are shown in Table 9 in Appendix B.
Table 3 lists the estimated potentials (film values) \(\hat{y}_f=10{\mathbb P}(V_f>c)\) and the realised film values \(y_f\) of Dutch films released in 2010. Overall, the estimated values are rather close to the realised values. Some films have an estimated film value more than two points below the realised film value; their performances are better than estimated. These differences are caused by debuting filmmakers in the film team. This is the case for the films New Kids Turbo (debuting director and screenwriter), Gelukkige Huisvrouw (debuting director and screenwriter), Dik Trom (debuting director), Loft (debuting director and screenwriter), and Ernst, Bobbie en het geheim van de Monta Rossa (debuting director and screenwriter). Our method estimates the film values based on past realisations of the filmmakers. Debuting filmmakers have no results yet, making it hard to estimate their results.
Further, two films have an estimated film value more than two points above the realised film value; their performances are worse than estimated. First, the film Het Geheim is a movie for children that did not attract as many visitors as expected. The film is based on an original story and was not based on a bestselling book. Hence, a good promotion was needed. Further, this is the third film of the scenarist, making him a beginning scenarist with limited experience. His potential is not easy to estimate with our method. Second, the film Vliegenierster van Kazbeck is a movie that was expected to win awards. Unfortunately that did not happen. Furthermore, this is the second movie of the director, making her a starting filmmaker. Therefore, it is not easy to estimate her potential.
To evaluate our estimations, we use the Theil U statistic (Theil 1961). This statistic is widely used to measure the accuracy of estimates. Since \(\hat{y}_f\) denotes the estimated value of film f, and \(y_f\) the realised value, the Theil U statistic equals \(U=\sqrt{\sum _f (y_f-\hat{y}_f)^2/ \sum _f y_f^2}\). The value U has the following meaning. If \(U>1\), then the estimate is not good. If \(U<1\), then the estimate is good, and the closer it is to 0, the better the estimate. In general, values of 0.55 or less are considered very good. For our estimated and realised film values, the Theil U statistic has the value \(U=0.40\), indicating that our estimates are very good. Hence, our method is a useful tool for more objective judgement of proposals for new films.
Besides, we use our model to evaluate the individual filmmakers. If, e.g. a producer was a director in the past, then this directing experience is not taken into account in the evaluation of the producer. We only consider the experience of a filmmaker per role since the experience gained in a film depends on the specific tasks and responsibilities related to that role. The resulting evaluations of the filmmakers are confidential. Therefore, we do not mention the names of the filmmakers, but we describe their results based on their past performances as indicated in Table 4. The ranked list of top 10 producers with largest evaluations is shown in Table 5. In the table, we list the evaluation values \(10{\mathbb P}(X_p > c)\) for each filmmaker p; these values may be interpreted as grades. The rankings of top 10 directors and screenwriters follow in Tables 6 and 7. These tables show that experienced filmmakers have large values. These rankings are concluded to be representative.
5 Conclusions
In this paper, we introduced a new numerical method to estimate the potential of proposals from collaborating professionals. Our method uses the past performances of the professionals to indicate their current potentials. These are combined to obtain an estimate of the potential of the proposed project by the collaboration.
We applied our method to estimate the potentials of proposals for Dutch films released in 2010. Our method is shown to obtain good results. Therefore, it is a useful tool for more objective judgement of proposals for new films. Besides, we rank producers, directors and screenwriters of Dutch films. These rankings are concluded to be representative. This application also shows that experienced filmmakers are highly valued, and that cooperation with new talented filmmakers is encouraged.
In general, our method may be used as a selection method for proposals that is more objective than reviewers expertise. It provides a tool for managers to estimate the potential of a proposal from collaborating professionals based on numerical data. Our model provides clear directives on which the estimate, and consequently the selection, is based.
Notes
Netherlands Film Fund website. http://www.filmfonds.nl/international.
References
Aitken A (1935) On least squares and linear combinations of observations. Proc R Stat Soc 55:42–48
Basuroy S, Chatterjee S, Ravid A (2003) How critical are critical reviews? the box office effects of film critics, star power, and budgets. J Market 67:103–117
Bergman MA, Lundberg S (2013) Tender evaluation and supplier selection methods in public procurement. J Purch Supply Manag 19:73–83
Bomsdorf F, Derigs U (2008) A model, heuristic procedure and decision support system for solving the movie shoot scheduling problem. OR Spectr 30:751–772
De Vany A, Walls WD (2002) Does Hollywood make too many R-rated movies? risk, stochastic dominance, and the illusion of expectation. J Bus 75(3):425–451
Ebbers JJ, Wijnberg NM (2012a) The effects of having more than one good reputation on distributor investments in the film industry. J Cult Econ 36(3):227–248
Ebbers JJ, Wijnberg NM (2012b) Nascent ventures competing for start-up capital: matching reputations and investors. J Bus Ventur 27:372–384
Eliashberg J, Shugan SM (1997) Film critics: influencers or predictors? J Market 61:68–78
Eliashberg J, Jonker J-J, Sawhney MS, Wierenga B (2000) MOVIEMOD: an implementable decision-support system for prerelease market evaluation of motion pictures. Market Sci 19(3):226–243
Eliashberg J, Elberse A, Leenders MAAM (2006) The motion picture industry: critical issues in practice, current research, and new research directions. Market Sci 25(6):638–661
Eliashberg J, Hui SK, Zhang ZJ (2007) From story line to box office: a new approach for green-lighting movie scripts. Manag Sci 53(6):881–893
Fizel J (ed) (2006) Handbook of sports economics research. M.E. Sharp, New York
Foutz N, Jank W (2010) Prerelease demand forecasting for motion pictures using functional shape analysis of virtual stock markets. Market Sci 29(3):568–579
Garfield E (1979) Is citation analysis a legitimate evaluation tool? Scientometrics 1(4):359–375
Ghiassi M, Lio D, Moon B (2015) Pre-production forecasting of movie revenues with a dynamic artificial neural network. Expert Syst Appl 42:3176–3193
Greene WH (1993) Econometric analysis, 2nd edn. Macmillan Publishing Company, New York
Guimerà R, Uzzi B, Spiro J, Amaral LAN (2005) Team assembly mechanisms determine collaboration network structure and team performance. Science 308:697–702
Hadida AL (2009) Motion picture performance: a review and research agenda. Int J Manag Rev 11(3):297–335
Hadida AL (2010) Commercial success and artistic recognition of motion picture projects. J Cult Econ 34(1):45–80
Holmstrom B (1982) Moral hazard in teams. Bell J Econ 13(2):324–340
Lampe HW, Hilgers D (2015) Trajectories of efficiency measurement: a bibliometric analysis of DEA and SFA. Eur J Oper Res 240:1–21
Lampel J, Shamsie J (2003) Capabilities in motion: new organizational forms and the reshaping of the Hollywood movie industry. J Manag Stud 40(8):2189–2210
Mestyán M, Yasseri T, Kertész J (2013) Early predictions of movie box office success based on Wikipedia activity big data. PLoS ONE 8(8):e71226
Palia D, Ravid SA, Reisel N (2008) Choosing to cofinance: analysis of project-specific alliances in the movie industry. Rev Financ Stud 21(2):483–511
Ravid SA (1999) Information, blockbusters, and stars: a study of the film industry. J Bus 72(4):463–492
Sharda R, Delen D (2006) Predicting box-office success of motion pictures with neural networks. Expert Syst Appl 30:243–254
Shugan S (1999) Pre-concept new product forecasting. Working paper, presented at Fall 1999 INFORMS Meetings in Philadelphia, College of Marketing Session
Shugan S, Swait J (2000) Enabling movie design and cumulative box office predictions using historical data and consumer intent-to-view. In: Working paper, University of Florida
Theil H (1961) Economic forecasts and policy. North-Holland, Amsterdam
Vogel HL (2004) Entertainment industry economics: a guide for financial analysis, 6th edn. Cambridge University Press, Cambridge
Walls WD (2005) Modeling movie success when ‘nobody knows anything’: conditional stable-distribution analysis of film returns. J Cult Econ 29:177–190
Author information
Authors and Affiliations
Corresponding author
Additional information
We thank Dr. Wilbert Kallenberg for inspiring discussions on the BLUE, and on statistics in general. We thank the associate editor and two anonymous reviewers for their helpful comments.
Appendices
Appendix A: Estimating the potential of a player
In this section, we elaborate on the technical details to estimate the potential of a player, as indicated in Sect. 3.
We derive the best linear unbiased estimator for the potential \(\mu _p\) of player p, \(p\in \mathcal{P}\). Let \(D_p=\{d_{p,f}: 0\le d_{p,f}\le 1,\ f\in \mathcal{F}; \sum _{\{f:p\in C(f)\}} d_{p,f} = 1\}\) be a set of coefficients for the projects of player p. Define the linear estimator \(\widehat{m(d_p)}\), \(d_p\in D_p\), of the potential \(\mu _p\) by
By (2) and (4), this is a linear unbiased estimator of the potential \(\mu _p\). As the variables \(U_{p,f}\), \(p\in \mathcal P\), \(f \in \mathcal F\), are independent, the variance of this estimator is
Using this, the estimator satisfies the following equation.
The best linear unbiased estimator (BLUE) \(\widehat{\mu }_p\) of the potential \(\mu _p\) is the estimator with minimal variance among the linear unbiased estimators \(\widehat{m(d_p)}\). The set of coefficients \(D_p^*=\{d^*_{p,f},~f:p\in C(f)\}\) that minimises the variance of \(\widehat{m(d_p)}\) solves
Then the BLUE may be written as \(\widehat{\mu }_p=\sum _{f: p\in C(f)} d^*_{p,f} X_{p,f}\). This coincides with the generalised least squares estimator in the generalised heteroscedastic regression model (Aitken 1935; Greene 1993).
1.1 Example continued: BLUE of the potential of a player
Using (12) in our example, the variance of the estimator \(\widehat{m(d_p)}\) is
Observe that the term \(\sigma ^2 / w^2_{p}\) does not depend on project f. Therefore, we obtain the BLUE of the potential \(\mu _p\) by solving
Lagrangian optimisation readily gives that there is a unique minimizer, namely
with normalising constant \(z=1/\left( \sum _{\{f:p\in C(f)\}}\frac{1}{v_{p,f}^2}\right) \). Thus, the BLUE of the potential \(\mu _p\) is
An unbiased estimator \(\widehat{\sigma ^2}\) for the variance \(\sigma ^2\) is readily obtained from (13), namely
\(\Box \)
Appendix B: Data of Dutch films in 2010
Our data have 1287 observations of filmmakers and their films. For each filmmaker, we estimated its expected potential by the BLUE (15). For each observation, the realisation of the noise is the difference between the realised potential and the estimated potential (2). This results in a sample of 1287 realisations of noise. In Fig. 1 a histogram and normal Q–Q plot of the noise are shown. As can be seen, the data do not strongly deviate from the normal distribution. Therefore, we assume it to be normally distributed, although the Kolmogorov–Smirnov test does not confirm this.
Table 8 shows the initial results of the film teams of Dutch films in 2010. Following the procedure in Sect. 4.3, for each film we first derive the expected potentials for each type of filmmaker. Thereafter, these are combined to obtain the expected potential of the film team for film f, \({\mathbb E}[V_f]\), and its variance, \({\mathbb V}\mathrm{ar}(V_f)\), using (10) and the independence of the types of filmmakers. The variance of a debuting filmmaker is set to 100. Finally, by (11) and the normal distribution of the film value, this results in the estimated film value \(\hat{y}_f\).
The subsequent Table 9 shows the number of visitors, the awards won, the artistic score \(c_{2f}\) of the awards, and the realised film values \(y_f\) of Dutch films in 2010. The realised film values follow from (7).
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Timmer, J., Boucherie, R.J., Lammers, E. et al. Estimating the potential of collaborating professionals, with an application to the Dutch film industry. OR Spectrum 40, 69–95 (2018). https://doi.org/10.1007/s00291-017-0492-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00291-017-0492-0