Breakthrough innovations and where to find them

Breakthrough innovations significantly depart from common practices and can potentially reshape existing markets, create new markets, and prompt the emergence of new technological trajectories. The crucial role that breakthrough innovations play in technological progress has stimulated a lively line of research investigating methods to identify them in actual empirical contexts. Despite this ongoing effort, the availability of data on breakthrough innovations is still scarce and seems to have prevented, at least so far, an integrated perspective comprising both their technical and economic significance. We address this limitation by developing a method that uses award-winning innovations which have been successfully commercialized to find breakthrough patents on a large scale. For the period 1976 – 2013, in a sample of 138,467 USPTO patents, we identify 17,176 break-through innovations. Relevant robustness checks support the validity of our classification. We then exploit this sample to assess the sources of breakthrough innovations.


Introduction
Breakthrough or radical innovations are generally regarded as ruptures along specific technological trajectories, possibly leading to shifts or transformations in the prevailing technological paradigm (Dosi, 1982).Thus, they play a crucial role in the "creative destruction" process that characterizes the long-run dynamics of technological evolution (Ahuja and Lampert, 2001).In contrast, the literature refers to continuous or incremental innovations when the outcome of the innovation process is an improvement of the existing technology (Garcia and Calantone, 2002).While incremental innovations occur, more or less continuously, breakthrough innovations are sporadic.As Fleming puts it, in the technology landscape, "almost all inventions are useless; a few are of moderate value; and only a very, very few are breakthroughs" (Verbatim).
Breakthrough innovations are therefore rare events that are inherently difficult to appraise and characterize.As such, they are better investigated by in-depth qualitative studies rather than by large scale quantitative studies that use noisy proxies of innovation output and innovation quality such as patent data.When studying breakthrough innovations on a large scale, the key challenge becomes the measurement of patent quality or value.In the welter of large patent samples, how can we identify those covering breakthroughs?The increasing appreciation of the high variability in the value of patents has led to the emergence of a lively stream of research investigating whether the information contained in patent documents would be suitable to construct reliable indicators of patent quality (Harhoff et al., 2003).So far, several proxy indicators of patent quality have been tested for their ability to capture the heterogeneity in the quality of the underlying inventions and reproduce the skewed distributions associated with it (see for example Harhoff and Reitzig 2004, Lanjouw and Schankerman 2001, Tong and Frame 1994, Trajtenberg 1990).
The identification of breakthrough innovations is mostly based on the technical merits of an innovation and most commonly carried out using patent citations.Dahlin and Behrens (2005) and Verhoeven et al. (2016) looked at the pattern of patents' backward citations to spot breakthrough innovations based on their ex-ante characteristics.The intuition is that breakthrough patents differ in how they source and recombine existing knowledge compared with previous patents in the same field.Other studies classify inventions as breakthroughs using forward citations, as they will be cited in many follow-up patents (Ahuja and Lampert, 2001;Fleming, 2001).For example, in light of the skewed distribution of patent value and the importance of patent citations in determining patent quality, a commonly adopted criterion defined by Ahuja and Lampert (2001) considers breakthrough inventions the top 1% most cited patents.Even if plausible, it is hard to tell how many genuine breakthroughs this approach would capture.
Despite these attempts, the actual empirical effectiveness of the proposed metrics remains uncertain.In a recent study, Higham et al. (2020) show how commonly used patent quality measures are generally not in agreement about what makes a "high-quality" patent.They tend to capture different aspects of technological discontinuities.Moreover, their ability to pinpointing the economic significance which emerges during the shift from invention to innovation is still questionable (Bessen, 2008).Information on the use of patented inventions is highly valuable for policymakers to the extent that patenting without commercialization has a limited societal impact (Higham et al., 2020).Extant evidence on this aspect has been collected via surveys (e.g., Giuri andMariani 2007, Webster andJensen 2011) or retrieved from virtual patent marking websites (e.g., de Rassenfosse and Jaffe 2018).In general, however, the scarce availability of data on breakthrough innovations seems to have prevented, at least so far, an integrated perspective capable of capturing both technical and economic significance.This paper seeks to address this gap in three main ways.First, we develop a method that uses award-winning innovations to identify breakthroughs on a large scale.Starting from a sample of patents protecting products that received a UK Queen's Award, we build a classifier that optimizes patent-based indicators' ability to classify award-winning patents correctly.We then use the same parameters to predict the probability of a patent to be a breakthrough on a larger scale.In comparison with existing approaches, our method proposes to shift away from arbitrarily chosen cutoffs and towards a more careful evaluation of the probability of an individual patent to be a breakthrough.Moreover, drawing on a sample of patents that proved to be commercially successful in the "real-world", we ultimately seek to identify breakthrough innovations, as opposed to inventions.While the paper presents an application using the Queen's Awards, the methodology is flexible and replicable in other contexts.
Our second contribution speaks to the scarce availability of data on breakthrough innovations.The output of our method is a dataset comprising 138,467 patents filed between 1976 and 2013, with 17,176 patents meeting our criteria for breakthrough innovations ( (Capponi et al., 2021).Relevant robustness checks confirm the validity of our method and the quality of the dataset.
Finally, we exploit this dataset to re-examine the sources of breakthrough innovations.Previous work has shown mixed evidence on the difference in technical merit and commercial success between inventions created by independent inventors and organizations.Dahlin et al. (2004) find that while independent inventors are over-represented among low-impact patents, they also hold the most influential patents.In contrast, focusing on the performance of "lone inventors", Singh and Fleming (2010) find that individuals are less likely to generate breakthroughs and more likely to invent failures.We test the ability of different actors to develop commercially successful breakthroughs and find that both individual inventors and public organizations are less likely than corporations to be the source of breakthroughs.
The remainder of this paper is organized as follows.Section 2 provides an overview of the literature on breakthrough innovation and measurement.Section 3 presents the Queen's Award prize scheme.Section 4 describes the process behind the dataset construction.Section 5 goes through the methodology.Section 6 reports the results and Section 7 shows the robustness checks to validate our procedure.In Section 8, we exploit our dataset to re-examine the sources of breakthrough innovations.Section 9 concludes.

Literature review
Breakthrough or radical innovations can be defined along different dimensions and several concepts have been proposed to assess their impact on firms, markets, and the economy at large.Focusing on technological importance, seminal work regards breakthrough innovations as discontinuous, disruptive.Utterback (1994) defines them as "change that sweeps away much of a firm's existing investments in technical skills and knowledge, designs, production technique, plant and equipment".Ahuja and Lampert (2001) describe technological breakthroughs as inventions that shape the development of industries by opening new paths for many subsequent technological developments.Looking at the skewed distribution of inventive value, Fleming (2007) refers to breakthroughs as the "long tail" of innovation.The critical role that breakthrough innovations play in technological progress generated a prolific line of research investigating ways to identify them.Most of the work in this domain relies on patent data and the measurement of patent value.
Patent value heterogeneity reflects the distribution of innovation significance.In the literature, we find two main approaches to its measurement: to estimate patent value directly or indirectly ascertain quality-adjusted measures of inventive output using information contained in the patent document (Bessen, 2008).Early contributions in the first group investigate the value of patents by using data on the stock market evaluation of firms (Griliches, 1981;Hall, 1993;Hall et al., 2007;Pakes, 1985).More recently, Kogan et al. (2017) provided individual patent market evaluation using assignee's abnormal stock return on the date of patent grant.With this approach, they can estimate the economic value of patents issued to US firms between 1926 and 2010. 1 A different line of literature uses renewal decisions to infer the value of patents (Lanjouw et al., 1998;Pakes, 1986).Alternative direct ways to estimate the economic value of patents include indicators based on patent transactions such as the decision to license (Gambardella et al., 2007;Sampat and Ziedonis, 2004) and the level of licensing revenues (Sampat and Ziedonis, 2004).Other studies exploit survey data to collect patents' expected sales value (Gambardella et al., 2008;Harhoff et al., 2003Harhoff et al., , 1999)).An important contribution in this respect is the Pat-Val-EU Survey which collected a direct measure of patent value from inventors' direct ex-post evaluation of the patent for a very large sample of European patents.
Considering the indirect approach, the count of patents' forward citations is by far the most popular indicator of technologically valuable patents (Albert et al., 1991;Carpenter et al., 1981;Trajtenberg, 1990).The number of forward citations relates to technological significance because it signals patents' role in establishing a successful line of innovation.Patents' backward citations have also been exploited to assess patent novelty.In this respect, even if the count of backward citations, in principle, ought to be considered as an indicator of incremental innovation (Ahuja and Lampert, 2001;Squicciarini et al., 2013), they can provide insightful information on the new recombinations of prior knowledge (Verhoeven et al., 2016).Following a similar logic, recent developments in text mining techniques are looking for new recombinations of knowledge within the patent document itself (Arts et al., 2019).Another variable related to patent value is the number of claims, insofar as "An inventor's invention is embodied in his or her claims" (Tong and Frame, 1994, pp. 134).Each patent contains some claims which disentangle the invention in different contributions, which represent built-in inventions themselves.Other studies exploit patent family size which refers to the number of countries in which a patent is filed and enforced.Since the geographical extension of patent coverage is costly, only the inventions with a sufficiently high expected value on the market will be protected abroad (Lanjouw et al., 1998).
While all these metrics have substantially contributed to our understanding of patent value and provided a meaningful ground for its measurement, there are still important pitfalls to address.First, these metrics tend to capture different dimensions of patent value and they can lead to contrasting results.It is becoming clear that a one-size-fits-all approach to using these indicators can be misleading (Higham et al., 2020).The second caveat that predominantly applies to these indirect approaches is the general detachment between the patent and the actual 1 Note that these data are publicly available and we will used them in Section 7.
G. Capponi et al. value of the invention in the real world.While the value distribution of patents reflects the distribution of inventions' significance, the commercial value of patents remains less explored.
In this paper, we seek to overcome some of these limitations using patent-based indicators and parameters externally validated by a list of patents linked to award-winning innovations.Economic historians proposed prizes as an alternative (or complementary) source of innovation data to address the shortcomings of patent data.For example, analyzing inventions promoted by the Royal Agricultural Society of England from 1839 to 1939, Brunt et al. (2012) found that patents are more likely to be renewed when linked to awarded inventions, indicating a positive relationship between the awards and the value of the patent.Moser (2005Moser ( , 2011) ) compiled a dataset of innovations using the catalogues of nineteenth-century industrial exhibitions and highlighted this type of data covers "economically useful innovations" rather than "inventions" (Moser, 2002, pp. 1-2).In this paper, we combine the use of patent-based indicators and prizes to identify breakthrough innovations.While we present an application using the Queen's Awards, the methodology is flexible and replicable in other settings.

The Queen's Award for enterprise: background
The scheme of the Queen's Awards was initially announced on February the 4th 1965 in the House of Commons by Prime Minister Harold Wilson, following the recommendations of a Committee chaired by HRH The Duke of Edinburgh.In a later speech in the House of Common, Wilson specified that: "The purpose of this new scheme is twofold: to reward and to stimulate.I hope that the award will encourage industry in its efforts to achieve the improvements in exports and the technological advance on which our national future so much depends". 2 To keep the scheme's parameters up to date, the first draft released in 1965 announced that "the working of the Award Scheme as presented in our report should be reviewed after five years so that any modifications which practical experience of its operation had shown to be desirable could be introduced" (Mountbatten- Windsor, 1965).In practice, apart from the first presentation draft, they made only three other official reports : in 1970, 1975 and 1999.In each review, the Committee in charge collected feedback from the applicants to assess whether and to which extent to amend the scheme.Over its lifetime, the nature of the prize remained largely unaffected and, still today, it represents the most prestigious award for innovation for individuals and businesses in the UK (Groom, 2015).Every year, major British newspapers such as the Financial Times and The Guardian publish the list of winners and specific articles devoted to comment a selection of them.
At present, what is known as "The Queen's Awards for Enterprise" recognizes achievements in three separate fields: innovation, international trade and sustainability (Department of Trade and Industry, 1999).There are no restrictions regarding sectors or predetermined patterns of regional allocation: all large, medium or small organizations that regularly operate as a "business unit" in the UK are eligible to apply.Through the years, another issue commonly discussed was whether the grant should be associated with tangible rewards, for example, special tax reliefs.However, these suggestions have never been endorsed as "their inclusion would detract from the dignity of the Honour" (McFadzean, 1970).Instead, "The Award signifies recognition by the Sovereign of striking achievement and we recommend that it should be represented by an emblem which the holder of the Award should be authorial to display in a wide variety of ways […] it may be displayed on flags, plaques, note-paper, packaging, and goods themselves" (Mountbatten-Windsor, 1965).To encourage further improvement, the founding Committee limited the currency of the award to 5 years, during which winners have the right to display the emblem together with the year of victory (Mountbatten-Windsor, 1965).
In this study, we only focus on the Queen's Award for Innovation (hereafter QAI).In the following part, we provide some details on the past and current eligibility criteria to give a precise idea of the parameters used to classify innovations as successful.

Eligibility, selection and relevance of the QAI
At the very beginning, what was known as the Queen's Award for Technological Achievement was rewarding "A significant advance, leading to increased efficiency, in the application of an advanced technology to a production or development process in British industry or the production for sale of goods which incorporate new and advanced technological qualities".Already in the first report in 1970, the Scheme Review Committee underlined the applied connotation of the definition by removing the word "advanced" before "technology", and emphasizing that "the timely application of established technology, as against advanced technology, may well be equally important and deserving of recognition, particularly in the less sophisticated sectors of industry" (McFadzean, 1970).
In the same spirit, in 1999 the Review Committee in charge drafted the final and most substantial reformulation of the criteria for the award.Above all, the name changed from "Queen's Award for Technological Achievement" to "Queen's Award for Innovation", to broaden the eligible set of subject matters to innovation in services and, in general, innovations which are not technology-driven.There are two main groups of eligibility criteria: -Outstanding innovation, resulting in substantial improvement in business performance and commercial success, sustained over not less than two years, to levels which are outstanding for the goods or services concerned and for the size of the applicant's operations, and arising in the fields listed below.Or: -Continuous innovation and development, resulting in substantial improvement in business performance and commercial success, sustained over not less than five years, to levels which are outstanding for the goods or services concerned and for the size of the applicant's operations, and arising in the fields listed below.
Achievements under either criterion may be assessed in any of the following fields: the invention, design, production (in respect of goods), performance (in respect of services, including advice), marketing, distribution, after-sale support, of goods or services (Department for Business Energy and Industrial Strategy, 2013, pp. 6).
The application procedure requires a detailed description of the innovation and evidence of its commercial success covering a period of two to five years before the submission.Specifically, candidates should support their application by providing the patterns of growth of earnings associated with the innovation, providing information on indicators such as profitability, market shares, and others.Because of the meritocratic nature of the award, the Committee refrained from fixing a number of prizes to be given each year (McFadzean, 1970).Initially, the body in charge of selecting winners was a mix between members from within and outside the government, forming an Advisory Committee which was assisting the Prime Minister by making recommendations about the winners.This group was also supported by two subordinate committees under the guidance of the Ministry of Technology (Mountbatten-Windsor, 1965).In 1999, the subordinate committees were replaced by appointed Panels of ad-hoc Judges with the relevant 2 Available as a transcript of a House of Common sitting at: https://api.parliament.uk/historic-hansard/commons/1965/aug/03/the-queens-award-to-industry G. Capponi et al. expertise, one for each Award category.The Panel of Judges examining the Queen's Award for Innovation is chaired by the Permanent Secretary of the Department for Business, Innovation and Skills.Applicants are initially screened by a group of contracted technical assessors, who provide the first round of recommendations (Department for Business Energy and Industrial Strategy, 2013).In September 2018, we had the chance to speak with two contracted technical assessors who provided more details on the selection process pipeline 3 and the criteria adopted to establish innovation quality 4 (personal communications with two contracted technical assessors, 28/09/2018) Despite the rigorous procedure, the evaluation is made on a pool of self-selected applicants.To mitigate potential bias problems, we crossvalidate the relevance of the Queen Award-winning innovations by comparing them with the well-known SPRU Innovations Database (Pavitt, 1983).The latter collects over 4000 technological breakthroughs introduced in Britain between 1945 and 1983, selected by experts' recommendations.The two dataset overlap for 18 years, in which the SPRU Innovations Database identifies 2,595 technological breakthroughs, while 424 innovations receive a QAI.Exploiting the names of the companies and the time references reported in both dataset, we considered a match to be successful if the innovator's name was the same and the time difference between the introduction of the innovation in the SPRU Database and the QAI recognition was at most five years.We found 144 matching observations, suggesting that 34% of the prizes awarded between 1966 and 1983 were awarded to innovations listed in the SPRU Database.In our view, this provides a very significant corroboration to the reliability of the QAI selection process in pinpointing breakthrough innovations. 5 Given the recognized significance of this accolade, some studies dealing with breakthrough innovations have used QAI winners as a source of data, relying mostly on information collected via questionnaires.The first book using the QAI as a source of data was the book Wealth from Knowledge.Studies of Innovation in Industry (Langrish et al., 1972) This work contains 84 case studies based on firms winning in 1966 and 1967.In particular, the authors look at "success stories" to capture the relation between technological innovation and organizational change (see Langrish et al., 1972, page 4).Other studies used the Queen's Awards to assess the relevance of informal external linkages in the innovation process (Conway, 1995), analyze the actual contribution of SMEs to innovation in Britain (Romijn and Albaladejo, 2002;Tether, 1998Tether, , 1996) ) and investigated the reasons behind the tendency of innovative firms to congregate in metropolitan regions (Simmie, 1998).
To the best of our knowledge, this paper represents the first attempt to systematically gather the QAI data over the entire lifetime of the scheme, match the winning inventions to patents and use QAI as a starting point for detecting breakthrough innovations.

Dataset construction
In this section, we first explain how we matched patents to Queen's Award winning innovations.We then construct an "in-sample" of QAI patents and controls that we will use to test which model better predicts the probability of a patent to be a QAI patent.The parameters providing the best estimate "in-sample" will then be used to identify breakthrough patents in the "full sample", a dataset that combines the in-sample patents with all the patents filed at the USPTO by a GB applicant over the same period.

Patent matching
Every year the full list of the QAI winners is published with the name of the company, the location and a brief description of the innovation, for a total of 1,475 prizes awarded between 1966 and 2015. 6For example, a typical entry taken from the 1993 list of winners looks as follows: BT Laboratories (BTL) Optical Research Division, Ipswich, Suffolk, Metal-organic vapor phase epitaxy for making semiconductor devices for the optoelectronics industry.
Private, public and not-for-profit entities are eligible for a QAI provided that the innovation contributes to industrial efficiency.In the case of joint development, the award is given to both entities.It was not before 1975 that the Review Committee in charge required sales figures in the QAI application to provide evidence of the economic value of innovation (Mountbatten-Windsor, 1975).For this reason, we consider only prizes awarded from 1976 till 2015, for a total of 1,234 innovations.Starting from this listing, we first matched the names of the winning firms to the unique code identifying them in AMADEUS Bureau van Dijk (BvD), a database containing comprehensive information on 21 million European companies. 7We matched 983 innovators (80% of our sample) for which we could collect the relevant firm-level information and the list of patents associated with each firm.We then looked for 3 The applications are allocated to the assessors depending on their area of expertise and the number of assessors depends on the number of applications.Assessors are qualified in science and technology, most of them are PhD graduates working in different industries.Assessors review the applications and produce a short list of recommended applicants for the Panel of Judges.Every application is appraised by two people independently.Their decisions have then to be discussed and approved by the lead assessor.After the first round of recommendations, shortlisted firms have to produce audits to confirm their figures.Afterwards, the selected applications are forwarded to a Panel of Judges.Panelists are appointed by consultations with stakeholders, both within government and externally.The panel is reviewed every three years, with no fixed term on membership.Candidates' due diligence and CSR practices are the main concerns at this stage and the retention rate tends to be high.The finalist has to be approved by the Prime Minister and its Advisory Committee. 4Technical assessors evaluate innovativeness by looking at competitors and current market offerings.They use the description of the nature of the product reported in the application, the narrative content explaining the innovation process and the challenges faced in developing new solutions.Patents are regarded as supporting evidence rather than a determining signal of novelty.Financial criteria are also important: there should be consistency between the numbers applicants claim, the statements, and the firms' size. 5It is worth noting that Queen's Awards are given yearly and as such recognize "contemporary" innovations; the SPRU approach is more longitudinal and recognizes innovations "ex-post".Focusing on the two decades of overlapping (1966)(1967)(1968)(1969)(1970)(1971)(1972)(1973)(1974)(1975)(1976)(1977)(1978)(1979)(1980)(1981)(1982)(1983)(1984)(1985), the proportion of Queen's Award innovations which are listed in both dataset is relatively higher in the first decade than in the second.When considering all the Queen's Award given between 1966 and 1985, 56% of the prizes are given in the first decade (1966)(1967)(1968)(1969)(1970)(1971)(1972)(1973)(1974)(1975) and 44% in the second (1976)(1977)(1978)(1979)(1980)(1981)(1982)(1983)(1984)(1985).When considering only the Queen's Awards which are also in the SPRU dataset, the share of prizes given in the first decade becomes 70%, with only 30% in the second decade.This evidence suggests that innovations awarded in the second decade might not have been established enough to be included in the SPRU dataset.Arguably, innovations awarded in the second decade might have been overlooked because experts' could benefit from hindsight to a lesser extent.
possible matches between the innovations winning a QAI and one or more patent family. 8We did not impose any restrictions on the application authorities, however, we limited the search to a certain time interval which varies depending on the rules governing the prize scheme in every period. 9Using this approach, we estimated that 32% of the innovations winning a QAI between 1976 and 2015 were protected by at least one patent at the time of the award.Specifically, we singled out 1, 468 patent families filed between 1976 and 2013 protecting 401 innovations. 10Table 1 reports the number of matched patent families having at least one application filed in a major patent office and the corresponding share calculated on all the matched patent families.We can observe that almost half of them have a filing at the USPTO and 38% at the EPO.

In-sample and out-of-sample composition
To implement our method and identify breakthrough innovations, we only focus on the applications filed at the USPTO.There are two main reasons behind this choice: first, since the winning companies are all based in the UK, by selecting a foreign authority we are implicitly raising the quality level of the group of valuable patents, possibly capturing the upper tail of the QAI patented innovations.Typically, companies file applications in foreign patent offices for inventions that are planning to economically exploit abroad.Second, we can easily retrieve the scores of the patent-based indicators from the OECD Patent Quality Indicators database, which provides details on roughly 7 million USPTO patents, with filing dates going as back as 1976 (Squicciarini et al., 2013).
We first exploit patent-based indicators to identify the QAI patents within a larger pool of control patents.We build the control group using all the patent applications filed at the USPTO by companies having at least one QAI patent.This sampling choice allows us to compare the awarded inventions with the other innovations developed within the same group of innovators, i.e. leveraging on the same set of resources and capabilities. 11This dataset consists of 9,638 patents, of which 524 QAI patents (5,4%) and 9,114 controls, we refer to this selection as the "in-sample". 12 To find commercially successful innovations on a large scale, we construct an "out-of-sample" as a comprehensive set of USPTO patents filed by GB applicants from 1976 to 2013.Starting from the full set of USPTO patent applications listed in the OECD Patent Quality Indicators database (around 7 million patent applications), we single out the patent applications associated with a GB applicant in the EPO's Worldwide Patent Statistical Database, henceforth PATSTAT.Excluding those reporting missing values on at least one of the indicators of interest, we end up with an out-of-sample of 128,829 patents.Table 2 reports the composition of the full sample.
Our method seeks to first identify the model which best classifies QAI patent in-sample, and then use the same parameters to estimate the predicted probability of being a breakthrough for all the patents in the full sample.

Methodology: how to identify breakthrough innovations
We start by comparing the ability of patent-based indicators to identify QAI patents in the in-sample.We estimate logit models specified as follows: Where p=prob(QAI patent=1).QAI patent is a direct measure of patent value indicating whether a patent protects a QAI winning innovation or not, X is a vector of patent-based indicators including renewals, 13 number of forward citations, patents' family size and number of claims.We have selected indicators that are ((1) widely adopted as measures of patent quality and (2) standardized and easily retrievable The OECD Patent Statistics Manual defines patent families as "the set of patents (or applications) filed in several countries which are related to each other by one or several common priority filings" (OECD, 2009, page 71).Our analysis is based on DOCDB families, which are expert-validated families.In principle, the technical content of patents protecting the same invention should be the same.The DOCDB definition builds on experts' assessment on whether the technical content of a patent matches an existing family or not (Martínez, 2011). 9A detailed explanation of the patent matching procedure is provided in Appendix A. 10 However rigorous, the method we followed looks into a specific and limited time window to find matching patents.Thus, it is possible that a few cases were not captured by our search.
11 The Queen's Award for Innovation strictly recognizes innovations developed in the UK.The innovator can be either a British firm or the UK branch of a multinational company. 12The construction of the in-sample requires access to the complete list of the patents associated with a firm, therefore we only considered the winning companies that we successfully matched on AMADEUS.An additional reduction to the sample size resulted from matching the patent applications to the OECD Quality database, which covers patent applications filed between 1976 and 2015 at the USPTO.To have consistent estimates across observations, we exclude patents having missing values on at least one of the indicators of interest.
13 Note that while the OECD Quality Database Squicciarini et al., 2013) includes a yearly measure of renewals, USPTO patents are renewed after 3.5 years and renewals fees are only available since 1981.From an e-mail exchange with the authors, we learnt that the renewal indicator for the USPTO patents in the OECD Quality Database reflects the lifetime of a patent and counts the number of years between the patent application and the latest date during which any of the following events occurredpayment of the renewal fees (not applicable before 1981), withdrawal, refusal, etc.To better capture the value associated with the payment of renewal fees, we define a variable Renewal USPTO that counts the number of renewal fees paid instead (0 if the OECD indicator is <4 years, 1 if [4, 8), 2 if [8, 12), 3 if >=12 years).In our dataset (138,467 observations) we also have 13,350 patents that have been filed before 1981 when the payment of maintenance fees came into force.While this makes the attribution of a Renewal_USPTO value problematic, the OECD indicator shows that 95% of the patents in this group were kept alive for less the 4 years anyway.In these cases, Renewal_USPTO can take value zero without loss of generality.As for the remaining 5% (696 patents), we assign the Renewal USPTO assuming that renewal fees would have been paid throughout the indicated patent lifetime.
from major databases.We also include sets of dummies controlling for patents' filing year and technology class. 14Table B.1 in Appendix B provides the list of variables and sources.
Working with binary models, the key issue we face is the definition of a threshold probability which best classifies patents as QAI patents or controls.We first evaluate different specifications using a Receiver Operating Characteristic (ROC) curve, which selects the best model based on its accuracy, i.e. the percentage of correctly classified patents at a certain threshold. 15The ROC curve estimates the optimal probability threshold as the one maximizing sensitivity and specificity with equal weight. 16While giving an estimate of the models' accuracy, this approach does not incorporate information on the consequences of misclassification and it fails to tell whether a specification is worth using at all.This concern is particularly prominent in epidemiology where a false negative is much more harmful than a false positive.In these cases, a model with a high level of accuracy but a level of specificity which is slightly higher than sensitivity would be a poor choice.Similarly, since we aim to identify breakthroughs out-of-sample, we would prefer a model which limits the false positive rate rather than a model which maximizes accuracy.
For these reasons, we select the best model adopting a Decision Curve Analysis (DCA), a comparative approach that incorporates information on the consequences of a misclassification (Vickers and Elkin, 2006).This method assesses the effectiveness of the prediction models by looking at the theoretical relationship between the threshold probability p t of being a breakthrough and the relative value of false positive and false negative.To explain the concept, suppose that a surgeon needs to intervene on a patient depending on a marker prediction on the patient probability to have cancer.On the one hand, an unnecessary intervention can have severe side effects, on the other hand, failure to intervene on a true positive case may lead to the patient's death.Assume that the surgeon would definitely proceed if the probability to have cancer is 30%, but he would not if the probability was only 1%.However, if the risk was 10%, he would be uncertain.To fix p t at 10% means that the surgeon considers the failure to intervene in a positive case to be 9 times worse than unnecessary surgery (Vickers and Elkin, 2006).This relationship is defined by the net benefit formula (first introduced by Peirce 1884): Where p t is the threshold probability.In this equation, (p t /(1− p t )) tells the relative importance of false negative and false positive results.The basic mechanism to follow would then be to (1) choose a meaningful range [0,1] for p t , (2) look at the net benefit associated with each point in the interval for the models we are comparing, (3) choose the optimal threshold and the best model accordingly.Finally, using the parameters estimated in-sample with the selected model, we predict the probability of a patent to be a commercially successful breakthrough on the full sample.The threshold probability indicated by the DCA will then be used to classify the patents as breakthroughs or not.

Results
As expected, the descriptive statistics within the in-sample show that forward citations, family size and claims present a right-skewed distribution, we therefore follow previous papers (see for example Singh and Fleming 2010) and use their logarithmic transformation in the analysis. 17The number of renewals is not affected by outliers as it only takes four possible values.Looking at the distribution separately for QAI patents and controls, we notice that the means for the QAI patents are higher for all indicators except for the family size (see Table B.2 in Appendix B).To better interpret this outcome, it should be pointed out that we set an explicit requirement on the breadth of control patents' 14 Based on (Schmoch, 2009) classification, which is reported in the OECD Quality database (Squicciarini et al., 2013)  15 The ROC curve is defined on a space having sensitivity as y coordinate and 1-specificity (fall out rate) as x coordinate.The ROC curve of a diagnostic test leading to perfect discrimination would be a vertical line from (0,0) to (0,1) joined with a line from (0,1) to (1,1), while the curve of a poor performing test would resemble the 45 lines, which is the chance level (Hajian-Tilaki, 2013).In this study, we adopt the empirical (non-parametric) ROC estimator, which constructs the curve by connecting the (1-specificity, sensitivity) points obtained at all possible cut-off values (Park et al., 2004). 16To obtain the optimal threshold for each model we adopt the Youden Index, which selects the point on the curve having the maximum vertical distance from the chance line, maximizing both sensitivity and specificity (Youden, 1950).
families by considering only USPTO filed applications.Also, jurisdictions vary in terms of market size and they should not be considered equally relevant (Van Zeebroeck, 2011).
We run the logit models to test the effect of each patent-based indicator on the probability of a patent to be a QAI patent.
The coefficients are positive and significant for all patent indicators in the six specifications we defined (see Table 3).However, the models' discriminatory power varies: a ROC curve approach would select Model 6 as the most accurate model, with 80% of the patents correctly classified.This evidence is in line with Higham et al. (2020) who describe patent quality as a multidimensional concept that cannot be captured by a single metric.Maximizing specificity and sensitivity with equal weight, Model 6 classifies a patent as 1 if its probability of being a QAI patent is above 7.8%.This cut off is rather low because of the strong presence of control patents in the sample (more than 90%).Indeed, the level of accuracy is heavily driven by the ability of the model to classify the zeros.Accordingly, Model 6 shows a relatively high level of specificity at the optimal threshold.For these reasons, the model may perform poorly when it comes to the identification of breakthrough innovations in the full sample.
Given the very small share of QAI patents in-sample (5.5%), suppose that we would classify a patent as 1 if its probability of being a QAI patent is at least 15%.Fig. 1 shows the decision curves for each model estimated in Table 3, the x-axis is p t while the y-axis is the net benefit, which indicates the percentage of true positive cases that a model captures at different probability thresholds (see Eq. ( 2)).
At each threshold probability, the model leading to the highest net benefit is the best.Looking at the y-axis, we can observe that the highest possible net benefit is the true prevalence of QAI patents in the sample.For example, identifying breakthroughs using a model that at a certain p t achieves a net benefit of 2% (0.02), is equivalent to a strategy that correctly picks 2,000 breakthroughs out of 100,000 patents.Note that this graph does not give any information about the number of false negatives and false positives. 18 In all models, as the threshold probability rises, we assign 1 to fewer patents, and the share of true positives that we capture tails off to zero.Instead, if we set the threshold probability to zero, the net benefit reached by the models approaches the prevalence.In general, the models we are testing achieve a higher or equal net benefit at each probability threshold, compared to the baseline strategies.At this point, we should identify a suitable range for p t and select the best model.Fig. 1 shows that for p t > 0.2 the net benefit is very low and we risk missing out on too many breakthroughs.Instead, for p t < 0.1 the net benefit increases but the risk of including a high number of false positives increases too.Fig. 2 zooms on this interval.
We can observe that Model 6 performs better than the other models for probability thresholds ranging from 13% to 19%.To have a better sense of the outcome for this model, we calculate the prevalence of false positive and false negative for p t =0.15, p t =0.16, p t =17, p t =0.18 and p t =0.19 based on the estimates obtained for Model 6 (in Table 3). 18The decision curve analysis enables us to compare the performance of the models vis-a-vis the two baseline strategies of "pick all", or "pick none".The horizontal line (pick none) corresponds to a strategy in which we assume that there are no breakthroughs in the sample.In this case, the net benefit would be zero, regardless of p t .Conversely, a "pick all" strategy assumes that all the innovation in the dataset are breakthroughs.This approach has a performance which is comparable with the models only if we set p t =0, that is if we do not attach any harm to the inclusion of false positive, while we are 100% committed to avoid false negative.In this case, the intercept point on the y axis is 5.5%, i.e. the QAI patents' prevalence.If we assign 1 to every patent in the dataset, than 5.5% will be true 1s, hence the net benefit, with 94.5% false positive rate.
Table 4 shows the results; as p t increases, the group of patents classified as QAI patents becomes smaller and the share of false positive decreases.
Opting for a conservative approach, we estimate the predicted probability in the full sample using Model 6 and setting p t =0.19.We then define a new variable predicted breakthrough which takes value 1 if the predicted probability calculated in the full sample is p>=0.19,0 otherwise. 19This method classifies 17,176 of the patents in the full sample as breakthroughs (12%).In the next section, we check the validity of our method testing both the economic significance and the technical merits of our predicted breakthroughs.

Relation with independent estimates of patent value
We first examine the correlation between our predicted breakthroughs and patent value.We use the estimates developed by Kogan et al. (2017) to measure the monetary value of patents.20By matching their data with ours, we could assign a monetary value to 2, 887 patents of our in-sample (29.9%) and 32,221 patents of our full sample (23.2%). 21he regressions in Table 5 examine the correlation between QAI patents, our predicted breakthroughs, and the log of patent value estimated by Kogan et al. (2017).In all regressions, we include year and technology dummies as controls.In some specifications, we also include the number of forward citations of the patent, to see whether the QAI patents or the predicted breakthrough variable remain significant while including one of the most used indicators of patent value.
Models 1 and 2 examine the correlation between receiving a Queen's Award and the patent monetary value in the original in-sample set of patents.We find that QAI patents have a positive and significant higher economic value as compared to similar patents owned by the same set of innovators.Models 3 and 4 report the correlation between predicted breakthrough and the patent monetary value in our full-sample set of patents.We find a significant positive correlation between our estimated variable and patent monetary value in Model 3. When including forward citations, predicted breakthrough is still significant, albeit at the 10% level (Model 4).
Finally, Models 5 and 6 show the results of an estimation similar to the one presented in Model 1 and 2 but performed on the full sample of patents.In this case, the correlation between receiving a Queen's Award and the patent monetary sample is negative but insignificant.This is a surprising, but interesting result.The regressions of Models 5 and 6 suggest that when studied against the background of a very large set of patents, the QAI patents are probably not characterized by monetary values that set themselves out from the "average" patent.In other words, the extension of the sample from Models 1 and 2 to Models 5 and 6 brings in several additional patents of relatively high monetary value, so  Note: The results are derived using the result estimation of Model 6 in Table 3. 19 The distribution of the predicted probability on the full sample is very skewed (mean=0,09, median=0,06), thus we can reasonably expect to capture high quality patents setting p t = .19that QAI patent becomes insignificant.However, if we employ our predicted breakthrough indicator in the same large sample, we find still find some effect.All in all, we interpret the findings of Table 5 as further corroboration of our approach.

Impact on innovations in the same technology class
The evolutionary economics literature makes a fundamental distinction between the search and selection of new technological paradigms and the technological progress along a specific trajectory (Dosi, 1982).In this context, breakthrough innovations have the potential to shift the incumbent trajectory and stir the direction of technical progress accordingly.We now investigate whether this property characterizes the breakthroughs identified by our method.
Following previous studies, we consider patents as belonging to the same trajectory if they belong to the same technology class (Andersen, 1999).To be very precise about the scope of the effect of a breakthrough, we define the technological domain at the highest level of disaggregation using the WIPO International Patent Classification up to the subgroup level. 22As regarding the timing, we examine the effect of each predicted breakthrough patent on its technological domain for the ten years after their filing, as compared to the previous ten.Our dependent variable is the number of patents filed in a given year, the independent variable of interest post predicted breakthrough is a time-variant dummy that takes value 1 from the year in which our predicted breakthrough patent was filed.We estimate OLS regressions testing the effect of the filing of a predicted breakthrough patent on the number of subsequent patents filed controlling for the year trend, the year trend squared, and the IPC subgroup.To address possible issues of autocorrelation of the residuals, we cluster the standard errors by IPC subgroup (Cameron and Miller, 2015).Table 6 shows that the effect is positive and significant.
To check the robustness and sensitivity of our results, we run the same models in a "placebo" setting, anticipating the year of the occurrence of the predicted breakthrough patent (t) by five years (t-5).Consider for example an IPC subgroup in which the predicted Table 5. Patent value (Kogan et al., 2017), QAI patents and predicted breakthroughs.  . 22The WIPO Classification separates the whole body of technical knowledge using the following hierarchical levels: section, class, subclass, group and subgroup.For example, the group "H01S 3/00" corresponds to "Laser", the subgroup "H01S 3/14" corresponds to "Laser characterised by the material used as the active medium" (the complete classification is available at https://www.wipo.int/classifications/ipc/en/).In the dataset we use for this exercise we have 4,043 technical subgroups.
G. Capponi et al. breakthrough was filed in 2000, the placebo model would replace the shock year (i.e. when post predicted breakthrough becomes 1) with 1995 and test the effect of this fictitious shock on the subsequent 10 years.Models 5 and 6 show that when arbitrarily changing the shock year (and controlling for the time trend) the effect is no longer significant.The positive relation between the breakthrough arrival and the development of more inventions in the same technological areas suggests that our method correctly identifies breakthroughs.

The sources of breakthrough innovations
From an evolutionary perspective, breakthrough innovations emerge from a process of recombinant search and selection: technical novelty is associated with the ability to find new recombinations of prior knowledge which are then evaluated in the selection phase.This mechanism has been investigated along different dimensions to improve the understanding of the sources of breakthrough innovations.An important debate is whether independent inventors are more or less likely to introduce breakthrough innovations than corporate inventors.On the one hand, being industry "outsiders" with no preconceived notions, independent inventors can be better equipped to come up with novel ideas.On the other hand, financial constraints coupled with the absence of a corporate setting and structured intellectual support may lead to the generation of inventions of little relevance (Dahlin et al., 2004).Singh and Fleming (2010) focus on the "lone inventor" as a possible source of breakthroughs and argue that individuals are less likely than teams to generate breakthroughs because they cannot benefit from the help of collaborators in sorting and identifying promising new ideas.
Within our full sample, 93% of the patents are assigned to at least one organization. 23To analyze which conditions favor the emergence of breakthroughs, we define three mutually exclusive dummy variables capturing whether at least one assignee is a public organization (6% among universities, government and hospitals), at least one private company and no public organizations (87%) or only individual inventors (7%). 24 The analysis is carried out at the patent level and we estimate the probability of a patent to be a predicted breakthrough depending on the type of assignee and the number of inventors.To account for the prior experience and resources of the assignee(s), we also include the cumulative number of patents filed over the ten previous years and the cumulative number of patents filed over the ten previous years in the same technology class. 25Finally, we include year and technology dummies. 26 Table 7 presents the results.Looking at the independent variables of interest, M1 shows that the odds of patents developed by individual inventors and the public sector to be breakthroughs are 35% and 20% lower than the odds for corporate patents.This evidence is expected given that our method seeks to identify inventions that became commercially successful, corporations are generally better equipped in this sense.In line with Singh and Fleming (2010), Model 2 shows that the team indicator is positive and significant, however, it does not improve the chances of a patent to be a breakthrough in these cases (Models 3 and 4).Our findings seem to suggest that individual inventors (or "garage inventors") are less likely to contribute to the upper tail of the distribution regardless of whether they work in a team or not.Intriguingly, the presence of a team seems to be detrimental when public organizations are involved.Arguably, in the context of public organizations, single individuals can perform better than teams because they can benefit from a higher level of intellectual freedom than in a corporate setting. 27At the same time, they can rely on the organizational structure that "lone inventors" miss.This evidence suggests that universities and more in general public institutions can potentially create a productive research environment for single inventors, to the extent that they offer a combination of formal support and intellectual freedom.Finally, it is interesting to notice that organizations' experience over the past 10 years is negatively associated with the probability of a patent to be a breakthrough.This evidence supports the incumbent curse argument whereby incumbents would be less likely than new entrants to come up with breakthrough innovations.

Conclusion
In this paper, we have introduced a method to identify breakthrough innovations on a large scale starting from award-winning innovations.The output of our procedure is a dataset comprising 138,467 patents filed over 37 years, of which 17,176 are classified as breakthroughs.We then exploit this dataset to investigate the sources of breakthrough innovations depending on the type of assignee and the number of inventors.
Our proposed method emphasizes the link between patent data and  1998). 24These information are retrieved from Table TLS206_PERSON in PATSTAT. 25Note that 74% of the patents have only one assignee, whenever the number is higher we take the average of the assignees' experience. 26Following the set up proposed by Singh and Fleming (2010), we also tested models controlling for the number of claims made by a patent, the number of backward citations and the number of non-patent references.The results (not shown) remain broadly consistent. 27In line with Singh and Fleming (2010), we find that working in teams further enhance the probability of a patent to be a breakthrough when the assignee is a private company.
innovations with demonstrated technical and commercial success.Through external validation of patent value, we seek to advance research in this field by addressing the often missing connection between measures of patent value and the actual use and commercial success of such patents.It is also worth stressing that future contributions in this direction could replicate, adapt or extend our approach taking as inputs other prize schemes or different evidence of innovations' economic and technical performance. 28This would in turn increase the stock of available data and our understanding of fundamental issues on the origin, characteristics and impact of breakthrough innovations.With this goal in mind, we also encourage the use of the dataset constructed in this paper and the further evaluation of potential differencesboth conceptual and empirical -between the breakthroughs identified with this procedure and those classified by other   This paper suffers from some limitations.First, while our procedure is flexible and can be applied in different contexts, it is quite laborintensive.The search for an external measure of patent quality remains a difficult task and the association with patent data can be timeconsuming.Also, the adoption of a binary classification oversimplifies the issue of patent value estimation, leading to an extreme classification whereby a patent is either valuable or not.Another word of caution stems from the fact that while we provide validation of economic and technical significance ex-ante, we cannot quantify nor disentangle these two aspects at the patent level.Finally, this study suffers from the common pitfall characterizing research that uses archival patent data, above all the fact that many important inventions are never patented.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.G. Capponi et al. patent characterizing the past few decades, which is extensively covered in the literature.In contrast, the dotted path shows that breakthrough patents emerged mainly in the 1980s and the 2000s.While they present an increasing trend as well, they show a tendency to cluster in time.
Focusing on breakthrough innovations alone, Fig. C.2 shows the absolute patent frequency by sectors over time. 29Patents in electrical engineering are by far the most frequent from 2000 to 2010.At a more disaggregated level, we observe that the sub-classes of computer technology, telecommunications and digital innovation prevail.In the most recent years, we find a relatively high share of breakthroughs in instruments (mainly measurement and optics).Also, a growing number of breakthrough patents are filed in what Schmoch (2009) categorized as "other fields", driven chiefly by innovations in civil engineering.The least represented classes are mechanical engineering, which in the graph is relatively more present in the 1980s, and chemistry.Interestingly, the patents that we do not classify as breakthroughs do not show the same tendency to cluster in time.
Table C.1 reports the scores on three widely adopted concentration indexes calculated on 35 technology fields. 30Breakthrough patents exhibit a higher level of sectoral concentration than the rest of the sample, with the top three fields covering 41% of the patents.
Finally, we classify the innovations following the Pavitt Taxonomy, a classification that describes the behavior of innovating firms depending on their technological competencies (Pavitt, 1984). 31The original taxonomy presented four main categories: supplier dominated firms which operate in traditional industries (ex.textile), specialized suppliers of equipment who work in close collaboration with their customers, science-based firms whose main source of knowledge is in-house R&D, and scale intensive firms operating in mass production industries (Archibugi, 2001).It is worth highlighting that Pavitt's seminal work is based on evidence emerging from the SPRU innovation database, which also includes innovations winning a Queen's Award.Hence, it seems to be particularly appropriate to analyze our sample using the Pavitt Taxonomy.Between 1976 and 1994, breakthrough innovations are relatively more frequent in specialized suppliers sectors.Pavitt recognizes core specialized suppliers in sectors like machinery and instruments, where innovation is driven by performance-sensitive clients valuing product design (Pavitt, 1984).This evidence is consistent with the significant proportion of successful innovative activities in the complex product systems (CoPS) branches of UK manufacturing during the 1980s and early 1990s (Hobday and Rush, 1999).
In the second period, between 1995 and 2013, we notice the role of science-based sectors as generators of breakthroughs.At a more disaggregated level, it emerges that breakthroughs are driven by innovative activities in the field of electronic engineering while the rest of the sample presents a higher share of patents in chemistry.Arguably, this trend captures the heavy digitalization process we witnessed in the past two decades, which shifted the locus of breakthrough innovations consolidating information and communications technology (ICT) as a new techno-economic paradigm (Freeman and Louçã, 2001).
Of course, this is just a set of exploratory stylized facts emerging from our new dataset of UK breakthrough innovations that could be further analyzed in future work.

Appendix D. Methodology steps
See Table D.1

G
.Capponi et al.   new methods (see for example Funk and Owen-Smith 2017).

Fig. C. 1 .
Fig. C.1.Frequency of breakthroughs and other patents over time.
Fig. C.3 shows the patents' distribution across the five classes, comparing the composition of breakthroughs and other patents over two time periods.

Table 1
Patents matched with QAI winning innovations by filing authority.

Table 2 .
Number of observations by sample type.

Table 4 .
Prevalence of false positives and false negatives at selected p t using the ROC and DCA approach.
OLS regressions with robust standard errors.In-sample refers to the USPTO patents filed by QAI winners (i.e. it includes the QAI patents and the controls).Full sample refers to all the patents in the in-sample and the patents in the out-of-sample (i.e.universe of USPTO patents filed by GB applicants).Technology dummies are defined at the IPC-class level.Legend: *** (**, *) indicate a significance level of 0.1% (1%, 5%).

Table 7 .
The source of breakthrough innovations.

Table B . 1 .
Overview of the variables used in Table3.
28We provide a detailed and generalized sequence of the steps we followed in Table D.1 Appendix D.

Table C . 1 .
Patents' concentration in technology fields.