An embarrassment of riches

Open innovation systems have provided organizations, ranging from businesses and governments to universities and NGOs, with unprecedented access to the "wisdom of the crowd", allowing them to collect candidate solutions, for problems they care about, from potentially thousands of individuals, at very low cost. These systems, however, face important open challenges deriving, ironically, from their very success: they can elicit such huge levels of participation that it becomes very difficult to guide the crowd in productive ways, and pick out the best of what they have created. This viewpoint article reviews the key challenges facing open innovation systems and issues a call-to-arms describing how the research community can move forward on this important topic.


Introduction
Open innovation systems represent an emerging collective intelligence success story. In such systems, a customer describes a problem they want to solve (e.g. "we want ideas for new beverage products") and provides an online tool that allows the crowd to submit proposed solutions, as well as rate (and sometimes critique) other people's proposed solutions. Many open innovation platforms have emerged (e.g. ideascale, spigit, imaginatik) and have been used widely in contexts that range from IBM to Starbucks, from the Danish central government to the White House. A recent survey (Thompson 2013) found that 1 in 4 companies plan to utilize open innovation systems within the next 12 months, and this figure is growing. Such systems have proven, time and again, that they can elicit substantive contributions at a very large scale and very low cost. In the early weeks of his first term, for example, President Obama's asked citizens to submit and vote on questions on his web site change.gov, and promised to answer the top 5 questions in each category in a major press conference. This initiative engaged over 100,000 contributors, who submitted over 70,000 questions and 4 million votes. Google's 10 to the 100th project received over 150,000 suggestions on how to channel Google's charitable contributions. In IBM's Idea Jam in 2006, 46,000 ideas for possible IBM products and services were generated by 150,000 contributors. Such large-scale participation enables, in turn, such powerful emergent phenomena as: • The long tail: crowds can generate a much greater diversity of ideas, including potentially groundbreaking "out of the box" contributions, than we could easily access otherwise.
• Idea synergy: crowds can rapidly develop huge volumes of novel ideas by re-combining and refining the ideas proposed by other participants. • Many eyes: crowd participants can check and correct each other's contributions, enabling remarkably high quality results very inexpensively. • Wisdom of the crowd: crowds can collectively make better judgments than the individuals that make them up, often exceeding the performance of experts Open innovation systems face, however, serious challenges that, paradoxically, are largely a result of how successful they have been at eliciting huge volumes of participation. In this article, we review these challenges and propose some promising directions for moving forward.

Challenges
The challenges faced by open innovation systems occur both with idea generation and idea evaluation. Key challenges with idea generation include: • Harvesting costs: open innovation engagements tend to generate idea corpuses that are large, disorganized, and highly redundant. Pruning such a list to find the best ideas can be a massive undertaking. Google's 10 to the 100th project, for example, had to engage 3,000 employees to prune the 150,000 ideas they received, putting them 9 months behind schedule. IBM flew 100 senior executives into New York from around the world to prune the results of their Idea Jam. • Unsystematic coverage: open innovation systems have no inherent mechanism for ensuring that the ideas submitted comprehensively cover the most critical facets of the problem at hand, so the coverage is hit-or-miss and may not align with the customer's needs.
• Shallowness: open innovation systems tend to generate large numbers of relatively shallow ideas. A major reason for this, we believe, is that collaborative idea development, and accurate credit assignment, is typically not well-supported in current tools.
Open innovation systems also face challenges with crowd-sourced idea evaluation: there is often a disconnect between what the customer wants and what the crowd selects. This can occur for several reasons: • Shallow evaluations: Little support is provided for the crowd building upon each other's evaluative expertise, since users usually do not examine and correct each other's facts and reasoning. • Rating lock-in: When there are thousands of ideas, many potentially valuable ideas may not end up being evaluated in sufficient depth, and the system can quickly "lock" into a fairly static, and arbitrary, ranking, where the winning ideas are inferior to others in the list.
Open innovation systems thus face critical challenges in terms of ensuring that the potentially massive contributions of the crowd provide high value to the customer without incurring prohibitive harvesting costs.

Promising Directions
How can we meet these challenges and more fully achieve the promise of open innovation systems? Progress will require, we believe, advances on the following two key fronts:

Better open innovation processes
New open innovation processes are needed that provide more guidance about how the crowd can best contribute, help crowd members build on each other's inputs, and make it easier to harvest their contributions, • collaborative idea definition: helping the crowd make more deeply-considered contributions will require progress on incentive schemes and collaborative authoring structures. Participants, for example, can be asked to structure their contributions as deliberation maps (Klein 2008), i.e. as trees made up of problems to solve, potential solutions for these problems, and the arguments for and against each potential solution, all single-authored. Participants can then compose proposals from the best solution ideas in each map. Credit assignment becomes straightforward because each proposal is built from components with clear authorship. • novel rating mechanisms can help ensure that the crowd evaluates ideas quickly and accurately with respect to the criteria the customer cares about. One possibility, for example, is to use a kind of prediction market where participants are given a limited budget to buy and sell stocks, each representing a different idea, and get a payoff when ideas they "bought" are successfully implemented by the customer. This gives users incentives to evaluate ideas carefully from the customer's perspective. Deliberation maps can also help here, by allowing crowds to check and build upon each other's reasoning by creating chains of supporting and rebutting arguments. • creativity enhancement techniques (which have been developed, to date, almost exclusively for face-to-face team settings) can be used offline to feed open innovation engagements as well as adapted to crowd-scale online contexts.

• interleaving ideation and evaluation across multiple open innovation rounds. The crowd
can be asked to create new ideas built upon those that survived the previous round of selection, so idea generation is more likely to focus on what the customer wants.
Deeper computer support Crowds (of people) and clouds (computers) have synergistic capabilities. Crowds are able to create, understand and evaluate ideas in ways that computers cannot match, but are best suited for performing relatively small and quick tasks that require little context. Computers, by contrast, excel at rapid analysis of large swaths of data to get "the big picture" of what is (and is not) happening in a crowd. Combining these strengths will require bridging the semantic gap between the natural language that crowds use, and the formal languages that computers require. Someday, this will be achieved by advanced algorithms that allow computers to deeply understand natural language. But this achievement seems to remain far off. In the meantime our goal, we believe, must be to find ways that crowds can do the minimum formalization needed to enable significant computer support, for example by: • semantic tagging: crowds can annotate natural language idea corpuses with semantic cues (e.g. idea and argument boundaries, topic keywords). This can be a mixed initiative process, wherein computers propose possible tags that are corrected by crowd members, and where machine learning can be used to improve the computer algorithms over time based on this human feedback. • design tools can allow users to express their ideas as semi-formal models built on domain-specific primitives, rather than just as natural language text. Design tools aimed at the masses (e.g. google sketchup) are already becoming ubiquitous, but have yet to be incorporated into open innovation platforms.
As the semantic gap between crowds and clouds narrows, we can create powerful new forms of computer support for open innovation, such as: • analysis tools that take advantage of semi-formal idea representations to help evaluate the strengths and weaknesses of contributed ideas • semantic compression algorithms that remove duplicates, and cluster related ideas, in order to compress idea corpuses • visualization tools that summarize what the crowd has done so far, so customers can determine what the gaps/promising areas are and use this information to guide future crowd contributions, for example via focused incentives.

A Call to Arms
Open innovation systems, as we have seen, have the potential to harness the collective intelligence of the crowd for problem solving in areas ranging from business to government, from science to education. This potential is far from fully realized, however, largely because of our inability to deal effectively with the massive levels of user contributions that these systems can elicit. Advances in this area will require contributions from many disciplines, including computer science, cognitive science, social psychology, computational linguistics, and economics. Will you join us in addressing these important challenges?