Improving defense innovation programs to enhance force readiness

Purpose – The purpose of this study is to evaluate Department of Defense (DoD)-backed innovation programs asa meansof enhancing theadoption ofnewtechnology throughout thearmed forces. Design/methodology/approach – The distribution of 1.29 million defense contract awards over seven years was analyzed across a data set of more than 8,000 DoD-backed innovation program award recipients. Surveys and interviews of key stakeholder groups were conducted to contextualize the quantitative results andgarneradditionalinsights. Findings – Nearly half of DoD innovation program participants achieve no meaningful growth in direct defense business after program completion, and most small, innovative companies that win follow-on defense contracts solely support their initial sponsor branch. Causes for these program failures include the fact that programs do not market participants ’ capabilities to the defense community and do not track participant companiesafter programcompletion. Practical implications – Because the DoD does not market the capabilities of its innovation program participants internally, prospective DoD customers conduct redundant market research or fail to modernize. Program participants become increasingly unwilling to invest in the DoD market long term after the programs fail to deliver theirexpected bene ﬁ ts. Originality/value – Limited scholarship evaluates the ef ﬁ cacy of DoD-backed innovation programs as a means of enhancing force readiness. This research not only uses a vast data set to demonstrate the failures of these programs but also presents concrete recommendations for improving them – including establishing an “ Innovators Database ” to track program participants and an incentive to encourage contracting entities and contractors to engage with them.


Introduction
The USA emerged from the Second World War as the world's leading economic, political and technological superpower, and this position remained largely uncontested for the duration of the twentieth century. Post 9/11, however, the global landscape began to shift. Intensified competition with rival powers including Russia and China, the advent of information warfare, a proliferation of threats, and the Global War on Terror and conflicts in the Middle East now pose acute challenges for both American hegemony and national security. Additionally, while in decades past Department of Defense (DoD) research often produced revolutionary technological breakthroughs for the civilian sector, commercial innovation now increasingly outpaces the DoD. The Department must therefore attempt to modernize as a whole, while it simultaneously competes to identify and integrate the most cutting-edge technological innovation. Furthermore, as elucidated in the 2018 National Defense Strategy, "Success no longer goes to the country that develops a new technology first, but rather to the one that better integrates it and adapts its way of fighting" (Department of Defense, 2018). To maintain a strategic overmatch demands mission planning and execution across the forces and unprecedented levels of Department-wide cooperation and communication. Just as innovation can serve as a force multiplier, it can also severely degrade military productivity and lethality if it is siloed within a single service branch or command structure. The DoD therefore faces a dual challenge today: it must not only identify and attract innovative and modernizing solutions providers but also integrate revolutions in military technology across the forces as quickly and seamlessly as possible.
The DoD has widely acknowledged that the prowess and success of America's armed services demand ongoing, supported collaboration with private sector innovators, and increasingly so in light of twenty-first-century military and national security challenges. For more than 60 years, DoD-backed innovation programs have played an outsized role in the narrative surrounding military innovation. They will consume almost $14bn of taxpayer dollars in FY19. They also enable thousands of disruptive technologies to enter the highly adaptive, risk-averse DoD ecosystem every year and produce a network of companies with a rare combination of innovative prowess and DoD past performance (Office of the Undersecretary of Defense, 2018). Given that a competitive advantage in today's mission environment demands rapid, force-wide integration of innovative technologies, it is essential that these programs also connect the capabilities that they foster to as many prospective DoD customers as possible. However, while these programs implicitly and explicitly market themselves as "points of entry" for small, innovative companies into the overall DoD market, limited scholarship exists to evaluate how participant companies perform in the defense sector in the years following program completion and if their capabilities are leveraged by the military at large. This research seeks to fill this gap by examining if and how DoDbacked innovation programs enhance the integration of new technologies.

Department of Defense innovation programs
To evaluate DoD-backed innovation programs as a means of enhancing the adoption of new technology across the forces, it is important to first understand their history, how the largest are structured and where they fit within the broader landscape of DoD innovation. The history of America's DoD-backed innovation programs begins in 1957, when the Soviet Union's surprise launch of Sputnik left the American people and its leaders fearful that the USA had lost its technological edge. President Eisenhower responded by creating the "Advanced Research Projects Agency" (later renamed the Defense Advanced Research Projects Agency or "DARPA") the following year. According to the DARPA website, it has a "singular and enduring mission: to make pivotal investments in breakthrough technologies Defense innovation programs for national security." Today, DARPA runs over 250 research and development (R&D) programs designed to further its core mission (DARPA, 2015). DARPA is unique in the DoD innovation community as it exists as a stand-alone research agency that funds innovative research within industries and also conducts its own research. DARPA uses contracting authorities to solicit solutions directly and participates in the Small Business Innovation Research (SBIR) program and the Small Business Technology Transfer (STTR) program to fund research conducted by small businesses.
The Small Business Administration (SBA) launched the SBIR program in 1977 to "support innovation through the investment of federal research funds in critical American priorities to build a strong national economy" (SBIR, 2011). It established its identically structured sister program, the STTR program, in 1992, designed to help further scientific research with potential for broader commercial benefit. SBIR/STTR programs run in three phases, all of which are restricted to small businesses of 500 employees or fewer. In Phase I, companies establish the technical merit, feasibility and commercial potential of their project over the course of one year in exchange for $150,000 in federal funding. In Phase II, participant companies continue to grow and develop their technology with additional funding not to exceed $1m over a two-year period. A product with "commercial potential" may then proceed to Phase III. SBIR/STTR does not fund Phase III directly, but many federal agencies with SBIR/STTR authority provide follow-on contracts or funding to support this additional development. Phase III is intended to be the primary means of transitioning new technologies into the broader service branches or agencies that need them. The armed services and defense agencies each run their own SBIR/STTR programs and define the topics for which they are seeking small business applicants.
Based on the success of SBIR/STTR and the need to accelerate the fielding of innovative technology, the DoD created the Rapid Innovation Fund (RIF) in 2011, marketed on the DoD's Defense Innovation Marketplace site as "a collaborative vehicle for small businesses to provide the Department with innovative technologies that can be rapidly inserted into acquisition programs that meet specific defense needs." Much like SBIR/STTR, the armed services and various defense agencies run their own RIF programs and define their own project specifications. However, unlike SBIR/STTR, and despite its mission statement, RIF permits companies of any size to participate. The RIF process starts with a Broad Agency Announcement and the request for companies to submit a white paper. The sponsoring agency reviews all submissions and invites shortlisted companies to submit a full-scale proposal. The company with the most competitive proposal receives the RIF award. One of the primary objectives of RIF is to better and more rapidly connect research and technologies developed in the SBIR/STTR program to a wider audience within the DoD. As such, from 2011 to 2015, RIF awarded more than half of its $1.4bn in contract awards to companies that had previously participated in SBIR/STTR (Bujewski and Purdy, 2017).
In 2015, Secretary of Defense Ashton Carter established Defense Innovation Unit Experimental (DIUx) in support of the Third Offset Strategy initiated by Chuck Hagel in 2014. DIUx, like its predecessors, is designed to fund innovative companies with the purpose of solving national defense problems. It determines project specifications and areas of interest in concert with DoD entities. Companies of any size can respond to a DIUx solicitation by submitting a solution brief. According to DIUx's website, briefs are typically evaluated within 30 days, and shortlisted companies are then invited to submit a full-scale proposal and begin negotiations for a pilot contract. Unlike the other DoD innovation programs, however, DIUx uses "Other Transaction Authority" (OTA), a contract vehicle that streamlines the funding process and, according to the DIUx website, enables them to fund projects in 90 days or less. According to US Code 10 2371b, OTA funds include a designation for prototype projects deemed critical to enhancing the mission effectiveness of the military or to improve systems already in use by the armed forces. The armed services may contract up to $250m annually for projects that meet OTA criteria, provided the primary awardee for a given project represents a "nontraditional defense contractor" or a small business. As such, OTA also affords DIUx greater latitude in allocating funding, making it more agile and more appealing to nontraditional solutions providers.
In addition to these 5 DoD-backed innovation programs, the DoD has approximately 50 additional external funding programs, 20 of which are designed to rapidly integrate technology (Small Business Technology Council, 2014). The DoD also continuously stands up new DoD-backed innovation programs and uses OTA funds independently to support internal innovation efforts. Since January 2015 alone, the DoD has created DIUx, the Warfighting Lab Incentive Fund, AFWERX (an Air Force innovation initiative), SOFWERX (SOCOM's innovation imitative), etc. (Gibbons-Neff, 2016;West, 2018;Kaplan, 2015). This process compounds across the DoD and innovative companies interested in pursuing the defense marketand the service branches and Departments that seek to collaborate with themface a complex web of prospective routes and access points, as shown in Figure 1.
This confusing and hyper-bureaucratic system also elucidates the Department's riskaverse culture, which is seen as one of the most acute obstacles thwarting DoD innovation efforts. Due in part to the consequences of perceived failure, decision-makers are disincentivized to experiment, collaborate and take risks. This culture has resulted in countless "stove-piped" initiatives that are developed without cross-communication and operate in parallel. Given the Department's ongoing struggle to modernize and remain innovative, it should invest in building robust, long-term relationships with the innovative companies that do manage to navigate this web successfully and achieve proven DoD past performance. Consequently, rather than evaluating the effectiveness of innovation programs as a means of addressing singular, branch-specific requirements, this research sought to evaluate if and how the largest and most prominent DoD-backed innovation programs improve the rate of adoption of innovative technology force-wide.

Program participants
To determine if the companies fostered in DoD-backed innovation programs achieved forcewide adoption, company performance in the defense sector after program completion was evaluated first. Although there are no data available to determine the results of these programs in general, one indicator of a company's DoD performance is the number of defense contracts that it wins. Thus, the number of defense contracts won by DoD-backed innovation program participants in the years following program participation was examined. Quantitative research was focused on SBIR/STTR and RIF for several reasons: they are the largest of the DoD-backed innovation programs; they publish complete lists of their program participants, which made it possible to gather a significant data set; and all branches of the armed forces and all defense agencies participate in both SBIR/STTR and RIF, making the data set not only large but also comprehensive.
First, the publicly available lists of RIF project award recipients between 2011 and 2015 (103) and DoD-sponsored SBIR/STTR award recipients between 2013 and2016 (8,158) from the SBA database were combined. Both RIF and SBIR/STTR are rife with serial users, which means that there were dramatically fewer unique companies in this ecosystem relative to total awards. As such, the 8,261 total awards were adjusted to control for repeat usage and isolate unique companies, which resulted in a data set of "program participants" that contained 1,140 companies ( Figure 2). To understand how these 1,140 companies performed in the defense sector subsequent to their program participation, more than seven years' worth of publicly available defense contract award data were scraped and filtered from FBO.gov from January 1, 2011, to January 15, 2018. This timeframe was selected to enable analyses of the most recent complete data sets, as at the time of data compilation, RIF awards were only publicly available from program inception from 2011 through 2015. SBIR awards are all publicly available, but the most recent complete data are from 2013 to 2016. Because FBO.gov only publishes unclassified prime contract awards, the data did not include classified contracts or information about companies' performance as a subcontractor or teaming partner on DoD contracts. Nevertheless, the set contained more than 1.29 million defense contract awards, which were then filtered to isolate the defense contracts specifically awarded to the 1,140 companies in the Program Participant data set: a total of 13,449 defense contracts (Figure 3).

Supplier retention
The distribution of these 13,449 contracts across the 1,140 Program Participant companies in the data set produced striking results, as shown in Figure 4.
A startling 26 per cent of the companies in the vast data setin other words, approximately 296 program participantswon zero defense contracts after completing their DoD-backed   Defense innovation programs innovation programs. Another 22 per cent of companies in the set (approximately 250 Program Participant companies) won only one single defense contract following their RIF/SBIR/STTR award. In other words, nearly half of the program graduates (48 per cent) received fewer than 2 per cent of the total 13,449 defense contract awards.
The format of publicly available data limited the ability to control for specific timeframes, as contract award data were fixed from January 1, 2011, to January 15, 2018 (per the parameters of the "scraper" used). The entire data set of participant companies over this period of time was assessed; thus, a company that completed SBIR/STTR or RIF in 2011 was evaluated over seven years, while a company that graduated in 2016 was evaluated over two years. However, all companies in the data set had at least 13 months to win defense contracts. Given the size of the overall data sets and the overwhelming share of companies that failed to win DoD contracts following program participation, it is unlikely that the timeframe limitation significantly affected the results. Additionally, small, innovative companies often lack the capacity and infrastructure to bid/perform as prime contractors and instead subcontract or team with large strategic integrators to expand their defense business. Unfortunately, innovation hub participants' performance as subcontractors or teaming partners could not be examined because no large, publicly available data sets exist containing that information. Presumably, a portion of program participants that won few or no defense contracts as a prime did achieve some growth in their defense business indirectly. However, the significant and stark results of the data incontrovertibly reveal a trend and thus remain deeply troubling. A significant share of participant companies went through the involved and drawn-out process of applying for an innovation hub, proved their relevance to a branch-specific requirement and received an award, and expended significant resources to develop the capabilities requested by their branch sponsor; and these efforts failed to bridge into meaningful growth in their direct defense business. The implications are perhaps most concerning with respect to force readiness. The capabilities of these approximately 547 companies were honed by initial branch sponsors based on critical requirements for which private sector collaboration was deemed essential. For almost 300 of these companies, their capabilities were never procured directly by other defense customers, neither from within the sponsor branch nor from other branches, and the other aproximately200 companies faced essentially the same fate.
The distribution of the 13,449 contracts across the 1,140 companies also made it clear that a small subset of companies won a disproportionate share of the remaining 98 per cent of contracts in the data set. While RIF describes itself as a "vehicle for small businesses," the participant data data set includes names such as 3M, BAE Systems, General Dynamics, Raytheon and other experienced contractors. To better understand the effects of these outliers on the data, companies from the participant data data set that had won 50þ DoD contracts from January 1, 2011, to January 5, 2018, were isolated. In total, 40 companies in the data set fit those criteria.
These 40 companies, or approximately 3.5 per cent of the total program participant data set, collectively received a staggering 80 per cent of the defense contract awards in the overall defense contract award data set, as depicted in Figure 5.
Removing these outlier companies from the data set, the percentage of the remaining 1,100 companies that won zero defense contracts after their SBIR/STTR or RIF participation also jumped by 10 per cent, from 26 to 36 per cent.
For the companies that stand to gain the most benefit from these programs, the realitythat the programs contributed to no meaningful growth in participants' direct defense businessis even more acute. Furthermore, these programs are designed to provide the forces with greater access to emerging technologies, whereas our forces are otherwise limited to the technologies that the established DoD contractors present to them. The data suggest that these innovation hub programs have in fact become yet another channel for legacy contractors to gain DoD market share.

Customer diversity
While the initial analyses demonstrated that innovation hubs fail to convert a large portion of participants into viable DoD suppliers, it was also important to assess, for emerging technology companies that did manage to survive the transition from DoD-backed innovation program into the defense market, how broadly their capabilities were integrated across the forces. To do so, participants that won contracts with customers other than their initial sponsor were examined. Limitations to the functionality of the data analytics tools meant that customer diversity for the entire data set could not be measured, so a subset of the participant data was created to include companies that won between 2 and 12 defense contracts subsequent to program completion: 360 companies, or 32 per cent of the participant data set. As 48 per cent of participants won zero or one defense contract after program completion and 3.5 per cent of participants accounted for a startling 80 per cent of all defense contract awards, the data left a narrow subset of companies that fell somewhere in the "middle"i.e. non-entrenched contractors that had successfully bridged the gap from program participation into a healthy, modest defense contracting business. By focusing on companies that won between 2 and 12 contracts, customer diversity could be tested using a sizable data set from this narrow middle.
The data indicated that for a striking 76 per cent of companies in this subset, all of their defense contract awards came from their initial sponsor branch.
In other words, 76 per cent of these companies achieved no customer diversity, suggesting that even when the DoD manages to retain a supplier honed in an innovation hub, it largely fails to integrate the company's capabilities across the forces.

Defense Innovation Unit Experimental: micro-analysis
DIUx has garnered widespread attention as an especially promising approach to the DoD's innovation problem (Kaplan, 2016;Williams, 2017;Maucione, 2017). Its funding is slated to Defense innovation programs increase by more than 70 per cent in fiscal year 2019, and Secretary Mattis recently lauded it, saying, "There is no doubt in my mind that DIUx will not only continue to exist, it will actually[. . .]grow in its influence and its impact on the Department of Defense." (Lanier-Chappellet, 2017). Although DIUx data was too limited to include in the broad quantitative analyses and despite the more acute impact of limitations on a small sample size, it is nevertheless important to determine if DIUx retains and disseminates its participants' capabilities more effectively than its predecessors.
A "DIUx participants" data set was created using the company names marketed on the DIUx website as of Q4 2017 (31 companies). Next the DoD contract data were filtered to isolate DoD contracts awarded to these 31 companies from January 1, 2015 (the year DIUx was founded), through January 15, 2018: a total of 440 defense contracts.
As demonstrated in Figure 6, entrenched government contractors have utilized DIUx as well, which profoundly affected the results. Specifically, Rockwell Collins, a legacy DoD contractor, won 408 of the 440 total defense contract awards from the FBO award data set. The distribution of the remaining 32 contracts across the 30 other DIUx participants in the data set reflected the same troubling trends identified in the SBIR/STTR/RIF analyses. Four DIUx participants won 3-6 DoD contract awards after program completion, and the remaining 26 companiesor 81 per cent of DIUx participants in the setwon zero or one DoD contract.
Despite its laudatory attention, DIUx, like its predecessors, does not appear to position private sector innovators for long-term success in the defense sector. Rather, DIUx engages with participants solely to address singular, branch-specific projects and does not disseminate supported capabilities to other prospective DoD customers thereafter.

Survey methods
While the quantitative analysis indicates that DoD-backed innovation programs are not effective distribution channels for small, innovative companies into the broader defense market, surveys and interviews with individuals from three key stakeholder groups were conducted to better understand potential causes of these program failures (Figure 7).

Survey and interview feedback
The anecdotal evidence consistently pointed to the same programmatic failures identified in the quantitative analyses and, most significantly, helped elucidate the reasons for these problems. The key issues revealed by the survey and interview data include the following: DoD-backed innovation programs do not educate participants on how to succeed in the broader defense sector. The majority of participants surveyed noted that their innovation program offered no instructions/education for how to identify or bid on government contracts after program completion and indicated that the program did not prepare them to support a broad base of DoD customers. Likewise, when program managers were asked how their program helps a participating company identify other relevant DoD requirements, it was clear that no formal, institutionalized process exists. For instance, one program manager explained, "We naturally become aware of [relevant opportunities] from our contractors, conferences, even our customers sometimes," and another stated that they rely on "collaborative conversations" to find additional DoD opportunities for participants with promising capabilities. Similarly, when asked how they would improve the DoD innovation program, one respondent recommended, "Provide some sort of bridge to help small business survive the gap between the end of the 'innovation contract' and the follow on sustainment work." Given that the innovation programs do not educate participants on how to find and bid on government contracts, it is not surprising that, as indicated by the quantitative data, nearly half of program participants win few if any defense contracts: DoD-backed innovation programs do not market participants' capabilities to the broader armed services community.
The quantitative results are further explained by the fact that the innovation programs do little if any marketing of participants' capabilities to the broader DoD community. Simply put, a customer cannot buy something it does not know exists. More than half of the program managers that provided feedback indicated that their programs have no formal process for circulating information about participants' capabilities to the broader armed services community. One of the most frequent comments from members of the DoD community was that they receive very few briefings on the projects their own branch funds and almost never receive information on the capabilities of companies funded by other branches. When members of the DoD do learn about the capabilities of participant companies, it is not because the programs are marketing them effectively. According to those surveyed, at best, they might hear about a company through a random, one-off initiative like a "quarterly update" mailer that references an innovative technology, or "infrequent[ly] hosted industry days." Usually, they learn about the technology through a chance meeting with a colleague who is familiar with it: DoD-backed innovation programs do not track the performance of participant companies in the years following program completion.
The qualitative research exposed the troubling fact that there is no centralized database containing information about companies that have participated in a DoD-backed innovation program; and DoD-backed innovation programs do not follow a systematic, consistent Survey and interview data Defense innovation programs process for tracking the performance of participant companies after program completion. Many DoD-backed innovation programs do not track program participants at all; others may track some companies inconsistently or on an ad hoc basis. Without tracking participants, DoD innovation programs cannot discern how these companies fare in the defense sector or at large. They do not see changes in a company's performance year to year; they do not receive updates on a company's capabilities developments; and if a company changes its name, changes its leadership or moves its headquarters, that information is not recorded in a central database. Perhaps most concerning, they do not monitor the long-term effectiveness of significant investments of public funds into DoD-backed innovation programs. The absence of a formal process for tracking the long-term performance of participants is the equivalent of a venture capital firm not tracking the performance of its portfolio companies. It elucidates these programs' egregious supplier retention problems and signals that these programs are not concerned with the long-term success of the companies that they fund: A company's failure to win DoD contracts after program participation does not necessarily correlate to a lack of demand for that company's capabilities.
Survey and interview feedback also demonstrated that while in some cases innovation program participants may not achieve widespread adoption across the DoD due to the specificity of a sponsor project, a company's failure to win DoD contracts after program participation does not necessarily correlate to a lack of demand. For example, the chief executive officer (CEO) of Monterey Technologies Incorporated (MTI), a company that develops mission planning software systems and has been an active defense contractor since 1984, provided valuable feedback. MTI received one Navy-sponsored RIF award in 2012 and six Navy-sponsored SBIR awards between 2013 and 2016, and between January 1, 2011, and January 15, 2018, MTI won three defense contracts, all from Navy customers. While MTI has only won Navy contracts, there have been 144 DoD contract awards for "mission planning services" over the past five years alone, signaling Department-wide demand for their solution. The CEO shared that recently, an Army Airborne officer serendipitously learned about MTI through a Navy contact. The Army had been handling mission planning by hand, and MTI had the potential to address this ongoing problem. He called MTI's CEO directly, and as a result of the ad hoc outreach, they are now collaborating on a pilot project. This example demonstrates not only bona fide demand for MTI but also the inadequacy of current communication channels across the services. The experience of another company in the participant data set, "Enomalies," tells a similar story. In 2016, Enomalies participated in Phase II of an SBIR project and, like many DoD-backed innovation program participants, has won no subsequent defense contracts. Enomalies specializes in advanced imaging and field-ready prototyping, and the Navy sponsored its SBIR project to further develop a Rapid Synthetic Environment Tool (SBIR. gov, 2014). The tool scans an area and quickly creates a three-dimensional (3D) model that strategists and planners can walk through and interact with. Enomalies's tools also have the ability to rapidly scan objects and print prototypes on 3D printers (Enomalies, 2017). Since 2016 alone, there have been 26 defense contract awards for 3D printing systems and 4 for 3D scanning services. According to their website, Enomalies supports a broad range of commercial customers and appears to remain a viable company. Presumably, then, it is neither a lack of demand nor an inability to perform that has kept Enomalies from winning a defense contract since its SBIR award. Instead, it suggests that DoD innovation programs fail to adequately connect their participants to prospective DoD customers.

Further analysis and recommendations
The fact that DoD-backed innovation programs fail to market their participants to prospective DoD customers has myriad consequences. It results in lost revenue for the innovative companies who, as the data illustrated, do not become robust DoD suppliers. Perhaps more concerning are the consequences for the strength and readiness of our forces at large. If the armed services are not made aware of the capabilities supported in DoD-backed innovation programs, they cannot adopt theminstead, they either do not modernize or conduct redundant market research. Furthermore, these programs introduce private sector innovators into the defense ecosystem, provide them with past performance and then fail to nurture them as long-term suppliers. These results are costly and damaging to the reputation of the Department: innovative suppliers undertake the complex, expensive and time-consuming process of participating in a DoD-backed innovation program only to find that it does not contribute to meaningful growth in their defense business. They are expected to undertake full lifecycle sales processes for each individual DoD customer, in contrast to how large commercial customers operate (where new vendors typically undergo a vetting period and if successful, their product or service is distributed across the organization's broader portfolio). If these problems are not addressed, innovative companies will increasingly forgo public sector opportunities altogether, and innovation programs will fail to attract top innovators and fail to realize their full potential as force multipliers. However, with the proper reforms, DoD-backed innovation programs do have the potential to drive vast improvements in the readiness of our armed forces. As stewards of significant tax dollars, the defense sector should focus on maximizing their investments in innovation and R&D over the long term. To do so first and foremost requires that DoD-backed innovation programs maintain consistent, clean and accessible records about its projects and program participants: First and foremost, the DoD must create an "innovators database"a central, searchable database containing information about all DoD-backed innovation program participants.
The innovators database would be populated by innovation hub program managers and the companies themselves and would contain company basics, information about the company's capabilities and details about the projects that the company has supported for both government and commercial customers. This platform would enable DoD-backed innovation programs to easily share information with stakeholders across the armed forces on the capabilities of their participant companies. These stakeholders could access the database directly and search its rich pool of proven solution providers to identify prospective suppliers. It would reduce redundancy in market research and requirements development and improve the likelihood of broader, more rapid integration of proven capabilities.
For an entity as rigid and adaptive as the DoD, simply making more information available to key stakeholders will not drive change. The Department must also encourage key stakeholders to better leverage DoD-backed innovation programs: The DoD should implement an incentive program to require defense contracting entities and large defense contractors to allocate a set percentage of business to "proven innovators" or companies that have graduated from a DoD-backed innovation program.
This incentive program would be similar to existing set-aside programs designed to increase opportunities in the federal market for various historically disadvantaged groups. Proven

Defense innovation programs
Innovators would earn their "set-aside" status upon completion of their DoD-backed innovation program. Like the existing cadre of set-aside programs, contractors and contracting officers would be required to achieve minimum engagement levels with proven innovators and would be motivated to do so through tax incentives and other benefits. This system would force broader adoption of leading technologies across branches, as the different branches (and the contractors that serve them) would be required to stay abreast of various sponsors' projects by using the innovators database, and it would force collaboration and cross-communication in an otherwise siloed environment. This incentive program would also add tremendous value to participant companies, lending them a concrete advantage in the broader contracting environment. Participants would see an increase in their federal business overall and an increased willingness on the part of large integrators to support and engage with them: DoD innovation hubs must also educate participants on the fundamentals of the defense contracting sectorhow to identify and bid on contracts, how to find and team with other firms, how to register for set-asides, etc.to make them more competitive.
Innovation hubs could potentially partner with the SBA, which already offers similar training. The more benefits DoD-backed innovation programs can offer, the better positioned they are to attract and serve the most discerning, talented technology companies.

Conclusions and future work
While the aforementioned recommendations are crucial first steps towards improving the efficiency and functionality of DoD-backed innovation programs, further research is required to fully understand participants' experience in the defense sector after program completion. First and foremost, additional research is required to further analyze the large data sets compiled. A more sophisticated analytics tool would enable additional conclusions, such as if and how non-DoD agencies leverage the technologies fostered by DoD-backed innovation programs and the impact of the rampant serial usage on the effectiveness of these programs as a whole. Additional data and analyses are also needed to evaluate participants' performance as subcontractors and teaming partners on defense contracts. Once DoD-backed innovation programs are reformed and improved, additional research is needed to understand the most effective ways to market-to and attract the best and brightest innovators into these programs. Additional research is also needed to better understand opportunities for streamlining, merging or eliminating redundant or ineffective entities throughout the vast, complex DoD innovation landscape, including the individual, specialized innovation programs within each branch and combat command. From the vantage points of both national security and efficiency, it is also critical that cutting-edge capabilities are integrated throughout the whole of the government. Therefore, further research is required to determine how to position participants in DoD-backed innovation programs for success as suppliers to both DoD and non-DoD customers.
It is essential for America's national security that the armed services have access to the best and brightest new technologies, and the continuous investment in DoD-backed innovation programs over the past 60þ years has resulted in a vast infrastructure of programs and access points for new, cutting edge solutions. DoD-backed innovation programs provide both participants and the Department with valuable benefits. Participant companies benefit tremendously from the funding these programs provide, which allows them to grow their business, develop new research and prototyping and commercialize. The programs also introduce participants to the various idiosyncrasies of the contracting space and to the unique challenges facing the armed forces. However, today's adversaries and threat environment demand unprecedented synchronicity and collaboration across the armed forces. It is therefore essential that all branches of the military adopt revolutions in technology as quickly and seamlessly as possible to ensure consistent standards in warfighting capabilities; to ensure fighters across the services can communicate and share information; and to ensure that the Department leverages its full potential and buying power to appeal to the small, private sector innovators it so desperately seeks to attract and retain. Unfortunately, DoD-backed innovation programs have failed to serve as viable entry points for emerging capabilities into the broader defense sector and have failed to enhance the integration of these emerging capabilities across the forces. A superior military not only serves domestic national security interests but also enables the USA to fulfill its role in safeguarding peace, prosperity and freedom. As such, the DoD must stop at nothing to keep the forces agile, modern and at the forefront of new technologies, and that includes adapting its existing resourcesin this case, its vast network of innovation hubsto be as effective as possible.