How do developers utilize source code from stack overflow?

Technical question and answer Q&A platforms, such as Stack Overflow, provide a platform for users to ask and answer questions about a wide variety of programming topics. These platforms accumulate a large amount of knowledge, including hundreds of thousands lines of source code. Developers can benefit from the source code that is attached to the questions and answers on Q&A platforms by copying or learning from (parts of) it. By understanding how developers utilize source code from Q&A platforms, we can provide insights for researchers which can be used to improve next-generation Q&A platforms to help developers reuse source code fast and easily. In this paper, we first conduct an exploratory study on 289 files from 182 open-source projects, which contain source code that has an explicit reference to a Stack Overflow post. Our goal is to understand how developers utilize code from Q&A platforms and to reveal barriers that may make code reuse more difficult. In 31.5% of the studied files, developers needed to modify source code from Stack Overflow to make it work in their own projects. The degree of required modification varied from simply renaming variables to rewriting the whole algorithm. Developers sometimes chose to implement an algorithm from scratch based on the descriptions from Stack Overflow answers, even if there was an implementation readily available in the post. In 35.5% of the studied files, developers used Stack Overflow posts as an information source for later reference. To further understand the barriers of reusing code and to obtain suggestions for improving the code reuse process on Q&A platforms, we conducted a survey with 453 open-source developers who are also on Stack Overflow. We found that the top 3 barriers that make it difficult for developers to reuse code from Stack Overflow are: (1) too much code modification required to fit in their projects, (2) incomprehensive code, and (3) low code quality. We summarized and analyzed all survey responses and we identified that developers suggest improvements for future Q&A platforms along the following dimensions: code quality, information enhancement & management, data organization, license, and the human factor. For instance, developers suggest to improve the code quality by adding an integrated validator that can test source code online, and an outdated code detection mechanism. Our findings can be used as a roadmap for researchers and developers to improve code reuse.

the top 3 barriers that make it dicult for developers to reuse code from Stack Overow are: (1) too much code modication required to t in their projects, (2) incomprehensive code, and (3) low code quality.
We summarized and analyzed all survey responses and we identied that developers suggest improvements for future Q&A platforms along the following dimensions: code quality, information enhancement & management, data organization, license, and the human factor. For instance, developers suggest to improve the code quality by adding an integrated validator that can test source code online, and an outdated code detection mechanism that can identify code for outdated software versions. Our ndings can be used as a roadmap for researchers and developers to improve future Q&A platforms.

Introduction
Technical question and answer (Q&A) platforms such as Stack Overow have become more and more important for software developers to share knowledge.
Developers can post questions on these Q&A platforms, which in turn are answered by other developers. These answers often contain source code snippets.
As of August 2017, Stack Exchange reports that there are approximately 7.6 million users and 14 million questions with 23 million answers on Stack Overow (Stack Exchange, 2017). Among those answers, 15 million (75%) have at least one source code snippet attached, which forms a huge code base for developers to reuse source code from.
However, reusing source code is not easy (Gamma et al., 1995). For example, these are two of the challenges that developers face when reusing source code from Q&A platforms: (1) It is dicult for developers to nd suitable source code based on their particular needs, such as language, functionality, and performance (Wang et al., 2014a). To address this challenge, a number of studies have been done to help developers to locate more relevant source code snippets (Ponzanelli et al., 2014a,b;Wang et al., 2014a).
(2) Even if developers are able to nd suitable source code, it may be dicult to integrate the code in their own projects. For example, parameters may need to be adjusted, or additional source code may need to be added (Cottrell et al., 2008). To address the challenge of code integration, various techniques have been proposed (Cottrell et al., 2008;Meng et al., 2011;Yellin and Strom, 1997;Meng et al., 2013;Hua et al., 2015), such as automatically renaming variables to make the code t in the required context.
Prior studies (Gharehyazie et al., 2017;Treude and Robillard, 2017;Xia et al., 2017;Barzilay, 2011) on reusing source code from Q&A platforms have mostly focused on helping developers to locate relevant source code, and on integrating that source code into their own project. In this study, we build upon these studies by investigating how developers utilize code from Q&A platforms. Such knowledge will help us better understand the potential barriers that developers face when reusing code from Q&A platforms. In this paper, we use this knowledge to provide a roadmap for improving next-generation Q&A platforms.
We rst conduct an exploratory study of 289 source code les from 182 open-source projects, which contain at least one link to a Stack Overow post.
We manually study each le and its linked Stack Overow post, to investigate how developers reuse code from Stack Overow. In particular, we focus on the following research questions: RQ1: To what extent do developers need to modify source code from Stack Overow in order to make it work in their own projects?
In 44% of the studied les, source code had to be modied before it could be used in the developer's own project. The required modication varied from simple refactorings to a complete reimplementation. This nding provides empirical evidence for the importance of prior studies on automatic code integration (Feldthaus and Møller, 2013;Alnusair et al., 2016;Wang et al., 2016b). In 12.5% of the studied les, developers reimplemented code based on the idea of a Stack Overow answer, which suggests that Q&A platforms should consider to summarize key points that are discussed in a post to give developers a quick overview of a question and its answers.
RQ2: From which part of the Stack Overow post does the reused source code come?
Developers reuse source code from non-accepted answers (26%) for several reasons, such as, the simplicity and performance of the source code. Some developers even adopt answers that are total opposites from what the original asker wanted but meet their needs. Hence, Q&A platforms should consider to improve the way of organizing answers, so that developers can nd the most suitable answers based on their requirements easily, such as voting on the dierent aspects (e.g., readability or performance) of answers or adding tags for answers.
To further understand the barriers that developers face when reusing code and to collect suggestions for improving the code reuse process on Q&A platforms, we conducted a survey of 453 open-source developers who are also on Stack Overow. In our survey, we focus on the following research questions: RQ3: What are the preferences of developers when it comes to reusing code? Slighty more participants prefer reimplementing source code slightly over reusing source code from Q&A platforms. The most occurring reasons that developers gave for their preference of reimplementation over reusing were having to make the code t in their own projects, and a low comprehension or low quality of the code from Q&A platforms. These ndings provide empirical evidence for the importance of research on code integration and code comprehension, and highlight the need of providing code quality indicators on next-generation Q&A platforms. The rest of this paper is organized as follows. Section 2 introduces the background and related work of our study. Section 3 introduces our research questions and describes our data collection process. Our exploratory study is described in Section 4. Section 5 presents the survey design. Section 6 summarizes and analyzes the survey results, and presents our roadmap for next-generation Q&A platforms. Section 7 describes the threats to validity.
And nally, Section 8 concludes the paper. to obtain knowledge about and nd solutions to their programming problems (Treude et al., 2011;Abdalkareem et al., 2017). Developers are allowed to post questions, answer questions and vote on questions or answers on Q&A platforms. In the questions or answers, developers often attach snippets of source code to explain their questions or answers along with the textual description. For example, Figure 1 shows an example of an answer that contains source code on Stack Overow. The asker asked how to get the HTML of a selected object with the jQuery library, and the answerer posted an answer that provides a solution in the form of the attached source code. Stack Overow allows askers to mark at most one answer as the accepted answer to indicate whether the answer meets his/her requirement (see Figure 1).

Stack
Overow allows developers to upvote or downvote a post (e.g., a question or an answer) to express whether the post is useful. The total number of up and downvotes that a post receives is displayed as a score next to the post.
For example, the score of the question in Figure 1 is 673. In general, answers and questions with a high score are usually regarded as high-quality ones.
Several studies have been done to investigate the quality of questions and answers on Q&A platforms (Sillito et al., 2012;Treude et al., 2011;Ponzanelli et al., 2014c;Wang et al., 2017b). Sillito et al. (2012) performed an empirical study on factors that make a good source code example on Stack Overow.
They found that explaining important elements and presenting a solution step-by-step make a good example. Treude et al. (2011) performed a study on Stack Overow to explore which questions are answered well and which ones remain unanswered. They found that source code is an important factor for code review questions to get a good answer. Ponzanelli et al. (2014c) studied the factors that potentially aect the quality of questions on Stack Overow. They showed that the attached source code is an important factor for question quality. In this paper we study whether developers tend to reuse source code from a high-quality answer (i.e., a high-voted answer).

Source Code Reuse from Stack Overow
Source code reuse can be commonly observed . Figure 2 shows an example of code reuse. This source code snippet, which is taken from a GitHub project 1 , is reused from the Stack Overow post shown in Figure 1.
To help developers reuse source code, several approaches have been proposed. Rigby and Robillard (2013) proposed an approach to extract code elements RQ1: To what extent do developers need to modify source code from Stack Overow in order to make it work in their own projects?
Prior research has proposed several ways to utilize the source code from Stack Overow (Rigby and Robillard, 2013;Ponzanelli et al., 2013Ponzanelli et al., , 2014a. For example, Ponzanelli et al. (2013) presented an approach to automatically construct queries from the current context in the Eclipse IDE and retrieve relevant code and its corresponding discussions from Stack Overow. However, there is no empirical evidence about the process of how developers are reusing the source code, e.g., whether they copy-and-paste the original source code without any modication or they need to modify it considerably.
Knowing how developers reuse source code from Q&A platforms will give us insights on how to make source code reuse easier in next-generation Q&A platforms.
RQ2: From which part of the Stack Overow post does the reused source code come?
Intuitively, we may expect that accepted or answers with a high score are the most useful. In this RQ, we study whether this intuition is correct.
In particular, we investigate why developers chose to reuse code from non-accepted or low-scored answers. By understanding why developers chose non-accepted or low-scored answers, we can provide insights to help Q&A platforms organize their answers better so that developers can nd solutions more easily.
RQ3: Do developers prefer reusing or reimplementing source code?
Intuitively, reusing source code will consume less eort than reimplementing, especially if the source code is well-tested. In this RQ, we survey developers about the correctness of this intuition. We also investigate which factors make developers prefer reimplementing source code over reusing it. RQ5: How can code reuse be improved in next-generation Q&A platforms?
Motivation: In this RQ, we elicit suggestions from the surveyed developers for improving next-generation Q&A platforms. We analyze and synthesize their suggestions to dene a roadmap for researchers and developers of next-generation Q&A platforms.

Data Collection
In the remainder of this section, we describe our data collection process for the exploratory study (to answer RQ1 and RQ2) and our survey (RQ3RQ5). All our studied data and the corresponding analysis are available from our online appendix (Wu et al., 2017).

Collecting Data for the Exploratory Study
The steps of our data collection process for our exploratory study on code reuse from Stack Overow in open-source projects (Section 4) are shown in Figure 3. First, we collected source les that contain an explicit reference to a Stack Overow post from searchcode.com (Searchcode, 2016b)  To search for such source les, we use searchcode.com (Searchcode, 2016b) as our search engine. searchcode.com has indexed over 20 billion lines of source code from 7 million open-source projects. With its API (Searchcode, 2016a), we were able to collect 4,878 les in total using stackoverow as the search keyword. We focused on les that are written in the ve most popular program languages on Stack Overow (Stack Overow, 2016) (JavaScript, Python, Java, PHP, and Objective-C).
Removing Irrelevant Files: The stackoverow keyword can match source les that contain the stackoverow keyword outside of a link to a post, e.g., in an API name. We manually removed such false positives in this step. In addition, not all projects in open-source repositories are interesting from a software engineering point of view. For example, GitHub contains many toy projects, from which we cannot extract knowledge that is representative of other projects (Kalliamvakou et al., 2014). Therefore, to mitigate the eects from small projects, we removed les that belong to projects with less than 1,000 commits and 10 contributors. We then remove the les that are duplicates of each other (e.g., because they come from forked projects). Finally, we ended up with 289 unique les, which belong to 182 open-source projects. Within these les, 321 Stack Overow hyperlinks were found.

Collecting Participants for our Survey
We used the dataset provided by Vasilescu et al. (2013) to get candidate participants. This dataset includes 93,771 email addresses from the intersection of users of GitHub and Stack Overow. We took a random sample of 6,000 users from this dataset and sent them email invitations for our online survey.
1,935 of the emails did not reach the survey candidates because the email address did not exist any more. In the end, we received 453 responses which equals a response rate of 11.1%.  personal projects are the dominant project types that the participants are involved in, followed by academic projects (see Figure 5). Note that a participant can work on more than one type of project.

An Exploratory Study of Source Code Reuse from Stack Overow in Open-Source Projects
In this section, we present and discuss the results of our exploratory study of  (Seaman, 1999;Seaman et al., 2008) for identifying the type of the code utilization. This process involved 3 phases and was performed by the rst three authors (i.e., P1P3) of this paper:  Results: 31.5% of the reused source code was modied in one way or another. Table 1 shows that 20.5% of the studied les reused source code without modication (C1). In 31.5% (C2 and C3) of the les, the source code required modication before it could be used. Type C4 (12.5%) indicates that it is not exceptional that developers converted the ideas written in natural language to source code from scratch. Type C5 (35.5%) indicates that there exist developers who use Stack Overow as a programming manual. The nding that 31.5% of the source code reuse required additional modication implies that nding the code is only the rst step for code reuse. More eort is needed to facilitate code reuse from Q&A platforms after retrieving relevant code from them, such as making the source code work in the required context.
In 10.0% of the studied les, developers make cosmetic modications when reusing source code, which may improve the readability or simplicity of the source code. In the Cosmetic Modication category, developers copy-and-paste the source code from a Stack Overow post and make modications to the source code which may not be necessary to make the source code work in the target project. In the example shown in Figure 7, the developer copied two lines of source code in the accepted answer from the Stack Overow post and renamed the variable name from re to emailRegExp.
In 12.5% of the les, developers wrote the source code from scratch based on the descriptions of the algorithm. In the example shown in Figure 8 and  Developers are the most likely to reuse code or ideas in JavaScript. Figure 6 shows the distribution of each type of source code utilization for each studied programming language. We observe that code and idea reuse was the highest in JavaScript (79.1% of the studied JavaScript les). One possible explanation is that Stack Overow provides an online running environment for JavaScript, which may make developers more condent about reusing code or ideas in JavaScript from Stack Overow than in other languages. source code from the project 5 is implemented based on the description of the algorithm from Stack Overow (see Figure 9). 31.5% of the reused source code required additional modication, which shows the importance of studies on automatic code integration. In 12.5% of the studied les, developers reimplemented code based on an idea, which suggests that Q&A platforms should consider to summarize the key points that are discussed in a post to give developers a quick view of the question and its answers.
1 Let the segment endpoints be p1 =( x1 y1 ) and p2 =( x2 y2 ). 2 Let the rectangle 's corners be ( xBL yBL ) and ( xTR yTR ). 3 4 Then all you have to do is 5 6 A. Check if all four corners of the rectangle are on the 7 same side of the line . The implicit equation for a line 8 through p1 and p2 is : 9 10 F(x y) = (y2 -y1 )x + ( x1 -x2 )y + ( x2 * y1 -x1 * y2 ) 11 12 If F(x y) = 0, (x y) is ON the line . 13 If F(x y) > 0, (x y) is " above " the line . 14 If F(x y) < 0, (x y) is " below " the line . 15 16 Substitute all four corners into F (x y ). If they ' re all 17 negative or all positive , there is no intersection . If 18 some are positive and some negative , go to step B . 19 20 B. Project the endpoint onto the x axis , and check if the 21 segment 's shadow intersects the polygon 's shadow . Repeat 22 on the y axis : 23 24 If ( x1 > xTR and x2 > xTR ) , no intersection ( line is to 25 right of rectangle ). 26 If ( x1 < xBL and x2 < xBL ) , no intersection ( line is to 27 left of rectangle ). 28 If ( y1 > yTR and y2 > yTR ) , no intersection ( line is 29 above rectangle ) . 30 If ( y1 < yBL and y2 < yBL ) , no intersection ( line is 31 below rectangle ) . 32 else , there is an intersection . Do Cohen -Sutherland or 33 whatever code was mentioned in the other answers to 34 your question . 35 36 You can , of course , do B first , then A.  Results: In 26% of the studied les developers chose a non-accepted answer and in 58%, those non-accepted answers were not the highest-scored ones. The results of the categorization are shown in Table 2. As we can see from the results, not all reused source code came from an accepted answer. In 48% of the studied les, developers chose source code from an accepted answer.
However, there are still a considerable number (26%) of les where developers choose the source code from non-accepted answers. Moreover, among those non-accepted answers, 58% were not the highest-scored ones, which indicates that developers certainly did not always choose source code from the accepted or highest-scored answer. In the remainder of this section, we discuss the

Dierent Requirements than the Question Asker
Description : Developers chose source code from a non-accepted answer because they had dierent requirements than the original question asker.
Example : A developer wanted to implement a method to generate GUIDs. The source code in this example is shown in Listing 1. This source code snippet is actually from a non-accepted answer 11 on Stack Overow which has 37 votes, while the accepted answer has 1290 votes. The source code provided by the accepted answer is shown in Listing 2.
According to the description in the answer that contains the source code, the algorithm in Listing 1 is simpler and has very good performance, but not compliant with the RFC 4122 standard. The author of this answer also attached a performance test result in which several algorithms that are mentioned in other answers of the Stack Overow post are compared, which shows that the algorithm in Listing 1 outperforms the others. Hence, one possible explanation is that the developer who adopted this low-scored answer prioritizes performance 10 https://goo.gl/Z1pRMS 11 https://goo.gl/aC4auZ 12 https://goo.gl/xpAcga and simplicity over other factors, such as whether the generated result is compliant with a standard.

Fixing Bugs
Description : Developers adopted source code that xes a potential bug of the accepted answer.
Example : A developer was looking for a method to draw a dashed line around a selection area in JavaScript 13 . The non-accepted answer 14 improves the accepted answer by utilizing the built-in transformation functionality of Canvas, and also handles special cases where the line is vertical, which was not addressed in the accepted answer.

Improving Speed
Description : Developers adopted source code with a better performance.
Example : A developer was looking for an algorithm that sorts an array by the Levenshtein Distance in JavaScript. According to the comments below the accepted answer, the implementation in the accepted answer performed better than the one provided by the original asker. However, a non-accepted answer provided an improved version of the accepted answer which was described as Most speed was gained by eliminating some array usages , which was reused by the developers in their project. Thus we believe this developers gave performance a higher priority. Phase II: the two authors used the revised schema to code the answers. Developers reimplement source code slightly more frequently than that they reuse source code. Figure 11 shows the comparison of frequency of reimplementing source code and reusing source code from Q&A platforms.
The number of participants who reimplement source code monthly (33.2%) and those who reuse source code monthly (29.2%) are close, while the dierence increases to 25.0% vs. 15.0% at a weekly frequency.
A majority of developers (65%) prefer reimplementing source code, due to the code modication that is required to make the code from the post work in their own project. Table 3 shows the reasons for choosing reimplementation over the reuse of source code. The top reason that makes developers prefer reimplementing source code is the code modication that is required to make the code from the post work in their own projects. This nding is consistent with our nding in RQ1 (i.e., most code needs modication before reusing) and also provides empirical evidence for the importance of research on code integration. Several studies have been done on automatically retrieving and integrating code in a user's project context (Alnusair et al., 2016;Galenson et al., 2014;Wang et al., 2016b). However, these approaches are not widely adopted by developers. Future studies should investigate which factors prevent such tools from being applied in practice.
Code comprehension ranks as the second most important reason that preferring reimplementation over code reuse. This nding is in line with the work by Xin et al. (2017), which showed that developers spend 58% of their time on program comprehension activities. Hence, next-generation Q&A platforms should investigate how to improve comprehension of the source code in a post to facilitate its reuse. Approximately one-third (32%) of the participants complained about the low code quality on Stack Overow, which highlights the need for next-generation Q&A platforms to improve or verify the code quality of source code snippets.
An interesting observation was that 17% of the participants stated that reusing source code takes more time than reimplementing it, which is against the common wisdom. One possible reason is that if the source code snippet is large or complex, it could take more time to comprehend it than to make it work in another context.
Developers reimplement source code slightly more frequently than that they reuse source code. The primary reason is that it would take longer to adapt the source code to work in their own projects, than to simply reimplement it.
Our observations provide empirical evidence for the importance of research on automated code integration and code comprehension, and highlight the need of improving the quality of code snippets on next-generation Q&A platforms.
5.2 RQ4: Is code license a barrier for code reuse for developers?
In 75.2% of the cases, participants do not have a good understanding of the license terms of Q&A platforms, which indicates that there may be license violation issues when developers reuse source code from Q&A platforms. Figure    concrete answer, e.g., Depends on the platform. Stack Overow is attribution-required, but the requirements of most other sites are vague or not generally known.
In 39.2% of the cases, participants are not sure whether the license of a Q&A platform is compatible with that of their own project. Figure 13 shows that an additional 12.1% of the participants (strongly) disagrees that the license of a Q&A platform is compatible with that of their own project. Hence, 51.3% of the participants may experience diculties (or cause a violation) when reusing code from Q&A platforms due to their license. These diculties were noticeable from the survey responses when the participants were asked why they preferred reimplementing over reusing source code. For example, one participant mentioned that licensing is sometimes an issue. . More than half of the participants (57.4%) think that having more information about the code license is (very) important (see Figure 14). Together with the nding that most participants do not have a good understanding of the code license of Q&A platforms, these ndings reveal a need for clearer information about the code license of Q&A platforms.
Generally speaking, participants did not have a good understanding of the code license on Q&A platforms. More than half of the participants believed that, or were unsure whether, there exist incompatibilities between the code license of their own project and Q&A platforms. Almost 60% of the participants thought that Q&A platforms should give more information about their code license. Based on these observations, next-generation Q&A platforms should have clearer license information and make that information more visible to developers.

A Roadmap for Next-Generation Q&A platforms
In this section, we summarize and analyze the results for RQ5 (How can code reuse be improved in next-generation Q&A platforms?). In total, we collected 150 responses for Q19 in the survey. 22 of these responses were not actually suggestions for next-generation Q&A platforms (e.g., Not much, quite happy with Stack Overow. ) and were excluded from the following analysis.
Each response can contain multiple suggestions. In total, we extracted 183 suggestions from the responses. Using our open coding-like approach (see Section 5), each suggestion was categorized into one of these ve categories: code quality, information enhancement & management, data organization, license, and human factors. We categorized suggestions that did not fall into one of these categories into a sixth other category. We highlight the ndings and discuss implications on future research of next-generation Q&A platforms of each category in the remainder of this section.

Suggestions on Code Quality
Next-generation Q&A platforms should integrate mechanisms for online code validation and detecting outdated code. Code quality is the most popular type of suggestion (35.0%) from the participants. Table 4 shows the categorization of the suggestions that participants made on improving the code quality on Q&A platforms. The two most important suggestions from developers on improving code quality were adding (1) an integrated validator (27 participants) that can test source code online and (2) an outdated code detection mechanism (19 participants) that can identify code for old software versions.
An integrated validator is a convenient way of testing source code snippets online to ensure the quality of the source code.  Overow. There are several challenges when it comes to online validation of all languages.
One of the biggest challenges is to make an incomplete code snippet run Providing an outdated code detection mechanism is the second most popular Next-generation Q&A platforms should allow tagging for answers.
Information enhancement & management (23.5%) is the second most popular suggestion for developers on Q&A platforms. Table 5 presents the results of suggestions related to information enhancement and management. Based on the observations, future studies should focus on recommending the tagging-like information for answers. These recommendations could be made automatically using, e.g., machine learning techniques (Wang et al., 2017a(Wang et al., , 2014bZhou et al., 2017), or aspect-mining techniques (Wong et al., 2008;Wang et al., 2010;Yu et al., 2011;Liu et al., 2015;Zhao and Li, 2009).

Suggestions on Data Organization
Next-generation Q&A platforms should better organize their data, for example by providing a better searching and indexing mechanism and better duplicate detection. As shown in In prior studies (Wang et al., 2014a;McMillan et al., 2011;Lv et al., 2015;Bajracharya et al., 2006;Searchcode, 2016b), researchers have studied    Ahasanuzzaman et al., 2016;Wang et al., 2017c;Zhang et al., 2017). The common way to identify duplicate questions is to measure such questions' similarity in terms of semantic meaning. Recently, deep learning has proven its power of capturing semantic meaning from natural language in several studies (Ganguly et al., 2015;Lai et al., 2015;Bian et al., 2014;Chen et al., 2016). Hence, future research could consider to employ deep learning to detect duplicate or nd similar questions.

Suggestions on Code License
Next-generation Q&A platforms should make their code licensing information clearer and more visible. In 12.6% of the cases, participants suggested to improve license-related issues, in particular to make the license more clear (16 participants). This percentage is in line with our earlier nding that 75.2% of the participants did not have a good understanding of the license terms of Q&A platforms (see Section 5.2). Table 7 shows the suggestions about the code license of Q&A platforms.
Participants requested that Q&A platforms provide a clearer explanation of their license terms: By far the most important requirement is clear licensing.
Much of the code provided on such platforms is not currently usable because the  user contributions licensed under cc by-sa 3.0 with attribution required . As such mismatches will further deepen developers' misunderstanding of license terms, we suggest that next-generation Q&A platforms explicitly describe their license terms for source code reuse in a consistent manner.

Suggestions on the Human Factor
Next-generation Q&A platforms should assign human experts to curate knowledge on the platform. In 10.4% of the cases, participants suggested to improve Q&A platforms in terms of the human factor. Table 8 shows that twelve participants suggested to have better curators to improve the quality of posts and help with marking good answers. One participant suggested: Pay some vetted, experienced developers to check the answers, instead of relying on gamication. Another suggestion emphasized the importance of collaboration within the community: Arriving at a 'most correct' solution should be a more collaborative eort with a clearly shown path of how it was Our ndings suggest that future studies are necessary on how to improve the gamication mechanism of Q&A platforms.
6.6 Other Suggestions Table 9 shows the 13 suggestions that did not fall into the other categories.
There were four suggestions that suggested that Q&A platforms are open-sourced.
In addition, there were three suggestions that were related to AI techniques.
For example, one of them suggested the use of an AI technique that can automatically produce source code while developers only need to review the source code. These suggestions support the current research on code generation, in which tools are developed to automatically generate code based on a natural language input (Yin and Neubig, 2017;Gu et al., 2016). The rest of the suggestions in the other category talk about dierent topics. For example, one of them suggests the integration of a tool that can automatically translate a source code snippet across programming languages.

Threats to Validity
External validity. Threats to external validity relate to the generalizability of our ndings. In this study, we used searchcode.com as our search engine and found 4,878 source les in total that contained links to Stack Overow posts. After removing the small projects and duplicate les, there were 289 les left. The number of les may not be large enough to represent all the cases in the real world. However, we think that the number is large enough for our exploratory study to nd out the core issues involving the process of code reuse from Q&A platforms.
For the survey, we only invited developers who are in the intersection of users of GitHub and Stack Overow. Hence, our results may not generalize to software developers who are not in this intersection, such as developers of closed-source software. Future studies should extend our exploratory study and survey to developers from other domains.
We focused the rst part of our study on the ve most popular programming languages, and as a result, our ndings may not generalize to other languages.
Future work is necessary to investigate whether our ndings hold for other languages.
Internal validity. Threats to internal validity relate to the experimenter bias and errors. In this research, we manually inspected the source code in the projects and in Stack Overow posts. We also manually categorized each case of how developers are reusing the source code. To mitigate the threat of bias during the manual analysis, two of the authors conducted the manual analysis and discussed any conicts with a third author until a consensus was reached.
When analyzing the results of user survey, we also manually categorized the open questions in the survey. We used Cohen's kappa (Gwet et al., 2002) to measure the inter-rater agreement. The kappa values ranged from 0.79 to 0.92, which implies a high level of agreement.
Another threat is that our ndings in RQ2 may not exactly reect the intent of a user who reused code from Stack Overow, since the code provided on the post to which the link points may not exactly be what the developer needs. To mitigate this threat, we tried contacting the developers about their intent, but received no response.
Construct validity. A threat to the construct validity of this study is that we used mostly closed-ended questions in our survey, which may aect the richness of the responses collected from participants. However, open-ended questions have several disadvantages (Reja et al., 2003): (1) Open-ended questions take much longer for participants to ll out, making it more likely that they do not ll out the survey at all.
(2) Open-ended questions have the problem of missing data, i.e., participants skipping or doing a poor job at answering a question (i.e., by giving an incomplete or invalid answer).
(3) Open-ended questions are much more dicult to code than closed-ended questions. We felt that for most of our survey, the advantages of closed-ended questions outweighed the disadvantages. Therefore, we chose to use mostly closed-ended questions, together with an Other. eld.

Conclusion
Prior studies focus on code reuse from Q&A platforms have focused on locating and integrating source code in a required context automatically. However, no studies have been done on how developers utilize source code from Q&A platforms. In this paper, we studied how developers reuse code from Q&A platforms and we identied the barriers that they faced during the code reuse process.
We rst conducted an exploratory study on 289 source code les from 182 open-source projects which contain an explicit reference to a Stack Overow.
Based on the result of this exploratory study, we surveyed 453 developers who are in the intersection of GitHub and Stack Overow users, to further understand the barriers that they face when reusing code and to obtain suggestions for next-generation Q&A platforms. The most important ndings of our study are: 1. Our exploratory study shows that 78.2% of the reused source code from Stack Overow had to be modied in order to work in the required context.
The required modication ranged from simple refactoring to a complete reimplementation.
2. Developers reuse source code snippets from non-accepted answers as well (26% of the studied cases).
3. Developers prefer reimplementing source code over reusing source code because of the diculty of integrating the code into their own project, and low comprehension or low quality of the code.
4. The most suggested improvements for next-generation Q&A platforms are about improving the quality of source code snippets.
5. Many developers do not understand, or do not seem to care much about the code license of Q&A platforms.
Our study shows that Q&A platforms are evolving beyond their traditional use of asking and answering questions. From our survey, we can conclude that code reuse from Q&A platforms is a real challenge for developers: judging by the suggestions that we extracted from the survey, many developers face similar barriers during the code reuse process. Next-generation Q&A platforms should integrate existing solutions that improve the quality, comprehension and organization of source code snippets to better facilitate code reuse. Researchers and developers of Q&A platforms can leverage the roadmap that is presented in this paper to remove many of these barriers in next-generation Q&A platforms.