Methodological challenges of research on crowdsourcing

Crowdsourcing is a relatively new concept and, despite the interest of researchers, still little is known about it. At the same time, one observes difficulties of a cognitive and practical nature. This has become a premise for a reflection on the methodology of research on this subject. The subject of the article is the identification of the existing procedures of studying crowdsourcing, with particular inclusion of the methodological challenges that researchers of this concept may face. The article was written based on a systematic literature review. Its results enabled the formulation of some methodological guidelines for further research. Research should be conducted taking into account three levels of crowdsourcing: organization, technology, and participant. In addition, a quantitative and qualitative approach will enable the expansion of knowledge about crowdsourcing.


INTRODUCTION
The notion of crowdsourcing appeared in the subject literature for the first time in 2006 owing to Howe.At first, crowdsourcing gained popularity in management sciences due to its potential in the scope of innovative problem solving (Afuah & Tucci, 2013).In the next years, researchers saw its benefits related to, inter alia: developing business processes, creating open innovations (Burger-Helmchen & Pénin, 2010), building competitive advantage (Leimeister & Zogaj, 2013), and accessing experience, innovativeness, and information (Aitamurto, Leiponen, & Tee, 2011).Crowdsourcing also enables crisis management, expands an organization's existing activity, creates an organization's image, improves communication with the environment, and optimizes the costs of an organization's activity.For these reasons it has become a megatrend in economic practice -more and more organizations reach for it taking into account just the potential business value alone (Leimeister, Huber, Bretschneider, & Krcmar, 2009).
Currently, one may observe a gradual interest in crowdsourcing, both of scientists and practitioners (West, Salter, Vanhaverbeke, & Chesbrough, 2014).It is possible to indicate many reasons for the growing attention to this process, which argue for the currently observed, and postulated in the future, intensification of the discourse on crowdsourcing.Brabham (2008) formulated an ascertainment that crowdsourcing is a new, exciting direction for research, which is of great importance to the whole organization (Colombo, Buganza, Klanner, & Roiser, 2013).It is considered in the literature as a rising phenomenon based on Web 2.0, which draws the attention of both practitioners and scientists.It is the possibilities and benefits that come from using crowdsourcing, which constitute the source of its popularity.There is even a conviction that crowdsourcing in the next years will be a dynamic and active area of research (Zhao & Zhu, 2014).Moreover, crowdsourcing is beginning to play a very important role in a number of fields (Tapscott & Williams, 2007).One observes the growing importance of these problematic issues in medical sciences (Callaghan, 2014), technical sciences (Halder, 2014), and management sciences.
Despite the importance and establishment of crowdsourcing in management sciences, it seems that it has not yet seen comprehensive and cross-case analyses.The existing scientific output is mediocre and its nature is mainly conceptual.Therefore, one may ascertain that the research field under consideration is in a phase of early growth -it also concerns the methodology of research.Crowdsourcing may be considered a highly topical area of consideration.
The article aims to identify the existing procedures of studying crowdsourcing, with particular inclusion of the methodological challenges, which the researchers of this concept may face.In addition, based on other researchers' recommendations, methodological guidelines for further research were formulated.The article was written based on a systematic literature review.The biggest, full-text databases, i.e.Ebsco, Elsevier/ Springer, Emerald, Proquest, Scopus, and ISI Web of Science, which include the majority of journals on strategic management were analyzed.In order to establish the state of knowledge and existing findings, a review of databases in Poland, BazEkon and CEON, was also conducted.54 elaborations of English language databases and 41 from Polish language databases, from the period 2006-2017, were analyzed.

LITERATURE REVIEW
The concept of crowdsourcing was introduced into economic literature by J. Howe, the editor of Wired magazine, in June 2006.In his article entitled "The Rise of Crowdsourcing," he describes various organizations making use of the Internet to establish cooperation with customers and engaging them in creating innovations.The definition of crowdsourcing proposed by Howe, after consulting with his editorial colleague M. Robinson, appeared one month after the article was presented in a blog run by the editor (www.crowdsourcing.com).He defined crowdsourcing in the so-called White Book as the "act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call.This can take the form of peer-production (when the job is performed collaboratively) but is also often undertaken by sole individuals" (Howe, 2006).The second definition proposed in the socalled "Soundbyte" considers crowdsourcing as the "application of Open Source principles to fields outside of software" (Howe, 2008).The author considered crowdsourcing as a tool or a way that helps organizations acquire free or inexpensive workforce.
Howe points out, that crowdsourcing is a notion which owes its beginning to Surowiecki.Howe exposes in crowdsourcing the importance of the crowd and the forces that activate it to take action.He assumes that the crowd is distinguished by wisdom and each of its members possesses knowledge or skills which may become valuable to someone.The basis here is collective intelligence and cooperation, which may contribute to creating values, choosing the best solutions, gathering opinions, and formulating judgments.
A continuator of Howe's concept is Brabham.He proposed the first definition following numerous publications in the years 2008-2012, in his book entitled "Crowdsourcing" in 2013.According to Brabham's, crowdsourcing is not "just old wine in new bottles."The author gives examples of open calls for solving difficult problems: creating the Oxford English language dictionary in 1800 by means of open discussions and the Alkali prize for developing an alkali method founded in 1775 by Louis XVI.In Brabham's opinion, they are not examples of crowdsourcing since it is only present when the organization has a task to be performed, whereas the online community carries it out voluntarily.A result of these actions there are mutual benefits for both parties.For Brabham, crowdsourcing is an Internet-dispersed model for solving problems and production, a tool for social participation, a planner for governments, and a method of building and processing a significant number of shared resources.
With time new definitions started to appear, which considered the role of the Internet as a characteristic moderator (Quinn & Bederson, 2011).It had become linked with establishing collaboration and relations with virtual communities (Yang, Adamic & Ackerman, 2008), by making use of their wisdom (Surowiecki, 2004) to solve problems (Vukovic, Mariana, & Laredo, 2009), and creating innovative solutions (Sloane, 2011) and open-source software.In crowdsourcing, it is the wisdom and collective intelligence that gain importance.The crowd becomes wise, rational, kind, and useful (Gloor & Cooper, 2007;Wexler, 2011).Most authors acknowledge that a crowd is a general group, usually an undefined, large group of people -an online public (Kleemann, Voß, & Rieder, 2008) which is often called users, consumers, clients, voluntary users, or online communities (Chanal & Caron-Fasa, 2008;Whitla, 2009).It is accepted that the crowd in crowdsourcing constitutes a group of amateurs, composed of students, young graduates, scientists, or organization members (Schenk & Guittard, 2009).Other authors point to network employees (Heer & Bostok, 2010) emphasizing their education and intelligence.
The definitions of crowdsourcing define a particular conceptual framework (Table 1), including common features of crowdsourcing, i.e.: (1) crowd (who forms it?, what is she/he supposed to do?, what does she/he get in return?);(2) initiator (who is it/she/he?, what does it/she/he get in return from the crowd?); (3) process (type of process, way of joining the crowd, way of mediation between the organization and the crowd).
The indicated levels correspond with the fact that crowdsourcing is a complex and multidimensional concept.Nevertheless, there are discrepancies when it comes to their number.Some authors indicate five levels (Leicht, Durward, Blohm, & Leimeister, 2015): organization, intermediary, system, user, and application and evaluation.Whereas Hetmank (2013) also identified four levels, but they were named in a different way: organization, technology, process, and human-centric.Based upon the research by Zhao and Zhu (2014) they combined all the findings of other researchers and indicated three levels of crowdsourcing: organization, participant, and system.The researchers' findings are reflected in the work of other authors (Vukovic et al., 2009;Zogaj, Bretschneider, & Leimeister, 2014).Moreover, so, the organization level refers to the premises for involvement of the organization in crowdsourcing (Schenk & Guittard, 2009), identification of the critical success factors, involvement consequences (Sims & Crossland, 2010), possible benefits to be achieved (Stol & Fitzgerald, 2014), conditions, and implementation barriers.A question appears at this point about the initiator of crowdsourcing, namely who it/she/he is?What are the benefits it/she/he obtains?(Burger-Helmchen & Penin, 2010).Assuming the participant's level means that, owing to cumulating knowledge and skills, the virtual communities are able to solve problems, design new products and services, large aggregate amount of data, or collect funds for a given goal.Crowdsourcing at a system level is a social and technical system which supports interactions and communication between the people and the organization.Apart from the technical or IT aspects, the issue of how the virtual community integrate their ideas with the organization's specificity (Steiger, Albuquerque, & Zipf, 2012) arises.Others think that understanding the motivational mechanisms of crowdsourcing (Archak & Sundararajan, 2009;DiPalantino & Vojnovic, 2009;Horton & Chilton, 2010;Wilcox, 2000) may contribute to the greater involvement of the virtual communities (Zhao & Zhu, 2012).

RESEARCH METHODS
The review of research on crowdsourcing was conducted based on the results of a systematic literature review.One of the main reasons for using this methodology is the need for a methodological regime, which is essential if we are willing to fulfill the rule of continuity.As opposed to traditional literature reviews, a systematic literature review avoids the dangers stemming out of subjectivism, the lack of a systematic approach, and prejudice.According to its methodology, the entire procedure includes three stages: (1) selecting databases and a collection of publications, (2) selection of the publications and development of a database, (3) bibliometric analysis, contents analysis, and verification of the usefulness of the obtained results for further research.
The first stage constituted a choice of the subject of research.This concerned specifying a collection of publications, which would be analyzed.The basis at this point was selecting the databases.The analysis covered full text, greatest databases which include the majority of journals dealing with strategic management, i.e.Ebsco, Elsevier/Springer, Emerald, Proquest, Scopus, and ISI Web of Science.In order to establish the state of knowledge and existing findings, a review of Polish databases, BazEkon and CEON, was also carried out.They were selected owing to their integrity and completeness.The reason for using several databases simultaneously is due to their diverse range and the gathered resources and sources.The principal issue in defining the collection of publications is the choice of keywords connected with the subject of the research, in order to identify potentially significant scientific articles from the point of view of the analyzed problematic aspects.In each of the databases mentioned above, keywords were used which met the following criteria for inclusion: "crowdsourcing," "crowdsourcing" in the abstract, title, and keywords.The base of publications obtained in such a way was further analyzed and selected in the next stages.As a result of searching through the chosen databases over 46,000 publications were obtained selected from English language bases and 388 selected from Polish language bases.
The second stage is based on imposing limitations and database selection according to the "snowball" procedure.Therefore, the following limitations were imposed on the identified articles: full text, reviewed publications and the area of management sciences.Publications related to IT, social, technical, mathematical, medical sciences, and humanities were excluded from the collection.Duplicating publications, books, dissertations, and book chapters were eliminated.Articles in their full version, published in journals and the so-called proceedings were included.
The third stage is the basis for identifying the areas for further research exploration, valuable from a cognitive point of view, and important for the development of the theory of management.At this stage, the usefulness of the obtained elaborations for realization of the research aims was verified.Those publications, which did not strictly concern crowdsourcing, but instead treated it as a secondary subject, were discarded.Only those publications, whose leading object of analyses had the term "crowdsourcing" placed in the title and keywords, were deemed important from a research point of view.As a result, a literature base was obtained in the form of 54 publications selected from English language bases and 41 publications selected from Polish language bases.In the next stage, that total of 95 publications were further analyzed using bibliometric techniques, including the frequency, number of publications, and citations.Also at this stage, an analysis of the contents was also carried out, which determined the findings of other researchers and their evaluation, and also organized the research findings.The results of this systematic literature review have been presented in the second part of this article.

ANALYSIS
Interest in the subject of crowdsourcing started in 2006 with the first publication by Howe entitled "The Rise of Crowdsourcing."Since then crowdsourcing has started to receive the attention of researchers.Most publications still refer to Howe and the continuator of his concept Brabham (2008).One may observe that J. Howe's publication deserves to be called a seminal study and, therefore, the leading one, constituting an inspiration for further scientific studies (according to Google Scholar the number of daily citations for 30.10.2016 was 3276).The conducted analysis of the number of publications devoted to crowdsourcing enables one to ascertain that this subject enjoys researchers' interest.The figure for the trend of publications contained in the English language databases is R2=0.668, which proves a growing tendency of the publications.In the case of Polish language databases, the figure for the trend of publications is R2=0.133.This result clearly shows that the number of publications in the last ten years has been going up slightly.However, it is difficult to consider this result as spectacular.
Based on a frequency analysis it was found that most of the contents of the publications which qualified for analysis were of a theoretical and review nature (22 publications-foreign bases, 41 publications out of 43 analysed-Polish bases).They were reviews of the definitions or the current state of knowledge on crowdsourcing.The remaining publications were articles that present the results of original research of an empirical nature, in particular, case studies or descriptions of events.This statement also concerns national publications: most of them, apart from the theoretical layer, which constitutes a literature review, included descriptions of examples of good practices or quoted data of the Central Statistical Office of Poland.
It is pointed out in the literature that the measurement of crowdsourcing is a great challenge for the researcher (Cullina et al., 2015) -some authors include the measurement of this particular term among the most difficult ones (Hirth et al., 2015).Despite the difficulties, it is an important and significant issue.As ascertained by Afuah and Tucci (Afuah & Tucci, 2012), studying crowdsourcing is promising and it may be a source of theoretical, empirical, and scientific knowledge.This is because the measurement will enable organizations to obtain different advantages (Malone, Laubacher & Dellarocas, 2010), but above all, it will contribute to understanding this phenomenon (Wilson, 2015).However, despite the recommendations and need for conducting research in this regard, in the existing research output there is a lack of conceptual coherence, an insufficiency related to a holistic perspective of crowdsourcing which hinders making comparisons, a diversity of methodological approaches, as well as inaccuracy and inadequacy in the measurement methods used.In the author's opinion there is a need for identifying the inaccuracies, and limitations should be one of the key guidelines taken into account when formulating the methodological indications and recommendations for future research projects.
The first problem is the nature of the studied phenomenon alone.The differences between the authors result from a number of difficulties, adopted definitions, or conceptualization.In addition, a significant part of the existing measurement proposals does not refer to the notion's conceptualization.It is only limited to the choice of variables, without indicating the Volume 14, Issue 4, 2018: 107-125 Regina Lenart-Gansiniec / multidimensionality or an ambiguous nature.The importance of the process of arriving at understanding the term, identifying, and defining how a given term is understood should be emphasized in the research procedure (Babbie, 2008).Conceptualization, which is discussed here, is a process of agreeing on the meaning of the terms.Its result is giving meaning to a term by means of indicators, and thus marks of presence of the notion studied.It constitutes the necessary condition for further actions in favor of operationalization and implementing empirical research on crowdsourcing.The most often quoted paper related to crowdsourcing defines it as "the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call.This can take the form of peer-production (when the job is performed collaboratively) but is also often undertaken by sole individuals" (Howe, 2008).In his, work Howe points out the affinity of outsourcing, peerproduction, and crowdsourcing.
Nonetheless, Howe's approach seems to have some constraints.The author indicates the similarity between crowdsourcing and outsourcing and peer-production.In outsourcing, a supplier, selected by the organization, carries out some specific actions in accordance with the requirements and agreement.Peer-production assumes decentralization of tasks, large dispersion of the team, independent choice of tasks based on a selfassessment of skills and interests and treating the products or services being created as common goods available for a wider circle of recipients.In crowdsourcing, we are dealing with the crowd which is difficult to specify or define.Concluding agreements may prove to be impossible.Crowdsourcing should, however, be considered as a broader term: the crowd can focus its actions also on other activities.In the author's opinion, Howe's concept may be assumed as the principal definition, but at the same time connecting crowdsourcing with outsourcing and peer-production should be rejected.
The next issue is the need for a holistic look at crowdsourcing -this is required by the high level of this term's complexity.What is more, the methodological rigor requires its development according to the rule of continuity, namely taking into consideration previous studies.The research on crowdsourcing is not made easier due to its multidimensionality and manysidedness (Cullina et al., 2015).The authors Zhao and Zhu (2014), based on a review of 55 articles from the years 2006-2011 devoted to crowdsourcing, ascertain that future research should be conducted taking into account three perspectives or levels: organization (acceptance, implementation, management, quality, evaluation), technology (incentive mechanisms, technological issues), and participation (crowd motivation, organization employees' behaviors).However, in all of the existing research, a simplified approach has been used which is limited to one level, selected by the researchers.Among the thirty-two research projects (foreign bases) which refer crowdsourcing, twenty-five take into account the participant level, especially the virtual communities, five take into account the technological level, and two the organizational one.In the case of domestic bases, two research projects consider the virtual community level.Such limitations omit the holistic look at crowdsourcing or even the relationships between each level.The organizational level cannot exist without the technological level.In turn, the participant's level may exactly be a result of the organizational level.Instead of studying each level separately, future research should expand the scope of study by introducing new measurement scales as well as new mediation and moderation variables.Only on this basis may one conduct a detailed study of the elements and their mutual relations.
The second problem concerns the methodological approach used by researchers.The existing research on crowdsourcing is conducted according to a constructivist belief.It assumes that the nature of social reality is subjective and it only exists owing to an agreement between people.From this point of view, reality is changeable and not durable.Organizational cognition results from constantly occurring activity of the organization's members in the reality which surrounds them.Organizations should be treated as systems of knowledge composed of the knowledge of the members of these organizations and the social interactions between them (Petit & Huault, 2008).Such an approach to the issue decreases the coherence of the empirical research conducted in the literature and it reveals a peculiar research gap, consisting in the lack of an overall look at crowdsourcing.
As a result, one observes a multitude of approaches, which have their limitations.In order to indicate them, two purposefully chosen articles were analyzed: one which takes into account the quantitative studies, whereas the second one the qualitative studies.For example, in the quantitative approach, the researchers search for a model of statistically proven relationship between the variables.A contribution to testing this doubt is the research on crowdsourcing conducted by Yejun Xu, Enrique Ribeiro-Soriano and Gonzalez-Garcia in 2013.The research sample included the biotechnical industry and telecommunications enterprises operating in the Chinese market.A questionnaire survey was conducted, composed of 8 questions.The sample size was 393 enterprises (201 from the biotechnological industry and 192 from the telecommunications market).Hypotheses were studied which concerned the relationship between crowdsourcing and innovative competencies, the key competencies according to J. Schumpeter's approach, continuous improvement of competences.Taking into account the lack of earlier research studies and by the same token lack of measurement tools -the authors used the Delphi technique.They invited 24 experts: company managers, biotechnological and telecommunications industry specialists, and professors who deal with research on crowdsourcing.They received a proposal of a survey questionnaire composed of 14 items.Finally, they were reduced to 8 items.A Likert-type scale was used, with a range of five points from 1 -"much worse" to 5 -"much better".The obtained results confirm the relationships between crowdsourcing and the dependent variables distinguished during the study.The quoted results of empirical studies have their limitations.Firstly, the test sample, which includes just two industries in China, and therefore, is potentially affected by factors specific to these types of entities.In addition, markets were considered in which expenditure on research and development is very high.Secondly, the respondents were managers of the highest level responsible for the whole enterprise, which could have distorted their perception of crowdsourcing in the direction of innovations or cost optimising.Thirdly, limiting crowdsourcing to 8 items may constitute another limitation.Asking the respondents if they possess a crowdsourcing platform and whether they possess security systems protecting against data leakage, seems inadequate.Fourthly, it needs to be borne in mind that the quantitative approach has some specific threats, Exploring Management Through Qualitative Research Marta Najda-Janoszka, Corina Daba-Buzoianu (Eds.)inter alia: it does not enable disclosing the best combinations and the most effective strategies.Testing hypotheses are connected with searching for existing models of dependencies and, to a limited extent, it formulates their practical implications.
It is pointed out in the literature that a case study may be used for identifying motivation in crowdsourcing, complementing internal competencies, acquiring ideas, solving problems, impact on business models, obtaining benefits for the organization and its clients, knowledge production, and collaboration with various entities (Flyvbjerg, 2004;Sarker, Xiao, & Beaulieu, 2013;Yin, 2013).One example is the studies by Schlagwein and Bjørn-Andersen conducted at LEGO.The study covered the LEGO Cuusoo platform.In the years 2010-2014, a total of 19 in-depth interviews were conducted with managers and 25 informal personal discussions or online discussions with 25 internal and external stakeholders.It aimed to identify the importance of crowdsourcing for organizational learning.The authors ascertained that the necessities connected with explaining research issues, complex cause and effect relations, the researcher's interest in a contemporary phenomenon and its context, not evident borders between them, lack of possibility to influence them, and the need to evaluate the studied phenomena, support the choice of a case study as the research method.The case study may constitute an answer to the arising problems related to measuring crowdsourcing, i.e., early stage of knowledge development, need to identify the phenomenon in a given context, unclear borders between the phenomenon and its context, developing the existing theory, explaining phenomena, which have not been identified so far, analysis of organization behaviors, testing theory and understanding the circumstances of events, processes without conducting any manipulation related to their course.Moreover, the case study is a useful research method when testing hypotheses, particularly the hypothesis, which supposes the existence of a necessary condition and sufficient condition.This means orientation on preparing the actions of the decision-maker, studying the issues connected with the context of a given phenomenon and people's behaviors participating in it.
In conclusion, a synthesis of the above-mentioned considerations and the identified methodological challenges enables bringing to light some methodological guidelines for future research on crowdsourcing: 1) The measurement tool should cover by its range all three crowdsourcing levels, i.e., organizational, technological, and participant.2) A quantitative-qualitative approach may make it possible to achieve testing and theory creating goals.For instance, a multiple case study may be a reference to the recommendations of other researchers identified, based on the systematic literature review.The quantitative-qualitative approach is recommended by Brabham and it will enable the expansion of knowledge on crowdsourcing.3) Research should be conducted taking into account the constructivist paradigm.Such approach to the issue increases the coherence of the empirical studies conducted in the literature and fills in the specific research gap, consisting of the lack of a comprehensive look at crowdsourcing.

CONCLUSION
Based on the conducted systematic literature review a few methodological guidelines for future studies on crowdsourcing may be proposed.Firstly, crowdsourcing is a relatively new concept.Focusing on theoretical considerations means that in the theoretical and practical aspects there is still "terminological chaos," that too high an advantage of theoretical approaches persists, and many areas are completely untouched or poorly clarified in the literature.Most domestic publications focused on an analysis of best practices, also showing the benefits coming from crowdsourcing and the possibilities of its use.
Secondly, the crowdsourcing analysis presented in this elaboration points to generally accepted convictions in the literature on the impact of crowdsourcing on innovativeness or competences.Nevertheless, despite the recommendations included in the subject literature, crowdsourcing is not formulated holistically.The empirical study of these relations focuses only on individual levels: organization, technology, and participant.Thus, the identified relations will take place only for one, chosen level.An answer to these problems may be the quantitative and qualitative approach -on the one hand; it will enable achieving the testing goals and, on the other, the theory genic ones.
Thirdly, the multitude of crowdsourcing definitions or interpretations does not facilitate the development of adequate measurement tools.Nevertheless, there is a need to analyze the existing methods of measuring crowdsourcing, elimination of limitations and inaccuracies of the present methodologies of measuring crowdsourcing -based on it developing an appropriate measurement method taking into account the realization of goals of other researchers' work.This is important due to the fact that a proper definition and, next, operationalization, constitute the basis for conducting a proper measurement of this interesting, although difficult, concept.