The influence of expert groups: a citation analysis

ABSTRACT Decision-makers rely extensively on expert groups and commissions for policy advice. Many see in this a growing technocracy or expertization of decision-making. Yet, we know little about the actual influence of these expert bodies in policy-making. When does expert group advice shape policy-making rather than being brushed aside? The article goes beyond existing research by measuring and explaining variation in expert group influence. Using the number of citations to expert group reports in government white papers and later expert group reports as measures of influence, the article analyzes citations to 1545 Norwegian advisory commission reports published 1972–2017. It concentrates on three dimensions of the organization of expert groups as potential determinants of influence: member composition, appointing ministry and resources. The results suggest that expert groups have greater influence when they include more politicians and have a bigger secretariat. The article contributes empirically and methodologically to research on expertise and policy-making.


Introduction
Many observers claim that policy-making has become increasingly technocratic or 'expertized', as elected leaders rely extensively on experts for policy advice and delegate ever more decisions to unelected officials (Habermas, 2015;Turner, 2003;Vibert, 2007).The need for specialized expertise has grown with the expansion of government responsibilities in the welfare state and economic regulation and increasingly complex technologies.During the covid-19 pandemic, this expert dependence was evident, as policy-makers worldwide appointed expert groups to give advice on how to contain the spread of the virus, produce and distribute vaccines, counteract the economic slowdown, etc.Yet, this is only the latest example of political leaders' reliance on non-elected expert bodies for advice.Across a broad range of issues, national governments and international organizations draw on commissions, expert groups, advisory councils and task forces to define policy problems, establish principles for policy and make specific recommendations for reform.
Yet, despite their importance, we have limited systematic knowledge about the actual policy impact of these kinds of expert groups.While advisory bodies sometimes appear to have considerable influence on the thinking of policy-makers and the content of policy, other times their advice is ignored.What determines whether the advice of an expert group ends up shaping policy-making or being brushed aside?In the literature, establishing and explaining the influence of policy advisory bodies has been described as 'notoriously tricky business' (Campbell & Pedersen, 2014, p. 278) and as 'the million-dollar question in policy advisory research' (Hustedt & Veit, 2017, p. 46).A few studies examine the impact of expert groups across a small number of advisory commissions or policy processes (Inwood & Johns, 2016;Metz, 2015).Yet, while they offer important insights, there is also a need for studies that investigate expert group influence across a large number of cases.
Answering this 'million-dollar' question is important for several reasons.From a normative point of view, the extent to which unelected experts influence policy matters for the state of representative democracy.Some observers speak of a 'scientization' or 'expertization' of modern politics (Turner, 2003), suggesting that the democratic chain of delegation from voters via elected governments to policy-making has come under strain.Others see increasingly evidence-informed policy-making as a guarantee for the quality of public policies (Head, 2016).Greater empirical knowledge about the actual policy influence of experts can inform these normative debates.Moreover, there is growing awareness among scholars and practitioners that how expert knowledge is organized into policy-making matters for effective and legitimate governance (European Commission, 2022;OECD, 2020;Parkhurst, 2017).Expert groups vary in their composition, resources, mandates and scope, and thus offer a promising setting for examining exactly how organization matters for the influence of these groups.
The article makes an original contribution by measuring and explaining variation in the influence of expert groups across a large number of Norwegian official advisory commissions.We understand the influence of expert groups as the extent to which expert groups shape the problem understandings and policy solutions adopted by decision-makers.This influence may be either direct, i.e., by providing advice directly to decision-makers that forms their thinking about policy problems and measures, or indirect, by shaping discourse and debate around a topic, which in turn may affect the understandings and solutions that decision-makers adopt (Campbell & Pedersen, 2014).
Indirect influence may run either through what Vivien Schmidt (2010) calls coordinative discourse -discussions among policy-making elites (academics, bureaucrats, key stakeholders, etc.) -or communicative discourse -public debate involving a broad range of actors.We examine both direct influence and indirect influence on debates among policy-making elites.
To capture the influence of advisory commissions we use two quantitative citation measures, which count the number of citations to a commission's report in government white papers (measure of direct influence) and in later advisory reports (measure of indirect influence on policy debates).Citation measures of influence are a relative novelty in research on expertise and policy-making (see Bornmann et al., 2016;Christensen 2023;Karseth et al., 2022).Yet, they are regularly used to study similar phenomena, such as the scientific impact of academic work (e.g., Wuchty et al., 2007) or the influence of cases in law (Fowler et al., 2007).Citation analysis bears great potential for large-n studies of expert group influence, but also has important limits.Acknowledging that citations are proxies for influence and that not all citations entail impact, we therefore also validate our measures qualitatively through in-depth analysis of a sample of citations.
To explain variation in expert group influence, we examine statistically how the number of citations to a commission report varies depending on three aspects of how the commission is organized: (1) its member composition, i.e., the relative share of academics, civil servants, interest group representatives and politicians on the commission; (2) the strength of the ministry appointing the commission; and (3) the resources at the commission's disposal.We expect expert groups to be more influential when they have more academics or more politicians as members, when they are appointed by the finance ministry, and when they have a large secretariat and more time at their disposal.
Our analysis draws on an original dataset of all Norwegian official advisory commission reports published 1972-2017 and all Norwegian government white papers published 1999-2017.The dataset includes information about the characteristics of each advisory commission (N = 1545) and data on citations to commission reports in government white papers and other commission reports.We use hierarchical regression models to examine the determinants of the number of citations to commission reports.We find that expert group composition matters: commissions with more politicians as members have greater direct and indirect influence, whereas groups with more academic members have greater indirect influence (but not more direct influence).Commissions are also more influential when they are supported by a bigger secretariat, whereas the strength of the appointing ministry is not associated with greater influence.
The article makes an empirical and methodological contribution to scholarship on expertise and public policy.Empirically, the findings provide novel and robust evidence on the variation in the influence of expert groups and the determinants thereof, which complements existing small-n studies.
Our study also goes one step further than existing quantitative analyses of expert groups, which have so far only investigated their composition and not their influence (e.g., Gornitzka & Sverdrup, 2011;Hesstvedt, 2022).Methodologically, the article introduces citation analysis as a new approach to measuring the influence of expert bodies, responding to recent calls for methodological innovation in expert influence research (Christensen 2023).
The article proceeds as follows: We first review previous research on the influence of expert advisory bodies, before presenting our influence concept and discussing potential determinants of influence.We then present and discuss the empirical context, data and measures, before presenting the empirical results.We end by discussing the findings and how they contribute to existing debates.

Previous research
An important trend over the last half-century is the growing reliance of political decision-makers on unelected experts (Turner, 2003;Vibert, 2007).This is seen as a function of the increasing complexity of government: Not only have governments taken on more tasks in welfare provision and economic regulation; policy issues have also become more uncertain and specialized and involve complex technologies.In response, politicians have delegated decision-making responsibilities to experts in regulatory agencies and central banks and turned to various types of expert bodies for policy advice.Some observers see this expertization as a sign that the power to decide over public policy has drifted away from citizens and their elected representatives, with serious consequences for our democracies (e.g., Habermas, 2015).Others see it as a largely positive development, which ensures that policies are increasingly informed by sound evidence (Head, 2016).
Part of this expertization trend is the use of expert groups, commissions and task forces, which play a major role in decision-making processes in many political systems.During several major crises of the past decades, governments have relied heavily on such bodies to evaluate, inform and reform policy.Think for example about the U.S. Financial Crisis Inquiry Commission or the numerous scientific advisory councils and task forces appointed during the covid-19 pandemic.Although less visible outside times of crisis, advisory bodies are also extensively used in day-to-day policymaking.The European Commission, for example, has appointed more than 900 permanent and ad hoc expert groups since 2014, and the Scandinavian countries each appoint on average between 20 and 100 ad hoc advisory commissions every year (Dahlström et al., 2021;Hesstvedt & Christiansen, 2022).
In this article, we define an expert group as a consultative institution that is appointed by cabinet or parliament on a temporary basis, and that is mandated with providing analysis, knowledge or recommendations to government.While expert groups are mandated by, designed by and report to the incumbent government, they usually include external members -such as academics or interest groups -whose primary organizational affiliation is outside the permanent government apparatus (Hesstvedt & Christensen, 2021).Since expert groups are established by the government and asked to give advice, policy-makers may in general be more receptive to their recommendations than to unsolicited advice from, say, universities, think tanks or interest groups. 1 But how does the influence of expert groups vary and what explains their varying influence?
Existing research offers valuable insights but has not fully addressed this question.The literature on 'policy advisory systems' (Craft & Halligan, 2017) and 'knowledge regimes' (Campbell & Pedersen, 2014) has highlighted how a wide array of advisory bodies -government research units, think tanks, advisory councils and commissions, etc. -provide decision-makers with policy-relevant knowledge.This work has also discussed conceptually how advisory bodies may have both direct and indirect influence in policymaking (see next section).Yet, when it comes to empirically investigating influence, Campbell and Pedersen conclude that 'determining which policy research organizations are the most influential is difficult if not impossible ' (2014, p. 277).Similarly, the introduction to a recent special issue on policy advisory systems ends with a call for research on 'the million-dollar question in policy advisory research, namely seeking to measure, assess or determine the influence of policy advice on policy output' (Hustedt & Veit, 2017, p. 46).
There are also several existing studies of expert groups and advisory commissions, including in the European Commission (Gornitzka & Sverdrup, 2011;Metz, 2015), Anglo-Saxon countries (e.g., Hunter & Boswell, 2015;Inwood & Johns, 2016;Rowe & McAllister, 2006;Zegart, 2004) and Northern European countries (Christensen & Holst, 2017;Hesstvedt, 2022;Krick, 2015).These studies have generated important insights about the appointment, organization, composition and operation of expert groups.A few of them have also examined the influence of expert groups: Metz (2015) examines qualitatively how European Commission expert groups are used in 48 EU legislative drafting processes.Although mainly focused on the different ways in which knowledge is used in these processes, she also seeks to measure the influence of expert groups.She finds that in many policy processes, expert groups were influential in the sense that their position was adopted by the European Commission.Inwood and Johns (2016) examine the impact of expert groups on policy change through a small-n comparative analysis of Canadian commissions of inquiry.They argue that some commissions contributed to transformative and direct policy change while other commissions had diffuse or marginal influence, and discuss factors that could account for the varying impact.
Building further on these insights, our article makes an original contribution by seeking to measure and explain variation in influence quantitatively across a large number of expert groups.Before presenting our empirical and methodological approach, we discuss our concept of influence and potential determinants of influence.

Conceptualizing the influence of expert groups
Existing literature employs different concepts to understand the use or influence of expert advice in policy-making (see Christensen, 2021).One literature discusses the different purposes for which knowledge is used in policy-making, including problem-solving (i.e., when policy-makers have a genuine need for knowledge to solve a problem), symbolic use (i.e., when knowledge is used to gain legitimacy), political-strategic use (i.e., using knowledge as political ammunition to support predetermined policy stances) and enlightenment (when scientific theories in the long run shape how people think about societal issues) (Boswell, 2008;Hunter & Boswell, 2015;Weiss, 1979).Other work focuses on the extent of knowledge utilization or expert influence in policy-making and the mechanisms through which this influence occurs (see more below) (Campbell & Pedersen, 2014;Haas, 1992;Landry et al., 2003).There is considerable overlap between these approaches, and we draw inspiration from both.Yet, we take the latter approach and construct our argument around the notion of influence, since we are interested in the extent to which expert advice is incorporated into policy-making.
We define the influence of expert groups as the extent to which the arguments and recommendations presented by expert groups shape the problem understandings and policy solutions adopted by decision-makers.Expert groups present arguments about policy problems and recommendations about appropriate remedies, which may draw on evidence on the nature of problems and the likely effectiveness of solutions or on considerations of administrative or political feasibility.Expert groups are influential when these analyses, arguments and advice shape the understandings of decision-makers about policy problems and solutions (i.e., How important is an issue?What is causing the problem?What are appropriate solutions?)and the policy decisions they adopt to address a policy issue (e.g., a new policy measure or changes to the funding, generosity, target groups or duration of a policy) (cf.Haas, 1992).This notion of influence is related to -yet distinct from - Weiss's (1979) problem-solving model of knowledge use. 2  Influence can be both direct and indirect (Campbell & Pedersen, 2014) (see Figure 1).Direct influence entails that expert groups provide information and recommendations directly to decision-makers through reports, formal testimony, etc., which shape their thinking and the policies they adopt.Indirect influence occurs when expert group analyses and arguments shape the discourse and debate about an issue, which in turn may affect the understandings and solutions that decision-makers adopt.Indirect influence may again be divided into influence on what Vivien Schmidt (2010) calls communicative discourse, that is, debate about a topic that involves a broad range of actors in the public sphere, and influence on coordinative discourse among policymaking elites (academics, bureaucrats, key stakeholders, etc.) about problems and solutions.
Furthermore, influence can be conceptualized either as influence on a specific decision (Dür, 2008) or on multiple decision-making processes over time (Landry et al., 2003).Landry and colleagues, discussing the influence of academic research, criticize the first approach for being simplistic, since 'research findings generate many effects, not a single effect' and since 'decisions do not depend on a single piece of research, but on a series of research results converging toward one direction ' (p. 193).We concur.We see policy-making as a cumulative process: the problem definitions, policy principles and policy recommendations presented by expert groups may inspire and be developed further by subsequent expert groups, and expert group reports may have immediate influence on a policy but may also shape many other policies in the subsequent years and decades.
In this article, we are interested in examining both the direct influence of expert groups on the problem understandings and policies adopted by decision-makers (relationship a in Figure 1), and their indirect influence on coordinative discourse, namely the ability of expert groups to shape debates among policy-making elites (relationship b).As we will explain in the Research Design section, we measure direct influence using the number of citations to expert group reports in government white papers and indirect influence by counting citations to expert group reports in other expert group reports.Acknowledging the limitations of these quantitative citation measures, we also supplement our quantitative analysis with an in-depth qualitative assessment of a selection of citations.

Explaining variation in the influence of expert groups
We know from qualitative studies that some expert groups have a profound and lasting effect on policy discussions while others receive little or no attention and follow-up (e.g., Hunter & Boswell, 2015;Inwood & Johns, 2016).Different types of factors may account for the varying impact of expert groups.Features of the political environment may be crucial, such as the degree of political support for the expert group and the salience of the issue it addresses.The organization of the expert group may also condition its influence, including who sets up the group, who is appointed to the group, how its terms of reference are defined and how much independence and resources it is given.Moreover, influence may depend on how an expert group solves its tasks, such as the role played by its leader or the degree of consensus vs. disagreement within the group.
We focus on the second set of explanatory factors, namely how expert groups are organized.Theoretically, institutional design of expert groups matters both because it defines participation in the group and sets the parameters for its work and because it sends signals to the outside world about the group's credibility (Hesstvedt & Christensen, 2021).As we will argue, both mechanisms may condition the influence of expert groups.Studying institutional design features is also interesting for a more practical reason: since these aspects of expert groups are subject to design, they can be used deliberately to shape governance processes (Egeberg & Trondal, 2018).
We concentrate on three dimensions of the organization of expert groups as potential determinants of their influence: member composition, appointing ministry and resources.
Member composition.Expert groups often have hybrid membership, bringing together academics, civil servants, interest group representatives and sometimes even politicians (Christensen & Holst, 2017;Krick, 2015).Yet, the relative share of these different actor types varies: while some expert groups are dominated by academics, others are composed mainly of bureaucrats and interest group representatives.These differences in member composition may condition the group's influence.Different types of actors possess different kinds of knowledge and information that may be useful for decision-makers and different levels of credibility and authority that may strengthen or weaken the group's influence (Gornitzka & Sverdrup, 2011).
Starting with academics, we may expect the participation of academic experts to increase the influence of advisory groups, since academics possess both advanced specialized knowledge and scientific authority.Under conditions of uncertainty, decision-makers have a genuine need for expert knowledge to interpret and respond to a policy problem (Haas, 1992).Compared to other actors, academics may be better able to provide new knowledge and perspectives on the problem, and more objective and unbiased knowledge, which may be of greater use to decision-makers in making sense of a problem and finding relevant solutions.Moreover, since scientists are usually regarded as credible and independent providers of knowledge, expert groups dominated by academics may be seen as more authoritative and legitimate than other expert groups, boosting their influence.
Expert groups that include politicians may also be expected to have greater policy influence.Not only does the participation of politicians on advisory groups make it possible to find political compromises that can serve as a basis for reform (e.g., Grødem & Hippe, 2019); it also signals that the group's advice should be taken seriously and enjoys political backing.
As for groups with more bureaucrats and interest group representatives, we have no clear expectations.These groups may be better able to offer policy-relevant knowledge.Bureaucrats often have extensive knowledge about which policies work and about what is administratively and politically feasible.Interest groups can offer unique information about conditions on the ground and grass-root preferences, which can be crucial to formulating policy proposals that enjoy broad support (Gornitzka & Sverdrup, 2011).The presence of interest groups may also increase the legitimacy of expert bodies, at least if the groups are perceived as representative of society.However, expert groups dominated by bureaucrats and interest groups may be less able to come up with innovative policy solutions and may be gridlocked by conflicting interests.They may also face questions about the independence and objectivity of their advice.These mechanisms may undercut their influence.
Based on these theoretical considerations, we hypothesize that: H1: Expert groups with a greater share of academics will have greater influence in policy-making.
H2: Expert groups with a greater share of politicians will have greater influence in policy-making.
Appointing ministry.The influence of expert groups may also depend on which ministry appoints the group.Some ministries are stronger than others: they have greater analytical, institutional and political capacities, which enhances their power in the policy process (Garritzmann & Siderius, 2022).We would expect expert groups appointed by strong ministries to be more influential.Strong ministries can use their greater in-house analytical capacities to support expert groups.This may produce reports of higher quality, which can be conducive to influence.Strong ministries are also more authoritative.The expert groups they appoint may therefore be seen as more important and have a greater chance of being listened to.The quintessential example of a strong ministry is the finance ministry, which typically not only controls the budget but also has coordinative powers and responsibility for a range of economic policies (Garritzmann & Siderius, 2022).We therefore hypothesize that: H3: Expert groups appointed by the Ministry of Finance will have greater influence in policy-making than groups appointed by other ministries.
Resources.Finally, an expert group's influence may depend on the resources at its disposal.Groups with larger budgets, more staff and more time to prepare their report can be expected to have greater influence in policy-making.Greater staff resources give groups greater capacity to review existing research, collect and analyze empirical information and develop well-founded policy recommendations.Larger budgets and more time also allows groups to commission external analyses and research syntheses, invite external speakers, organize study trips, etc.This can result in reports that are more thorough, comprehensive and have a stronger knowledge base, which may be conducive to influence.Although we lack data on budgets, we would expect budgets and staff resources to be strongly correlated.We therefore hypothesize about the effect of staff resources and time on influence: H4: Expert groups with larger secretariats will have greater influence in policymaking.
H5: Expert groups with more time to write their report will have greater influence in policy-making.

A new approach to studying expert group influence
Existing studies of the influence of expert bodies rely mainly on processtracing or surveys.Following recent calls for methodological innovation in expert influence research (Christensen 2023), this article employs a novel approach for studying the influence of expert groups: citation analysis.Citation analysis involves examining relations between documents in the form of citations.The problem understandings and policy solutions adopted by decision-makers are typically expressed in policy documents that make the case for policy action and present concrete policy measures.These documents often use citations to build their argument and support their proposed measures, either through academic-style referencing or by mentioning sources in the text.Citations therefore provide information about what knowledge, arguments and recommendations a policy document builds on (Karseth et al., 2022).References to expert group reports in policy documents can thus be seen as an indicator of influence.
We here draw inspiration from citation analyses of related phenomena: Citation analysis is a mainstay of studies of the scientific impact of academic publications (e.g., Wuchty et al., 2007).It has also been employed to study the legal influence of different court rulings, which offers a relevant analogy since court opinions cite previous case law to build a legal argument (Fowler et al., 2007).Some recent studies use citation analysis to map the use of academic articles and other types of knowledge in government publications and impact assessments (Bornmann et al., 2016;Christensen, 2018;Karseth et al., 2022;Pattyn et al., 2021;Vilkins & Grant, 2017).But to our knowledge, ours is the first study to use citation analysis to examine the influence of expert bodies.Before describing and discussing our citation data and measures, we briefly present the empirical context.

Empirical context
We examine the influence of expert groups by analyzing data on Norwegian Official Commissions (Norges offentlige utredninger -NOU).These are temporary commissions appointed by the government -either by Cabinet decision or by a ministry.Most of these commissions have a policy advisory function: their task is to analyze a policy issue and offer policy recommendations.They are usually appointed early in the policy-making process, before the government has presented concrete policy proposals.For instance, a commission may be asked to analyze the challenges of growing international mobility of companies for the tax system and to propose reforms to address these problems.Yet, about one-quarter of commissions primarily have a law-drafting function (lovutvalg): they analyze and propose legal text for new or revised laws.
Commissions work outside the regular bureaucratic structure.Their work is defined by the terms of reference written by the government, which identifies the issues and questions to address (and not) and the type of advice sought.Commissions are made up of a chair and regular members, which are formally appointed by the government.They are supported by a secretariat, which reports to the chair.Commission members are drawn both from the public service and from outside organizations such as interest groups, academic institutions, private companies and political parties.Outside members participate as independent experts and cannot be instructed by the government.Public servants make up a large but deceasing share of commission members (see Figure 2).By contrast, the share of academics has increased considerably, whereas the share of politicians has dropped.Interest group representatives also make up a significant portion of commission members.The secretariats are predominantly staffed by civil servants.
Commissions on average work for about two years (recently somewhat less), and deliver their analyses and recommendations in a public report.Commission reports can form the basis for the subsequent steps in the policy-making process, namely the formulation of government white papers (stortingsmeldinger) and draft resolutions and bills (stortingsproposisjoner).White papers are documents submitted by the government to parliament where the government describes and justifies its policy plans within a specific area.These plans are then discussed in parliament.White papers do not contain proposals for new laws but often form the basis for subsequent legislative proposals.
The rationale for studying commissions is their central place in the Norwegian policy-making system.Norway has a tradition of knowledge-intensive policy-making, with a bureaucracy with high levels of specialist expertise and low levels of politicization and a thorough policy preparation process involving expert input.Official commissions are a 'cornerstone' of this governance model and a vital part of the decision-making process on major policy issues (Arter, 2008;Christensen & Holst, 2017).The number of official commissions appointed annually has decreased since the 1970s, but still remains substantial with about 20 commissions appointed per year in the 2010s (see Appendix Figure A1).This makes the Norwegian commission system far more extensive than for instance the systems of royal commissions/commissions of inquiry in the Westminster countries (Craft & Halligan, 2017).Moreover, commissions have had significant influence in many areas of Norwegian public policy, such as tax reform and climate policy (Lie & Venneslan, 2010;Tellmann, 2016).

Data
The analysis draws on unique data from three databases: (1) A database with information about all Norwegian Official Commissions that submitted a report between 1972 (when the report series began) and 2017, comprising 1545 reports. 3The database is compiled from pdf versions of reports available on the websites of the Norwegian government (regjeringen.no)and the Norwegian National Library (nb.no).It contains information about each commission (e.g., type of commission, appointing ministry) and its members (i.e., affiliation of chairperson, members and secretaries).(2) A database containing all citations in Norwegian Official Commission reports published 1972-2017.Citations were harvested from the pdf version of the reports.If the report had a reference list/bibliography, all citations in the list were collected.If not, all unique references in the footnotes of the report were gathered.We use only a part of this citation database, namely all citations from one commission report to another commission report (a total of 2611 citations).(3) A dataset containing all citations in Norwegian government white papers published 1999-2017, comprising 877 white papers.Since white papers present the rationale for new/revised policies, we look for traces of expert group influence on policy in white papers rather than in draft bills.The dataset was compiled from pdf versions of white papers available on the Norwegian government website (regjeringen.no).White papers are only available in digitized version from 1999 onwards, which explains the shorter data period.Citations in reference lists or footnotes were harvested in the same way as described above (dataset 2).In addition, we collected data on mentions of commission reports in the text of white papers.This was done through a pdf text search for the word 'NOU', which is the identifier for Norwegian Official Reports and is widely used to refer to these reports.There were 1907 citations to commission reports (including mentions) in the white papers.

Dependent variables
The unit of analysis in our study is advisory commissions.The outcome of interest is whether the advisory commission is cited by other commissions and white papers.We constructed two dependent variables to tap into the two dimensions of influence: (1) the number of white papers that cite or mention a given commission report (measure of direct influence) and (2) the number of other commission reports that cite a given commission report (measure of indirect influence on debates among policy-making elites).The measures are analogous to a Google Scholar count of how many times an academic publication is cited.For instance, a commission whose report is cited by two government white papers and five subsequent commission reports gets the score 2 on DV1 and 5 on DV2.Both dependent variables are count variables.Our measures do not distinguish between how many times a commission report is cited by a given white paper or report.Although the number of times a commission is cited by a given report could provide additional information about its influence, we do not have this information since we mostly collect citations from reference lists. 4 We further discuss and assess the validity of our citation measures in the sub-section 'Validity and validation of citation measures'.

Independent variables and controls
We include three sets of independent variables in our analysis.First, we include variables for the share of different actor types among the members of a commission: share of academics, share of politicians, and share of interest group/business representatives. 5These are continuous variables.An academic is anyone working in an academic position at a university/university college or as a researcher at an independent research institute.People working for think tanks or consultancy firms are not counted as academics.We define a politician as a person who is elected to office at the national/regional/municipal level, and/or who is listed in the commission report as a representative of a political party.Interest group/business representatives are defined as persons representing organized interests or a company.'Members' here include the commission chairperson and regular commission members, but exclude members of commission secretariats.
Second, to examine the effect of ministry strength on expert group influence, we include a dummy variable for commissions appointed by the Ministry of Finance.The Ministry of Finance enjoys a powerful position in the Norwegian government, given its broad portfolio -which includes budget policy, economic policy, financial market regulation and parts of administrative policy -and the weak coordinating role of the Prime Minister's Office (Hesstvedt & Christensen, 2021;Lie & Venneslan, 2010).
Third, we include two variables that measure different types of resources: secretariat size, measured as the number of secretaries supporting a commission, and the time allotted to the commission to prepare its report, measured as the number of years between the dates of appointment and report submission.A deadline for report submission is usually stipulated in the terms of reference, although this deadline is sometimes extended.
We also include controls for possible confounders.First, we control for whether the commission produced a law proposal, as the influence of lawdrafting commissions may differ systematically from policy advisory commissions.Second, we control for whether a commission was appointed by royal decree.Such commissions are approved at the cabinet's weekly official meeting and often deal with major issues that cut across ministerial boundaries, and may thus be more likely to have influence.Finally, we control for whether the commission produced multiple reports, since these commissions may differ systematically from one-off commissions in influence.
Descriptive statistics are provided in Appendix Table A1.Note that whereas the analysis of commission citations covers all commissions that delivered a report between 1972-2017 (1545 commissions), the analysis of white paper citations only includes commissions that published a report between 1998-2017 (450 commissions).On average, commissions were cited by 3.01 white papers and 1.69 other commissions reports.The frequency distribution on both dependent variables is highly right-skewed and includes many zero values (i.e., many commissions are never cited) (see Appendix Figures A2 and A3).

Models and analytical strategy
The regression analyses are based on hierarchical negative binominal logit models.First, since the dependent variables are count variables with many zero values, we use negative binomial regression to estimate the effect of the independent variables.Second, we use time-hierarchical models that enable control for spurious effects related to time trends (see Appendix A for details).We thereby filter out variance that is due to between-year differences and only look at variance between reports published in the same year (this is a common approach in citation analyses, e.g., Wuchty et al., 2007).

Validity and validation of citation measures
Our citation measures have some distinct advantages: they are objective measures of influence rather than relying on perceptions of influence; they allow for comparison across a large number of commissions; and they allow us to examine the cumulative influence of a commission report on multiple white papers and subsequent commission reports, rather than on one specific policy decision.
Yet, using citations to measure impact has well-known limits (Vilkins & Grant, 2017;Christensen 2023).First, citations do not tap into the substance of the recommendations of expert advisory reports or the policy decisions adopted.Citations also provide little information about the nature or strength of the relation between two documents.White papers may cite advisory reports because they are genuinely inspired by them or just to give the impression that policies are evidence-based (cf.Weiss, 1979).And a white paper may be profoundly influenced by some of the reports it cites and marginally influenced by others.Citations may even be used to reject the recommendations of advisory reports.Conversely, an advisory report may influence a policy decision without being cited in a decision document, for instance if influence runs through more informal channels such as the consultative processes of ministries.However, aggregate citation measures like our dependent variables are less sensitive to many of these problems than single citations.
To address these concerns, we carried out additional analyses to validate our citation measures (see Adcock & Collier, 2001): First, since our two dependent variables are meant to tap into two dimensions of the same concept (influence) that we expect to be positively correlated, we examined the correlation between these variables.Remember that our two citation measures are based on data from different sources, so there is no common source bias.The two dependent variables are strongly positively correlated (0.58) (see Appendix Figures A5 and A6), which strengthens our confidence in the validity of the measures.
Second, and more importantly, we qualitatively analyzed a sample of citations to examine how citations to commission reports are actually used in citing documents.We drew a random sample of 50 citations to commission reports in government white papers and 50 citations to commission reports in other commission reports.We then searched the citing documents for every mention of the cited report in the text, and for each mention we coded inductively how the citation was used.The codes were then aggregated for each of the 100 citations.See full list of codes with frequencies and examples in Appendix Table A2.
We find that about half of citations to commission reports in white papers were used to describe government adoption or follow-up of commission recommendations (24 out of 49 citations) (see Figure 3).Most other citations (22 citations) referred to recommendations, arguments, empirical findings and factual information from a commission report without explicit information about government follow-up.Only 3 citations were casual references, and no citations were clearly used to reject commission recommendations.As for citations to commission reports in other commission reports (Figure 4), nearly half of citations (21 out of 47) were used to describe that the cited commission's recommendations were adopted by the citing commission and/or had been adopted in policy measures or laws.A further 15 citations referred to arguments, factual information or empirical findings from  reports, whereas 9 citations were casual references and 2 citations were used exclusively to reject the cited commission's recommendations.
While this analysis shows that citations to commission reports are used in manifold ways, it indicates that citations often tap into the kind of influence we seek to examine, namely that expert group arguments and recommendations shape the problem understandings and policies adopted by the government.

Results
Moving to the results of the quantitative analysis, we first look quickly at which commissions are cited the most (see Figures 5 and 6).Most frequently cited in white papers are the two reports from the Productivity Commission, a recent commission that advised the government on how to maintain economic growth, productivity and welfare.This commission was appointed by the Ministry of Finance, as was the third most cited commission, the Commission on Redistribution and Social Inequality.The Power and Democracy Study from 2003, which analyzed the state of Norwegian democracy, is most often cited in other commission reports.Also extensively cited is the 1989 commission on public sector reorganization, which introduced New Public Management in Norwegian governance and was named the most significant commission report ever in an informal ranking by the Norwegian weekly Morgenbladet (Lien & Gundersen, 2014).Commissions on topics like criminal sanctions, health care, and higher education policy are also near the top of the list.We then turn to the analyses of whether member composition, resources and ministry strength matter for expert group influence.We first present scatterplots showing how the number of citations to a commission report varies with the commission's member composition, by year of report submission (Figures 7 and 8). Figure 7 shows the number of citations received by commissions with and without academic members.It suggests that commissions with academics have more influence on both policy proposals (panel A) and policy debates (panel B).For instance, panel A shows that academic commissions were cited about twice as often by white papers than other commissions during almost the entire period 1998-2017.A similar pattern applies to commissions containing politicians, which are more often cited in both white papers and other commission reports than commission without politicians (Figure 8).
Scatter plots of citation numbers depending on appointing ministry and resources suggest that whereas Finance-appointed commissions are not more cited than other commissions, commissions with larger secretariats and more time at their disposal tend to be cited more often (Appendix Figures A7-A9).
Next, we present the results from the regression analyses (Tables 1 and 2).We first run separate models for each set of independent variables (models 1then a model with all the independent variables (model 4) and finally a model that also includes controls (model 5).We also visualize the results in a coefficient plot (Figure 9), which displays the results from the full regression model with control variables (model 5).Coefficients whose confidence  interval does not cross the dotted line (0) are significant at the 5 per cent level, indicating a relationship between the variables of interest.In the text, we discuss coefficient size in terms of predicted probabilities.We expected that commissions with a greater share of academics would have greater policy-making influence (Hypothesis 1).Regression model 1 shows that commissions with more academics are cited significantly more often, both in white papers and other commission reports.For example, while the predicted number of citations to a commission without academics in other commission reports is about 2, it is 4 for a commission consisting solely of academics. 6In the full model including controls (model 5), the positive association between academic membership and the number of citations remains strong and significant (p < 0.001) for citations in other commission reports, but is no longer significant for citations in white papers.H1 is thus not fully supported.
We also expected commissions with more politicians to have greater influence (Hypothesis 2).Regression model 1 shows that commissions with a greater share of politicians are cited significantly more often in white papers and other commission reports.A commission that consists solely of politicians has a predicted probability of receiving 6 citations by other commission reports, compared to 2 citations for commissions without politicians.These effects remain statistically significant (p < 0.05) in the full model (model 5).H2 is thus supported.Furthermore, the share of interest group/ business representatives on a commission is negatively associated with the number of citations in later advisory reports but has no effect on the number of white paper citations.
In Hypothesis 3, we expected commissions appointed by the Ministry of Finance to have greater policy-making influence than other commissions.We do not find support for this hypothesis.Even though Finance-appointed commissions are cited significantly more often by other commission reports in model 1, this effect almost disappears in the full model with controls.And finance ministry commissions are not cited more often than other commissions in white papers.
The size of the secretariat, by contrast, matters greatly for commission influence (Hypothesis 4).Commissions with larger secretariats are cited Note: Level 1: commissions, level 2: years.DV: number of citations to a commission in subsequent commission reports.Standard errors in parentheses.*p < 0.05, **p < 0.01, ***p < 0.001 more in both white papers and other commission reports, and this correlation is strong and statistically significant (p < 0.001) across all models.For example, a commission with one secretary has a predicted probability of receiving about 1 citation by later commission reports, compared to 3 citations for a commission with 8 secretaries.These results support H4.
Time granted to a commission (Hypothesis 5) has a positive and statistically significant effect (p < 0.001) on citations in later commissions reports across all models, but there is no statistically significant relationship between time and citations in white papers.H5 is thus not fully supported.
Our regression models also show that commissions appointed by royal decree are cited significantly more often, as expected.By contrast, law-drafting commissions and commissions that published multiple reports are not cited significantly more or less than other commissions.

Discussion and conclusion
This article has sought to measure and explain the varying influence of expert groups on policy-making.Using the number of citations to expert group reports in government white papers and other advisory reports as measures of influence, it has investigated the determinants of influence through a quantitative analysis of Norwegian advisory commissions.The analyses show that the influence of expert groups varies greatly, and that this variation to some extent can be explained by differences in the organization of these bodies.
First, we find that the member composition of expert groups matters, although the empirical picture is somewhat mixed.The share of academics on Norwegian expert groups has increased sharply over time, and our results show that commissions with more academics are more likely to be cited by other commission reports.Yet, they are not significantly more likely to be cited by government white papers.This suggests that while scientist-dominated groups have major influence on policy debates among elites within the commission system, this influence fades in the later stages of policy formulation.
The participation of politicians on a commission seems to generate both direct and indirect influence, as it has a significant positive effect on citations both in white papers and in other commission reports.It is therefore paradoxical that political membership on commissions has become increasingly rare in Norway: politicians account for only 2 per cent of members over the last 20 years.This may suggest that while commissions with politicians are a powerful tool to shape policy through political consensus-building (Grødem & Hippe, 2019), such commissions are only appointed on select issues and when the political environment is favorable.As for commissions with more interest group/business participation, they are cited significantly less by other commission reports (but not significantly less by white papers), which suggests that interest group arguments carry less weight than scientific arguments in the coordinative discourses among policy-making elites.
Overall, these results do not support the argument that only expert groups with scientific knowledge and authority will be listened to in policy-making.This finding should assuage fears about an excessive 'scientization' of policymaking that crowds out other legitimate concerns and interests in a democracy.Even if scientists have become more numerous on advisory commissions and shape discussions among policy-making elites, their input is balanced with other concerns when decisions are made.But neither do the findings support the opposing 'two communities' view that scientists and policy-makers are unable to communicate with each other (Newman et al., 2016).The results suggest that advisory commissions to a considerable extent manage to bridge the gap between science and policy-making, which is reassuring for those who see evidence-informed policy-making as a guarantee for high-quality public policies (Head, 2016).
Second, the results do not support our expectation that commissions appointed by the finance ministry are more influential.This may be because we are looking at average effects across many rather different commissions.For instance, while the three commission reports most frequently cited in white papers were appointed by the finance ministry, the ministry also set up many rarely-cited commissions dealing with narrow technical issues.
Third, we find that resources matter for the influence of expert groups, especially staff resources.More staff gives commissions greater capacity to collect and analyze data and write thorough reports, and the results suggest that this is crucial for both direct and indirect influence.We also see that commissions with more time at their disposal have greater indirect influence on policy-making debates, but not more direct influence.While the positive relationship between resources and influence is not surprising, it illustrates that organization matters for the influence of expert advisory bodies.From a design perspective, it implies that resourcing can be an important tool for governments to boost (or limit) the impact of an expert group.
These findings nonetheless come with limitations.The results may partly reflect confounding factors that affect both the design of commissions and their influence, such as the salience of the policy or the government's reform willingness.For instance, governments may be more likely to appoint academics and politicians to commissions dealing with salient issues, and commissions on salient issues may be more likely to have influence.Yet, we partly address this problem by controlling for whether the commission is appointed by full Cabinet decision, since such commissions usually deal with major policy issues that have the priority of the government.
Another issue, as discussed above, is the validity of our citation measures.While these validity concerns cannot be eliminated, we have tried to mitigate them in three ways: by using citation counts rather than single citations; by triangulating two citation measures of influence drawn from different data sources; and most importantly, by validating our measures through a qualitative analysis of a sample of citations.The results of these analyses strengthen our confidence that the citation measures actually tap into expert group influence.
The article has made two original contributions to research on expertise and public policy-making.First, empirically, it is the first quantitative analysis of the determinants of expert group influence, drawing on unique large-scale data on expert groups.Our findings complement insights from qualitative studies of commissions and their influence.They also go beyond existing quantitative studies that have investigated what explains access to expert groups but not what explains their influence.
Second, it has offered methodological innovation.Existing research has repeatedly called for analyses of the influence of advisory bodies but has also highlighted the methodological difficulties involved (Campbell & Pedersen, 2014;Hustedt & Veit, 2017).Seeking to overcome some of these obstacles, the article has introduced a new citation measure of influence.Citation analysis offers a promising way forward for systematic, large-n analyses of the policy impact of expert bodies (Christensen 2023), which can easily be applied to expert bodies in other geographical settings.Policy documents are often readily available online, making it possible to collect and analyze citations to publications from various advisory bodies.
A question for future research is therefore whether the findings from the Norwegian setting can be replicated elsewhere.Some of our findings may be transferable.For instance, we would expect greater staff resources to increase the potential influence of expert groups regardless of the setting.Other findings may reflect specific features of the Norwegian policy-making system and are not easily generalizable.For example, whether academic participation helps or hinders expert group influence depends on how receptive policy-makers are to scientific knowledge.Norwegian policy-makers seem particularly likely to listen to scientists, given Norway's tradition of knowledge-intensive policy-making, specialist merit bureaucracy and low levels of politicization.Policy-makers are likely less receptive to academic input in countries with more politicized and less expert bureaucracies, such as Italy or Spain.
Another lesson from our study is the importance of methodological triangulation for identifying influence (Dür, 2008).To advance research, it is crucial to further triangulate citation measures of influence with measures based on qualitative interviews, survey data or qualitative or quantitative text analysis.Another promising avenue for future research is to examine expert group influence through citation network analysis (see Fowler et al., 2007).We used simple counts of inward citations to measure the influence of reports, discarding all other information about the citation links between documents.Network analysis offers multiple measures that can exploit this information to capture the importance of a given document in a citation network.

Notes
1.At least in countries with well-established expert group systems.2. In Weiss's problem-solving model, 'research provides empirical evidence and conclusions that help solve a policy problem' and 'clarifies the situation and reduces uncertainty, and therefore, it influences the decision that policymakers make' (Weiss, 1979, p. 427).However, our notion of expert group influence differs from Weiss's model in some important ways: First, our concept is not about the influence of single research studies but about the influence of expert group advice that is based on a range of evidence and other considerations.Second, it does not assume that expert groups somehow 'solve' a policy problem simply by providing the information that decision-makers miss.Rather, we argue that decision-makers may or may not listen to the advice of expert groups for a variety of reasons (see subsection 'Explaining variation...'). 3. The Norwegian Inquiry Commission database is available for download from the Norwegian Agency for Shared Services in Education and Research (see Hesstvedt & Christensen, 2023).Complete replication material for this article is accessible from Harvard Dataverse, see https://doi.org/10.7910/DVN/XTUYE0.4. Additional analysis of a sample of 100 citations showed that commission reports were cited 1-20 times per citing white paper or commission report, with an average of about 3 times.5. To avoid overdetermining the model, we do not include a variable for the final major category of members: public servants.6.All predicted probabilities based on model 1 in Table 2. more recent expert group reports and white papers are far more citation-intensive than earlier ones, which pulls in the opposite direction.
The values on key independent variables are strongly time-dependent, too: The share of academics on commissions has increased markedly, while the share of politicians has plummeted.Commission secretariats have expanded, while the time commissions have to prepare reports has become shorter.This means that the year of report submission may confound the relationship between our independent and dependent variables.
We therefore apply a mixed-effects multi-level model where commission reports are nested within the year of submission.In this way, we filter out variance that is due to between-year differences and only look at within-effects between reports published in the same year (studies of citations of academic articles take a similar approach, cf.Wuchty et al., 2007).

Figure 2 .
Figure 2. Share of Norwegian Official Commission members from different member categories.Annual averages with trend line (lowess).N = 1545 commissions.

Figure 3 .
Figure 3. Qualitative validation of citation measures: citations of commission reports in white papers.N = 50.

Figure 4 .
Figure 4. Qualitative validation of citation measures: citations of commission reports in other commission reports.N = 50.

Figure 5 .
Figure 5. Commission reports most frequently cited in white papers.

Figure 6 .
Figure 6.Commission reports frequently cited in other commission reports.

Figure 7 .
Figure 7. Citations in white papers (panel A) and commission reports (panel B) of academic commissions (at least one academic -solid lines) and non-academic commission (no academic members -dotted lines).Average number of citations per commission, per year of commission report submission.Lowess trend lines.

Figure 8 .
Figure 8. Citations in white papers (panel A) and commission reports (panel B) of commissions with political members (at least one politician -solid lines) and commissions without political members (dotted lines).Average number of citations per commission, per year of commission report submission.Lowess trend lines.

Figure A1 .
Figure A1.Number of Norwegian Official Commissions appointed annually.Scatterplot with trend line (lowess).

Figure A4 .
Figure A4.Number of citations over time.X-axis = year of submission of report, y-axis = number of citations, averaged per commission per year.Scatter plots with fitted trend lines (lowess).

Figure A5 .
Figure A5.Number of citations to a commission in white papers and commission reports (DVs).Scatter plots of commissions (N = 532) with fitted trend line.Total number of citations in commissions on y-axis and white papers on x-axis.

Figure A6 .
Figure A6.Correlation (Pearson's r) between the number of citations to a commission in white papers and in commission reports (DVs).Correlation per year (annual average) with fitted trend line using lowess.

Figure A7 .
Figure A7.Are commissions appointed by powerful ministries cited more?Citations in policy reports (A and B) and white papers (C and D) of commissions appointed by the Ministry of Finance and other ministries.Average number of citations per commission, per year of commission report submission.Trend line with locally weighted averages (lowess).

Figure A8 .
Figure A8.Are commissions with more staff resources cited more?Citations in white papers (panel A) and commission reports (panel B).Number of secretaries per commission on x-axis, citations on y-axis.Trend line with locally weighted averages (lowess).

Figure A9 .
Figure A9.Are commissions that are granted more time cited more?Citations in white papers (panel A) and commission reports (panel B).Time from appointment to submission of commission report (in years) on x-axis, citations on y-axis.Trend line with locally weighted averages (lowess).

Table 1 .
White paper citations.Results from hierarchical negative binominal regressions.

Table 2 .
Commission report citations.Results from hierarchical negative binominal regressions.

Table A2 .
Qualitative analysis of citations: codes, frequencies and examples.The commission delivered two NOU reports … The commission's recommendations and proposals are part of the basis for government white paperSt.meld.30(2003St.meld.30( -2004))'.

Table A2 .
Continued.The County Governor has since 2009 carried out additional audits of support and child houses.The Country Governor has found unsatisfactory conditions in 13 cases.These are particularly linked to handling of medications, competence and training of personnel, lack of leisure activities and the state of the buildings (NOU 2016: 17)'.Casual reference 9 Commission only listed in reference list, not mentioned anywhere in the text.Rejects argument/recommendation 2 'There are arguments for not taxing normal returns, cf.the Bergo Commission's report(NOU 1996: 17).On the other hand, there are also arguments for maintaining taxation of normal returns on capital … The commission [citing commission] has concluded that it is not an option to remove taxation of normal returns on capital in Norway'.