A Three-Stage Evaluation of a Deliberative Event on Climate Change and Transforming Energy

Formulating deliberative evaluation tools indicates an important new field for the practitioner of deliberative design due to the increasing adoption of deliberative policy mechanisms by governmental and non-governmental organisations. Evaluation tools aid the design, effectiveness and deliberative integrity and legitimacy of these mechanisms. Relatively few studies though have reflected on how to conduct evaluation. Here, we report on our formulation of a three-stage approach to deliberative evaluation, which we applied in an independent evaluation of a Courageous Conversation on Climate Change and Transforming Energy (March 2007), designed by the Ethos Foundation, Queensland, Australia. Overall, we found our schema successful in identifying both the positives and negatives of the design, of what could be applauded and what needed revision. More broadly, we advance that it offers a useful approach for practitioners to develop further, especially in getting the balance right on evaluating process and outputs, to which we have added inputs in reflection of deliberative design advances.

Deliberative experiments, building on a long history of civic engagement (for example, Wagenet and Pfeffer 2006), are now emerging for critical analysis and evaluation (Hindmarsh and Du Plessis 2008).They reflect new forms of enhanced citizen involvement in decision-making.The participatory or deliberative 'turn' for good governance-especially concerning science and technology management and environmental sustainability-has especially been popularised over the last decade (Dryzek 2000, Jasanoff 2004, Hartz-Karp 2006, McCallum et al. 2007, Dryzek 2006, Rose 2007).
In early 2007, an opportunity arose for us-as an informal research group focusing on environmental deliberative governance-to undertake an evaluation of a participatory engagement technique that seemed to fit the transition to deliberative design in its title and advertised approach stressing inclusiveness, dialogue and community participation.Called 'Courageous Conversation 2007: Transforming Energy-Inverting Power, Transitioning to Renewables and Preparing for Climate Change' (26-30 March 2007), 1 the technique was designed by the Ethos Foundation (South East Queensland), a non-profit ecological adult learning organisation.
Our evaluative approach was based on assessing the quality of the overall deliberative (or 'inclusive participatory') design, 2 informed by principles, ideals and practices of participation and deliberation, coupled to the debate in the participatory literature about the need for balanced evaluative criteria, and by the particular design of the Courageous Conversation technique.The structure of the paper is organised as follows.We set the scene by introducing the Ethos Foundation and the Courageous Conversation multi-layered design.Second, we discuss deliberative design evaluation, and outline our evaluative framework and criteria.Third, we discuss our application of those criteria to the Climate Change and Transforming Energy Courageous Conversation and report our findings.
However, before continuing, it is apt to stress that our evaluation was conducted independently of the Ethos Foundation.We approached the Ethos Foundation with the idea, and the Ethos Foundation welcomed the opportunity for their design to be independently evaluated as part of design process.The only financial support for the exercise came from a Griffith University research grant, which funded accommodation and meals in attending the five-day Conversation program.The first opportunity the Foundation had of learning more about our evaluation was when key Foundation officers were given a copy of this manuscript to comment on.Then, and prior to that, no attempt was made to influence the independence of our evaluation or the reportage here.

THE ETHOS FOUNDATION AND COURAGEOUS CONVERSATIONS
To cultivate new thinking and values and develop action for an ecologically sustainable future, the Ethos Foundation (2006a) applies interactive and holistic approaches to learning, education and sustainability for education programs; government, industry and community forums; learning networks; conservation and restoration projects; special events in the creative arts; and consulting services for business and government.
Government, industry and community forums centre on the Ethos Foundation's development of an engagement practice or technique called 'Courageous Conversations', aiming to bring together a rich diversity of social actors to develop inclusive partnership approaches to decision-making.The practice is dialogue based and multidisciplinary, and offers a first (major) step to facilitating broader public deliberation exercises.Typically, Courageous Conversations bring together leaders, emerging leaders, key players and stakeholders, from government, business, science, policy, the arts, education and the community to discuss big environmental and social policy issues facing South East Queensland (SEQ) and Northern New South Wales (NNSW), which together form a bioregion (Ethos Foundation 2006a, 2006b).
The Conversations are widely advertised in local newspapers, flyers in the SEQ and NNSW areas, word of mouth, and through various networks.A limit of sixty participants is invited to a Conversation, which involve five-day programs aiming to engender 'deep dialogue and meaningful interaction' (interactivity).Key participatory concepts include 'multidisciplinary dialogue', 'critical and personal reflection', 'strategic conversation' and 'community building'.As part of our program researching deliberative governance, which is especially interested in whole-of-society approaches to deliberation, our intention was to evaluate this first major step of the Ethos Foundation process, and later, evaluate any downstream community engagement that might be an outcome of this process.
Reflecting its big policy intent, the immediate aim of the Climate Change and Transforming Energy Courageous Conversation was to create/encourage 'a collaborative, high level, multidisciplinary regional taskforce to plan for regional climate change impacts, and an effective transition process towards deep sustainability' (Ethos Foundation 2007c).In addressing this aim, the design elements for the weeklong exercise involved both deliberative and nondeliberative elements.Our evaluation focused on the deliberative elements, which occupied most of the program.The first deliberative element comprised 'Inquiry Groups' held on the Monday, Tuesday, Thursday and Friday.To ensure a diversity of participants was represented in each group, the organiser assigned participants into the small self-managed dialogue groups of six-eight members, with also featured a continuity of core membership and facilitator over the week.
The second element, held on the Wednesday only, comprised a special multi level, actor and disciplinary 'think tank' forum to deliberate upon the adverse effects of climate change and how they might be addressed downstream at regional, sectoral and community levels.The forum was targeted specifically to industry, government and community leaders (Ethos Foundation 2007c), but involved all members of the Inquiry Groups as well.'Think Tank Working Groups' (of 15-20 participants) were a key feature of the forum.These elements typically had co-convenors with responsibility for focusing and guiding discussion towards outcomes reflecting the views of all participants.The third element, held on the Thursday afternoon, comprised Open Space working groups (of 3-10 participants), meeting around topics proposed by participants to attract other participants to join in deliberation on them.The non-deliberative elements, held at the start or end of each day included Opening the Circle sessions, Guided Reflections, a Community Choir and Community Gatherings.These were designed to stimulate inclusiveness, community building, trust, and equitable and informed knowledge processing and interactivity.
Figure 1 shows diagrammatically the arrangement of the key elements, especially those pertaining to deliberation, as well as the process linkages between them (in the form of arrows) in the weeklong program.The linkages indicate the daily progress of the sessions of each type of element, and each session's influence on downstream sessions.For example, the daily engagement of the Inquiry Groups and their outputs were reflected upon in a guided reflection space at the start of most days.Guided reflection provided opportunity for participants to better understand each other's positions and identities. 3The catalyst presentations of key speakers (from science, engineering, and renewable energy) provided participants with stimulating and necessary background information to enable 'informed' deliberations.Because Day 3 was somewhat separate (as convening the one-day 'think tank' forum), the dashed lines shown in the figure (from Day 3 to Day 4) indicate that while there was some connection to Day 4 elements, these were not as strong as other connections.For example (as shown in the figure), the main influence for Day 4's Inquiry Group (Round 3) was Day 3's guided reflection session, which, in turn, was influenced/informed by Day 2's Inquiry Group (Round 2) and catalyst panel sessions, and so on.

DELIBERATIVE DESIGN EVALUATION
The starting point for designing our evaluative framework was to reflect generally on the academic literature and theory about deliberation, deliberative design, deliberative governance, and, more specifically, deliberative engagement process.
In emphasising 'meaningful public participation ', McGurk et al. (2006: 810), highlight that deliberative processes diverge sharply from traditional civic engagement, which usually involves an over reliance on passive involvement techniques (such as surveys, reports, press releases, and news conferences, which do not promote communication and deliberation), insufficient resources for participants, lack of early involvement in the planning cycle, information and communication deficiencies … [to] insufficient breadth of involvement.
Tracing what deliberative process and evaluation meant chronologically, the deliberative democracy theory of Cohen (1989) was initially instructive, being characterised by five main features: deliberation is ongoing; all members of a 'democratic association' work within a framework that makes deliberation possible, as free deliberation endorses legitimacy of process; deliberation is pluralistic and consists of a diverse range of people, ideas and preferences; and the focus is on process rather than on gaining results, as deliberation characterises legitimacy, not the results.Finally, all members recognise that each member has deliberative capacity, and therefore are equals in a process that seeks rational consensus.
Following Cohen, Fiorino (1990) advanced a 'preliminary' evaluative framework of relevance to deliberation.Fiorino evaluated several institutional mechanisms for citizen participation in environmental and risk policymaking including public hearings, public surveys and citizens' review panels.From participation theory, Fiorino devised four 'democratic process criteria' that were to be seen in continuum: (1) access for direct participation of amateurs (citizens) in decision-making, ( 2) the extent to which citizens are enabled to share in collective decision-making, (3) provision of a structure for face-to-face deliberative discussion over a period of time, and ( 4) capacity of citizens to participate on some basis of equality with administrative officials and technical experts.However, Fiorino's assessment guided by these criteria was more a critical discussion than rigorous application of the criteria and there was no specific evaluation of deliberative process aspects of the mechanisms.
In 1999, participatory design and evaluation became more informed by environmental decision-making researchers Tuler and Webler (1999), who-from an analysis of forest policymaking that asked participants about their ideas of 'good' public participation process-identified seven categories of evaluative principles: access to process, power to influence process and outcomes, access to information, structural characteristics to promote constructive interactions, facilitation of constructive personal behaviours, adequate analysis, and enabling of future processes.By the early 2000s, though, most research on evaluative criteria had come to emphasise process and outcomes, although tempered by a paucity of empirical examples in the academic literature.Rowe and Frewer (2000: 10) also found that most criteria was procedural rather than substantive in relating to what made for effective process, rather than how to measure effective outcomes.In turn, Beierle and Konisky (2000: 589) found most evaluations either assessed the quality of participatory process or 'the extent to which particular parties achieved their own specific goals in participatory decisionmaking'.
As part of a push to improve both the substantive and procedural quality of participatory decision-making (Beierle and Konisky 2000: 587), Rowe and Frewer (2000) set about defining better what was meant by 'effectiveness' in participation.They specified a set of generic criteria on participatory processes, categorising them into public acceptance and good process criteria.Public acceptance criteria included representativeness of the population of the affected public and their views; independence where participation should be conducted in an independent and unbiased way; early public involvement as soon as value judgments became salient with regard to the issue under question; influence where the output of the procedure genuinely impacted on policy; and transparency to both involved and interested parties.
In turn, good process criteria included resource accessibility, where public participants had access to appropriate resources-including information, material and time-to enable them to achieve objectives; and task definition, which amounted to specified well-defined tasks for participation with regard to the scope of the exercise, its expected outcome/s, and mechanisms of procedure.In addition, structured decision-making focused on appropriate organisational and procedural mechanisms for structuring and displaying the decision-making process, including efficiency and transparency of process and documentation of process.For the latter, Rowe and Frewer cited a range of mechanisms based on deliberative approaches centred on dialogue, with the role of facilitation emphasised (see also Horlick-Jones et al. 2006).
However, by 2007, little advancement in deliberative evaluation had occurred.Rowe et al. (forthcoming) conjectured two main reasons.First, that public engagement was more often seen by agencies as an end in itself (for example, in reflecting regulatory policy or organisational accountability) than as a means to an end.An example of the latter, Beierle and Konisky (2000: 589) suggested was a better functioning environmental management system.Later, Rowe et al. (forthcoming) commented: 'It is perhaps no surprise that when evaluations are conducted, they are often done in a rather informal and subjective manner, in which the evaluators (often the same people as those conducting the exercise) limit themselves to commenting upon apparent positives that emerge from the considered process.'The second reason given was that evaluation was difficult, especially in the absence of a widely accepted evaluative framework.
Against this background, we found the Ethos Foundation's Climate Change and Transforming Energy Courageous Conversation comprised a three-stage design of inputs (information), process (an ensemble of deliberative and nondeliberative elements and processes) and outcomes.For our proposed evaluation, we aligned to this three-stage design-especially utilising Agger and Lofgren's (2006) three-phase model focused on assessing the democratic effects of networks.To do this, we first reorganised Rowe and Frewer's criteria.Where Rowe and Frewer cast inputs as part of public acceptance criteria, we aligned inputs to the first stage in the Courageous Conversation approach.We then pulled apart the intermingling of process and outputs in Rowe and Frewer's framework, to rearrange process as a distinct phase in the middle of our framework, between inputs and outputs.Specific process criteria focused on participant's interactions in deliberation (see also Renn et al. 1995).Although some might describe process as the whole engagement procedure, we found our three-stage partitioning provided more direction and scope for evaluation.It also addressed the gap in the literature on how to conduct empirical evaluation involving designs clearly incorporating different stages of deliberation, while also recognising that the three stages we had identified, in following the design under investigation, represented an interactive process.As the Ethos Foundation stated: 'Transforming Energy was designed to weave a holistic balance between diverse learning processes that stimulate and support each another.'

EVALUATIVE CRITERIA
In devising our criteria we drew on deliberative design literature, but because this literature was limited we also drew on public and planning participatory literatures.We formulated seven criteria for the inputs stage, twenty-six for the process stage, and four for the outputs stage (see Appendices A and B for detail).Although this weighting suggests some imbalance, our process criteria reflected the difficulty of formulating process criteria, vis-à-vis inputs and outputs criteria, because of the complexities of human interactions in undertaking deliberation.In other words, process is a lot harder to measure.
Our first key criterion for input into deliberative processes measured diversity of participants (for example, in demographics) (Sharp 2002).Diversity provides a multiplicity of knowledge and also works against the possibility that one interest group may dominate engagement with their views, values or bias.Other input criteria measured opportunity for participation, and the changing influence of new arrivals throughout the week.In association, four key criteria, relating to both the inputs and process phases measured: (1) ownership of the agenda through participants' direct involvement in agenda setting; (2) participants' shared commitment to appropriate terms of deliberative association, though regular attendance in process engagement, attempting to uphold deliberative ideals, and not in disrupting process (Cohen 1989); (3) facilitator training, as facilitators have significant influence over deliberative process especially inclusion and equality of participants (Cohen 1989, Flyvbjerg 2001, Sharp 2002, Agger and Lofgren 2006, Hendriks 2006); and (4) availability of relevant, open and understandable information (Agger and Lofgren 2006).
Turning more to process criteria, a key criterion measured the capacity building of participants to understand background information and each other's information (Gonzalez and Healey 2005).Other process criteria included measuring engagement in collective thinking, dialogue, and the 'discovery' of creative solutions (Hitchcock et al. 2001).This was furthered through evaluating participants' engagement with understanding the underlying causes and structural issues of the problem at hand (Pinto 2003), which, here, related to scientific and political aspects of climate change with regard to transforming energy systems (that is, from fossil fuel driven ones to renewable ones).Such engagement improves awareness of participants' own thinking, reasoning and value-positions (Gonzalez and Healey 2005), which heightens collective thinking and dialogue.Further process criteria measured participant attendance and inclusion, information access and translation, participant understanding and learning, and the free consideration of proposals and creative solutions.The final key criterion measured to what extent, trust was generated between participants.Trust is closely related to effective and open dialogue (Innes and Booher 2004).Finally, outputs criteria measured whether creation of new discourses, networks and coalitions resulted, and whether outcomes had an ability to influence policy or organisations (Rowe andFrewer 2000, Carson andHartz-Karp 2005).

EVALUATION PROCESS
To undertake the evaluation, we arranged with the Ethos Foundation that our evaluation team, comprising four members of our group (the senior students4 ), would engage in participant observation and action research throughout the weeklong Conversation program.Before starting the exercise, the team carefully studied the evaluation method and criteria.Briefings discussed the theoretical basis for the criteria selected, and the observable 'indicators' for each criterion (see Appendix A).To measure each criterion, each evaluator independently applied a five-point rating Likert scale and wrote down their observations (see Appendix B).However, despite a guidance sheet aiming to calibrate ratings, the Likert scale proved cumbersome due to the many criteria and their sometimesimprecise nature for quantification, where many criteria were also only subtly different to each other.This meant that a statistical inter-rater reliability of the evaluative ratings was virtually meaningless.Instead, to formulate rating reliability, we established the extent of evaluator consensus on each criterion by collating our comments and carefully scrutinising them for evaluative patterns, which in turn produced our findings.However, despite what we consider to be the good reliability of our findings, we also believe that our criteria and ratings schema need a good revision for future use.In that regard, Steenbergen et al. 's (2003) discourse quality index may offer promise as a quantitative measure of discourse in deliberation with a focus on observable behaviour. 5eturning to our process, two of our evaluators engaged directly in the process as facilitators of Inquiry Groups (in addition to being observers for other activities).The other two engaged as free roaming observers of all activities held throughout the week.To monitor the team's ongoing engagement, and address any unforeseen problems that might arise, the team met each evening for a general discussion (or debriefing).This was guided by a set of questions developed for that purpose (see Appendix C).Importantly, the evaluators did not discuss their ratings and evaluative comments with each other, in order to ensure the independence of evaluations for later analysis.
Turning to the arrangement of our evaluation criteria, inputs criteria measured selection of participants, especially diversity; opportunity for participation in deliberative processes, including daily agenda setting; the changing influence of new arrivals throughout the week; and perhaps most importantly, information.The two major information input stages, at the start of the week and mid-week, were evaluated at those times.Adequacy of facilitation was also evaluated with regard to facilitator training prior to the exercise, and facilitator performance in relation to the deliberative quality of Inquiry Group discussions held throughout the week.
Process criteria were evaluated though the daily deliberative elements.Criteria, to reiterate, measured participant attendance and inclusion, information access and translation, participant understanding and learning, the free consideration of proposals and creative solutions, ownership of the daily agenda, building of trust, and participant awareness and understanding of each others' thinking and reasoning.Finally, with regard to the outputs phase, we first evaluated each group's output (in the form of a well-displayed poster) at the end of each Inquiry Group session, especially in relation to the creation of new discourses.Second, we measured any immediate creation of visible local networks or connections for plans of action, and later reflected on any eventuation of downstream ones following the exercise, and whether any outcomes influenced policy or organisations.
Finally, because of time constraints in setting up the evaluation at short notice, we were only able to informally survey some of the participants during the event for their evaluative feedback about the Conversation exercise.Such feedback is an important form of evaluation, and in future exercises we plan to incorporate a more formal process, the value of which is well acknowledged (for example, Duram and Brown 1998, McGurk et al. 2006, Midgley et al. 2007).However, to provide more sense of the overall level of engagement, participant attendance and behaviour were also reflected upon.

INPUTS
Participants attracted to the Climate Change and Transforming Energy Courageous Conversation included professional and citizen representatives of the environmental, local and state government, academic, scientific, urban and rural planning, engineering, low emissions and renewable energy, media, business and finance, sectors.With most being professional stakeholders (including activists), a lack of both younger and aged people was apparent; as was a diversity of ethnic groupings.One omission was representation from the coal industry, although invited; however, the focus of the workshop was on renewables.Overall, the knowledges present were representative of relevant societal sectors in relation to the topic under consideration.
With regard to access, the Ethos Foundation provided grants to potential participants in vulnerable or disadvantaged socio-economic groupings, thus providing some equity of access.However, like most events held over five days (especially over the working week), some would-be participants were likely disadvantaged from attending.However, this appears as a general problem of timing for all events whether of a public or private nature.Additionally, the location of the Ethos Foundation's exercise was off the beaten track, not accessible by public transport, which may have deterred some from attending.But again, we note the Courageous Conversation was targeting certain participation publics (who would not be so alienated by such considerations) as a first major (formulating) step towards downstream broader deliberative events on the topic of transitioning to renewables and preparing for climate change.
Turning to input for its agenda, the Ethos Foundation drew on scientific, social and political contexts, with provision for participants to take more ownership of the agenda through the daily deliberative elements.Informing would-be participants of the agenda was information provided on the Ethos Foundation website and through its email communications and registration package (Ethos Foundation 2007a).When the program began, the expert (catalyst) presenters gave informative contextualising presentations on the first two nights of the program, with similar ones given at the Think Tank on the Wednesday (see Figure 1).
Training of the Inquiry Group facilitators was informed by ( 1) a brief given to facilitators about what was expected of them a short time prior to the event (which did not offer much time for clarifications), and (2) a short face-to-face training exercise on the morning before the event began.We found this inadequate for the weighting placed on facilitation in the literature for good deliberative process, and because it became evident in some Inquiry Groups that poor facilitation affected some elements of process.However, this is not a strong criticism of the organisers as this event represented what we came to understand as being more a transition to a deliberative approach than a genuine deliberative design, as determined by a 'hybrid process design context' as an Ethos Foundation organiser put it later.In addition, the organisers encountered unexpected problems with two trained facilitators unavailable at short notice.
More positively, the instructions given to the facilitators worked to encourage dialogue.A 'light handed' approach assisted participants to take more ownership of the agenda and self-organise discussion, which nurtured 'deep' dialogue and interactivity, and in doing so also met goals of the organisers.This was also facilitated by a strongly shared commitment of participants to appropriate terms of association with all participants initially attending the various deliberative and non-deliberative elements.However, participation declined gradually, though minimally, over the week as participants made their own choices to attend or not attend some activities, which takes us to process.

PROCESS
Participant inclusion is emphasised in the literature as most important for deliberative process (Rowe and Frewer 2000).Overall, the inclusion of a relevant diversity of participants, as well as participation in the Inquiry Group dialogue, was well achieved, despite some dwindling of participant numbers towards the week's end, which in a minor way lessened the deliberative capacity of the Inquiry Groups.Also noticeable were limitations in some facilitator's ability to facilitate 'deliberatively'.This was because some participants were observed to dominate discussion at times.This was particularly the case when expert presenters joined the groups, which more reflected specialist narratives of persuasion rather than ones of deliberation (see Hitchcock et al. 2002: 4).Obviously, this raises questions about expert inclusion in the process stage.
Another problem was that some participants encountered difficulties in understanding scientific information about climate change and transforming energy systems.What this highlighted was that even the relatively well-informed participants needed more time and information resources to better understand the highly complex topic.In acknowledging this problem, Inquiry Group facilitators spent considerable time trying to ensure that participants understood each 'community's jargon'.Nevertheless, this learning process diminished as time went on in order to devote enough time to the broader issues posed by the Courageous Conversation's agenda.This saw a lessening of the deliberative capacity of some participants.
Also noticeable was that some participants were initially hesitant to start learning from each other, in being competitive, yet this receded with familiarity and the sharing of stories and experiences.Trust thus became facilitated through 'personalised dialogue'.It was also interesting that while scientific rationalism was initially very strong in providing understandings of the topic, with increasing participant sharing, a notable turn occurred towards holistic, innovative, creative and artistic learning styles and understandings, which increased as the week progressed.The latter included interdisciplinary approaches, creative media of drawing, modeling and designing, and prose and poetry, thus allowing pluralism in scientific and non-scientific understanding.Collective thinking flourished.
In other words, as deep dialogue became more evident, this led to greater mutual understanding of the multiple positions and knowledges that participants brought to the Inquiry Groups, with general all-round enhancement of participants' capacities to engage in deeper levels of rational and creative thought.That reflects a key characteristic of deliberative engagement (Cohen 1989, Sharp 2002, Agger and Lofgren 2006, Hartz-Karp 2006, Hendriks 2006).Participants were also able to control the daily agenda to a large extent, albeit, within the overall broad framing of the event.Building on the first day of 'scoping', where each group explored weeklong directions for their Inquiry Group, increasing group ownership of the daily agenda was reflected in the topics discussed, and the diversity of daily outputs.Overall, dialogue facilitated meaningful and considered interactivity, which, again, was a planned design feature of the Courageous Conversation program.
Somewhat detracting from the process stage, however, was the very busy and intensely engaging program.Given the many complementary activities, and some poor facilitation coupled to the Inquiry Groups' increasingly deep dialogue, with sessions going overtime, a significant decrease in free time for daily reflection and regeneration resulted.This led to some participants soon speaking of 'being overwhelmed'.Overall, though, the deep dialogue reflected success of the program in deliberative engagement terms, and in the obvious enjoyment of most participants to engage in deep deliberation.The most interesting deliberative element, for the number of creative solutions generated, was the free form Open Space element where participants created entirely their own agenda.Perhaps increased agenda setting control facilitated this, but this also reflected the personalities and knowledges of the participants involved, the facilitation skills present, and the structured lead-in over the week.
In sum, the process phase appeared very engaging and reflective of deliberative ideals, principles and practices in its mix of deliberative elements and procedures, but with some areas clearly needing deliberate design learning, including facilitation, program content and structure, time and information resources, and the role and input of expert presenters, as we have indicated.

OUTPUTS
By the week's end, Inquiry Groups were presenting increasingly creative solutions in their daily output posters and conversations, which also reflected a greater understanding of the structural issues and underlying causes of climate change and transforming energy systems.Such engagement and understanding clearly built connections and networks for change, with much multidisciplinary collegiality evident as the week went on, within deliberative and non-deliberative sessions, and at meal times and breaks.
Another indicator of building connections was where the organisers received enthusiastic feedback following the event through the event's email list.Some participants proclaimed new policy directions they planned to undertake in their organisations.A number of networks and coalitions also immediately resulted, which included community participation units or elements of the Commonwealth Scientific and Industrial Research Organisation (CSIRO, Brisbane), community alliances (such as the Wollumbin Collaboration of SEQ and NNSW), the Ethos Foundation, environmental organisations (like the Sunshine Coast Environment Council), a number of local government councils, and researchers from Griffith and Sunshine Coast Universities.
But a central output aim of the Courageous Conversation was to create/encourage the formation of a collaborative, high level, multidisciplinary regional taskforce to plan for regional climate change impacts and an effective transition process towards deep sustainability.That gained momentum and legitimacy about five months after the Courageous Conversation through a large participatory gathering called the Leading for the Future roundtable.Held at Queensland's Parliament House (29 August 2007) it was hosted by the Minister for Education and organised by the Ethos Foundation in partnership with Green Cross Australia, GeoLink and other organisations.One hundred and sixty leaders of community, business and local government joined policy analysts and senior researchers to develop proposals for climate change leadership initiatives for SEQ and NNSW.A central aim was to develop further networks and partnerships for broader community engagement.Elements of the formative networks from the Climate Change and Transforming Energy Courageous Conversation were evident in contributing to the roundtable and two of the invited keynote speakers were experts in community deliberative engagement.Following these events, as part of addressing climate change, in 2008, the Ethos Foundation with the Food Connect Foundation-Australia's largest community supported agriculture program-plan to undertake a series of Local Food Value Chain community educative and participatory workshops in the SEQ and NNSW bioregion.
Such outputs signal the Ethos Foundation, through the Climate Change and Transforming Energy Courageous Conversation, enjoyed some success in putting the issues more on the policy agenda, in creating new partnership discourses, networks and coalitions, and in building a platform for broader societal engagement and inclusive participation in climate change transitions.

CONCLUSIONS
Overall, we found our schema successful in evaluating the design of the Ethos Foundation's Climate Change and Transforming Energy Courageous Conversation.It identified both the positives and negatives of the design, of what could be applauded and what needed to be revised.This is the more powerful because it was an independent evaluation.More broadly, we advance that our three-stage approach to deliberative evaluation offers a useful approach for practitioners to develop further.This is because it addresses the dire lack of evaluative frameworks to both assess and improve the effectiveness, legitimacy and usefulness of deliberative approaches, especially in a time when deliberative design is evolving rapidly.Also, because it addresses productively the debate in the literature about getting the balance right on evaluating process and outputs; to which we have added inputs, in reflection of deliberative design advances.In sum, we believe the time has come for deliberative design and evaluation to better merge, for which our three-stage approach offers one contribution.

Facilitator encouraged deliberative dialogue
To what extent did the facilitator encourage dialogue?
The process was documented thoroughly So that the process can be used for reflective learning, the proceedings need to be thoroughly documented, through transcription, photo-documentation, and statements from the participants themselves.To what extent did this occur?There is opportunity for genuine reflection each day

FIGURE 1 :
FIGURE 1: OUTLINE OF THE FIVE-DAY CLIMATE CHANGE AND TRANSFORMING ENERGY COURAGEOUS CONVERSATION Time must be allotted to permit participants to reflect on the issues, ideas and solutions discussed.*To what extent did this occur?* This was a criteria aspect included in reflecting the Ethos Foundation's specific design.OUTPUT CRITERIA (SEE APPENDIX B) APPENDIX B: LIKERT SCALE AND COMMENTARY EVALUATION SCHEMA EXAMPLE: EVALUATION OF OUTPUTS TO THE PROCESS Ethos Foundation Transforming Energy Courageous Conversation March 26-30