The City as Living Laboratory : Empowering Citizens with the Citadel Toolkit

The roll out of high-bandwidth connectivity and the growing adoption rate of mobile technologies such as smartphones and tablets are said to be transforming the public realm and the way we live and interact in urban areas. These and other digital technologies, such as wireless sensor networks and network-based applications, have begun to cover the city and have started to form the backbone of a large, intelligent infrastructure (Schaffers et al., 2012). Through these rapidly advancing technological capabilities, citizens are increasingly able to access real-time information about the city environment anytime, anywhere they want. However, at the same time, many cities are confronted with a wide range of challenges such as the environmental pollution, traffic jams, governance, etc. More specifically, city governments seem to struggle to meet the demands for improvement in public service delivery associated with the quality of urban life – while facing the prospect of ever-diminishing resources (Gudeman, 2008). In this regard, new technologies can help to map and understand information about the city dynamics and to deliver more effective services.


Introduction
The roll out of high-bandwidth connectivity and the growing adoption rate of mobile technologies such as smartphones and tablets are said to be transforming the public realm and the way we live and interact in urban areas.These and other digital technologies, such as wireless sensor networks and network-based applications, have begun to cover the city and have started to form the backbone of a large, intelligent infrastructure (Schaffers et al., 2012).Through these rapidly advancing technological capabilities, citizens are increasingly able to access real-time information about the city environment anytime, anywhere they want.However, at the same time, many cities are confronted with a wide range of challenges such as the environmental pollution, traffic jams, governance, etc.More specifically, city governments seem to struggle to meet the demands for improvement in public service delivery associated with the quality of urban life -while facing the prospect of ever-diminishing resources (Gudeman, 2008).In this regard, new technologies can help to map and understand information about the city dynamics and to deliver more effective services.Furthermore, bottom-up processes are being increasingly considered for sensing the dynamics of cities based on the participation of citizens.Citizens are becoming actively encouraged to see the city as something they can collectively "tune", in a manner that it is efficient, interactive, adaptive, and flexible (Arup, 2010).By performing a multiple case study analysis of four collaborative smart city initiatives in Europe, namely Ghent (Belgium), Issy-les-Moulineaux (France), Manchester (UK), and Athens (Greece), we seek to yield insights into how bottom-up processes within smart city initiatives can be facilitated, with a particular focus on the role of the different stakeholders in the ecosystem and the civic capacities to participate.
To reach this objective, the article first discusses the smart city concept and the civic capacities to engage in Lately, the concept of smart cities has been changing from a top-down and mostly technological-driven approach, towards a bottom-up process that facilitates participation and collaboration among city stakeholders.In this latter respect, the city is an ecosystem in which smart applications, open government data, and new modes of participation are fostering innovation.However, detailed analyses on how to manage bottom-up smart city initiatives, as well as descriptions of underlying challenges and barriers, are still scarce.Therefore, this article investigates four collaborative smart city initiatives in Europe to learn how cities can optimize citizen involvement in the context of public sector innovation.The analytical framework focuses on the different stakeholder roles in the ecosystem and the civic capacities to participate in the innovation process.The findings illustrate how more inclusive citizen involvement can be realized by providing different tools that align with the specific capacities and skills of the citizens.Furthermore, through specified workshop formats and peer learning, citizens lacking technical skills were also enabled to participate in the evolution of their cities, and to generate solutions from which both the city and everyday urban life can possibly benefit.
The tools bring the citizen to the forefront of democracy.
A citizen of Manchester in the Citadel on the Move project

" "
The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit Carina Veeckman and Shenja van der Graaf the public domain, followed by an introduction of the living lab framework as a possible facilitator of bottomup innovation.Next, the research design is presented with some additional information about the four cases.Finally, we discuss the interplay between the living lab methodology and the development of the toolkit, and how these were aligned with the capacities of the citizens.

The Smart City through Open Data and Mobile Apps
Over the past few years, many smart city projects and initiatives have popped up as a seeming answer to challenges that cities are facing (Pallagst et al., 2009 ment, 2014).The local development path, the interpretation of the concept, and place-specific characteristics can thus explain the various implementations of smart cities.
Among these different definitions and implementations, we see that, on the one hand, ICT plays a dominant role in becoming more intelligent, interconnected, and efficient (e.g., Hall et al., 2000), while on the other hand, a broader perspective with social and economic factors is incorporated in the definition of the smart city concept.In this article, we follow the definition of Caragliu and colleagues (2009) as it balances between economic and social demands, and links up to democratic processes.According to the authors, a city may be labelled smart "when investments in human and social capital and traditional (transport) and modern (ICT) communication infrastructure fuel sustainable economic growth and a high quality of life, with a wise management of natural resources, through participatory government" (Caragliu et al., 2009).
Nowadays, the ICT layer underpinning the smart city concept relates to smart embedded devices ranging from smartphones to sensors, smart meters, and other instrumentation that sustain the intelligence of the city (Schaffers et al., 2011).Data coming from these sensors, or integrated networks, can provide citizens with realtime and location-based information.For example, sensors can monitor the air quality or detect patterns of movement of people in the city.These data, and information stemming from these datasets, can help governments in better understanding the city environment (e.g., improving urban planning) and in creating and delivering new effective services.Additionally, we see that more and more government entities are opening up their data, meaning "data produced or commissioned by government or government controlled bodies, which can be freely used, reused and redistributed by anyone" (Open Government Working Group, 2015).These data are made available at no cost to the public, so that, for example, (citizen) developers or startups can add relevance and value to the information and develop a service based on the data.In this respect, de Lange and de Waal (2013) consider cities as information-gathering systems in which data commons arise: "As these data are being aggregated, they may become a 'data commons': a new resource containing valuable information for urban designers".The availability of data and access to it, along with the skills of citizens to use the data in a meaningful way, are hereby two preconditions to establish a data commons (de Lange & de Waal, 2013).
In this context, urban competitions on open data, or hackathons, are increasingly being organized to stimulate the development of mobile applications.For example (Baccarne et al., 2014) illustrated that the goal of these hackathons is to stimulate both citizens and professionals to work with open government data, with the belief that it will result in more efficient and user-centric applications.
Smart city applications thus form a new digital layer of the city, in which citizens are not only invited to participate in the data collection (e.g., crowdsourced information about air quality), but also in the actual ideation and development process of the services.In this view, the services are not only thought to make the city smarter, but also to serve the mobile citizen in a better way (Hielkema & Hongisto, 2013).
The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit Carina Veeckman and Shenja van der Graaf

Participation, Citizen Involvement, and Civic Capacity
In the literature, some authors (e.g., Baccarne et al., 2014;Schaffers et al., 2011) are still rather hesitant about the value of ICT-enabled smart city solutions, while others clearly express their beneficial use (e.g., European Parliament, 2014;Hancke et al., 2012).According to the latter view, the use of ICT makes a city a "smart" city, because it improves the efficiency and effectiveness of the city processes, activities, and services.Despite these clear-cut opportunities, there is also the belief that, without engaging citizens about the role and impact of technology in their cities, the smart city vision will fail (FutureEverything, 2013).If cities want to reinvent themselves, solely pushing out highly technical solutions will not work, because new forms of digital divide can be created.Instead, a good balance between bottom-up processes (i.e., including the voices of citizens), and the technology push is desirable (Pallot et al., 2011).The mapping study of smart cities by the European Parliament (as referred to earlier) showed that one of the success factors for smart cities is "people", or the involvement of citizens in the creation and realization of the smart city vision (European Parliament, 2014).This form of participation shifts the role of the citizen from a mere passive subject into an engaged actor (Schaffers et al., 2012) and promotes the view of a "participatory governance", or as it is also called, "empowered participatory governance" (Abers et al., 2003).This democratic reform is called participatory because it relies "upon the commitment and capacities of ordinary people to make sensible decisions through reasoned deliberation" and it is empowered because it attempts to tie "action to discussion" (Abers et al., 2003).
However, meaningful participation will largely depend on the specific capacities and skills of the citizens (Wagemans, 2002).In this regard, Saegert (2004) speaks of civic capacities or "the ability to participate in public life with the result of more democratic governance at various scales".Moreover, Stembert and Mulder (2013) speak of different "participation parameters" to facilitate participation and co-creation between citizens and local governments.In their study, they focused on three parameters to investigate citizen participation in the public domain: ability, motivation, and satisfaction.The first parameter, ability, stresses the importance of guiding and supporting the users in a positive and obstructive way.Not everyone has the ability to easily express themselves or to imagine a proposed solution.Therefore, the authors' advice is to communicate in a "common language".For example, generative tools reveal a "new" language that is predominantly visual and they make use of a large set of components that together form "creative toolkits" that people can use to express their thoughts, feelings, and ideas (Sanders, 2000).These toolkits help to bridge the gap between developers and users.Besides providing the right tools and techniques, the users' motivation is another crucial parameter.Malone and colleagues (as cited in Stembert & Mulder, 2013) relate motivation to the goal users pursue: "money, love, and glory".However, public governments cannot reward participants with money generated by taxes and would be better off triggering citizens with "the motivator of love or glory in the form of creativity" (Leadbeater, 2006).Last, satisfaction refers to how the participation process is perceived as satisfying by the user.
These different parameters should thus be taken into account when seeking citizen involvement in the public domain.Furthermore, participation will always lead to some unintended consequences; there will be always some citizens that will be included, while others will be excluded (Turnhout et al., 2010).

The City as Living Laboratory: An Ecosystem to Foster Innovation
One way to organize bottom-up processes within smart city initiatives is by applying the living lab approach.
Living labs can be regarded as "physical regions or virtual realities where stakeholders form public-private-people partnerships (4Ps) of firms, public agencies, universities, institutes, and users, all collaborating for creation, prototyping, validating, and testing of new technologies, services, products, and systems in real-life contexts" (Westerlund & Leminen, 2011).The living lab concept appeared in academic discussion in the 1990s, but really took off in 2006 when the European Commission initiated projects to advance, coordinate, and promote a common European innovation system (Dutilleul et al., 2011).According to (Pallot et al., 2011), living labs are a good way to bridge the gap between technology push (i.e., solution developers) and application pull (i.e., user communities), because they bring the necessary combination of digital skills, creativity, and innovation methods together.Coenen and colleagues (2014) describe living labs following a "meet in the middle philosophy", an approach "for involving both the voice of citizens and local grassroots organizations to represent the bottom-up perspective and the voice of government and companies to represent the top-down view".Schaffers and colleagues The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit

Carina Veeckman and Shenja van der Graaf
(2012) take one step further, stating that "cities are becoming a living lab itself, a playground of innovation and transformation", exemplified by the emerging ways of collecting and using urban data.Living labs can thus be regarded as an effective means to facilitate bottomup processes within smart city initiatives, as they promote multi-stakeholder collaboration and consider users as innovators (von Hippel, 2005).
Living labs can have different thematic focuses and interests, such as focusing on innovation in health, media, smart grids, etc.In this article, we focus on urban living labs that specifically involve citizens in city development to make urban areas better suited to their needs (Juujärvi & Pesso, 2013).Obviously, the goal of urban living labs differs fundamentally from more ICT-oriented living labs, which tend to be rooted in commercial contexts; here, the generated public value will be more of concern than the economic value (Baptista, 2005).
Regarding the key participants and their roles, Juujärvi and Pesso (2013) found that the role of citizens in urban living labs is more comprehensive than in other types of living labs.They discovered that citizens can have multiple roles in urban living labs, ranging from a mere informant to tester as well as contributor and co-creator in the development process.Furthermore, the motivation to participate can also be different, because cit-izens can have a natural motivation to participate in shaping their environment through a "sense of place", "a sense of being at home in a town or a city" (Horelli, 2013).Last, the role of the city can be described here as the "enabler" or "mediator" in the ecosystem, bringing everyone to work together effectively (Ratti & Townsend, 2011).
Table 1 provides more information about the different actor roles in living labs (Leminen & Westerlund, 2012) and specifies the role for each stakeholder in urban living labs (Juujärvi & Pesso, 2013).

Research Approach
As part of the Citadel on the Move project (Box 1), this research was conducted by the iMinds-SMIT research organization (iminds.be/en) at the Vrije Universiteit Brussel in Belgium.The pilot project initially focused on a network of four smart city initiatives in Ghent (Belgium), Issy-les-Moulineaux (France), Manchester (UK), and Athens (Greece), where citizens were engaged to participate in the design of a toolkit to build mobile applications.At the same time, the four cities were opening up their data and transforming it into a publicly usable format.Citizens were invited to provide suggestions for new datasets or to convert the dataset by themselves.

Carina Veeckman and Shenja van der Graaf
In this article, a multiple case study analysis of the four smart city initiatives is described.In the analysis, we focus on the following levels: i) the actor roles in the ecosystem and ii) the required civic participation capacities.These specific dimensions were chosen to provide an overview of the different stakeholders, to analyze the role they play, and to reveal how participation and collaboration is set up between the stakeholders in order to involve citizens.
In these different cases, Ghent and Issy-les-Moulineaux mainly focused on the delivery of better services within the tourism domain, whereas Athens and Manchester sought new services within the transportation domain to enable citizens to overcome health challenges and adopt more active lifestyles.In the latter two cities, sensor networks to measure air quality were also installed.By tapping into the innovation potential of citizens and by facilitating collaboration, these cities were interested in gaining better insights into citizens' needs and establishing a better communication with the citizen.
In early facilitated workshops, five main themes were identified from the discussions: i) environmental information, ii) parking in the city, iii) events in the city, iv) points of interest in the city, and v) crowdsourced information.Based on these themes, so-called mobile "templates" were created that citizen (developers) could use to quick-start the mobile application development process.The source code of the templates, together with guides, were made available on the project platform and GitHub (github.com/citadel-eu).
For our analysis, we used the user feedback collected from the living lab experiments of the four cases.These experiments were set up in an iterative and gradual approach, which aligned with the maturity of the mobile application development toolkit.In total, four iterative testing cycles were set up involving self-reporting methodologies (e.g., diaries), participatory methodologies (e.g., design charettes), and observational methodologies (e.g., participant observation in the city).By deploying this multi-methodological approach, feedback about various aspects of the toolkit was collected from the early stages of the project until the eventual selfgovernance of the toolkit.The chosen methodological approach was designed to test, evaluate, and co-create the toolkit with the citizens.These findings were used to investigate how bottom-up processes can be set up between the city and its citizens, and how hurdles can be tackled concerning the civic capacities of the participants.

Findings
In this section, we first provide an overview of the different stakeholders in the innovation ecosystem, together with a role description.Next, the user feedback of the four cases is discussed along the different living lab testing cycles to formulate conclusions on how citizen involvement can be optimized.

Actor roles in the innovation ecosystem
According to the typology of Leminen and Westerlund (2012), we identify the following roles in the ecosystem: the city as enabler, the citizens as users, and the research organization as provider.The role of utilizers is not present within this ecosystem, because the scope of the initiative is more oriented towards generating public value.Figure 1 illustrates the different stakeholders and exemplifies the role they play within this particular ecosystem.
The four local governments play the role of enabler in this ecosystem as they set out the smart city objectives, provide the necessary resources, and bring the different stakeholders of the living lab network together.In all cases, the city promoted the networking among citizens, the developer community, students, small and medium-sized enterprises, etc. to increase awareness about open data and to enable cooperation among the

The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit
Carina Veeckman and Shenja van der Graaf different stakeholders.During the living lab experiments, these stakeholders were brought together to both brainstorm and build around new ideas for applications with help from the provided toolkit.
A research organization (iMinds) was also involved in the ecosystem to provide innovative research and development methodologies.The organization had previous experience in the design and implementation of userdriven methods in living labs, and could thus accumulate knowledge over the long term.In this instance, the research organization did not have direct contact with the living lab participants, because such interaction would have a negative impact on the community building and citizen-government relationship.Instead, the research organization provided diverse protocols and guidelines to the cities on how to set up the living lab experiments.Afterwards, both the city and the research organization collaboratively assessed the results.
The last role is that of users, who were invited to provide feedback and participate in the co-creation processes.In this ecosystem, users were defined as citizens, (citizen) developers, and professional developers who were interested in using and creating innovative applications in the domain of tourism and transport.These different groups were segmented into different categories based on their level of skills and technical knowledge: none, limited, or high.The following sections show the importance of categorizing users based on skill level.

Testing, evaluating, and co-creating mobile (template) applications
In this section, the results are presented from the early user requirements workshops until the last iteration cycle of the mobile application templates.The findings show how cities organized the bottom-up processes and how civic involvement was accomplished.
In the summer of 2012, a first workshop was organized in each city to gather preliminary thoughts and expectations about how the creation of applications in the transportation and tourism domains could be facilitated.Various stakeholders were invited to these workshops to discuss new ideas based on some predefined paper mock-ups.These mock-ups described some basic application features, and mostly served to define the first user requirements.
In the next phase, the user requirements were taken into account to develop a first version of the application templates.Based on the stakeholder feedback, five mobile application templates were created, focusing on the following aspects: i) environmental information, ii) parking in the city, iii) events in the city, iv) points of in- The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit

Carina Veeckman and Shenja van der Graaf
terest in the city, and v) crowdsourced information.These templates were working mobile web applications based on HTML5 and PHP.JavaScript and JSON were also used to enhance the user experience and allow the communication with the application's back-end and data respectively.By providing these templates, the cities facilitate mobile application development, as anyone is able to download the source code from the platform.This way, citizens are able to personalize the application templates in order to meet their needs.For example, citizens are able to combine multiple templates, add or remove features and datasets, etc. Figure 2 shows a first version of the templates.
Through these standard templates, cities are providing an easy way for citizens to start creating their own public services, and it makes the development processes less time-consuming and more cost-effective.Furthermore, when citizens can easily access open data, the innovative potential of citizens becomes stimulated as citizens themselves can determine the mobile applications they want and need.
To gather user feedback and iterate the development, the applications templates were launched into the living labs networks of the four cities.In total, four iteration cycles took place in order to optimize the use of the templates.
In the first testing cycles, the cities agreed to only recruit "citizen developers", because these are the citizens who have some development skills as well as innovative ideas for new applications.In total, 25 citizen developers were carefully selected and tested the first version of the application templates.Feedback was collected from interviews, focus groups, and journals.This latter method could foster the self-reporting of citizen developers about the experiences and activities with the toolkits.Also, logging provided substantial information about the number of downloads, error information, etc.
After two testing cycles, the results showed that about half of the citizen developers had been intensively adapting the templates over a period of one or two days.
The parking application and the crowd-sourcing template were perceived as most interesting, whereas the urban planning template was perceived as rather useless due to a lack of data.In general, the citizen developers found the application templates easy accessible.Because the templates had been developed using cross-browser HTML5 technology, there were no problems in using these templates on different types of mobile devices or operating systems.The user interface was rather well received and many suggestions were made to improve it.
Despite this positive feedback, none of the citizen developers actually started developing their own application, even after many technical difficulties were resolved after the first iteration.Furthermore, it became clear that the feedback differed depended on the skill level of the citizen developers.Some citizen developers perceived the download and installation pro- The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit Carina Veeckman and Shenja van der Graaf cesses as rather easy: "I found and downloaded the app files fairly easily and had to make some changes to the config file, which I am used to doing" (Manchester, December 2012).On the other hand, it seemed that some citizen developers were not familiar with these processes and stopped using the templates after downloading them: "We should have something very basic, like for example, the framework of Wordpress, where you find some boxes to fill in -some drag-and-drop elements.This is clearly what I expected to see, not some coding lines" (Issy-les-Moulineaux, December 2012).
In practice, less experienced citizen developers did not succeed in installing the templates, even with the help of others or when consulting the documentation.Instead, they evaluated the templates through the online demo website and stopped using the templates.In contrast, more experienced citizen developers were able to install and customize the templates.Based on this feedback, the cities and the research organization decided to implement a different approach based on the skill levels of the users.
After gathering feedback and iterating two testing cycles, none of the recruited citizen developers had created their own application.To increase usage and improve participation (regardless of skill level), additional tools were developed.The application templates would still remain available to the more skilled citizen de-velopers and professional developers, however, a new tool, called the "App Generator Tool" (Figure 3) was made available to ordinary users.With this tool, citizens with limited-to-no technical knowledge could participate more easily in the application development processes.This way, cities guarantee that every citizen, including those lacking specific capacities, is able to become involved and be heard.
The role of the App Generator tool is to allow users to combine various datasets of a city and build an application online without having to write a single line of code.
In order to generate a new application, users simply need to fill in a form.Several fields should be filled in, for example, to select a city and (one or more) dataset(s), to define the theme colour and fill in a title for the application.When the application is created, a unique identification number is assigned, and the application can also be shared with others.
Besides creating a more accessible tool, a separate evaluation track based on the level of skills was set up by the research organization.This step was necessary because, in the upcoming testing cycles, not only citizen developers were involved, but also a larger number of ordinary citizens.Therefore, separate surveys were programmed: one evaluating the application templates through the demo website for non-technical participants and one survey that guided the more experi- The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit Carina Veeckman and Shenja van der Graaf enced users through the download and installation processes.Furthermore, a participatory design workshop was organized in which different stakeholders (e.g., citizens, professional developers, data enthusiasts, thematic experts) were invited.To bridge the "gap" between technical and non-technical participants, simple and creative communication tools were used.In this way, people could easily express themselves by using visual aids, drawings, and so forth.At the end of the sessions, some paper mock-ups were presented that were based on several scenarios.These paper mock-ups were given as an inspiration for the (citizen) developers to start developing new applications (Figure 4).
This tailor-made approach was proven very successful: more citizens were being able to participate and to provide custom feedback.To further engage citizens in the development process, specific "Apps4Dummies" workshops were also organized.In these workshops, a demonstration was given of the different tools, and knowledge was shared about open data and coding in general.At the end of the living lab experiments, 80% of the key users stated that they had learned something new about creating applications in general, and half of them expressed that they are eager to learn more about the topic (e.g., data formats and conversion of datasets).The main conclusion was that the transfer of knowledge and skills proved to be more empowering that just the provision of tools.

Conclusion
This article discussed the findings of four smart city initiatives in Europe, with a specific focus on citizen engagement and the capacities to participate in the public domain.In conclusion, we identify three key lessons learned through this study: 1.The living lab approach facilitates participation After describing the various roles in the ecosystem, it became clear that the living lab approach played a central role in bringing different stakeholders together.By facilitating collaboration, stakeholders came together to jointly create new services, citizens made contact with their administrations, and mutual understanding was created.At the end of the testing cycles, citizens clarified that they better understand the challenges their city is facing, and that they would like to further contribute to the process of opening up data and building applications.
The living lab approach also entailed iterative testing and feedback.In the beginning of the project, citizen developers tested and positively evaluated the application templates, although none had the actual intention of developing their own application.This outcome did not fulfill the expectations of the cities: they had hoped to stimulate application development by providing standardized building blocks through the templates.
The analyses of the user feedback showed that the users' motivations and the abilities to participate were not fully satisfied.At first, the less skilled users were excluded from the development process, because they did not have the proper skills.Therefore, the cities, as well as the research organization, decided to develop different tools and a more targeted user recruitment and evaluation methodology to optimize the citizen involvement.This targeted approach seemed successful, given that all target groups started using the tools to create new applications.The user feedback was also The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit Carina Veeckman and Shenja van der Graaf more satisfactory: the tools were perceived as easier to use and more useful.Furthermore, after the end of the living lab experiments, two-thirds of the users were still using the tools to explore open data or to further improve their application idea.Here, we see that the iterative living lab approach not only proved its beneficial use in bottom-up co-creation, but also in validating the evaluation methodology and monitoring the participation parameters.

Co-creation processes can both include and exclude
Next, it became clear that facilitating co-creation processes between citizens and government entities could include some citizens and exclude others.This result was also found in (Turnhout et al., 2010), as one of the unintended consequences of participatory governance.Although it is impossible to involve everyone, the results here showed that, if different tools are aligned with the specific capacities and skills of the users are provided, more chances are created for users to become heard and take part.Interplay could be detected between the collected user feedback and factors influencing the civic capacities to participate.First of all, the iterative testing cycles made it possible to quickly respond to some technical issues and develop a better solution in the next phase.Participants are often frustrated when technical issues occur, and this frustration could evoke a decreasing interest in the long term (with possible drop-outs).But, more importantly, listening to the user feedback and taking the users' abilities and motivations into account, overcame possible failures or low-usage intentions in relation to the technical solutions provided.The development of the App Generator Tool enabled ordinary citizens to easily create applications, and technical skilled users no longer dominated the development processes.
To optimize the involvement, it was also necessary to develop a separate evaluation track for each of the targeted user groups.By making specific questions that matched the profiles of the citizens, the data collection methods were perceived as rather adequate, and not too easy or too difficult to respond to.The creative tools in the participatory design workshops were also very successful in creating "a common language" for communication between the different stakeholders.

The approach empowers citizens
One of the most important outcomes for the cities is that, by providing and co-developing the toolkit, citizens were given the opportunity to contribute to the opening-up process of data and to the building of service applications.What in advance was limited to only a few, can now be done by anyone.Citizens acknowledge that, by participating in the diverse evaluation activities and workshops in their cities, they have learned new skills and knowledge, and they can now independently create an application.For the more skilled citizens, the hackathons and other application competitions provided opportunities to network, to disseminate their work, and to exchange experience.For this target group, the motivation of "playfulness" and the opportunity to showcase their expertise and creativity prevailed.
Last, the organization of workshops and peer-learning activities in the community were also vital in supporting the citizens.In the beginning, we observed users who only used the App Generator Tool and then, along the testing cycles, acquired more skills and started to execute more advanced operations.On the platform, the more skilled users also helped the less experienced ones.Catalyzing this mutual support and connecting people with different perspectives thus strengthen civic engagement and the opportunities for creating innovative solutions.
This study was a joint effort funded under CIP-ICT-PSP.2011.5.1 "Open Innovation for Future Internet-Enabled Services in 'Smart' Cities" (no.297188).Our gratitude goes to the entire Citadel on the Move consortium.
In particular, we would like to thank the pilot cities for their efforts and support.
An earlier version of this article was presented at the 2014 International Conference on Engineering, Technology, and Innovation (ICE), which was held from June 23rd to 25th in Bergamo, Italy.The ICE conference discusses systems engineering as a socio-technical task, with a focus on design of products and services, and the entrepreneurial innovation process for its adoption in society and the economy.

Figure 1 .
Figure 1.Actor roles in the innovation ecosystem

Figure 2 .
Figure 2. Screenshots of the parking and points-of-interest application templates

Figure 3 .
Figure 3. Screenshots of the App Generator Tool

Figure 4 .
Figure 4. Paper mock-ups of mobile applications from Athens and Issy-les-Moulineaux

the Authors Carina Veeckman is
a researcher at the Vrije Universiteit Brussel, Belgium, where she started working for the iMinds-SMIT research group in 2011.Until March 2013, Carina was responsible for the living lab methodology within the Flemish Living Lab Platform, which included numerous projects within the smart grids, smart media, and smart cities domains with a test panel of 2,000 users.Her current research and interests are related to open data and the co-creation of mobile applications within a smart city context, measuring related impact and outcomes, and monitoring the willingness to share personal data when using these applications.Currently, she manages and conducts user research in the following smart city projects: Citadel on theMove (2012-2015), Open Transport Net (2014-2016), and the European Cloud Marketplace for Intelligent Mobility (ECIM)(2014)(2015)(2016).Shenja van der Graaf (PhD, LSE) heads the Code,Commodification & the City (Digital Cities) cluster at iMinds-SMIT at the Vrije Universiteit Brussel in Belgium.She is a researcher at the London School of Economics and Political Science in the United Kingdom, an honorary fellow at MIT Media Lab ID³ Hub in the United States, and a Futures of Entertainment fellow, also in the United States.Her current work is concerned with social, economic, and policy issues arising from innovations associated with the ICT.Specific lines of inquiry include the integration of new technologies into society; management of technological innovation in firms, cities, and communities; (new) media users and "cultures of expertise"; mediation of social and economic life, theoretical perspectives; and cybersecurity. www.timreview.caAbout

City as Living Laboratory: Empowering Citizens with the Citadel Toolkit
Carina Veeckman and Shenja van der Graaf