Advanced Engineering Informatics

Design is a balancing act between people ’ s competing concerns, design options and design performance. Recently collecting data on such concerns such as sustainability or aesthetics has become possible through online crowdsourcing, particularly in 3d. However, such systems rarely present more than a single design alternative or allow users to change the design and seldom provide quantitative design analysis to gauge design performance. This precludes a more participatory approach including a wider audience and their insight in the design process. To improve the design process we propose a system to assist the design team in exploring the balance of concerns, design options and their performance. We augment a 3d visualisation crowdsourcing environment with quantitative on-demand assessment of design variants run in the cloud. This enables crowdsourced exploration of the design space and its performance. Automated participant tracking and explicit submitted feedback on design options are collated and presented to aid the design team in balancing the demands of urban master planning. We report application of this system to an urban masterplan with Arup.


Introduction
Urban master planning is a balancing act between many inter-related design parameters and performance goals. Some goals are explicit and quantitatively measurable (e.g. water consumption), other aspects are implicit and not easily measurable with many being dynamically driven by human preference and market forces (e.g. aesthetics or city zoning ordinances). Optimization alone will not help find the most appropriate solutions as it does not known all of these implicit constraints. Indeed evolving a design toward optimal is a task beyond any one architect or engineer as different disciplines and stakeholders each have their own constraints and design options. To improve the design process we propose crowdsourcing augmented with design performance analysis to explore these aspects and guide us to a set of the most synergetic solutions.
The design process involves experts from many disciplines making a plethora of decisions which will affect the urban environment and those who live and work within it. The design space has a very large scope including not only choice of architectural forms but also a huge number of design decisions which must be made to implement the vision of the development. Such design decisions range from the number of parking spaces to the amount of green space included these create a vast number of design scenarios to be explored under which the design's performance will vary across many KPI's.
Aside from such design decisions the threat of climate change imputes a requirement to minimize the carbon emissions of a development by the adoption of mitigation strategies. Sustainability strategies such as the adoption of solar thermal panels or the use of water efficient fixtures and fittings have impact across multiple disciplines and Key Performance Indicators (KPI's) and require multidisciplinary collaborative exploration to enable their implementation. For example the implementation of a district energy system, where heating is provided by a central plant, will impact upon carbon emissions, water consumption and the urban landscape as well as the mechanical systems required for each building. Each aspect requires a different expert to collaborate to identify implications and benefits across the design space.
Such strategies are, of course, constrained by cost both in capital and operating expenses as well as impacts upon other resources such as water or energy consumption, creating a challenging optimization problem, particularly as many of the constraints are either undefined or cross traditional discipline boundaries. Such optimization of the design for an urban masterplan is rarely conducted with detailed quantified assessment neither is that assessment repeated for more than a couple of design cycles [1] neither does it often involve the key group of stakeholders the people who will live and work in such an environment.

Related work
In this context we seek to apply the crowdsourcing of design feedback to enable wider participation in the design process [2]. Crowdsourcing is the approach of seeking to involve a large number of individuals to solve a set challenge [3]. The advantage of this is to leverage the "Wisdom of the Crowd" [4] with many views and unique experience the problem is approached from more angles and different challenges and solutions may be offered [5]. Since the internet enabled its rise [4] in the early 2000's Crowdsourcing has be used extensively [6] to gather data most notably in the open street map project [7], to conduct large-scale user studies such as how people perceive different types of visualization [8] or to solve particular puzzles [9] for example in solving math problems [10]. One notable example is the use of large groups of people contributing data in disaster zones [11] with reports of forest fires, property and risk to life being collected by the crowd.
Crowd sourcing in design is thought to be particularly effective if conducted early in the design process through participatory design [12][13][14][15]. In recent years online web-based tools [16][17][18][19] are enabling residents and members of the public to submit their feedback. This enables many people to provide comment on design choices, contributing their insight by geospatial tagging [16,3] to highlight design issues. This technique has also been used to gather feedback on changes in the rural landscape [20] as well as the more general planning process [21].
As outlined in [5] who try to provide a unified definition of crowdsourcing it is important that participants feel there is a mutual benefit to the contribution of their time and expertise, this is particularly important in urban design [22]. In the context of design we see a positive benefit that the participants proposals may be included in the design for its betterment also negatively the participant may hope that the worst impact of a design or urban development may be avoided -for example in the placement of a windfarm [23]. Additionally one would hope that in the realm of urban planning citizens civic pride would help them to invest in the development of their local environment. An increasing literature suggests this future for urban planning [24] and for local government more widely [25].
Crowdsourcing is a good technique for this as it can help explore distant parts of the design space -see [26]. Each participant explores in their own direction based on their own experience and in aggregate we hope will explore a wider area of the design space than a professional design team could feasibly explore. A number of researchers have recommend the application of crowdsourcing in planning practice [27]. However, in general such systems are used to gather feedback on a single design option or a small number of design variants [20]. This presents a clear and constrained choice to the participants and focuses input to decisions of interest to the design team. However, the design space is dramatically larger than that exposed through these tools as there are many more design variants and improvements which could be made. To enable this the crowd would need to be given tools to explore the design space and assess the performance of design variants. This is the approach we introduce in this paper which as in [21] is unified by visualization.
The fusion of crowdsourcing into urban planning remains an active research front [28,2,27,25,23,24] and the authors hope this paper demonstrates the value of a computational design analysis framework to empower crowdsourcing as a technique to benefit urban planning.

Approach
In the context of this multidisciplinary design optimization problem, with its large design space, and the need to involve a range of stakeholders in design decisions we propose to augment a 3d visualization environment with a quantitative design assessment by providing online assessment of design choices in the cloud. Crowdsourced feedback is recorded and analyzed to provide a guide to the design team to improve decision making. This is enabled by an online workflow system capable of running design analyses automatically.
Real-time automated assessment enables a more flexible design space to be explored in directions not thought of apriori by the design team. This flexibility enables gathering of intention, for example, city planners using this tool may explore the implications of mandating electric vehicles for a campus, something not explored by the design team. On the other hand residents may be interested to learn the tradeoff between density of urban form and carbon efficiency, enabling participation in exploring design options. We hypothesise that viewing a wide range of design options and their performance is likely to lead to greater acceptance of design variation and hopefully create a consensus.
Specifically, we propose the following technical systems approach, summarized in Fig. 1: 1. An interactive, web-based, 3d environment for exploring proposed designs, annotated with design performance and enabling exploration of the design space for carbon mitigation strategies. Many aspects of design in the urban environment are inherently geometric (e.g. line of sight, shadowing) and are best understood through 3d visualisation. 2. We couple this with a detailed analytical model [29] and a computational framework for design exploration [30] providing realtime quantitative analysis of the masterplan's performance across a range of KPIs. 3. This enables exploration of the design space by a large number of participants each able to explore the implications of adopting carbon mitigation strategies on the macro (masterplan) and micro (building) level across the development. 4. Such crowdsourcing of the design space is tracked providing a quantified insight into how the design space is explored and areas of interest or otherwise identified by the participants. This may be done by implicit tracking or by explicit design feedback, as encouraged by [28].
This system enables quantified participatory design by a wide range of stakeholders and the public. The design process may by improved by providing the design team with more insightful data during the design process. This may include automated tracking of stakeholders through their interaction with the design, their crowdsourced insight and voting together with quantitative assessment of designs as well as how the design space is navigated by user demand. This system is fully implemented and this paper reports initial results from small-scale studies.

Hypothesis
Before exploring the engineering approach of the system proposed in this paper it is helpful to explore the hypothesis driving this approach.
We believe that the best use of computing in design is to guide the design team in their journey through the design space. We believe that the design challenge facing urban masterplanning is too complex for any one individual to grasp and the design space too vast for exaustive human exploration and too complex for soley machine led optimisation exploration. This is due to the huge range of design decisions made toether with its multidisciplinary nature coupled with the fact that many design constraints are not explicit (e.g. aesthetics or non-obvious complex engineering interactions). Hence the best design teams consult widely with a range of stakeholders and experts to gather best practice and design insight. Further since many of the design challenges encountered are performance based tools for assessing this performance must be made available early in the design process and to as many of the stakeholders as possible.
Hence the hypothesis of this work is that by providing tools to enable a much wider range of stakeholders to engage in exploring the design space we can gain their insight to help guide the masterplan design team to toward a better design. Further we believe that this must be done by providing non-experts with access to quantitative design assessment tools so they may understand the implications of their decisions. Finally we believe there is insight for the design team in analysing the crowd's design decisions and their collective paths through the design space as they each evolve the design subject to their personal experience and priorities.

Comparison to genetic algorithm optimisation
In a nutshell the approach we are taking might be compared to a genetic algorithm. Rather than an artificial population of design variants being bred together randomly we have a collection of intelligent agents -the crowd. Critically each of the agents participating in this exercise is different -in background, in domain knowledge and in their knowledge of the implicit and unstated design constraints. For example an engineer trained to assess the wildland fire risk will look at an urban masterplan very differently from an architect seeking to create forested campus.
Further in contrast to a genetic algorithm where a single fitness function is used to assess every design; every agent (human) in our approach has their own (possibly evolving) fitness function for the design based upon their domain knowledge and priorities. This means that rather than optimisation moving through the design with a single (perhaps multi-criteria) objective in mind the design space is explored in multiple directions simultaneously with many different views of an "optimal design" in mind. We believe that this form of exploration is much more likely to cover a wider portion of the design space and provide the insight necessary for improving the design. Further as the design space is explored participants are able to submit comments and feedback such as effective design strategies or noticed design challenges (e.g. a poorly performing building). Further by automatically tracking user interactions and their paths through the design space we gain a database for further analysis to guide the design team on their path through the design space.

Architecture
To explore our proposed approach we have created a system which enables crowdsourcing of design input backed by the automatic analysis of design scenarios. The architecture of the system is shown in Fig. 2 and builds upon the Hiersynth workflow system [30] and is extended by the Concinnity analysis platform [31].
The Concinnity analysis platform provides a workflow development environment for engineers and architects to configure and create analysis workflows interactively. The interface shown in Fig. 3 provides infrastructure for configuring suites of analyses to the specific design site and scenario with a library of standard and partially configured analyses. The platform uses the HierSynth engine [31] to provide a multi-scale analysis capability, with the ability to run analyses at different scales of design. By this we mean whole city, suburb or building level, each of which will be of interest to a different audience. In this paper we consider a site wide analysis and a per-district analysis, this is shown in Fig. 3 by the use of design queries reading all districts in a Geographic Information System (GIS) file documenting the design.
Once the analysis workflow is created, a visualization interface may be created interacting with the workflow system through a RESTful API server. We utilise the Unity3d game engine to create this interface due to its platform independence and ability to run online using WebGL. The visualization environment enables participants to explore the design spatially and to comment upon it with geo-located comments. Such feedback should aid understanding spatial analysis results, for example a lighting analysis or per district carbon emission analysis. In such cases physical form or location will impact upon performance, it is also likely that particular locations in the design will be of greater interest or concern to certain groups of participants. Participants are provided with an easy to use interface shown in Fig. 4 to create design scenarios for analysis and view the results of such analyses at both an aggregate and per district level. Requests for analysis of design variants, such as the calculation of carbon emissions, are submitted to the API server for analysis, this server is then polled for the results to be displayed. During this process tracking data of the user's movements in 3d, spatially located comments and voting on preferred KPI's are recorded automatically in a database together with their design space exploration history. More details of this recorded data are shown in Table 2 the size of the data collected does not exceed 1 Mb per user session.
Once a request for analysis of a design scenario is submitted it is parsed and validated against the workflow and analysis configurations using the Hiersynth engine to ensure that all inputs are valid. Valid design analysis requests are logged in a database as jobs for worker nodes to analyse. This provides a scalability point by running many worker nodes to support crowdsourcing as while a web-server may scale easily, analysis of designs is compute intensive. We deploy our infrastructure to virtual machines on a private cloud using Openstack. Once a job is received worker nodes, running an execution service, load and run the Hiersynth engine which manages the execution of a workflow, pulling all required analyses, design files and configurations (such as the type of analysis to perform) from a MongoDB database onto the local computer. The engine then expands the structure of the workflow to reflect the design and scenario based upon queries against design files (e.g. one analysis per district in a GIS databases). Then with concurrent execution the system runs the required analysis models with the design and scenario data. A local caching system of previous analysis results and design queries provides a significant performance improvement. It should be noted that the system provides scaling both in using multiple compute threads (CPU cores) and by replicating analysis worker virtual machines. Once computed analysis results are postprocessed, for example aggregating the performance of design elements which are then stored in a database ready to be returned to participants through the RESTful API and to be used for subsequent design space analysis. We note that this system enables the security of analysis models and detailed design data to be protected and remain confidential as it is not included in the visualization tool but resides solely upon the cloud system [32].
Finally, throughout this process metrics on the participant's interaction with the design and requests for analysis of particular designs are stored for subsequent analysis. By enabling this crowdsourced data to include quantitative analysis of the design space we believe greater insight can be gained than by crowd exploration of a single or small set of proposed designs.

Case study and analysis models
To investigate what can be learned through our crowdsourcing approach we sought to apply our framework to an industrial case study. Partnering with Arup North America we apply our framework to a coastal development developing a new business district. The case study site comprises 29 districts over 39 hectares with a proposed total Gross Floor Area (GFA) of over 300,000 m 2 . The key challenge we investigate in this development is how to achieve a more sustainable development. This presents a complex optimisation problem for which we require a number of analysis models to provide quantitative design insight. We use a range of models because there are many aspects of sustainability, ranging from models calculating carbon emissions to those understanding water consumption or waste production. The first of the models used in this casestudy is an Integrated Resource Management (IRM) model [29]. IRM models provide an integrated sustainability assessment of a masterplan design by coherently combining assessment models from several engineering disciplines. Discipline models contained within Arup's IRM model include carbon accounting, energy consumption, water, waste, transport, materials and economics. Critically, these sub-models are tightly coupled with calculations and inputs in one model affecting calculations in the others providing a greater fidelity of analysis. This allows the assessment of sustainability across a number of Key Performance Indicators (KPI's). Typical IRM models will contain several thousand design inputs and calculation assumptions, ranging from the site size to the carbon emissions for travelling 1 km by bus. The models calculate several hundred KPI's which are summarised in 8 KPI's shown in Table 1. The key benefit of such an integrated model is that it enables capturing of the interrelationships and side effects of the implementation of certain policies. For example water efficiencies may also reduce electricity consumption as there is less need to pump water. Understanding these implications in these KPI's is the insight we provide to participants to enable quantitative assessment of the design and potential changes.
The IRM model typically provides site-wide analysis. As described in [30] the Hiersynth engine enables running of the IRM model at a per district level for a greater level of detail providing spatially decomposed results for analysis within a 3d environment. It should be noted that the IRM model does not assess geometric form other than assessing quantitative statistics about the development arising from its geometry, such as Gross Floor Area. To demonstrate the potential of our approach we include a geometric analysis model. We incorporate an analysis of the lighting conditions within the proposed masterplan's massing model by incorporating analysis using the open source ray-tracer Radiance [33]. Using this model we calculate an industry standard BRE209 [34] daylight analysis, the results of such analysis can be seen in Fig. 5.
Arup's IRM model also enables assessment of the impact of over 40 sustainability strategies. These strategies range from adopting low flush fixtures and fittings to water efficient landscaping and district energy supply networks. The impact of these strategies is calculated across a number of the integrated models, for example a district energy supply may reduce energy and carbon consumption but may increase water consumption as heat is transferred between buildings as hot water. Capturing these interactions is the key strength of this modeling technique and supports exploration of this optimisation problem as a support to the design process. Design is primarily concerned with such trade-offs such as between increased build density which is good for lowering carbon emissions and increased green space which improves comfort and biodiversity.
These are the key optimisation problems and design challenges which we seek to gain crowdsourced insight upon within our 3d visualisation environment. Further discussion is provided in Section 7.1.

Crowdsourcing environment
A 3d visualisation environment provides an intuitive environment for exploring a spatial design [35,16]. We provide a 3d web-based and desktop visualisation environment through the Unity3d game engine. The environment is augmented with a heads-up display containing the performance of the whole masterplan design according to the 8 KPI's in Table 1. When the participant clicks on any KPI the per-district performance is shown geospatially as a billboard above each district coloured on a red-green scale according to the relative performance amongst its peers in each scenario as shown in Fig. 7. Participants are able to interact with a list of sustainability scenarios which may be toggled on and off to propose a new design scenario, once complete this may be submitted for analysis which will take around 10-30 s  At any time participants may submit their comments in a number of ways as shown in Table 2 including submitting a geolocated textual comment as in [16], voting for particular KPI's and for or against particular sustainability strategies. Participants are also invited to vote for a particular design scenario once they have had it analysed.

Design space exploration
Specifically, we ask participants to identify combinations of 49 sustainability strategies which should be applied to the masterplan to improve its sustainability, primarily through carbon emissions, while minimizing operational and capital expenditure to acceptable limits. This presents a design space of 2 49 design options, which is infeasible for exhaustive exploration by human or machine methods. However, we believe (see [36]) that the key finding from a crowdsourcing study is not an optimal design but rather feedback on directions and the most important strategies to be applied according to different stakeholders.
Such large design spaces have traditionally been explored by optimisation algorithms [37]. While such techniques promise to explore the design space towards a specific optimal design criteria which are set apriori they do not provide a wider understanding of the design space and take into account only one possible design optimality criteria. While a number of approaches have been proposed to compose a variety of views into account when designing such multi-criteria optimisation algorithms [38] they are unable to represent a wide range of stakeholder opinions and priorities. Particularly as these opinions and priorities may change as a result of learning whilst exploring the design space and design performance.
Alternatively design space exploration has been done through systematic design space exploration techniques such as Design of Experiments and Sensitivity Analysis [39]. These methods aim to map the design space by identifying the most sensitive levers for moving within the design space toward particular goals. For example they will identify the design variables with the greatest scope to change a given KPI. While these techniques have [30] been applied to this design space problem we believe that involving a wider range of participants within the design space exploration should lead to insight in the best synergies and approaches to improve the design.

Implicit and explicit user feedback
It is worth noting that much of the crowdsourced data we capture in Table 2 is done implicitly without the users' explicit action as encouraged by [28]; by this we mean that the data recording is done automatically by the 3d environment. Explicit actions such as voting on the importance of particular KPIs or strategies or the submission of geolocated comments has been shown to be an effective mechanism for crowdsourcing [16] insight. However, we believe additional insight can be generated by implicitly tracking users interaction with the crowdsourcing environment. For example automatically tracking the motion of participants as they move through the 3d environment [36] provides a rich dataset which may be post processed to identify points of interest in the design, through the use of a heatmap. Such data can then be correlated with the design analysis currently displayed. For example if one building is seen to be particularly viewed via tracking participants 3d movement then it may be that it is performing particularly poorly on the chosen KPI and would warrant further investigation.
Another implicitly collected dataset is the selection of strategies the user chooses to be applied to the data set, we collect events based whenever a strategy is selected or deselected. This may provide insight into the order of the choice of strategies. From this the relative

Table 2
Data recorded during crowdsourced quantitative exploration of the design space and its potential insight.
Crowdsourced data source Potential insight 3d motion path at 1 s interval including look direction Geospatial recording of areas of concern or interest. Key routes and paths through the development. preference of choice may perhaps be inferred and the more "obvious" or generally accepted interventions in the masterplan identified. Implicit and explicit feedback may of course complement each other. For example when a geolocated textual comment is explicitly submitted it may be useful to view the implicitly collected the 3d path which the user has taken over the past few minutes. This would provide further context to participant's comment as the design team try to understand the collated feedback. For example it may be that the 3d motion path shows the line of sight from the residents house will be affected by the new development.

Results
In conjunction with Arup we applied this methodology and framework to an urban masterplanning project in North America. Here we present the results of our application of our proposed crowdsourcing system. The goal of this pilot application was to demonstrate the potential of the insight which may be gained for the design team. Over 1500 interaction events were collected and 839 design analyses were performed. The intention of our proposed system is to inform the design process within insight sourced from a wider range of stakeholders not normally part of the design process. This insight might be explicitly submitted as textual comment or by analysing the users interaction with the system, that is by analysing the data collected while participants generate and analyse different design variants and in so doing explore the design space.
Participants in the study were a mix of researchers and Arup engineers. Participants were encouraged to interact with the online crowdsourcing environment which may be viewed at https://www.doc. ic.ac.uk/db805/Arup/CrowdSourcing/. The environment contains full instructions asking participants to complete the following tasks: 1. Identify themselves and their stakeholder type. 2. Vote on the most important Key Performance Indicators. 3. Seek to explore the design space by creating and submitting a series of design scenarios by choosing a set of sustainability strategies to apply with a goal making a better design -the definition of this is deliberately left to the users intuition so as not to bias the design exploration.
Participants were also asked to submit textual comments at any point and explore the design spatially, they were also able to vote for particular design scenarios and individual sustainability strategies. Fig. 6 provides a list of the most voted for KPIs by the two stakeholder groups, we note a strong correlation between the researchers and engineers priorities. Carbon emissions and transportation carbon remain the biggest concern with other resource consumption metrics (energy and water) also being important. We notice that Arup engineers have a greater interest in operational expenditure than the researchers exploring the design. Table 3 shows the most commonly applied sustainability strategies as submitted in design scenarios, on average 9 strategies were chosen per design variant. Most of these strategies are aimed at reducing energy consumption or improving the use of renewable energy sources. Other popular strategies are around land use.
This mix is reflected strongly when considering the sustainability strategies voted for by the stakeholders groups shown in Table 4 where researchers tend to prioritise land use improvements while Arup engineers chose a wider spread of sustainability strategies, particularly around waste reduction. Most of the applied strategies are straightforward to implement and do not require site wide design changes such as a district heating system which is more rarely applied but may have a larger impact upon the sustainability criteria.
In comparison the implicit tracking of which KPI's participants seek to view provides a quantitative understanding of their priorities which may be different from their stated preferences, for example we might  Table 3 The top ten most commonly applied sustainability strategies.
find a user who states they are most concerned about carbon emissions but never view carbon emissions across the development and instead focus on operating expenditure. It is striking that 38% of requests seek to view the carbon implications of design decisions. Next most popular is an interest in operational expenditure which varies dramatically depending on the implemented schemes, it is noticeably more of interest to Arup engineers. We would suggest this data could be used to create a weighting function for combining the KPIs into a single objective figure for subsequent optimization. Such functions could be created for different groups of participants for example residents living within the development site compared to those adjoining. Fig. 9 shows the results of 86 design analyses performed collected via crowdsourcing during our study. Encouragingly these results show a wide potential to affect each of the KPI's considered. There is a clear trend within the design space to trade improvements in KPI's such as a reduction in carbon emissions at the cost of an increases in operational expenditure. Whether this is a good approach or not is very much in the eye of the beholder, as increased operational expenditure might be good for the residents but not for the city council who is paying the cost. The role of the design team is balance these competing concerns which are highlighted quantitatively through out approach.
In Fig. 7 we are able to understand the groups of strategies used by different stakeholders to achieve relative design performance improvements. It is noticeable that researchers focus strongly upon land use and energy strategies while Arup participants take a more even approach considering waste and water strategies to achieve an overall improvement in sustainability. It would be interesting to see whether these differences are reflected by local residents.
Another data stream collected is motion tracking of users through the environment prior to the submission of new scenarios. From this we might infer whether participants were seeking to tackle a localized performance challenge such as a poorly performing building or a blocked line of sight. An example of this localisation can be seen in Fig. 8 where "flight paths" toward a poorly performing building can be seen.
Interactive segmentation of the analysis results according to specific groups of design decisions (strategies) would provide an interesting tool for designers to understand the implications of adopting that strategy and the design space around it. This would enable leveraging the wisdom of the crowd by asking how other participant's designs performed when they used a similar set of design decisions. For example if the strategy Maximise Green Space is enabled we could filter the analysis results to identify only scenarios where participants included this strategy and identify the best performing designs found within this reduced design space. By repeating this process with all of the most voted for analyses a popular and more constrained design space could be identified. This analysis could also be segmented to find the most popular design decisions made by different stakeholder groups -for example residents may favour more green space while clients may prefer reduced operating expenditure and city planners reduced carbon emissions.
These preliminary results show the potential of the type of insight which may be gained from involving a wider audience with tools to quantitatively explore the design space, particularly with larger numbers of participants through an online system.

Conclusions
In this paper we have proposed a system for augmenting crowdsourcing with quantitative design analysis computed on demand in the cloud. We believe that this will enable inclusion of a wider range of participants in the design process and help them to explore the quantitative implications of design decisions. This is especially helpful as urban master planning is the process of balancing of design decisions and performance implications across many design disciplines. This system has the potential to provide an order of magnitude improvement in the number of design variants analysed in a typical project [1,30]. It has also enabled over 800 design analyses to be performed on demand by the crowd.
Further we believe that tracking participants through their interaction with a design and the design space will lead to further insight for the design team. This can be done both explicitly through voting for design variants, particular elements and important KPI's or implicitly through tracking a 3d motion path through the 3d environment to highlight places of design interest. This complements traditional crowdsourced geospatial commenting systems.
As discused in our Hypothesis (see Section 4) we believe that  Fig. 7. The sustainability strategy groups used by different stakeholder groups to achieve their design performance improvement. Scale is the relative number of times these classes of strategies were applied.  8. Tracking users in 3d provides insight to interesting areas of the design and highlights potential design challenges such as the two poorly preforming buildings on the right hand image (as indicated by the orange/red markers above them). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) combining these two techniques of crowdsourcing and on demand quantitative design analysis can provide guidance to the design team as they evolve the design through the design space. This insight will be formed of their stakeholders' views and quantitative advice on the performance of design variants the crowd has explored within the design space. For example we have seen that different groups of participants have different priorities and achieve design improvements through differing means. We believe this will enable more informed balancing of the competing decisions inherent within urban masterplanning. Such techniques should be applicable to wider design problems common in architecture and engineering practice.

Further work
This paper presents a working crowdsourcing environment augmented with online computation of design analysis in the cloud. The key next step is to run a field trial of the system during the development of an urban masterplan. This would enable architects, engineers and policymakers access to the system to explore the design space in real time enabling quantified testing of hypotheses. We would further seek to open the platform to use by the general public in the area of the proposed masterplan to seek wider views. Large-scale testing would enable the implicit tracking of users to generate aggregate insight on the most interesting or contentious areas of the design and the most valued KPI's.
One limitation of the current system is that it does not allow variation of physical geometry, this could be amended by provision of design tools within the game engine environment. To complement this further analyses which compute upon physical form (e.g. walking distances) could easily be added to the Hiersynth engine. The potential of this can be seen in Fig. 5 with the integrated Radiance lighting analysis. This would also enable moving beyond the massing model level of detail to provide greater geometric fidelity and design variance.
One further limitation of the system is that it is dependent upon the runtime of the underlying analysis which if complex may take several minutes to run. While most analysis runs take a few seconds more complex geospatial analyses such as ray-tracing are more computationally intense and take longer to run. This lack of immediate feedback does limit interaction with and exploration of the design space. The fidelity of the analysis models could be traded with computational runtime to address this issue.
Finally, we would seek to explore the possibility of adding an element of gamification to the crowdsourcing environment. This would enable participants to view each other's design scenarios and share designs which are local optima in the design space via a multi-parameter leader board system. This element of competition would gamify the experience and potentially induce the crowd to further explore the design space, identifying new improved design solutions which push the Pareto Front toward better optimality. We believe that the lessons learned through this system will be valuable for many engineering practitioners where similar multi-disciplinary optimization challenges are experienced.