Evidencing the impact of online youth co-production on mental health research: Findings from the MindKind study

Background: Incorporating members of the public as partners in research is increasingly prominent and is broadly defined as co-production. The inclusion of youth in mental health research co-production is particularly relevant for research on topics that directly affect them. Co-production can involve stakeholder contribution at every stage of the research cycle but how to evidence its impact on research is not well defined. Objective: To evaluate the impact of substantial online youth involvement within the MindKind Study and categorise levels of their co-production. Methods: Virtual young people’s advisory groups (YPAGs) were set up as part of the MindKind study (MindKind Consortium, 2022) to fulfil co-production principles and involve youth in the research process. YPAGs were set up at 3 sites (India, South


Original Manuscript
Introduction Increasingly, participatory and collaborative research methods are employed to decrease the gap between evidence and implementation. One way to do this is to involve target groups in research, using co-production processes, particularly the involvement of youth to reflect their needs in research [1], [2]. Co-production is "an approach in which researchers, practitioners and the public work together, sharing power and responsibility from the start to the end of the project, including the generation of knowledge" [3]. It is a type of participatory research which ensures that the knowledge generated is informed by the needs of the target group, is relevant to them, and has a greater applicative value in their local setting [4]. Thus, such active involvement of knowledge-users (e.g., young people with mental health difficulties) in research, who are experts of their experience, enhances the quality and utility of research. In this paper we aim to describe how the impact of coproduction can be measured when conducting mental health research with youth (aged between 16 and 24 years).

Co-production in mental health research with youth
Different forms of mental health researcher and knowledge user partnerships have grown in popularity, primarily in the high-income settings, especially among young people [5]. With most mental health conditions having an onset by the mid-20s [6], there is a vital focus to understand youth mental health needs better, and curate services tailored to their needs. Several benefits of such youth-researcher collaboration have been identified. For example, co-producing with youth increases relevance of research aims [7], facilitates recruitment [8], [9], and helps produce richer and more reliable data [10]. Researcher-intensive tasks like data analysis, presentation of findings and dissemination also benefit from youth involvement [11]. Furthermore, co-production also enables incorporating lived experiences effectively; with regard to user outcomes, co-production helps young people develop transferable skills [5], [12], research skills [7], and even understand their own mental health better [13].

Complexities and challenges of undertaking co-production with young people
Though co-production of knowledge may seem like an elegant solution to ensure implementation of evidence-based practices, it is in fact a highly complex, context-specific experimental process that is often time and resource intensive [14]. Effective co-production can require collaborative efforts at every stage of the research process [15]. The environment is important, as are researcher traits such as openness, tolerance and flexibility [16], and organisational qualities like building trusting relationships and innovative methodologies [17], [18].
The complex process of co-production with youth comes with specific challenges as complicated power dynamics might need to be identified, acknowledged, and addressed. Researchers need to address youth-adult hierarchies [19] that can disrupt effective collaborative work. If such traditional patterns are not disrupted, tokenism as well as difficulties in conflict resolution and shared decisionmaking between youth and researchers can ensue [20]. Finally, research processes that ensure coproduction with youth require additional time and resources due to cultural, ethical and safeguarding considerations such as assent-consent procedures for legal minors [21], and ensuring safe practices while also recognising their agency [2].

Measuring the impact of co-production
Understanding and evidencing the impact of youth co-production remains an area within the literature in need of further elucidation. The current literature on monitoring and evaluation of young people's involvement in research is diffuse, dispersed and does not have standard taxonomy or methodologies (Wilson et. al., 2020). Moreover, much of the focus has been on "measurable, economic and quantifiable impacts", which does not take into account the many dynamic processes of co-production [22]. The diversity of approaches, evaluative frameworks, and depth of discourse has made it difficult to measure the impact of co-production on research processes and outcomes. There is a general scarcity of literature on reporting the impacts of youth co-production [23] and the need to map complex processes of collaboration and the non-linear impacts for stakeholders [24]. Given these gaps, we evaluate the impact of co-production within a study of mental health among young people. We utilise the impact framework proposed by [22] and its definition of impact categories. The framework uses a multi-faceted lens to understand the breadth of co-production at different levels, the interactions between these levels and emergent mechanisms. The emphasis within this framework is to "consider longer term developments, wider social changes, any unintended consequences and how co-produced research might affect and be affected by different power dynamics" [22].

The MindKind Study
The MindKind study used mixed methods to elicit young people's views on data governance and to assess the feasibility of setting up a user-controlled global mental health databank (see MindKind protocol [25]). The study was conducted in India, South Africa, and the UK. Participants aged 18-24 years (16-24 years in the UK) were enrolled on a separate website and randomised into one of four data governance options [25]. Using an app designed for the study, participants were invited to share data on key active ingredients of mental health, such as sleep, body movement, social connections and positive experience over a 12-week period. Study participants in the qualitative arm took part in country-specific and multinational group deliberative democracy sessions [26] focused on building consensus pertaining to governance of a future mental health databank. Each arm of the study had a strong co-production focus. Ethical approval for the main study was obtained from each of the country sites and the US.

YPAG overview
Each study site employed a full-time youth lead; a 'professional youth advisor' (advisor) to convene and run the young people's advisory group (YPAG) and advise on all aspects of the study. In addition to the in-country YPAGs, a multinational YPAG composed of young people from the three sites also advised on study decisions. Each advisor had one vote on the study's Steering Committee and therefore actively participated in study decision-making. Sessions conducted with the in-country and international YPAGs were planned collaboratively by the research teams. Sessions were recorded and feedback shared by YPAG members was passed on by the advisor to the research team via an online platform called AirTable. Researchers were then required to respond to feedback in AirTable clarifying whether the feedback has been actioned, and if not, why. Throughout every stage of the project, the work conducted was described to the YPAG with constant feedback on their input. We have described youth involvement in MindKind in Figure 1.

Data collection
The study was conducted between September 2020 -July 2022 with youth co-production taking place throughout. Data collection to evaluate the impact of co-production was conducted sequentially and separated into four sets of stakeholder groups: youth (advisors, YPAG members); researchers (all staff who contributed to research design, data collection, and analysis); decisionmakers (principal investigators, funders); and support and administrative staff (defined widely to include anyone who undertook significant administrative and support responsibilities as part of their role). First, we collated various project documents generated between September 2020 -January 2022, such as meeting recordings and minutes, project descriptions, and work plans, to determine coproduction outcomes relevant to all stakeholders. A total of 17 documents were downloaded from the project database for analyses. A summary of this is available in Table 1. The outcomes identified at this stage allowed us to refine subsequent data collection on significant changes for stakeholder groups. Second, building on the identified outcomes, we generated data on main changes to study activities that resulted from youth co-production for all stakeholder groups, based on the "Most Significant Change" technique (MSC) [27]. The MSC approach involves gathering stories of the most significant change from key stakeholders via open questions to distil key learnings about the study and recommendations for the future. We implemented the MSC technique synchronously via Zoom with in-country and international YPAGs. Here, in-depth group discussions were conducted by researchers and advisors experienced in qualitative data collection and analysis and YPAG facilitation respectively. We also collected data on significant changes from other stakeholder groups by way of open-ended online surveys. The survey solicited respondent views on significant changes across 4-5 specified outcomes for each stakeholder group. This online survey was emailed to all the stakeholders in the study and completed by 15 participants. All survey responses were de-identified prior to analysis. << Insert Table 1 about here>>

Data analysis
Analysis across the whole dataset focused on building a picture of key changes that occurred for different stakeholder groups as a result of implementing youth co-production in the MindKind study and understanding how stakeholders perceived its functioning and progress. We used a qualitative approach to collate and synthesise all data collected. This took place at two-time points; December -February 2022 and March -May 2022.
The first stage consisted of document analysis and was performed by the lead author with inputs from team members. All study documents pertaining to youth co-production were categorised under individual, group, organisational, infrastructural, and paradigmatic for all stakeholders to align with the impact framework proposed to be used [22]. Following discussions with other researchers and advisors a framework in MS Excel was created (see Table 2). The second stage consisted of analysing the MSC data collected both synchronously and asynchronously with all stakeholders. This analysis was performed collaboratively by three researchers, three advisors and seven YPAG members from the UK and India who had expressed their interest in being involved in data analysis. The analysis consisted of transcribing Zoom recordings of YPAG sessions, and categorising these into defined topic areas for the youth stakeholder group. It also involved categorising open-ended questionnaire data into defined topic areas for other stakeholder groups.
To increase the reliability of findings, a final stage of analysis was undertaken where the MSC data was then co-analysed using an iterative voting approach [27]. This involved all analysts discussing the key impacts, identifying the most impactful quotations shared by participants from each stakeholder group, discussing reasons for their choices, and then voting again. A preliminary version of Table 3 was then developed and revised after discussion with advisors and YPAG members. << Insert Table 2 about here>>

Overview
A total of 31 stakeholders, working across all MindKind study sites, belonging to four broad stakeholder groups participated in interviews : 11 youth (3 advisors, 7 members across YPAGs in India, South Africa, the UK, and an International YPAG, 1 lived experience consultant at the funding organisation); 11 researchers in India, South Africa, the UK, and USA; 6 decisionmakers (individuals responsible for key study decisions, such as principal investigators and youth) and 3 support and administrative staff. They had between <1 to 25 years of health research experience. Where stakeholders belonged to more than one group, the primary group affiliation was used for the analysis (e.g., a professional youth advisor could belong to the youth, researcher, and decisionmaker stakeholder groups). Participants were based in both high-and middle-income settings.
We found that impact occurred at five levels detailed in the framework developed by [22], namely at the paradigmatic, infrastructural, organisational, group, and individual levels. Table 3 about here>> Paradigmatic impact: Big-picture learnings for co-production

<< Insert
Paradigmatic impacts are those impacts of co-production with potential to modify ways of understanding and shift frames of reference [22]. We identified two paradigmatic impacts of coproduction in the MindKind study: first, co-production enabled new and unexpected ways of doing research, and second, co-production modified research priorities.
With regard to the first impact of new ways of conducting research, we observed that co-production prompted stakeholders to develop innovative methods to engage and recruit youth from diverse contexts to participate in the study. To ensure diversity was maintained, youth were deliberately recruited to represent a varied background in either research experience, ethnicity, geographical location, gender or disability status. Such innovation relied on improved digital/technological flexibility which co-production encouraged.
A key impact was the introduction of AirTable, weekly youth-focused meetings, and relationshipbuilding among individual site teams. These helped to improve the process of co-production by broadening opportunities to participate with separate spaces for youth to share feedback, creating frameworks for all stakeholders to engage with one another, and setting up multiple avenues for examining how co-production goals were being met. As one stakeholder put it: "We have to create spaces to explicitly solicit young people's feedback; just "having them in the room" may be too overwhelming an environment to engage" (Researcher).
Accountability mechanisms were also important in tracking the implementation of ideas generated. With regard to the second impact of modifying research priorities, we found that through coproduction we used points of dissonance between stakeholders to improve how study activities were decided, key research decisions made, and research knowledge generated to align more with youth priorities, life experiences, and contexts. Due to this, we decided to integrate more capacity building for youth to effectively advise and participate in research implementation.
These impacts were not straightforward to achieve. A shared mission and common goals were important in guiding the overall direction. Timing was critical, as a number of early decisions had to be taken (due to logistical and funding constraints) before the co-production systems had been set up highlighting the importance of timing and early engagement as otherwise there is a risk of disengagement among youth stakeholders: "If one is going to be doing research around youth, then involve youth from the beginning right through to the end. [I think MindKind didn't quite do this as we had already developed the App and the areas to focus on before we started the youth advisory groups… we did a bit of "roughshodding" and hoped they would agree with us!" (Decisionmaker) Capacity building for youth in order to advise on research implementation and dissemination needed significant additional time and resources, especially from advisors and early career researchers, this was exacerbated by tight funding timelines. These constraints meant that there were contexts where co-production was limited, and the study had to consider meaningful ways of course-correcting. Our systems for reflection in multiple teams, forums, and for different groups were critical for ensuring that meaningful engagement was possible in the key decisions at each time point whilst acknowledging that some decisions had to have been taken before all the youth systems were in place, sometimes requiring course correcting along the way.
"Many discussions around involvement have revolved around equality. However, I thought it was important to emphasise equity. This meant taking a unique approach which focused on youth capacity development and ongoing reflexivity rather than assuming involvement just meant a seat at the table." (Youth Stakeholder)

Infrastructural impact: Constraints and opportunities in existing systems
Infrastructural impacts include wider social, economic, policy and political impacts (Beckett et al., 2018). We considered infrastructural impacts to include the research, administrative, and funding infrastructures we work under.
An infrastructural impact was that we experienced a constant tension between project timelines and deliverables and co-production aims. To undertake co-production fully, we needed to invest significant time and effort on capacity building, tailoring existing systems, and mentorship, but these were not always compatible with study timelines. Furthermore, pressure to produce reports or updates in line with project milestones were often at odds with the time that was needed to effectively co-produce that specific aspect of the study, whether that was about the design, data collection, or analysis.
Factors that facilitated positive infrastructural impact included greater funder and administrative support to ensure these tensions were addressed appropriately. Flexible timelines and timely involvement of youth were also critical in ensuring that youth stakeholders could be fully onboarded and included in the project as equal partners in research. This also means that research funding was critical to timing, as funding timelines exerted pressures on studies and co-production itself needs resources available at the funding application stage to be done appropriately. Moreover, pressing study times lines affected capacity building, tailoring existing systems, and mentorship.
Infrastructural impacts varied across contexts. For example, South African advisors frequently faced planned electrity outages or 'load shedding', which hampered their ability to meet virtually on a regular basis. YPAG members also did not consistently have access to the internet themselves and their agreed remuneration were delayed due to wider issues with financial administration, both of which negatively impacted their ability to engage. On the other hand, advisors in India and the UK faced fewer infrastructural issues, which aided regular engagement and advice from YPAG members. These findings underscore the importance of infrastructure in a wider sense, including access to the internet, and responsive administrative systems that were essential in facilitating effective coproduction.
Co-production had positive impacts on research dissemination, for both formal (journal publications, conference presentations) and informal (blog posts, seminar discussions, internal presentations) outputs. For example, we co-analysed and co-wrote our youth co-production publications which were shaped in distinct and valuable ways by youth stakeholders. We also shared ongoing learnings of co-production throughout the study with other researchers and practitioners.

Organisational impact: Differentiated communication practices and the feedback loop
Organisational impacts include impacts on rules, norms (culture), practices, and organisational structures [22]. We found that the most critical organisational impact was on (i) ensuring that a feedback loop was in place to communicate with stakeholders, especially youth, and that this loop was closed in a way that stakeholders felt heard; and (ii) implementing differentiated communication practices for different stakeholders. With regard to the feedback loop, we set up an AirTable system (as mentioned above) to collate feedback where feedback from YPAG members and advisors were recorded, research and project teams would then action the feedback. If implementing the feedback was not possible, they would explain why it could not be done. While this was an encouraging start, we found that the critical component was ensuring that feedback loops needed to be closed in a way that youth felt they had been addressed. Our data collection on AirTable also had to be made more conversational to engage with youth, so our formal data collection platforms were supplemented by meetings where youth could share their feedback verbally in addition to written feedback.
Another key learning was to establish differentiated communication structures and practices to account for the diversity of stakeholders: "Communication must be done continuously and through multiple channels. Because everyone learns and contributes differently" (Decisionmaker).
Given that effective communication meant something different for each stakeholder group, our communication practices needed different frequencies and detail levels as well as a more inclusive information-sharing strategy to include youth. This sometimes meant that given the large number of stakeholders to consult with, decisions needed more time to be taken. We also implemented other changes to address these issues; for example a weekly digest email was introduced to include important announcements, quick links to project documentation, and a list of the upcoming week's meetings, including any scheduled YPAGs. Importantly, the digest was designed to ensure that everyone including the advisors were informed about decisions to be made, who was making them, and when/in what format feedback would be welcomed. These digests became an important tool to increase transparency and inclusivity.

Group impact/interpersonal relationships: Authenticity and cycles of engagement
Group level impacts constitute interpersonal and stakeholder relationships within a system [22]. We observed the following group impacts: (i) authentic relationships between advisors, and advisor and panel members; (ii) YPAG members feeling engaged and invested, and at other times, feeling disengaged; and (iii) long-term relationships and collaborations between researchers and advisor. Firstly, with respect to authentic relationships among the advisors, this was aided by creating multiple channels of communication with them and among them. The advisors and the study team met every week during the course of the study to discuss updates, youth integration, challenges and learnings. The advisors also met each other, together with an external 'youth involvement advisor' (working for the funder) for regular check-in meetings. They also had a monthly scheduled check-in with each other. Secondly, the advisors were responsible for leading their own YPAG and encouraged to foster authentic relationships with the members. They not only met the YPAG as a group on a fortnightly basis, but also conducted individual check-ins with the members to understand any concerns or feedback from them. Overall, the project team treating the advisors and YPAG members with curiosity and professionalism was helpful in fostering those relationships. Providing the YPAG members opportunities to upskill, spending time and resources on engagement, for example, actively soliciting youth feedback, trying innovative ideas, and offering multiple avenues for youth to engage and contribute to the study beyond the role of a youth advisor (capacity building activities, involvement in manuscript ideation and writing) helped increase engagement. Further, closing the feedback loop in a way that youth felt it had been closed was helpful, but when this was not possible, it could lead to disengagement if not properly explained. "Some key things to consider when engaging and retaining youth in the MindKind Study -Providing due credit to young people where necessary and allowing them to be key decision makers who hold equal power as adult staff on the team; Providing compensation/reimbursement/honorarium for the time that youth engage with us; Providing young people with constant opportunities to upskill (through workshops, manuscript involvement, consultancies on projects aside from MindKind); Ensuring that young people are given support when needed (emotional/support with work etc.)" (Youth) Collaborations between researchers and advisors were established over time and were beneficial to overall youth engagement. This was done through multiple channels, such as organising check-ins between researchers and advisors, as well as directly involving advisors in all research related activities.

Individual impact: better skills, knowledge, and capacities
According to Beckett et. al. (2018) [22], individual impact constitutes characteristics of stakeholders, including biological and psychological aspects, like improved mental or physical health, improved practice and skills for practitioners. For advisors and YPAG members, major individual impacts included greater insight into one's own mental health, increased knowledge of research concepts and processes, opportunities to harness lived experience of mental health challenges, remuneration for time and skills, and new academic, presentation, and study skills. The advisors, as well as YPAG members, reported developing better insights into their own mental health by participating in project activities and having discussions about a future global mental health databank. Further, they reported increased skills and knowledge development of research related concepts.
"One of the most significant changes for me is probably the importance of research and studies which, which I mean, are now involved in but just in general, for the understanding of mental health… it's helped me look at mental health as more collective thing, rather than being more focused on the individual just because, like we, we convene, we talk on a weekly basis, but also the moving parts and the systems which have to be in place for some sort of change to happen. And some sort of, like, insight into mental health and research to actually come to fruition." (Youth) These impacts were mediated by the following factors: having site-specific and international YPAGs led by advisors; capacity building for advisor to independently coordinate and lead several study components; regular capacity building for YPAG members to advise on study; and dedicated spaces for youth to interact with each other. This form of capacity building for example involved dedicating sessions to demonstrate research in practice. Customised support for advisors to suit their backgrounds and interests was also critical in enhanced outcomes.

"Young people's social and cultural contexts affect their knowledge and opinions on mental health. While this was something that I had some idea about prior to the study, having regular conversations with advisors and researchers across the sites led to interesting learnings on how social and cultural contexts play a role in shaping a young person's perspective and knowledge on mental health (for example, what mental health means to a young person in a HIC vs what it means to a young person in an LMIC." (Youth) Further, there was an increase in understanding of young people's life experiences and contexts by decision makers and researchers:
"I think we learned a lot about how youth wanted to be engaged… the decision making on the side of the research team needed to happen at a much faster pace than the youth were comfortable (or able) to make decisions. As a result, we sometimes just "told" rather than "asked". I think the youth forced us to slow down and demanded inclusion, which was a very powerful change to the study." (Decisionmaker) This was mediated by ongoing reflexivity by the research team and decision-makers, as well as course-correcting based on feedback by multiple stakeholders.

Summary of findings
Our findings suggest that youth co-production in the MindKind study had significant positive impacts for all the stakeholder groups (youth, researchers, decisionmakers, administrative and support staff). In addition, having an online YPAG allowed engagement from a diverse group of panel members, allowing for a wider reach and conducting sessions at convenient times. We found that co-production impacted on all five domains of practice: paradigms (new/unexpected ways of doing research and modified research priorities), infrastructure (constant tensions between study timelines and co-production, stresses on institutional infrastructure, co-production dependent on context, the messy doing of co-production), organisational (differentiated communication needs, practices, and outputs, greater emphasis on closing the feedback loop, better access between youth and research institutions), group (more authentic relationships between youth and between youth and other stakeholders, ebbs and flows in engagement), and individual (increased skills and capacities for all, opportunity to use lived experience of mental health for youth, increased knowledge of youth life experiences for researchers, decisionmakers). Furthermore, we found that some impacts not only occur during the research project but beyond and are therefore harder to measure and document, but cannot be ignored. For example some paradigmatic changes also impacted other levels (e.g., individual, group) and influences can be bidirectional.

Implications and recommendations
In unpacking the impact of co-production on the wider paradigm or conceptual or theoretical insights, we found that involving youth led to better outcomes overall not only for youth, but for other stakeholders and the research as a whole. Undertaking the MindKind study in partnership with youth constantly challenged stakeholders on their assumptions of what research could look like. This aligns with the wider literature, which emphasises that involvement of stakeholders in research is linked with greater effectiveness, given the articulation of a shared mission and goals [28]. We encountered several barriers in implementing co-production according to its tenets and in line with our shared mission, a process characterised as antagony [29]. Our systems for reflection and discussion were critical in addressing these barriers effectively, which is a characteristic of effective partnerships [28]. Our findings also point to a need for ongoing monitoring and evaluation of impact but also to ensure that this is set up as early as is practicable, in the research cycle. Our experience of co-production as an active process of valuing all types of knowledge and experiences and creating a balance between different stakeholders aligns with others' experiences [30]. We also found that co-production was enacted, implemented, and received differently by stakeholders in India, South Africa, and the UK (MindKind study sites). Equally, public and institutional infrastructure (e.g., access to internet) also played a significant role in the way coproduction was implemented and received in the different contexts. While such challenges broadly map on to the literature on co-production in low-and middle-income countries (LMIC) versus highincome (HIC) countries [23], we found that there was variation within and among specific LMIC contexts in our study, such as between South Africa and India. Our findings underscore the concept that co-production is highly 'place-based' and takes place in particular social, economic, and ecological contexts [31]. An important organisational impact we noticed was the change in communication practices and outputs needed to undertake co-produced research and ensuring that there was a feedback loop in place and stakeholders (especially youth) felt that this was closed in an appropriate way. Not only does this cover onboarding but this also includes understanding varied communication needs and changing communication and decision-making practices. These findings are consistent with previous literature which highlight that effective co-production is achieved when there are positive working relationships within and between teams, stakeholders feel heard, and differentiated communication helps create shared meanings of concepts [28]. Our key mechanisms of achieving this impact included establishing feedback loops and testing these to ensure they were fit for practice and supporting administrative staff to implement these communication practices. These practices are characterised as 'maintenance tasks' in the wider literature which support the functioning of partnerships by addressing core administrative and support needs [28].
We also found individual and group-level impacts that extended to increased knowledge, skills, and capacities for youth and other stakeholders, youth being able to use their lived experience of mental health and feeling heard, authentic relationships between YPAG andadvisors, and ebbs and flows in engagement over the project lifecycle. Many of these findings align with the wider literature on the impacts of youth co-production [1], [23], [32].
Our experience highlighted the importance of inclusion and representation which means including youth of diverse backgrounds and experiences, which we found greatly enhanced outcomes. We also found the presence of dedicated advisors to be critical in achieving these impacts, which we recommend as good practice in future studies, if that aligns with study aims and is facilitated by funding.
The key infrastructural impact was the constant tension we experienced in ensuring co-production was conducted effectively when there were time-specific project deliverables. Our experiences, while not directly aligned with the following, broadly map on to practical costs and challenges of coproduction such as large administrative burdens, increased researcher time and resources, lack of training on implementing co-production, and insufficient funding and resources to effectively undertake co-produced research [33]. While we found significant benefits of undertaking coproduction, we found that grappling with these time and financial costs could lead to a diluted or 'coproduction lite' version. Despite this focus and funding support, our experience nevertheless highlights entrenched structural challenges in undertaking co-produced scientific research more broadly [34]. Our findings highlight the need for co-creating the infrastructure needed to undertake co-production, resourcing it appropriately, and recognising the time and skills needed to co-produce research [33].

Youth involvement in this paper
In writing this paper, we were committed to ensuring meaningful involvement of young people, but the aforementioned funding and timeline constraints meant that such involvement needed to be proportional and pragmatic. We therefore involved the YPAG in areas where they had particular knowledge, interest, or skills, specifically data collection, analysis, and interpretation of findings. We conducted the data collection synchronously with domestic and international YPAGs, and asynchronously with advisors. We analysed the data in collaboration with advisors and YPAG members who were interested in doing so, and jointly interpreted findings and decided on specific findings to be presented in the paper. YPAG members contributed significantly to this process. In relation to dissemination, we also met with the YPAG members who had participated in data analysis and interpretation to understand how they would like to be acknowledged in the paper (in this case, as co-authors).

Conclusion
While the resources involved in undertaking co-production are significant, they yield significant benefits for varied stakeholder groups and across several domains. We have attempted to capture these impacts and their enabling or disabling factors in further detail, in the context of the MindKind study which focused on young people's views of mental health data governance in India, South Africa, and the UK. We found that "effective co-production emerges in practice" [35] and numerous opportunities and challenges arise in the 'doing' of co-production. No one size fits all co-production efforts. In sharing the impact of co-production on varied stakeholders and outcomes, we hope to have additionally demonstrated the value of measuring and reporting outcomes associated with youth co-production [23]. We recommend that monitoring, evaluation, and learning systems are designed and implemented early in co-production studies to enable more systematic reporting on youth co-production and its impacts. Future research should also examine how we could standardise the reporting of youth involvement in research; the GRIPP guidelines provide a useful starting point [36]. More frequent and detailed reporting of the impact of youth co-production will likely challenge current funding and research infrastructure constraints and enable better health research outputs.

Figure 1
Youth involvement in the MindKind study.  Table 1 Summary of data source Table 2 Outcomes intended to be achieved Category Outcomes

Paradigmatic
Better conceptualisation of study questions, measures, and design; iterative study design; reorienting common research practices and assumptions.

Infrastructure
Research that values youth voices; insights into co-produced research, conducted across sites with diverse youth; exploration of youth preferences and attitudes on a range of topics pertinent to data governance and mental health; youth-informed dissemination strategies; empowerment of young people as data users.

Organisation
Established communication channels between all stakeholders; accountability mechanisms for receipt of feedback; increased connection between youth and research institutions; differentiated information outputs based on different information needs of stakeholders; time and effort invested in team building to fully integrate youth.

Group
Engaging and authentic relationships between stakeholders, especially between youth and other stakeholders; youth-driven rules of engagement for advisors and YPAG members (for youth); feedback from youth sought and responded to.

Individual
Increased knowledge and capacity of research concepts and processes (for youth) and contexts of youth lives (for other stakeholder groups); adequate compensation for time and effort (for youth); networking and skill development opportunities (for youth); insights into own mental health experiences (for youth); feeling heard (for youth).

Table 3
Impact, process, and mechanisms of youth co-production in the MindKind study

Level
Research process Impacts Key mechanisms or elements

Select quotations
Paradigmatic Adhering to the principles of coproduction Impacts at a conceptual or theoretical level: New and unexpected ways of conducting research: (1) Developing innovative methods to engage and retain youth panel members in diverse contexts; (2) Incorporating accountability mechanisms into research; (3) Acknowledging that co-production is messy but creating spaces to learn and reflect; (4) Promoting digital/technological flexibility in research Factors which assisted impact at this level: (1) Good or bad timing can greatly impact outcomes, so coproduction must be timed appropriately; (2) Costs and benefits of co-production are both high, so finding the right balance between barriers and facilitators is key; (3) Coproduction is messy, and there are times we will fall short, so making space to reflect and learn is critical; (4) Shared mission and goals that are clearly articulated "We need timely involvement and for this to happen in this project it would have not been feasible as the delays to the project would have been too great and so we need to be mindful that we do not make the barriers to coproduction so high that no research gets done, so finding ways to make this feasible for other projects-as not all projects will have recourse to the funding but might be impactful. It is also quite a substantial investment and probably better to have YPAGs that are shared across projects to make them more sustainable." (Decisionmaker).
1. "It is important to build a trustworthy and meaningful relationship with the youth when soliciting feedback from them. 2. Young people must be provided with opportunities that will provide them with an advantage/incentive. Engaging them on only one aspect/part of the study will lead to a dip in engagement. 3. Use simple and easy language when communicating with the youth. Avoid research jargon.

Understand a young person's context and how it affects their communication and participation in research.
Ensure that enough support is provided to them to meaningfully participate in discussions (emotional Modified research priorities: (1) Understanding dissonance or points of divergence between youth and other stakeholders; (2) Incorporating capacity building initiatives in research; (3) Incorporating youth priorities (may/not include research) in study design support, capacity building for research etc.)" (Youth) "...Once the project was up and running I took a step back and continued to support advisor with reflecting and building community around their work. My monthly spaces with advisor were a chance to debrief, reflect and connect with one-another as well as share knowledge and learnings. These spaces were a brilliant opportunity to link involvement across sites, especially in a project with many meetings and high levels of complexity. Overall, this project is an opportunity to push conceptions and imaginations around involvement. Holding complexity and equity across multiple country sites. It has been a fantastic learning experience for myself, and I hope it has been impactful for other members of the project team." (Lived Experience Advisor)

Infrastructure Infrastructural actors involved and how
Higher education institutions in sites and USA; non-profit organisations in South Africa and USA; charities in the UK

Infrastructure impacted:
(1) Project timelines and co-production aims: constant tensions between the two; (2) Knowledge outputs (presentations, journal articles, blog posts) coproduced with youth; Co-production efforts sometimes fail, so important to create Factors which impact at this level: (1) Administrative support that is resourced appropriately; (2) Tension between project demands (funding, timelines) and co-production is a reality; planning for this and constantly reviewing this is key; (3) Timely involvement "The project was designed to move fast -faster than a study usually would. But meaningfully involving youth is done by creating time and space for those engagements. There was a constant tension between meeting deadlines and involving youth meaningfully". (Decisionmaker) "Supporting meaningful youth involvement is timeconsuming, and therefore costly. Significant time and effort must be expended on capacity building, contextualising, and mentorship if youth will be expected to contribute honest and actionable feedback as members of the team. It is not possible to meaningfully incorporate youth leadership into a project by simply including the youth in the project meetings". (Support staff) (1) Increased emphasis on closing the feedback loop and ensuring that youth feedback is addressed (or reasons provided for why it could not be addressed); (2) Increased bridging of gaps between youth and research institutions; (3) Differentiated communication

Factors which assisted impact at this level:
(1) Establishing processes for closing feedback loops; (2) Understanding communication preferences of different stakeholders; (3) Supporting administrative staff to create customised reporting and communication structures "The major learning for me is the importance of ensuring youth feel that the feedback loop has been closed. We did a poor job of this early on in the project and I think our AirTable forms, as well as the weekly digest emails from [Staff] and the structure of our Monday meetings helped in closing the feedback loop and ensuring things did not fall through the cracks. But this is still a work in progress!" (Researcher) "By virtue of the fact that we were brought on board as panelists, I think, the way we gave inputs, it was almost immediately inculcated or noted down. And in the next meeting we would see the suggestions being incorporated. I definitely felt like I was a part of the research group. Even when we were making decisions about the study, I think we were making informed decisions, since we were presented with certain existing data to take a call before sharing any suggestions. We shared from our personal experiences too, but along with data informed decisions." (Youth) (1) Authentic relationships between advisors, and advisors and panel members; (2) YPAG members feeling unheard when feedback was not incorporated; (3) YPAG members feeling engaged and invested, and at other times, feeling disengaged; (4) Long-term relationships and collaborations between researchers and advisors Factors which assisted impact at this level: (1) Treating advisors and YPAG members with curiosity and professionalism; (2) Opportunities for youth to upskill; (3) Closing the feedback loop in a way that youth feel it has been closed (or not doing this, which led to disengagement); (4) Spending time and resources on engagement e.g., actively soliciting youth feedback, trying innovative ideas, and offering multiple avenues for youth to engage; (5) Providing non-youth staff with resources and time to build relationships "Some key things to consider when engaging and retaining youth in the MindKind Study -Providing due credit to young people where necessary and allowing them to be key decision makers who hold equal power as adult staff on the team; Providing compensation/reimbursement/honorarium for the time that youth engage with us; Providing young people with constant opportunities to upskill (through workshops, manuscript involvement, consultancies on projects aside from MindKind); Ensuring that young people are given support when needed (emotional/support with work etc.)" (Youth) "I think researchers could benefit from being digitally/technologically flexible. Platforms like WhatsApp, Miro, Zoom chat, etc. have provided rich opportunities for exchange, and especially in an online-based study, "traditional" researchers need to adapt to the tools that work." (Researcher) (1) Site-specific and international YPAGs led by advisors; (2) Capacity building for advisors to independently coordinate and lead several study components; (3) Capacity building for YPAG members to advise on study; (4) Dedicated spaces for youth to interact with each other; (5) Customised support for advisor to suit their backgrounds and interests; (6) Ongoing reflexivity to coursecorrect "It has been really interesting to see the level of knowledge that YP have about data use, which reinforces the need to provide a solid education about these things. For example, a lot of young people involved in research were not aware that data is generally pseudonymised and made very difficult to identify individuals. I have also become a lot more aware of barriers faced by young people in low-and middle-income countries -including lack of internet access, load shedding, and difficulties accessing public facilities because of safety concerns." (Researcher) It has really helped with school, sometimes learning things during the panel. Having an external knowledge source. (Youth) I think we learned a lot about how youth wanted to be engaged… the decision making on the side of the research team needed to happen at a much faster pace than the youth were comfortable (or able) to make decisions. As a result, we sometimes just "told" rather than "asked". I think the youth forced us to slow down and demanded inclusion, which was a very powerful change to the study. (Decisionmaker) 21 contexts (researchers, decisionmakers)