Participants
Of the 63 authors from the publication references and 8 who were recommended by informants (some of whom were also in the reference list), 15 agreed to participate. These 15 researchers (n=13 women) represented eight international studies involving eight countries. We were successful in recruiting two researchers from each study, at least one from a more managerial or senior researcher position, apart from one study for which only one informant could join. All interviews were one-on-one, except one in which both researchers decided to perform the interview together. In the results, we denote an informant by the number of their study (1-8) and either A or B, which only denoted which interview was performed first, not their position within the research team. Additional information about each informant is presented in Additional File 3.
Themes and paper structure
Five overarching themes were identified: 1) Project Team, 2) Study management, which included management technologies, 3) Study plan, 4) Intervention, 5) Participants. This paper will focus on the first two main themes. Sub-themes included: Roles and responsibilities, Methods, Changes, Challenges and solutions and Expectations vs. reality.
Figure 1 illustrates these two main themes, which research question they answer and the order of sub-sections within the Results. The sub-themes are not explicitly listed here because one sub-theme could apply to several main themes. It was therefore difficult to report them separately or in sub-headings. Instead, sub-themes are represented within “comprehensive insights”, as it applied to the research questions. See Additional File 4 for a description of how sub-themes are represented as “Insights”, which form the Results sub-sections.
Figure 1. Illustration of Research questions, Themes, Insights (i.e. Results sub-sections), and their order within this text.
Theme 1 - Project team
Insight 1: Gaining experience on the job
Informants were asked several questions about their intention for the study, motivating factors about working in their project and any challenges that they experienced personally.
Success was seen as different concepts [Informant 3A: “we had all done clinical trials before so we knew the process of applying to the ethical committee…how to run it… in what order and how to have quality in our work”]. However, one-third of our informants were not experienced in digital health studies (Additional File 3). Some were not versed in the specific role they performed, while others were foreign to the research area [Informant 1A: “In a way it was…not at all my research area and of course that was also a bit difficult for me during the project…leading something that I did not know so much about”]. Project management was seen as a difficult task for many researchers [Informant 1A: “we are not educated to do [project management]”].
Not knowing a field or role was not a reason to not try [Informant 1A: “I am not a technology person. I am a nurse…I was in a way just put into this leading role so, I think in the start I did not know very much about how I should do it…it was trying and failure”]. The researcher role was described as more of a continuous learning opportunity – an opportunity to be a perpetual student [Informant 8B: “I am not an expert on [statistics]. I have to read a lot of this and study it a lot of this. I still have some things that I don’t understand fully”].
Informants’ motives for joining an unfamiliar territory were the potential of technology in health and the opportunity to work with interesting colleagues [Informant 6B: “[I] learned about [PI’s] team and this new avenue of digital medicine…got really excited about it and decided to transition into that role where I could work on a really cool team and also be at the forefront of technology”].
Insight 2: Building the research team
When asked to describe the team that they worked with, informants focused on the structure and intention behind the project teams they hired and worked with. Ensuring that the project team had the necessary competencies happened during hiring and recruitment [Informant 4A: “when bring new people in, I know what we need to train them in and where to put their skills. We will usually have a mix. They need good administrative skills for the project management and then they need good research skills for different components”].
Most informants noted the importance of their team members [Informant 4A “I think it is the people who makes [a project] successful”]. The diversity of team members’ backgrounds was considered a strength to interdisciplinary projects. Two common challenges to this ideal team building situation were finding unique and necessary competencies in the relatively new field of digital health, and time.
In more immediate situations, such as when the need for additional staff is emergent, time to perform such thorough vetting is not always available. In Study 6, the unexpected need for patient support services required hiring, training and receiving ethical approval for additional staff mid-study.
Insight 2.1: Impact of team members’ attitudes and support
Informants were asked about the interpersonal relationship within the team [Informant 4A: We had very different disciplinary lenses - we had very positivist thinkers and very constructivist thinkers…Still being able to bridge that epistemological divide was really encouraging. I think it is because those people were just so invested and wonderful and wanted to learn.”]
Setting aside personal ego or professional agenda had a positive effect on the team’s ability to work toward a common goal. The concept of having “common ownership” meant considering the project as the team’s project, and not only the principal investigator’s (PI).
Support from one’s team members could make up for the challenges of interdisciplinary work. Despite being confused about the project, support from team members kept some informants engaged.
Informant 7B: “I found that [the PI] was really receptive to my thoughts and ideas…that can be a frustrating experience, when you feel your voice is not heard, but… [The PI] deferred to my judgement 90% of the time… and that just felt really nice…to have my voice and my expertise and my opinion and perspective be valued by the team. So that was probably what kept me going”
Insight 2.2: Team member engagement, not guaranteed
When asked if team members knew what was expected of them and when they were expected to do it, Informants described a sort of “team member attrition” over the course of a project. By the end of a project, most informants described that the responsibilities fall on the shoulders of a few individuals.
Reasons for decreased engagement were often competing responsibilities or being offered competing opportunities. One informant noted strong team member engagement at the beginning of Study 1. However, the other noted also the challenges to recruiting collaborators before the study.
Informant 1A: “Some were on sick leave... Other persons, they had new positions..[or] not very into it and were engaged in other things and were busy… engaged in teaching and other projects. So, it was not easy to get them to have time…both municipalities [for] long periods, it was difficult to get in touch with people at all”
Insight 2.3: Roles and responsibilities: the importance of one person, one role
As a follow-up question, we asked informants to elaborate on the realities of working with or managing a team in which many aspects of the project changed over time. In Study 3, dividing responsibilities amongst the team was seen as valuable and effective [Informant 3B: “[Technology manager] was amazing in this project, so it was extremely important to have a dedicated resource for the technical support. So [they] provided technical support to our patients here and also…to the local teams in the other countries”].
Study 8 was an iterative study, making it uniquely capable of learning what to change from previous iterations in order to optimize resources and workflow.
Informant 8A: “we rigged an organization around it in a different way. We had a project manager that was a professional project manager. We had me as kind of a daily manager…to do the logistics of the participants, to manage the staff…And we had one person who worked on HR, just recruiting new people and doing kind of the administrative work around that. And one financial adviser…Was a major job. And then we had one person to work, working on mediation. Recruitment of participants and outgoing information to the public”
While some argued for a dedicated person for major tasks, others realized that this was not feasible [Informant 6B: “I quickly realized that I was getting overwhelmed. Trying to keep up with sending…100 devices per day…The project manager…had a lot of experience with running big clinical trials…I think she realized too that I was getting a bit overwhelmed].
Some informants noted their own challenges of holding dual roles [Informant 3B: “The optimal situation would have been to have…a project coordinator who doesn’t have the scientific responsibility,… There are a huge amount of activities which maybe one tends to underestimate when you start”]. Many faced the same barrier to hiring such a diverse research team - a limited budget.
Insight 3: The work environment
Informants were asked to expand upon what contributed or hindered effective teamwork. Again, the concept of putting the project’s needs first was beneficial.
Informant 8A: “The environment that you work in is very… an essential thing. You have to have trust between the technicians [research staff]…the feeling of one pulling in the same direction, is important. Instead of “this is me and mine” and “I want to leave early” and “I don’t care if this one has to work two hours more because I leave”. So, you expect a lot from the technicians that work there. Then you have to lay a very good foundation”
Work environments were benefited by the project leaders’ acceptance of the reality of the research situation and value of research team members.
Informant 6B: “I think it really helped, [saying] “…you can all talk in our team meeting…We are all kind of learning together…You can offer some really valuable insights coming from an administrative point of view… that would be beneficial for the study,” I was letting people know that their talents were really valuable…I think was important in terms of setting the stage and making sure everyone feels comfortable in raising issues”
Creating a positive work environment was also challenging for others and a trial-and-error process that was developed over time. [Informant 7B: “hearing them out, but politely saying “No, we are not doing that right now,” or “…maybe we can incorporate that into the next phase,”…I started to really listen...I found my little ways to kind of incorporate their views…that helped to relieve a lot of the tension as well”].
Insight 4: Open communication and trust
Informants were asked to describe the overall communication and efficacy of their collaboration during main project activities. Communication and trust contributed a healthy work environment [Informant 8A: “a safe environment for the technicians to bring up anything with any of us…[the project manager] was very good at…accepting whatever they had to bring in, listening to them and doing what she could about whatever they brought up”].
A common consequence of a lack of communication and common understanding were confusion and frustration.
Informant 7B: “[I] was kind of given a very “lucy goosey”…review what the project was about, but to be honest I had a really hard time at the time…fully understanding exactly what they were trying to do…There is still a part of me that still doesn’t even know what the whole full objective was…in my opinion it was kind of a very dysfunctional team…. the full proposal…was not shared…I have tried asking for it a few times…[the PI] seemed to hold that proposal…very closely”
A lack of clear communication resulted in inconsistent engagement by external team members during meetings. Subsequently, team members lacked understanding of the purpose of study tasks and provided unconstructive feedback. [Informant 7B: “I do think that the entire team sensed the lack of direction, and it was highly transparent in all of our team meetings.”].
In Study 6, people were encouraged to be honest about their work capacity so that tasks could be reallocated and workload shared. However, asking for help was not natural for everyone [Informant 6B: “I think maybe too often… being able to ask for help was really difficult… I said “I’m fine, I am just really beat”… I was not fine. I was so stressed… it was not normal to go at the pace that I was going…[if additional help was not hired] I would probably have a nervous breakdown”]. One reason for this challenge was a sense of duty, perception of expectation and responsibility to the project. Understanding work capacity was described as a process, and one that required the support of others.
Insight 4.1: But we don’t even speak the same language!
Different “languages” can mean different work cultures, vernaculars or spoken languages. Informants were next asked about specific challenges experienced during collaboration.
Informant 1B: “Because of the background that the work-package-leaders…they were mainly from nursing… [and] It is very important [to them] to have a baseline. You need to understand the current situation…While in information systems…we also need to design a solution…where the actual resources are required, and understanding the problem is an iterative process. Because as we design something, we test it and then we also learn about the problem again”
When research team members’ agendas did not align, the goal of meetings became “managing personalities” instead of study tasks.
Informant 7B: “They truly are like the [Country’s] experts. They would always speak up… they were really adamant about doing it through their lens. But, again, when you have like six people and they were all really adamant that their way was the best way…it was more just… managing the bold perspectives and personalities of the team. I think that´s what made it difficult”
Insight 4.2: Collaboration, challenges, and compromise
Informants were asked to exemplify how such interdisciplinary miscommunications were resolved. Conflicts were described during every stage of the study. Often these conflicts took time and several meetings to clearly communicate both parties’ perspectives of research.
Informant 6A: “There was only one conflict…with the sponsor team of scientists…They were very very used to their trial…[they] expressed their concern about how slowly we were enrolling…it seemed that they were missing the whole idea of this iterative [recruitment] process and learning as we go…We ended up having two meetings together. The first meeting went poorly because it was them expressing their concern and me getting angry because they didn’t understand. At the second meeting…we explained the whole vision for the trial…And kind of proving our ability to them”
Sometimes it was not always possible to identify the root cause of miscommunications during the course of the project [Informant 6A: “the person who put his reputation on the line to sponsor it…was told by many people, including his boss, that there was no way the trial was going to work…that we were going to be stuck with an unsuccessful trial and be blamed for things”].
Lengthening the recruitment time was another tactic taken when a partner was uncompromising in their enrollment goals. However, budget was one challenge that could not be changed. Some informants had to remain flexible and use outside resources to compensate for unfinished tasks that resulted from such miscommunications.
Informant 1B: “We had some issues with one of the [development] companies…I think it was a fuzzy responsibility issue… Basically, the project stopped, or at least the development part of the project. And in addition, the company had not really communicated about the hours they reported working. So, all of a sudden, their hours were done… They did not even have any documentation to provide us on the development process…we needed this technology development, but then we did not have the money”
Theme 2 - Study management technologies
Informants were asked what technologies they used to manage their projects, including team management, intervention design and participant coordination. Note that we expect that the management technologies mentioned during the interviews is not an exhaustive list and there were more that were used but not included in informants’ overview of used tools. A summary of the technologies and how they were used are described in Table 1.
Table 1. Digital and analogue tools used for managing digitally enabled health studies
Insight 1: “I prefer my own technology, thank you.”
An individual’s work technology choice was less of a choice and more a matter of availability and budget.
Informant 6B: “It was a lot of trial and error in terms of what worked best and what made the most sense at the time …what software we had available and…not going to be super expensive…Because we did not budget for them in advance… and could not realistically go back to our sponsors and say that “We need XYZ thing and this is what it is going to cost in terms of money. Please help us!””
Researchers at universities reported that they were required to use certain platforms because their organization had a subscription. These were seen as not ideal but mandatory. [Informant 7A: “[Microsoft] Teams it is a nightmare, because I have five different [Microsoft] Teams accounts”].
Several informants noted challenges when merging communication processes between interdisciplinary teams [Informant 4A: “I can adjust my internal team to use new technology, but when you are working with an interdisciplinary team… asking them to shift their process is going to be a non-starter. If they work on e-mail, that is how they work”].
Using different technologies often led to siloed work procedures. However, when others tried to impose a more “commonly understandable” and “accessible” workflow on some informants, the informants still maintained their own workflow and work separately. The mentality was that it was only necessary for it to make sense to them, not others.
Insight 2: Which is preferable – control or ease of data collection?
When asked to describe the process and resources used during data collection, informants described the wide range of possible data sources and necessity of compromise. Informants described how some resources such as paper questionnaires were simultaneously challenging, as they required more time, and beneficial, as they gave a better opportunity to clarify responses while the participant was still there. Informants believed that having digital questionnaires meant having less control over data collection.
The use of third party partners’ own databases, such as insurance claims, allowed some studies to perform participant identification, as part of recruitment, much faster than if they did it “from scratch” [Informant 6A: “We had a pool of about a hundred thousand people we could reach out to who we knew were very eligible for the study”]. However, miscommunication can affect the project at any time, requiring informants to redirect their focus and delay research activities [Informant 6A: “we purposively were going to send 1000 [recruitment] e-mails per week and learn…It was not just the fact that [the sponsoring insurance partner] thought the enrolment was slow, but it seemed that [the sponsoring insurance partner] was missing the whole idea of this iterative process”].
Many informants were additionally affected by the General Data Protection Regulation (GDPR) requirements, resulting in unexpected delays when transferring data from external registries such as electronic health records or national health registries. Study 6 also experienced an unexpected challenge when, as required to ensure privacy, participants returned the intervention device to the manufacturer for data extraction and anonymization. The anonymized data was then forwarded to the research team. However, the research team discovered that if there was missing data when participants did not use the intervention device, they could do nothing about it by the time their received the device data.
Some even considered the privacy and security measures for accessing and extracting data as unnecessarily cumbersome [Informant 8B: “It took more that 1,5 years to get the complete data”]. For example, in Study 8 accessing data from an external database with high privacy restrictions required the research team to apply for permission each time they wished to access the data, let alone export it. Electronically exporting the data required permission. The solution to extracting the data was to use the level of access that they did have permission for [Informant 8B: “I decided to just copy it by hand because I could literally see the code on the screen so I could just copy everything”]. In addition, metadata – a sort of code list – was required to translate and interpret the raw data into useful information. Only then could they perform the required analysis.
Even strategic and careful planning was no guarantee that protocols would go as planned [Informant 3B: We started all this process very early…the application to the ethics committee and everything was there and done…But then [the ethical review committee] were overloaded…too many requests… very long processing time before we got an answer”].
Insight 3: Frequency and effectiveness of communication platforms
Informants were asked about different technologies used during day-to-day communication, and to comment on their effectiveness. E-mail was the most common form of day-to-day coordination, although the amount of e-mails and managing information exchange via e-mail was challenging.
Sometimes, the amount of e-mail communication was too little. For example, it was not always successful at prompting efficient responses, especially from outside teams with whom informants had infrequent or indirect communication. It could take sending several e-mails to a partner before receiving a response with the necessary information.
Relying on E-mail to exchange information could result in missing details [Informant 4A: “using platforms like Freedcamp to allow for better information sharing, because the e-mailing got a little hairy internally, especially as we had turnover of coordinators and assistants and students…some of the data management got a little hairy as we had handovers”]. When it was important to have the most recent version of a document to which so many made edits, E-mail was not the most effective platform [Informant 4A: “There were a couple of authorships “snafu’s” where people lost a thread in an e-mail and wanted to be on something, and then you are two-thirds of the way through, and then you have to figure out a way to get them back on to it”].
Just because a technology is easy to use does not make it ideal. WhatsApp was used for regular communication of nonsensitive information within research teams in two studies. While this type of platform offered ease of use and access to colleagues, it could be overwhelming when messages came at any time of the day or night. Instead, informants opted to “drop out” of some chat groups.
Some had used platforms that only allowed one person to access a collaborative document at once, which required additional energy and communication [Informant 6B; “trying to figure out the right version of documents in the software that we were using…PM-ing [private messaging] someone like “Hey, I am using the tracking document now. Let me know when you are off””]. Informants from almost all studies mentioned using Google Drive, Google Docs and/or Google Meet. Google Drive was beneficial when team members were able to work simultaneously on collaborative documents, it was efficient and reduced the potential for miscommunications.
However, using these platforms to share and simultaneously update documents only works when the research team members use it [Informant 7B: “everybody had access to [them], but nobody, I don’t think had even logged into it”].
Insight 4: Needs for future management tools: feasible or a pipedream?
Informants were asked to reflect on what they would change if they were to do their studies again, given their experiences. Project management software, such as Basecamp, Freedcamp or Google Docs were preferable for more formal information sharing whereas Slack, WhatsApp or Discord were suggested for more continuous communication. It was also important to clarify which platforms would be used for which task [Informant 4A: “making sure you are clear on the process for communication… You need to have different processes for communicating with the investigators who are still part of the team, but you are not going to change the way that they operate on one of twenty different things they are working on”].
When hiring new personnel, whether it be because of turn-over or team expansion, it was especially complicated to train someone new on a technology or workflow that was used and designed by the previous employee. [Informant 6B “before it was just me…And then adding more people, we had to figure out a new tracking system…where everyone could see the status and updates of each participant in the study…[so] we could all be on the same page at the same time”].
Planning for which technology to use and for which stage was something that many valued but few were able to implement. All informants mentioned, at one point or another during their interviews, that automatic data extraction would had reduced time and energy.
Informant 5A: “If the [blood test] data had been extracted automatically it would have been brilliant…[and] online questionnaires that they could have done at home… [And] if we could have replaced some of the physical consultations with a video call, it might have resulted in spending less time reorganizing appointments”].
However, automatic data extraction is not always the best option, especially when interaction with the participants is highly valued and/or necessary for the intervention [Informant 5A: “I would have lost the opportunity to talk to them and get close relationships with the participants”].