Enhancing pediatricians’ engagement on social media: the role of board style questions

This article was migrated. The article was marked as recommended. Background: Social Media is used among medical professionals for collaborative education. Little is known about how case discussions prompt engagement. Objective: To determine the association between item characteristics of board exam-style questions to social media engagement. Methods: This was a prospective cohort study through the American Academy of Pediatrics (AAP) PediaLink FaceBook page, conducted in 2018 over 9 months. Items from the 2017 PREP® questions were ranked in difficulty, then rated in relevance to general pediatrics through content-expert consensus. Thirty-six questions were randomly posted on FaceBook and Twitter weekly. Independent variables included item difficulty rank, difficulty level (easy vs hard), relevance to general pediatrics, and word count. Outcome variables included percent correct responses and total comments under the post. Results: More difficult questions were associated with fewer comments (rho=0.63, p<0.001) and lower correct response percentages (rho=0.39, p=0.02). Easy questions garnered more comments than hard questions (median 18 IQR 13-23 vs median 9.5 IQR 5-14, p=0.001). Correct response percentage was lower for hard questions (90% IQR 85-95% vs. 77% IQR 60-94%, p=0.04). Relevance to general pediatrics and word count did not affect engagement (p > 0.1). Conclusion: Easier practice test items attracted more responses from pediatricians on social media, increasing engagement.


Introduction
Social Media (SM) is used in medical education at the undergraduate and graduate medical education levels, using popular platforms including Twitter and FaceBook (Admon et al., 2019;Cheston, Flickinger and Chisolm, 2013;Sterling et al., 2017;Nicolai et al., 2017).Medical education efforts using these platforms have centered around their ability to create forums for online community discussion (Nicolai et al., 2017;Ranginwala and Towbin, 2018).Twitter has been used often at live conferences to spur synchronous or asynchronous discussion in an open, inclusive manner (Desai et al., 2018), while FaceBook is more visible in the medical literature as a medium for exclusive community discussions in closed groups (Wright et al., 2019).
The American Academy of Pediatrics (AAP) provides a variety of educational content, including board review-style questions within a program called Pediatric Review and Education Program (PREP).Since 1979, PREP has provided in print form, these questions and their detailed answers as self-directed learning materials.In 2000, PREP moved to an electronic version, and in 2017, SM campaigns to spur its use and awareness were introduced.
FaceBook has been used as an educational adjunct in healthcare simulation (Ranginwala and Towbin, 2018), medical student training (Sterling et al., 2017;Nicolai et al., 2017), and in a variety of disciplines including radiology (Ranginwala and Towbin, 2018).Within the platform, an asynchronous discussion of relevant topics can be initiated either by faculty and attending-level physicians or by peers.The quality and depth of engagement within a discussion group dictates the richness of content in such social media platforms (Rishika et al., 2013).However, the factors that facilitate participant engagement in SM groups in a medical education setting are poorly understood.
The purpose of our study was to identify factors that enable enhanced engagement of participants in a social media learning platform.We hypothesized that greater the cognitive difficulty of the practice questions, the greater the engagement with the materials as evidenced by more participant comments and discussion.We also hypothesized that items with more relevance to general pediatricians would elicit more engagement as the majority of AAP members are general pediatricians.

Population and Setting
This was a prospective observational study conducted between February to October 2018 using the FaceBook and Twitter accounts for the AAP PediaLink group.PediaLink provides oversight over e-learning and educational technology offerings for the AAP; the SM team uses the PediaLink account to post information about AAP educational products including PREP.PREP, a practice board-style questions and answers set is updated each year for use by general pediatricians and pediatric subspecialists.All PREP questions are framed as a realistic case vignette followed by a multiple-choice answer format.As part of the SM promotion efforts, the AAP has been posting online through FaceBook and Twitter sample PREP questions since 2017.Subscribers to the PediaLink FaceBook Page or the PediaLink Twitter account are free to respond as they choose.Figure 1 depicts a screenshot of a posted PREP question.
All pediatricians who had subscribed to the FaceBook group during the study period were eligible to participate in the study.There were no exclusion criteria.The study was deemed exempt by the AAP Institutional Review Board.

Curation of PREP questions
We used the General Pediatrics PREP questions from 2017 for this study.There were 280 multiple-choice case vignette items developed and curated by the standard peer-review process through the AAP PREP Editorial Board.All test items had response data from more than 10,000 respondents for each item from previous subscribers or users of PREP.This enabled us to determine the percentage of correct respondents for all 280 items.Each PREP question was then ranked in difficulty from 1-280, corresponding from the lowest correct response rate to the highest.Thus item 1 was the hardest and item 280 the easiest question.
We randomly selected 70 PREP test items from the overall 280 using stratified randomization.From each heptile rank in difficulty, 10 PREP questions were randomly selected using a random number generator (http://www.random.org/);as an example, ten questions came from ranks 1-40, another ten from 41-80, etc.Then all 70 questions were randomized in order.The test item order is presented in Table 1.
Then, each item was assigned a scale on relevance to a general pediatrician.This was a de novo 7-point Likert scale for each item to quantify whether the case vignette portrayed a common, typical case seen in a general pediatrician's office.Seven physicians -4 general pediatricians and 3 pediatric specialists -rated all 70 item vignettes independently.The mean of all 7 rater scores became the relevance score for each test item.

Protocol
An AAP PediaLink staff member (LD) posted a weekly PREP question in predetermined order beginning in February 2018.The PediaLink FaceBook page also had additional direct marketing information about AAP and PediaLink offerings, but no other case vignette discussions nor test items were posted during this study period.A pediatric specialist (RA) posted an open-ended, single sentence comment on the content of that question on a weekly basis; her identity as a study author was not known to the subscribers.Her comment was a 'catalyst.'The answer to the PREP test item was posted the following week alongside the new question for that week in the pre-determined order.The PediaLink Twitter page also mirror-posted the PREP questions but replies and re-tweets were not tracked for this study.No active recruitment strategies to either the FaceBook Page nor the Twitter account were added during this study period.
An interim analysis was planned at Week 36, and authors agreed a priori to stop the study if the analysis yielded significant results.Otherwise the protocol would continue until Week 70 with the final answer revealed on week 71.

Variables of Interest
The primary independent variable was item difficulty, expressed both as an ordinal rank variable from 1 -280.A rank of 1 was the most difficult, and 280 the easiest.Item difficulty was also dichotomized to difficult vs. easy, in which difficult items represented ranks 1 -140 and easy 141 -280.Secondary independent variables included the relevance to a general pediatrician score, expressed as a continuous variable between 1 and 7.
Our primary outcome variable was the total number of Face Book comments per item.We decided that this would be a quantitative marker of participant engagement with the posted material.Comments include responses of any length, including a simple letter response "c." or a detailed explanation or question on either FaceBook or Twitter.If one participant posted 5 comments, all 5 comments were included.The catalyst comment from the study author was not included, and neither were comments posted after the answer was revealed.Secondary outcome variable included the correct response rate, calculated by dividing any correct responses divided by the total number of comments in both social media platforms.

Data Analysis
We first documented a weekly temporal trend in the total number of comments on both FaceBook and Twitter for overall social media activity using Spearman Rank.Descriptive statistics were used to characterize comments and correct response rates for the items.Spearman Rank correlations were performed for continuous and ordinal independent variables of difficulty and relevance to a general pediatrician.A Mann-Whitney U test was done for the dichotomous independent variable: item difficulty.A mixed-method average measures intraclass correlation using a consistency type was performed for the 7 raters involved in the relevance to a general pediatrician score and 95 th confidence intervals reported (Koo and Li, 2016).

Results/Analysis
As described in the protocol, the study was discontinued following significant results recognized during interim analysis in week 36.Figure 2 depicts overall social media activity during the weekly PREP test item protocol from February to October 2018; there was only a trend towards significance in increasing SM engagement (rho = 0.3, p = 0.07), which paralleled the gradual increase in social media followers: 4,999 to 5,322 Twitter followers and 3,862 to 3,914 FaceBook subscribers.
Table 1 details the type of questions posted, the number of comments and the general pediatrics relevance score for all the test items.For the 36 test items posted on social media, the median number of comments was 12.5 (IQR 8.5 to 18.25) and the correct response rate had a median of 90% (IQR 74.1 to 100%).Among the 2017 PREP cohort whose correct response rate contributed to the difficulty rank assignments, the median correct response rate was 76% (IQR 67.25 to 83.5%).Our SM cohort had a significantly higher correct response rate than the 2017 PREP cohort (p = 0.005).
More difficult test items were strongly associated with fewer comments (rho = 0.63, p < 0.001), and a correlation was found between difficulty rank and the correct response rate among 2018 social media respondents (rho = 0.39, p = 0.02).
When dichotomized, hard test items attracted a median of 9.5 (IQR 5 to 14) comments, while easy test items attracted significantly higher median of 18 (IQR 13 to 23) comments (p = 0.001) per item.Even among the social media respondents, hard questions yielded a 77.4% (IQR 60 to 94%) correct response rate, which was significantly lower than the easy questions' 97.5% rate (IQR 92.5 to 100%, p = 0.04).
Among the 7 raters for the 70 selected items, the relevance to a general pediatrician score demonstrated a strong ICC of 0.910 [95%CI 0.874 to 0.939].However even with such reliable ratings of relevance to a general pediatrician, we found no significant associations between items that were more relevant to a general pediatrician and to either total number of comments (rho = 0.3, p = 0.1) or to the correct response rate (rho = -0.2,p = 0.2).Incidentally, word count of the items was inversely associated with relevance to a general pediatrician (rho = -0.35,p = 0.003), indicating subspecialty questions were wordier.

Discussion
Our data show that easier case vignette questions posted among pediatricians yielded more responses and engagement than more difficult questions.SM has been used in educational endeavors as a method to transcend temporal and geographical constraints and to facilitate an ad hoc community (Admon et al., 2019;Cheston, Flickinger and Chisolm, 2013;Sterling et al., 2017;Nicolai et al., 2017;Melvin and Chan, 2014).
While the characteristics of the comments was not analyzed for this study, the majority of engagement to the multiplechoice case vignettes used in this study consisted of posting the letter choice of a response.On occasion, we saw dialoguebased conversations, but these tended to be rare, particularly for more difficult questions.
Our hypothesis was incorrect.We had anticipated that more difficult questions would engender greater conversations and dialog among the participants; but they did not.The behavior that was demonstrated on our SM platform is consistent with the literature on SM and social comparison theory (Vogel et al., 2014).While the content in our study was not particularly charged or controversial, the significant drop in comments with more difficult questions suggest that our adult learner subjects may have been reluctant to post an incorrect response to potentially avoid social embarrassment or judgment.Studies on anonymity in classroom settings indicate that anonymity may lower the barriers for reluctant students to speak or to actively participate (Barr, 2017).Given that FaceBook subscribers were not anonymous, the lower comment rate seems indicative of reluctance to err in such a public forum.Another interesting observation was that the correct response rate among our SM cohort was substantially better than the rate for the 2017 PREP test takers.There are several possible reasons for this finding.It may suggest self-censorship among participants who did not post a comment to avoid social embarrassment as discussed earlier, thereby driving the correct response rates higher.Another possible reason for the higher performance by the SM group might be their open access to the web where they could potentially search for the right answers before posting on the platform.As savvy users of online learning, this group is more likely to be engaging in such "just in time learning" compared to the traditional test takers.
We did not find an association with relevance to a general pediatrician to the level of engagement and responses on SM.While we do not know the specialization of the AAP PediaLink subscribers, based on AAP data, general pediatricians consist of approximately 75% of AAP members.The literature on SM suggests that a complex relationship exists between SM engagement and the degree of controversy of a topic (Garimella et al., 2018).Relevance and 'topicality' is known to affect responses and comments online outside of medical education (Chen, Shang and Li, 2014); however, we did not see substantial changes in comments based on relevance to general pediatrics.It is also possible that general pediatrics questions would have been more highly commented if we had collected a larger sample size.To find any significant association at a predicted rho of 0.28 would have required a sample size of 97, which would have taken almost 2 years to collect in this study design.
There were several limitations to this study.First, the study team did not have control over any other social media posting, including marketing or event notification posts.While no particular marketing 'surges' were noted, it is possible that extraneous materials influenced participation outcomes.The number of subscribers during the time period of the study was projected to increase, but the actual subscriber number was not available to us.Finally, although the data were aggregated from two social media platforms, we did not examine whether certain characteristics influenced commentary depending on the platform.

Conclusion
Enhancing social media engagement by participants is a complex multifactorial process.As the difficulty of posted test items increases, the correct response rate among pediatricians on social media and the number of posted comments and engagement decreases significantly.SM education groups -in pediatrics or in any specialty or discipline -can purposefully curate test item postings to garner different levels of engagement and discussion among participants.

Take Home Messages
• Easier board exam-style questions posted on social media were associated with more responses or comments posted from pediatricians, increasing engagement significantly from engagement with harder questions.
• Harder board exam-style questions posted on social media were associated with lower correct response percentages from pediatricians.
• Question relevance to general pediatrics and question word count did not affect the number of comments or level of engagement seen from pediatricians on social media.
• Medical professionals aiming to improve engagement and discussion among participants on social media for collaborative education using test questions should consider the level of difficulty of the questions.

Helena Filipe
Hospital of the Armed Forces/PL-EMGFA This review has been migrated.The reviewer awarded 4 stars out of 5 Thank you for the opportunity to review this paper, which I read with great interest.There has been an increasing attention towards the use of social media (SM) in medical education, recently propelled for the pandemic.These challenging times have underscored the need to develop collaborative, supportive, informal, safe social learning spaces.Fostering wellbeing and interpersonal support within a community of learning has been prioritised to learn effectively.Kudos for a step forward on advancing consistent ground for the use of SM in medical education.The use of widely known daily life social applications sprung have far from the academia accurate requirements.The authors make the case, that more than a compartmental field, the scientific method can have multiple applications and be the selected methodology for any field of inquiry.Thank you for describing comprehensively to the point of replication, your research.The results are quite interesting and open the wide range of potential applications of SM in knowledge building in contemporary innovative learning environments.Educators, planners, and providers will be interested to read this piece, which highlights the importance of their own CPD to effectively embrace and apply new facilitation skills to meet learners needs (1).They are expected to gear the rich opportunities offered by technology towards building multifaceted learning experiences.We tend to highlight and wonder about three results of this investigation:-While engendering less responses, and still considering the anonymity issue, could more difficult questions suggest coupling a small-groups (guided) reflective learning space, as a follow-up?-Asan effectiveness determining factor for CPD activities, practice-based learning has been associated with adult learning principles.Results invite to better understand the absence of a proportional engagement level with topics' relevance.This could start by expanding the sample, as Authors propose.-While yielding a higher level of engagement, might be interesting to explore the potential use of easier questions as drivers to build supportive and safe digital learning environments and how this could relate with learning improvement.The 4 messages left by the Authors summarise well the study purpose, methods and what was found.1. Filipe HP, Mack

Figure 1 :
Figure 1: Screenshot of a typical case vignette

Figure 2 :
Figure 2: Overall social media activity

Table 1 :
Characteristics of the 36 questions used in this study Kim Little-Wienert, MD, MEd is Assistant Professor of Pediatrics and Pediatric Emergency Medicine Fellowship Associate Director at Texas Children's Hospital and Baylor College of Medicine in Houston, TX.Todd P. Chang, MD MAcM is Associate Professor of Pediatrics at the University of Southern California and Children's Hospital Los Angeles (CHLA) in Los Angeles, CA.He serves as the Director of Research and Scholarship within the Division of Emergency Medicine and Associate Medical Director for the Las Madrinas CHLA Simulation Center.Rita Agarwal, MD is Clinical Professor of Anesthesiology at the Stanford School of Medicine in Palo Alto, CA and the President of the Society of Pediatric Pain Medicine.Rachel E. M. Cramton, MD is Associate Professor of Pediatrics and Pediatric Residency Associate Program Director at University of Arizona and Banner University Medical Center in Tucson, AZ.Using Twitter in clinical education and practice.Journal of Graduate Medical Education, 6(3), pp.581-582.Reference SourceNicolai, L., Schmidbauer, M.,Gradel, M., Ferch, S., et al. (2017)Facebook groups as a powerful and dynamic tool in medical education: mixed-method study.Journal of Medical Internet Research, 19(12), p. e408.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.