Article Text

Original research
Quality and reliability evaluation of online videos on carpal tunnel syndrome: a YouTube video-based study
  1. Donghee Kwak1,
  2. Jong Woong Park1,
  3. Yousun Won2,
  4. Yeongkeun Kwon3,
  5. Jung Il Lee4
  1. 1Department of Orthopedic Surgery, Korea University Anam Hospital, Seoul, South Korea
  2. 2Department of Radiology, Spine Love Hospital, Goyang, South Korea
  3. 3Division of Foregut Surgery, Korea University College of Medicine and School of Medicine, Seoul, South Korea
  4. 4Department of Orthopedic Surgery, Korea University Guro Hospital, Seoul, South Korea
  1. Correspondence to Dr Jung Il Lee; osjungil{at}gmail.com

Abstract

Objectives With the increasing popularity of searches for medical information on YouTube, the availability of videos concerning carpal tunnel syndrome (CTS) is increasing. This study aimed to evaluate the quality and reliability of YouTube videos on CTS.

Setting and participants No participants were included.

Primary and secondary outcome measures We searched YouTube on 1 April 2021 using the keywords “carpal tunnel syndrome” and “carpal tunnel release” and evaluated the first 55 retrieved videos. We summarised the video characteristics including Video Power Index (VPI), which was designed to evaluate video popularity based on the number of likes and views. We categorised them based on source and content. Video quality and reliability were evaluated using the Journal of the American Medical Association (JAMA) benchmark criteria, the Global Quality Score (GQS) and the Carpal Tunnel Syndrome-Specific Score (CTS-ss) .

Results The mean (range: minimum–maximum) of JAMA scores, GQS and CTS-ss were 2.13 (1–4), 2.69 (1–5), and 5.0 (1–15), respectively. The most common source of video was from allied health workers, and academically sourced videos had the highest JAMA score and GQS. Three scores were significantly correlated with each other. Multiple linear regression analysis showed that a higher JAMA score was associated with a higher likes ratio, and a higher GQS was associated with a longer video running time and greater number of comments. However, a higher VPI was not associated with higher video quality or reliability represented by the three scores.

Conclusions YouTube videos on CTS have low quality and reliability. Video popularity was not significantly correlated with quality or reliability. Our findings suggest that expert groups should provide and promote high-quality video content to YouTube users and patients.

  • Hand & wrist
  • NEUROLOGY
  • REHABILITATION MEDICINE
  • EDUCATION & TRAINING (see Medical Education & Training)
  • Neurological pain

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • Various characteristics including number of views, number of likes, Video Power Index and video uploader in the YouTube videos about carpal tunnel syndrome were investigated.

  • The reliability and quality of videos were investigated using three scoring systems: Journal of the American Medical Association benchmark criteria, Global Quality Score and Carpal Tunnel Syndrome-Specific Score.

  • Although these scoring systems are subjective and unvalidated, the scoring systems were independently assessed twice by the two raters, which showed intraobserver and interobserver agreements determined by intraclass correlation coefficients.

  • A multiple linear regression analysis was performed to identify video characteristics affecting the reliability and quality of videos.

Background

With the internet penetration rate exceeding 50% worldwide,1 searches for health information on the internet have become common. According to recent studies, 80% of internet users searched for health information online,2 and up to 30% of orthopaedic patients searched online for disease information.3 Furthermore, well-designed videos of disease information positively affect treatment outcomes by improving patient comprehension.4 5 However, most online information is not regulated, resulting in the spread of inaccurate and low-quality data among patients.6–10 Therefore, physicians should properly evaluate such information and help patients obtain accurate information and appropriate treatment.

YouTube, which has over 1 billion users watching over 1 billion hours of videos each day, is a source of representative video-based educational content.11 Although some high-quality orthopaedic content is uploaded by qualified experts on YouTube, most of the related content is uploaded by unqualified individuals, providing patients inaccurate and erroneous information. In previous quality-evaluating studies in the orthopaedic field, YouTube video accuracy and quality were low.1 10 12–14

According to previous studies that investigated the quality of carpal tunnel syndrome (CTS) information provided by internet search engines,15–17 the quality of online information has improved over the past decade but remains low. These studies reported that there was significant scope for improvement. In contrast, recent studies18 19 reported that most YouTube videos and websites that provide information on CTS can reinforce misconceptions. Two quality-evaluating studies on CTS information available on YouTube20 21 focused on video quality and reliability, and neither examined the relationship between characteristics such as video popularity and quality.

The current study aimed to (1) evaluate the quality and reliability of YouTube videos concerning CTS; (2) investigate the video characteristics, sources and contents; and (3) determine the relationship between video characteristics and quality.

Methods

Patient and public involvement

No patients were involved.

YouTube search design and study setting

The YouTube online library (https://www.youtube.com) was searched on 1 April 2021 using the terms “carpal tunnel syndrome” and “carpal tunnel release”. The first 50 videos retrieved based on each keyword and sorted by ‘view count’ for a total of 100 videos were selected for review. Of them, 45 were excluded (duplicates, 39; non-English, 3; information on cubital tunnel syndrome, 2; and soundtrack with no mention of carpal tunnel, 1). Thus, 55 YouTube videos found using the keywords “carpal tunnel syndrome” and “carpal tunnel release” were analysed (figure 1). The URLs of each video are listed in online supplemental table 1.

Figure 1

Search methodology for carpal tunnel syndrome-related YouTube videos.

Data on the following video characteristics were collected from each YouTube video: (1) title, (2) channel name, (3) number of subscribers, (4) video running time, (5) number of views, (6) number of comments, (7) video source/uploader, (8) content type, (9) days since upload, (10) view ratio (number of views:days since upload), (11) number of likes, (12) number of dislikes, (13) likes ratio (likes×100/[likes+dislikes]) and (14) Video Power Index (VPI). The VPI was calculated using the following formula: like ratio×view ratio/100. This value is an index designed to evaluate video popularity based on the number of likes and views.1

Video sources/uploaders were categorised as follows1 10: (1) academic (uploaders affiliated with universities or research groups); (2) physicians (individual physicians or physician groups not affiliated to a university or research institute); (3) non-physicians (allied health workers such as alternative medical providers, physiotherapists and occupational therapists); (4) trainers; (5) medical sources (animations or related content from health websites); (6) patients; and (7) commercial. Contents were categorised as follows: (1) exercise training; (2) disease information; (3) patient experience; (4) surgical technique; (5) non-surgical management, such as chiropractic treatment; and (6) advertisement.

Evaluation of video quality and reliability

The quality and reliability of YouTube videos were assessed using three scoring systems: the Journal of the American Medical Association (JAMA) benchmark criteria, the Global Quality Score (GQS) and the Carpal Tunnel Syndrome-Specific Score (CTS-ss). The JAMA criteria enable a non-specific assessment of content reliability and include four criteria (table 1).22 Each criterion is assigned 1 point for a maximum total of 4 points. A score of 0 indicates low video reliability and accuracy, whereas a score of 4 indicates high video reliability and accuracy. The GQS1 10 23 consists of five grades and provides a non-specific assessment of health-related website quality (table 2). The total GQS ranges from 1 to 5, with a higher score indicating better educational quality. To better evaluate quality and accuracy of YouTube videos concerning CTS, we employed the new CTS-ss, which consists of 20 items. We generated this scoring system based on recent review articles24–26 and guidelines published by the American Academy of Orthopaedic Surgeons,27 which were considered reasonable in previous studies.9 10 The CTS-ss evaluates information on (1) patient symptoms and population, (2) carpal tunnel anatomy, (3) CTS diagnosis and evaluation, (4) treatment options, and (5) postoperative care and course (box 1). One point was given for each of the 20 items for a total maximum of 20 points. Higher scores indicated higher CTS-specific educational value.

Table 1

Journal of the American Medical Association benchmark criteria22

Table 2

Global Quality Score criteria1 10 23

Box 1

CTS-ss for video content

Patient presentation

  • Describes symptoms (eg, nocturnal paraesthesia, loss of sensation, thenar muscle atrophy).

  • Describes patient population, especially high prevalence in older women.

Information about carpal tunnel syndrome

  • Describes carpal tunnel anatomy and/or function.

  • Mentions caused by nerve compression.

  • Describes risk factors (eg, diabetes, hypothyroidism, pregnancy and repetitive use).

Diagnosis and evaluation

  • Mentions physical examination and findings (eg, Tinel’s sign and Phalen’s manoeuvre).

  • Discusses electrophysiological tests.

  • Discusses additional diagnostic tests (eg, ultrasound and MRI).

  • Mentions patient-centred measures (eg, the Boston Carpal Tunnel Syndrome Questionnaire).

  • Discusses differential diagnosis (eg, cervical radiculopathy).

Treatment

  • Describes non-surgical treatment, especially changes in habits.

  • Mentions that laser therapy is one of the non-surgical options.

  • Mentions pharmacotherapy (eg, local corticosteroid injection, NSAIDs).

  • Mentions musculoskeletal manipulation and/or splinting.

  • Describes surgical treatment that is the most effective treatment.

  • Mentions open carpal tunnel release.

  • Mentions endoscopic carpal tunnel release.

Postoperative care

  • Describes complications and outcomes (eg, CRPS, scar tenderness, reoperation).

  • Mentions need for postoperative physical therapy.

  • Outlines return-to-function timeline.

Intraobserver reliability and interobserver agreement assessment

All three scoring systems (JAMA, GQS and CTS-ss) were independently assessed twice, 30 days apart, by two raters, consisting of one orthopaedic surgeon (DK) and one family medicine doctor (YK). Intraobserver and interobserver agreements were determined using intraclass correlation coefficients (ICCs). ICCs for absolute agreement with a single measurement were used to identify intraobserver reliability with a two-way mixed-effects analysis of variance models. ICCs for absolute agreement with a single rater were used to identify interobserver agreement using two-way random-effects analysis of variance models. A guideline28 for evaluating ICC values was adopted: excellent (>0.90), good (0.75–0.90), moderate (0.50–0.75) and poor (<0.50). In cases of disagreement, all authors re-evaluated the video in question until consensus was reached.

Statistical analysis

Continuous variables are presented as mean±SD. Differences in the JAMA score, GQS, CTS-ss and VPI according to (1) video upload source and (2) category of video contents were evaluated by one-way analysis of variance tests (for normally distributed data) and Kruskal-Wallis tests (for non-normally distributed data) followed by post hoc tests using the Bonferroni method. A Spearman correlation analysis was used to assess the correlation between scores and between video characteristics and scores. A multiple linear regression analysis was performed to identify video characteristics affecting the JAMA score, GQS, CTS-ss and VPI. All reported p values were two-sided, and those <0.05 were considered statistically significant.

Results

Video characteristics and quality scores

The mean JAMA score, GQS and CTS-ss were 2.13, 2.69 and 5.0, respectively, indicating low reliability and educational quality (table 3). Raw scores of JAMA score and CTS-ss are shown in online supplemental table 2. Non-physician video sources accounted for the largest share (29.09%), while commercial sources accounted for the lowest share (5.45%) (figure 2). Disease-specific information accounted for the largest share (32.73%), while patient experience accounted for the smallest share (3.64%) (figure 3). The video title, YouTube channel name, JAMA score, GQS, CTS-ss and VPI of the top 55 videos are listed in order of the number of views in figure 4.

Table 3

Characteristics of 55 YouTube videos about carpal tunnel syndrome

Figure 2

Categorical distribution of video source.

Figure 3

Categorical distribution of video content.

Figure 4

Data-bar visualisation of the top 55 carpal tunnel syndrome and release videos with the highest number of views. CTS-ss, Carpal Tunnel Syndrome-Specific Score; GQS, Global Quality Score; JAMA, Journal of the American Medical Association; VPI, Video Power Index.

Differences in video reliability and quality by source and content

The JAMA score (p<0.0001) and GQS (p=0.0004) differed significantly among the seven groups of video sources, with videos from academic and physician sources having the highest mean JAMA scores and GQS (table 4). The JAMA score (p=0.0077) and GQS (p=0.0018) differed significantly among the six groups of video content, with videos about surgical technique and disease-specific information having the highest mean JAMA scores and GQS. However, the CTS-ss and VPI did not differ significantly between the groups based on video sources and contents.

Table 4

Mean quality and reliability scores per video source and video content variable

Factors affecting video quality and popularity

JAMA, GQS and CTS-ss significantly correlated with each other (JAMA score vs GQS, p<0.001; JAMA score vs CTS-ss, p=0.001; GQS vs CTS-ss, p<0.001). However, the VPI was not significantly correlated with the three scores. Multiple linear regression analysis showed that a higher JAMA score was associated with a higher likes ratio of an academic or physician upload source compared with a patient upload source (table 5). A higher GQS was associated with a longer video running time; greater number of comments; and higher probability of academic, physician, non-physician, medical information and commercial upload source than of patient upload source. A higher CTS-ss was more associated with academic, physician, medical information, and commercial upload sources than patient upload sources. However, a higher VPI was not associated with higher video quality or reliability scores.

Table 5

Multiple linear regression analysis of correlations between video characteristics and the VPI, JAMA score, GQS and CTS-ss

Intraobserver reliability and interobserver agreement assessment

The intraobserver reliability of the two raters was excellent for the JAMA score, GQS and CTS-ss. The interobserver agreement between raters was good for the JAMA score (ICC 0.881, 95% CI 0.804 to 0.929), good for the GQS (ICC 0.881, 95% CI 0.804 to 0.929) and excellent for the CTS-ss (ICC 0.941, 95% CI 0.898 to 0.966).

Discussion

This study demonstrated that the reliability and quality of YouTube videos concerning CTS were low. This result was consistent with that of other previously conducted YouTube video quality evaluation studies.1 10 13 20 21 29–31 Mert and Bozgeyik20 evaluated the quality of CTS videos on YouTube and reported that the video reliability and quality were low. They presented no significant relationship between video characteristics, reliability and quality evaluation scoring systems. Radonjic et al21 also evaluated CTS videos on YouTube and showed low reliability and quality and found that videos uploaded by physicians had significantly higher reliability and quality evaluation scores than those uploaded by non-physicians. Goyal et al18 reported that YouTube videos of CTS have low information quality. They determined that the potential reinforcement of misconceptions is prevalent in YouTube videos on CTS.

Although the overall reliability and educational quality of YouTube videos were low, those of videos from academic and physician uploaders or about surgical techniques and disease-specific information were significantly higher than those of other video sources and contents. This is because the main purpose of these video sources and contents is to educate doctors, medical students and patients. In contrast, the CTS-ss did not differ significantly among the video sources and contents because YouTube videos focus on specific topics, such as symptoms and surgical technique or rehabilitation after surgical treatment, and deliver the content within a short running time. Additionally, some specific channels, such as the ‘Bob & Brad’ channel, posted videos in four series about CTS and release. Casual YouTube viewers cannot obtain sufficient content on CTS and release in only one or two posted videos, but an entire series can provide most of the content. YouTube uploaders usually post short videos of less than 10 min to maximise the number of views and user interest; thus, they split the content into several videos.

Most of the videos had low reliability and educational quality, but some videos had useful practicality and educational information. The ‘Carpal Tunnel Syndrome - Everything You Need To Know - Dr. Nabil Ebraheim’ video of the nabil ebraheim channel explains the overall symptoms, anatomy and risk factors of CTS. In the ‘Surgery Video: Carpal Tunnel - MedStar Union Memorial’ video of the MedStar Health channel, the surgical procedure and method of endoscopic carpal tunnel release were shown in detail. The ‘How to Determine If You Really Have Carpal Tunnel Syndrome - Dr Mandell, DC’ video of the motivationaldoc channel shows the physical examination required for CTS diagnosis.

In this study, video popularity showed no significant correlation with reliability or quality. Popular videos that casual YouTube users and patients frequently watch do not have good quality and reliability. Interestingly, YouTube videos of expert groups that are expected to have high reliability and quality, such as the American Academy of Orthopaedic Surgeons or Federation of European Societies for Surgery of the Hand, were not included in the top 55 videos. A manual search identified only about 1600 views for the carpal tunnel release video uploaded to the American Academy of Orthopaedic Surgeons YouTube channel (https://www.youtube.com/watch?v=eemuH5UYElo). Additionally, the Federation of European Societies for Surgery of the Hand and the British Society for Surgery of the Hand channels have no CTS-related videos and only 154 and 575 subscribers, respectively. It is necessary to promote an expert group’s YouTube videos and channels and try to provide accurate medical information by uploading a high-quality video and exposing it to casual YouTube users and patients.

In a previous study on the meniscus,10 video dislikes were described as predictors of YouTube video reliability, but this was not the case in this study. The independent predictor of the JAMA score in this study was the likes ratio. Furthermore, independent predictors of GQS were video running time and number of comments, suggesting that videos with a longer running time and greater number of comments are independently and significantly associated with a higher GQS. The longer the video running time, the greater the amount of information it contains; therefore, its educational quality also increases. For GQS, a greater number of comments contain more useful information for users who watched the video. Regarding the CTS-ss, compared with patient upload sources, academic, physician, medical and commercial upload sources are associated with a higher CTS-ss. However, unlike the JAMA score and GQS, CTS-ss showed no significant association with video characteristics except for video source.

Our study has several limitations. First, we searched the top 50 videos for “carpal tunnel syndrome” and “carpal tunnel release” on YouTube in the order of popularity. This search strategy missed certain videos with low views or hits but with potentially high quality. Although our search strategy could miss high-quality videos that are less ‘popular’, this strategy is the actual method by which casual YouTube users obtain information. Second, YouTube video metrics such as the number of likes and views are constantly updated; therefore, these study data are accurate only on the date of the search. Third, the assessment scoring systems that we used (the JAMA score, GQS and CTS-ss) are subjective and unvalidated. Because the JAMA benchmark criteria were developed to assess medical information on the internet website rather than video information, the criteria may not fit YouTube videos. The CTS-ss includes many contents of CTS, but almost YouTube videos have a short duration of about 10 min or less. Thus, it tends to be difficult to present all checklist of CTS-ss in short videos. Because some criteria in the JAMA benchmark criteria and CTS-ss were unsatisfied in most of videos, total score may be mainly influenced by some criteria; thus, all criteria have no equal weight. Nevertheless, we have no choice but to use these scoring systems due to lack of validated scoring system for evaluating the quality and reliability of medical information in YouTube videos. The excellent interobserver and intraobserver reliability were confirmed using ICCs to redeem these shortcomings. In addition, the GQS may be highly subjective; thus, we tried to resolve the subjectivity by having two independent authors perform each evaluation twice. Fourth, one video entitled ‘Podcast: See a Live Surgery for Carpal Tunnel Syndrome’ has the dominant number of views (66.5%), so the average views and VPI values tended to increase. We tried to buffer this dominance by analysing 55 videos.

Conclusions

This study demonstrated that YouTube videos of CTS showed low reliability and quality. Video quality is significantly associated with content and upload source. Video popularity was not correlated with video reliability or quality, which suggests that a good content quality does not guarantee video popularity. The impact of videos on patient care cannot be underestimated. To ensure the spread of accurate information, it is necessary to have YouTube videos published by expert groups and to strive to provide high-quality video materials that can assist with patient diagnosis and treatment.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information.

Ethics statements

Patient consent for publication

Ethics approval

No human participants included, but ethical approval for this study was obtained from the institutional review board of Korea University Guro Hospital (registration number 2021GR0314).

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors DK: design of the study, acquisition of data, interpretation of data, statistical analysis and writing of initial draft; JWP: interpretation of data and critical revision of the manuscript; YW: acquisition and analysis of the data; YK: interpretation of data and statistical analysis; JIL: supervised the study and critically revised the manuscript; all authors: read and approved the final manuscript. JIL is guarantor of this study.

  • Funding This works was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2020R1C1C1004851).

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.