Participants
Participants spanning all phases of the assessment tool development consisted of kindergarten, first and second grade students among two different school districts in North Central Texas. Five elementary schools were used to collect a variety of observations during recess. The schools and districts utilized were chosen due to accessibility of locations to the research team and approval of administration and principals to observe recess in the schools. Three phases were completed using the five schools randomly to confirm that this assessment tool was viable to use on playgrounds during recess with elementary school children.
Phase 1: Identifying the movements observed in recess
The purpose of Phase 1 was to study the types of activities children participated in at recess and categorize them into unilateral, bilateral, contralateral, and “no movement” categories. To accomplish this task, two researchers observed four elementary school recesses consisting of first and second-graders to document any physical movements observed. These grades were chosen because the students fit the elementary student criteria and the recess times available for the researchers to observe. A scan from left to right was conducted of the entire play area as all activities observed were recorded by hand. Hash marks were used to note repeated movement activities. Examples of activities recorded included skipping, sliding, hanging on bars, moving across the monkey bars, kicking or dribbling a soccer ball, using a jump rope to jump, climbing up a climbing wall, and running. The left-to-right scan was continued for the entirety of the scheduled 20-minute recess). Once two researchers observed four recesses each the listed activities were accumulated and analyzed. Each activity was grouped into unilateral, bilateral, contralateral, or no movement categories. All recorded activities fell into one of the four categories; therefore, the three limb movements were confirmed as the complete list of limb movements to assess during recess. With this established, an observation tool and protocol were created next to efficiently capture and record the limb movements of children during recess.
Phase 2: Development of a Limb Movement Assessment Tool
Phase 1 led to the validation of the different activities and limb movements observed on the playground above and the creation of the first iteration of this new tool named the Movement Pattern Observation Tool (MPOT). The first MPOT (Fig. 1) was designed for two observers to scan one-half of the playground each. Each observer would write the movements they observed in the chart. This tool was used to observe one elementary school recess and the recordings were analyzed. The analysis showed that the form did not capture what the individual child was participating in but rather what equipment or section of the playground was getting the most use. For example, the monkey bars may have been traversed thirty times within the ten minutes, with thirty hash marks, but the same five children may have completed the thirty traverses.
Therefore, revisions were made to the first iteration, and a second version of the MPOT (Fig. 2) was created to capture the individual activity per child with a three-minute “snapshot” scan. This “snapshot” format was included to observe the entire play area within three minutes. Three minutes were chosen through trial and error in the field. Two minutes did not allow enough time to complete observations, while five minutes seemed too long. A snapshot scan was accomplished by observing a designated section of the play area and scanning that section from left to right, spending no more than 20 seconds per section. A specific recording pattern was established among structures, swing sets, and open field areas for time efficiency. The observer recorded all activities seen as if a photo was taken of the scanned section. If the child moved from one activity to another within the designated section, the observer recorded the greater of the two movements.
The head researcher used this second MPOT version for two elementary school recess periods on two separate days for a total of four observations. The use and ease of the form were analyzed. The primary benefit of the snapshot scan style of recording was that it allowed the observer to traverse the entire play area efficiently, allowing more individual students to be observed. Although some students moved from one play area to another during observations, less repetition of the same children was observed with this format, and a greater percentage of the children were recorded with the different types of activities than with the original form. Although the snapshot scan produced a more accurate method of recording individual movements, another modification of the form design was necessary to create efficiency of commonly recorded activities.
Modifications for the final observation tool stemmed from the feedback of a new group of researchers/observers (focus group) to create an observation tool that would be usable and efficient across various researchers for future study purposes. The first focus group consisted of two new observers whom the master observer trained in the use of the MPOT. Then they used the MPOT at one elementary school for three different grade-level recesses in one day. The observers gave feedback after the observations were completed that helped to determine efficient recording mechanisms and column construction. The most common activities recorded in all previous tools were noted and grouped into categories to be noted with a hash mark instead of writing.
Once the modifications were made to the MPOT, the second focus group, comprised of the same two observers and the lead researcher, observed three recesses (kindergarten, 1st, and 2nd grade) from one school in one day. After each recess, the group discussed any questions about usability and suggestions for the tool's efficiency or the observation instructions. Modifications from their feedback included grouping the activity categories by the limb movement required, moving "run" to the top of the form since it was the most commonly used activity, and differentiating between gender within each category. The three, three-minute scans were modified to two, four-minute scans. Although the lead researcher could complete the observations within three minutes due to many familiarities, the focus team needed closer to four minutes due to inexperience in labeling the movements. As one uses the tool more often, efficiency can increase, but for training and usability purposes, four minutes was more appropriate. Finally, standing had been recorded in earlier MPOT versions as bilateral activity, but through further discussion, standing was shifted to the non-movement category. Though standing creates a bilateral load on the musculoskeletal system, this study is focused on limb movements and the neurological development that occurs when utilizing them, hence the label change.
The MPOT final version (Fig. 3) was constructed from the observer focus group feedback given above. The master observer practiced observations with the final MPOT version in two or more kindergarten, first-grade, and second-grade 15-minute recesses. Within each school district, two MPOT observations were completed per kindergarten, first and second grade, for a total of six observations per district and 12 observations overall. The number of children observed ranged from 90–130 per recess, depending on the grade level of the participating school. One school averaged 90–100 children per grade level, whereas the other averaged 110–130 children per grade level. Eighty-three percent of the total children attending recess were observed with the MPOT. The other 17% were not captured as they may have been transitioning from one playground section to another. After each observation, the master observer analyzed the data by categorizing the activities into unilateral, bilateral, contralateral, and no-movement categories. The total number for each category was recorded in Excel for each grade and recess individually to gather raw data totals. From the master observer’s perspective, the final observation tool was easy to use, captured the children in different areas of the play spaces, and quickly identified unilateral, bilateral, and contralateral movements throughout each recess period.
Phase 3: Interrater Reliability of the MPOT
The tool was now ready for interrater reliability analysis through the completion of multiple observations, focus group feedback, and the development of the final MPOT version in phase two. Thirty-five observations of kindergarten through fifth-graders were completed to test the interrater reliability. The master observer and two other trained observers (observer 2 and observer 3) participated in the interrater reliability assessment at an elementary school that none of the three observers had previously assessed. The master observer and observer 2 used the MPOT for 23 four-minute comparison observations, and the master observer and observer 3 used the MPOT for 12, four-minute comparison observations. Statistical analyses were run to determine internal consistency between observers. As evidenced below, the difference in the number of observations between observers did not appear to impact the interrater reliability, possibly alluding to reliable use without extensive practice. This statement is just an inference and would have to be tested to conclude statistically.
Interrater Reliability Results
Interrater Reliability was calculated using Interclass Correlation Coefficients (ICC) and their 95% confidence intervals using SPSS statistical package version 28. The results were based on a mean rating (k = 2), absolute-agreement, and a 2-way mixed-effects model (Table 1). ICC values less than 0.5 are indicative of poor Reliability, values between 0.5 and 0.75 indicate moderate Reliability, values between 0.75 and 0.9 indicate good Reliability and values greater than 0.90 indicate excellent Reliability [16, 17, 18]. The ICC of the master observer and observer 3 value was .898, and the ICC of the master observer and observer 2 was .885. Therefore, based on statistical inference, it was concluded that the Reliability is “good” approaching “excellent.”
Table 1
Interclass Correlation Coefficient
|
95% Confidence Interval
|
|
Average Measures
|
N
|
Intraclass Correlation
|
Upper Bound
|
Lower Bound
|
Sig
|
Master Observer and Observer 2
|
23
|
.898
|
.757
|
.957
|
< .001
|
Master Observer and Observer 3
|
12
|
.885
|
.599
|
.967
|
< .001
|