Introduction

Design considerations for an ASD early-intervention clinical trial must take into account (1) the trial duration, (2) number of participants, and (3) the quality of participant assessment. A short clinical trial of an early therapeutic intervention in 2–3 year old children can easily miss a target, as an improvement of symptoms may not emerge until children reach the school age. Small numbers of participants can easily skew the data as ASD is known to be a highly heterogeneous disorder. Longer clinical trials, with a greater number of participants, provide a better test for any intervention. Increasing the trial duration and the number of trial participants, however, raises the demand for regular assessment of participants by trained psychometric technicians. Furthermore, to attain the larger number of trial participants, clinical trials must accept participants across a large geographical region. The logistical issues associated with such an endeavor come at immense cost. As a result, large numbers of ASD clinical trials working under a limited budget suffer from short duration and low participant number, often compromising the trial objectives (e.g., Drew et al. 2002; Whitehouse et al. 2017).

A parent-completed Autism Treatment Evaluation Checklist (ATEC) assessment tool was in part designed to circumvent these problems (Rimland and Edelson 1999). If caregivers could serve as psychometric technicians and conduct regular evaluations of their children, the cost of clinical trials would be substantially reduced while simultaneously allowing for longer trial duration. There is an understanding in the psychological community that parents cannot be trusted with an evaluation of their own children. In fact, parents often yield to wishful thinking and overestimate their children’s abilities on a single assessment. However, the pattern of changes can be generated by measuring the score dynamics over multiple assessments. When a single parent completes the same evaluation every 3 months over multiple years, changes in the score become meaningful. ATEC was specifically designed to measure changes in ASD severity, making it useful in monitoring behaviors over time as well as tracking the efficacy of a treatment. ATEC is comprised of four subscales: (1) Speech/Language/Communication, (2) Sociability, (3) Sensory/Cognitive Awareness, and (4) Health/Physical/Behavior. The subscales provide survey takers with the information about specific areas of behaviors which may change over time.

The current observational study was initiated nearly two decades ago when one of the authors (Stephen M. Edelson of Autism Research Institute) distributed ATEC questionnaire to parents of children with ASD. Initially, ATEC evaluations were distributed as hard copy. In 2013 the online version of ATEC was developed. The current study analyzed data reported by participants using the online version of ATEC over a 4-year time span (2013–2017). The goal of the study was to characterize the typical changes in ATEC score over time as a function of children age, sex, ASD severity, and country of origin in a large and diverse group of participants.

Methods

ATEC Evaluation Structure

The ATEC is a caregiver-administered questionnaire designed to measure changes in severity of ASD in response to treatment. A total score and four subscale scores are reported. Questions in the first three subscales are scored using a 0–2 scale. The fourth subscale, Health/Physical/Behavior, is scored using a 0–3 point scale. ATEC can be accessed online or in hard-copy format.

The first subscale, Speech/Language/Communication, contains 14 items and its score ranges from 0 to 28 points. The Sociability subscale contains 20 items within 0–40 score range. The third subscale, Sensory/Cognitive awareness, has 18 items and scores range from 0 to 36. Finally, the Health/Physical/Behavior subscale contains 25 items. The scores from each subscale are combined in order to calculate a Total Score, which ranges from 0 to 179 points. A lower score indicates a lower severity of ASD symptoms and a higher score correlates with more severe symptoms of ASD.

Collection of Evaluations

ATEC responses were collected from participants voluntarily completing online ATEC evaluations from 2013 to 2017. The ATEC questionnaire was not actively advertised and use primarily originated from online searches. Participants consented to anonymized data analysis and publication of the results.

Evaluations of ATEC Score Changes Over Time

In order to study how ATEC scores change overtime and whether those changes vary within different ASD subgroups, the concept of a “Visit” was developed by dividing the 2-year-long observation interval into 3-month periods. All evaluations were mapped into 3-month-long bins with the first evaluation placed in the first bin. When more than one evaluation was completed within a bin, their results were averaged to calculate a single number representing this 3-month interval. It was then hypothesized that there was an interaction between a Visit and a given subgroup category (age, sex, ASD severity, and country of origin). Statistically, this hypothesis was modeled by applying linear mixed effect (LME) model with repeated measures, where an interaction term was introduced to test the hypothesis. This, in turn, enabled generation of pairwise differences between modeled subgroups at different visits. Participant specific variability was accounted for by introducing random effect into the model.

Pairwise differences were computed by applying “LS Means” and “contrast” functions to a generated LME model. For each ATEC score and a given subgroup, the following output was generated:

  1. 1.

    General ANOVA summary of the model itself, including p values for each covariate and the interaction term among them

  2. 2.

    LS Means computed for a given category at each visit (with 95% confidence interval)

  3. 3.

    Pairwise differences between categories at different visits with p values adjusted for multiple comparisons testing using Tukey method.

Participants

Participants were selected based on the following criteria:

  1. 1.

    Completeness: Participants who did not provide a date of birth (DOB) were excluded. As participants’ DOB were utilized to determine age, the availability of DOB was necessary.

  2. 2.

    Consistency: Participants had to have completed at least three questionnaires within 2 years and the interval between the first and the last evaluation was 1 year or longer.

  3. 3.

    Maximum age: Participants older than 12 years of age were excluded from this study.

    As diagnosis was not part of ATEC questionnaire, some neurotypical participants could be present in the database. To limit the contribution from neurotypical children, we excluded participants that may have represented the neurotypical population by using the minimum age and the minimal ATEC severity criteria.

  4. 4.

    Minimum age: Participants who completed their first evaluation before the age of two were excluded from this study, as the diagnosing of ASD in this age group is uncertain and the parents of some of these young cases may have completed the ATEC because they wanted to check whether their normal child had signs of autism.

  5. 5.

    Minimal ATEC severity: Participants with initial ATEC scores of less than 20 were excluded.

    After excluding participants that did not meet these criteria, there were 2272 total participants.

Age Groups

Participants were grouped based on age, calculated from the date of birth at the time of the first completed evaluation. The three age groups were: 2–3 years of age (YOA), 3.1–6 YOA, 6.1–12 YOA (Table 1).

Table 1 Characteristics and baseline measures for age groups
Full size table

Autism Severity Measurements

The initial ATEC total score was used as proxy for ASD severity. Participants were organized into three groups: mild (initial ATEC total score 20–49), moderate (initial ATEC total score 50–79), and severe (initial ATEC total score > 80) (Table 2).

Table 2 Characteristics and baseline measures for ASD severity groups
Full size table

Country Groups

Participants were split into two groups based on their country of origin. The developed English-speaking nations included participants from the United States, Canada, United Kingdom, Ireland, Australia, and New Zealand. Participants from other countries were grouped together as “the non-English-speaking countries” (Table 3). In the non-English-speaking countries group, only 53 participants (4%) were from Japan, France, Germany and northern Europe. The majority of participants were from Latin America (859; 63%), southern Europe (182; 13%), and India (70; 5%).

Table 3 Characteristics and baseline measures for the English-speaking countries and the non-English-speaking countries
Full size table

Sex Groups

Data were stratified based on sex. 83% of the 2272 participants were males (Table 4).

Table 4 Characteristics and baseline measures for gender groups
Full size table

Results

The least squared means (LS Means), as well as the dynamics of score changes (LS Means differences) over time (between visits) for each group are presented in the supplementary materials. The fitting of LME model allowed us not only to assess the temporal dynamics of the scores, but also to evaluate the “tightness” of each individual mean value by generating 95% confidence interval. There was a high degree of data consistency, similar to what was reported by Magiati et al. (2011). The differences between participant subgroups at different visits, and differences between the first and the last visit per subgroup are discussed in greater detail below.

Effect of Sex on Longitudinal Change of ATEC Scores

The interaction term between Sex group and Visits was not statistically significant (at the α  =  0.05 level of significance) for either the ATEC total score or any of the subscale scores (Table S1).

Longitudinal Change of ATEC Scores as a Function of Age

The significance of interaction term between Age group and Visits shows that the dynamics of ATEC total score as well as scores in the Communication, Sociability, and Sensory subscales vary within different Age groups (Table S2). Table 5 shows LS Means for age groups at the initial and the last visits. Reduction in ATEC total score (showing the degree of improvement) was inversely related to age (Table 5). Over the 2 years, the 2–3 YOA group improved by 28.35 units (SE = 1.30, p < 0.0001), the 3–6 YOA group improved by 19.73 units (SE = 0.72, p < 0.0001), and the 6–12 YOA group improved by 13.80 units (SE = 0.96, p < 0.0001).

Table 5 LS Means for various age groups
Full size table

For age group comparisons, three pairwise comparisons were made (Table 6). Neither pairwise comparison reached statistical significance at Visit 1, but all three pairwise comparisons in ATEC total score yielded statistically significant differences in LS Means at Visit 8 with younger children improving more than the older children. The difference in ATEC total score for the 2–3 YOA group relative to the 3–6 YOA group was − 2.26 units (SE = 0.83, p = 0.6501) at Visit 1 and − 6.36 units (SE = 1.44, p = 0.0042) at Visit 8. ATEC total score difference for the 2–3 YOA group relative to the 6–12 YOA group was 2.44 units (SE = 0.93, p = 0.7182) at Visit 1 and − 12.12 units (SE = 1.57, p < 0.0001) at Visit 8. ATEC total score difference for the 3–6 YOA group relative to the 6–12 YOA group was 0.17 units (SE = 0.71, p = 1.0000) at Visit 1 and − 5.76 units (SE = 1.16, p < 0.0003) at Visit 8. These observations were recapitulated in the Communication and Physical subscales (Table 6). For the Sociability subscale, only the 2–3 YOA group vs. 3–6 YOA group and 2–3 YOA group vs. 6–12 YOA group yielded a statistically significant decrease in score at Visit 8 (Table 6). For the Sensory subscale, only the 2–3 YOA group vs. 6–12 YOA group yielded a statistically significant decrease in score at Visit 8 (Table 6).

Table 6 LS Mean differences between age groups
Full size table

Country Effects on ATEC Scores

Surprisingly, a comparison of developed English-speaking nations (the United States, Canada, United Kingdom, Ireland, Australia, and New Zealand) to non-English-speaking countries demonstrated greater improvements in ATEC total score and all subscales in the non-English-speaking nations. The significance of interaction term between Country group and Visits shows that the dynamics of ATEC total score and all subscales varies in different Country groups (Table S13). Table 7 shows LS Means for Country groups at the initial and the last visits. Reduction in ATEC total score was greater in non-English-speaking nations group (Table 7). Over the period of 2 years the participants in the English-speaking nations group improved by 16.70 units (SE = 0.80, p < 0.0001), and non-English-speaking nations group improved by 21.58 units (SE = 0.70, p < 0.0001).

Table 7 LS Means of English-speaking countries group and non-English-speaking countries group
Full size table

The difference in ATEC total score for the English-Speaking Counties group relative to the non-English-speaking countries was 0.83 units (SE = 0.61, p = 0.9937) at Visit 1 and − 4.05 units (SE = 1.02, p = 0.0056) at Visit 8 (Table 8). This statistically significant difference at Visit 8 indicates that children in the English-speaking countries improve their symptoms to a smaller degree than children in the non-English speaking nations. Dissection of the ATEC total score into subscales indicated that children in the non-English speaking nations demonstrated greater improvements of their symptoms in all subscales with the Sociability and Physical subscales having the greatest contribution to the difference between the groups. The difference in the Sociability subscale score for the English-Speaking Counties group relative to the non-English-speaking countries was 0.13 units (SE = 0.19, p = 1.0000) at Visit 1 and − 1.37 units (SE = 0.32, p = 0.0023) at Visit 8 (Table 8). The difference in the Physical subscale score for the English-Speaking Counties group relative to the non-English-speaking countries was − 0.32 units (SE = 0.28, p = 0.9990) at Visit 1 and − 1.60 units (SE = 0.47, p = 0.0619) at Visit 8 (Table 8).

Table 8 LS Mean differences between the English-speaking countries group and non-English-speaking countries group
Full size table

Change of ATEC Scores as a Function of ASD Severity

The significance of interaction term between ASD severity group and Visits shows that the dynamics of ATEC total score and of individual scores within all subscales differs between severity groups (Table S24). Table 9 shows the LS Mean calculations for the three severity groups (mild, moderate, severe) at Visit 1 and Visit 8. Reduction in ATEC total score was directly related to severity (Table 9). Over the 2 years, the mild group improved by 11.20 units (SE = 0.87, p < 0.0001), the moderate group by 20.56 units (SE = 0.76, p < 0.0001), and the severe group by 29.52 units (SE = 1.10, p < 0.0001).

Table 9 LS Means for various severity groups
Full size table

In comparing the difference in LS Mean for ATEC total score at Visit 1, all three pairwise comparisons between severity groups yielded statistically significant differences (Table 10). This is in contrast to Visit 8, at which point none of the comparisons reached statistical significance (Table 10). The results for all four subscales mirrored those of ATEC total score, showing no statistically significant differences between severity groups at Visit 8 (Table 10). This may simply be an artifact of the definition of ASD severity, which is based solely on a child’s initial ATEC total score independent of child’s age. Therefore, a different approach to definition of ASD severity groups was investigated.

Table 10 LS Mean differences between severity groups
Full size table

According to ATEC norms, ATEC total scores in young children decrease exponentially with age, with a time constant of approximately 3.3 years regardless of the initial ATEC total score (Mahapatra et al. 2018). Thus, participants could be divided into three approximately equal groups using exponents decaying with a time constant of 3.3 years. The exact parameters of exponents were determined using best-fit trendlines to ATEC norms (Mahapatra et al. 2018), (Table 11).

Table 11 Severity group definition based on initial ATEC total score and age
Full size table

Group differences were reassessed using the severity group definition based on both the initial ATEC total score and age as specified in Table 11. The significance of interaction term between ASD severity group and Visits shows that the dynamics of ATEC total score and individual subscale scores varies between different severity groups (Table S35). Table 12 shows the LS Mean calculations for the three severity groups at Visit 1 and Visit 8. Reduction in ATEC total score was directly related to severity (Table 12). Over the 2 years the mild group improved by 16.46 units (SE = 0.92, p < 0.0001), the moderate group improved by 21.27 units (SE = 0.90, p < 0.0001), and the severe group improved by 20.48 units (SE = 0.90, p < 0.0001).

Table 12 LS Means for various severity groups defined based on initial ATEC total score and age
Full size table

In comparing the difference in LS Mean for ATEC total score at Visit 1, all three pairwise comparisons between severity groups yielded statistically significant differences (Table 13). This is in contrast to Visit 8, at which point none of the comparisons reached statistical significance. For the Communication subscale all pairwise group differences were statistically significant at Visit 8, confirming the advantage of severity group assignment based on both initial ATEC total score and age and indicating that the mild group improved more than the moderate group and the moderate group improved more than the severe group (Table 13). There were no statistically significant differences between severity groups at Visit 8 in the Sociability, Sensory, or the Physical subscales.

Table 13 LS Mean differences between severity groups defined based on initial ATEC total score and age
Full size table

Discussion

The regular assessment of temporal change in symptoms of children with Autism Spectrum Disorder (ASD) participating in a clinical trial has been a long-standing challenge. A common hurdle in these efforts is the availability of trained technicians needed to conduct rigorous and consistent assessment of children at multiple time points. If parents could administer regular psychometric evaluations of their children, then the cost of clinical trials will be reduced, enabling longer clinical trials with the larger number of subjects.

The ATEC was developed to provide such a free and easily accessible method for caregivers to track the changes of ASD symptoms over time (Rimland and Edelson 1999). Various studies have sought to confirm the validity and reliability of ATEC (Al Backer 2016; Geier et al. 2013; Jarusiewicz 2002), yet none to date have assessed longitudinal changes in participants’ ATEC scores with respect to age, sex, and ASD severity. One trial conducted by Magiati et al., aimed to comprehensively assess ATEC’s ability to longitudinally measure changes in participant performance (Magiati et al. 2011). That study utilized ATEC to monitor the progress of 22 schoolchildren over a 5-year period. ATEC score was compared to age-specific cognitive, language, and behavioral metrics such as the Wechsler Preschool and Primary Scale of Intelligence. The researchers noted ATEC’s high level of internal consistency as well as a high correlation with other standardized assessments used to measure the same capacities in children with ASD (Magiati et al. 2011). Charman et al. utilized ATEC amongst other measures to test the feasibility of tracking the longitudinal changes in children using caregiver-administered questionnaires and noted differential effects across subscales of ATEC, possibly driven by development-focused vs. symptom-focused subscales that are conflated in the ATEC total score (Charman et al. 2004). Another study assessing the ability of dietary intervention to affect ASD symptoms also utilized ATEC as a primary measure (Klaveness et al. 2013), concluding that it has “high general reliability” coupled with an ease of access. Whitehouse et al. used ATEC as a primary outcome measure for a randomized controlled trial of their iPad-based intervention for ASD named TOBY (Whitehouse et al. 2017). This trial was conducted over a 6-month time frame, with outcome assessments at the 3- and 6-month time points. Although the study did not demonstrate significant ATEC score differences amongst test groups, the researchers reaffirmed their use of ATEC, noting its “internal consistency and adequate predictive validity” (Whitehouse et al. 2017). These studies support the viability of ATEC as a tool for longitudinal measurement of ASD severity which can be vital in tracking symptom changes during a trial.

The current study analyzed data reported by participants using the online version ATEC over a 4-year time period from 2013 to 2017. Assessing these data permitted insight into the effects of age, sex, country of origin, and ASD severity on the longitudinal changes in ATEC score with all of these factors (save for sex) showing statistically significant differences affecting ATEC score dynamics. These findings identify specific variables capable of altering the developmental trajectory of children with ASD and indicate possible avenues of future investigation of causal relationships related to changes in ASD severity.

Sex Does Not Affect ATEC Score

The prevalence of ASD is strongly male-biased, affecting four times as many males as females. Accordingly, we were interested in differences in the rate of improvement between participants of different sexes. No significant differences in improvement of ATEC total score were observed. This suggests that the rate of improvement of ASD symptoms remains similar in males and females.

Effect of Age on ATEC Score

The participants’ age was a significant modulating factor in determining the rate of their improvement. Younger children demonstrated greater improvement in ATEC total score. This phenomenon was recapitulated across subscales, with differences between the 2–3 YOA group and 3–6 YOA group reaching statistical significance for the Communication, Sociability, and Physical subscales and differences between the 2–3 YOA group and 6–12 YOA group reaching statistical significance for all subscales (Table 6). This finding is consistent with other ATEC longitudinal studies: younger children showed greater improvement in ATEC total score compared to the older children (Magiati et al., Charman et al., Whitehouse et al., Table 14).

Table 14 Comparison of the annualized decrease of ATEC score across multiple studies
Full size table

The magnitude of the annual decrease of the ATEC score was also found to be roughly similar to other reports across the studied age range. For the younger children the reduction of ATEC score seen in this study is in between those of Whitehouse et al./TOBY trial and Charman et al., Table 14. For the older children, the reduction of ATEC seen in this study is somewhat similar to that reported by Charman et al., Table 14.

The small differences between the studies can be attributed to differences in study design. In particular, the current study (1) had significantly more participants, (2) was based on greater number of ATEC evaluations, and (3) was conducted over the longer period of time than all the others discussed herein.

Effect of ASD Severity on ATEC Score

In comparing the difference in LS Mean for ATEC total score at Visit 1, all three pairwise comparisons between severity groups yielded statistically significant differences (Table 10). This is in contrast to Visit 8, at which point none of the comparisons reached statistical significance (Table 10). The results for all four subscales mirrored those of ATEC total score, showing no statistically significant differences between severity groups at Visit 8 (Table 10). This may simply be an artifact of the definition of ASD severity, which is based on a child’s initial ATEC total score. This method groups children with the same initial ATEC total score together independent of age. Thus, children who score 80 on their initial evaluation at the age 10 are grouped together with children who score 80 on their initial evaluation at the age 2. According to ATEC norms (Mahapatra et al. 2018), these children will score 70 and 25 respectively at the age of 12, and therefore clearly belong to different severity groups. This inconsistency in definition of ASD severity solely based on the initial ATEC total score independent of age may explain the observation that none of the group comparisons reached statistical significance at Visit 8.

The definition of ASD severity groups based on two parameters—the initial ATEC total score and age—yielded somewhat superior results compared to defining ASD severity based solely on the initial ATEC total score. While both definition methods showed no statistically significant differences between severity groups at Visit 8 in ATEC total score (Tables 10, 13), the former method showed statistically significant pairwise differences between all the groups at Visit 8 for the Communication subscale, indicating more improvement in children with milder ASD and confirming the advantage of severity group assignment based on both initial ATEC total score and age.

Role of Country of Origin

Conventional wisdom may suggest that the increased access to resources, including government-provided therapy for ASD, should lead to greater improvements. English-speaking nations (the United States, Canada, the United Kingdom, Ireland, Australia, and New Zealand) lead the world in government spending on therapy for children with ASD (Ganz 2007; Horlin et al. 2014; Paula et al. 2011) and therefore would be expected to produce superior outcomes of ASD therapy. Surprisingly, a comparison of English-speaking nations to the non-English-speaking countries demonstrated greater improvements in ATEC total score as well as in each subscale within the non-English speaking nations (Table 8).

This observation runs contrary to conventional thought and underscores the consensus that there is a potential for improving the treatment of children with ASD in the developed world. While it is difficult to speculate on the reason for this disparity between developed English-speaking countries and non-English-speaking countries, it is notable that child treatment is more often outsourced in the developed English-speaking countries compared to more traditional societies where grandparents are more commonly available and mother is more likely to stay at home to personally take care of a child (Fetterolf 2017). Other factors, such as differences in diet (Adams et al. 2018; Rubenstein et al. 2018), reliance on technology (Dunn et al. 2017; Grynszpan et al. 2014; Lorah et al. 2013; Odom et al. 2015; Ploog et al. 2013) and prescription medications (Lemmon et al. 2011) could also play a role.

Limitations

Participant selection presents a novel challenge in a study focused on caregiver-administered assessments. In the selection of participants for inclusion in this study, a baseline of ASD diagnosis could not have been established as child’s diagnosis is not part of ATEC questionnaire. Thus, it is not impossible that some of the participants did not have ASD diagnosis altogether. E.g., parents of a neurotypical toddler worried for any reason about an ASD diagnosis could have decided to monitor toddler’s development with ATEC evaluations and thus inadvertently added their normally developing child to the ATEC collection. As neurotypical children develop faster, the presence of neurotypical children in the dataset would have artificially increased the magnitude of annual changes of ATEC scores, predominantly for younger participants.

It is unlikely though that there were many neurotypical participants in our database. First, ATEC is virtually unknown outside the autism community. Second, there is little incentive for the parents of neurotypical children to complete multiple exhaustive ATEC questionnaires (unless one of the children was previously diagnosed with ASD). Third, as described in the “Methods” section, to further limit the contribution from neurotypical children, participants possibly representing the neurotypical population were excluded: those with an initial ATEC total score of 20 or less (7% of all participants) and those who completed their first evaluation before the age of two (3% of remaining participants). Despite this effort, the reported data may over-approximate the magnitude of annual changes of ATEC scores, especially in the younger participants.

As noted by other groups (Whitehouse et al. 2017; Charman et al. 2004), the use of ATEC as a primary outcome measure has some inherent drawbacks. While the ATEC is capable of delineating incremental differences in ASD severity amongst participants, the variety of measures amongst its subscales fails to differentiate developmental-specific changes from symptom-specific ones. This aspect of the ATEC may introduce a confounding variable when participants are at different developmental stages and follow unique developmental trajectories during a study. To mitigate these effects, trial designs must accurately separate participants based on developmental stage. This is most often accomplished by using age as a proxy for developmental stage.

Conclusions

This manuscript attempts to characterize the typical changes in ATEC score over time as a function of children age, sex, ASD severity, and country of origin in a large and diverse group of participants. In doing so, it lends support to the efficacy of caregiver-driven psychometric observation, which, when applied at scale, may be a viable alternative to using licensed technicians to assess the children.