Next Article in Journal
Optimized Neural Network Based on Genetic Algorithm to Construct Hand-Foot-and-Mouth Disease Prediction and Early-Warning Model
Next Article in Special Issue
Characterizing Breakthrough Cancer Pain Using Ecological Momentary Assessment with a Smartphone App: Feasibility and Clinical Findings
Previous Article in Journal
Client-Centered Breastfeeding-Promotion Strategies: Q Methodology
Previous Article in Special Issue
Effects of the Physician–Primary-Healthcare Nurse Telemedicine Model (P-NTM) on Medication Adherence and Health-Related Quality of Life (HRQoL) of Patients with Chronic Disease at Remote Rural Areas
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 18
Issue 6
10.3390/ijerph18062958
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Effectiveness of a Web-Based Intervention on Parental Psychological Flexibility and Emotion Regulation: A Pilot Open Trial

by
Juan M. Flujas-Contreras
1,2,*,
Azucena García-Palacios
3,4 and
Inmaculada Gómez
1,2
1
Department of Psychology, University of Almeria, 04120 Almeria, Spain
2
Health Research Centre (CEINSA/UAL), University of Almeria, 04120 Almeria, Spain
3
Department of Psychology, University Jaume I, 12071 Castellon, Spain
4
CIBER of Physiopathology of Obesity and Nutrition CIBERobn, CB06/03 Instituto de Salud Carlos III, 28029 Madrid, Spain
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(6), 2958; https://doi.org/10.3390/ijerph18062958
Submission received: 1 February 2021 / Revised: 3 March 2021 / Accepted: 11 March 2021 / Published: 13 March 2021
(This article belongs to the Special Issue Improving Health Management and Well-Being for Young and Older Adults with eHealth and mHealth)
Browse Figures
Versions Notes

Abstract

:
“Parenting Forest” is an informed contextual therapy parenting program for improving parental emotion regulation strategies and psychological flexibility. The aim of this study was to evaluate the preliminary effectiveness of a self-guided web-based intervention of the Parenting Forest program. The intervention program consists of six self-applied sequential modules that use strategies from contextual therapies for providing a parenting style open to experience, mindful and committed to its actions. A pilot controlled open trial was conducted. Eligible parents (n = 12) enrolled in the web-based intervention completed baseline (T1) and post-intervention (T2) assessment instruments. Parental psychological flexibility, avoidance, emotional regulation, parental stress, satisfaction with life, children’s psychological adjustment and client satisfaction were measured to assess the effects of the intervention. Mood, coping, and value-related actions were assessed as measures of progress. The results showed positive effects on the parents’ psychological flexibility and emotion regulation. Parents’ mood and coping skills improved throughout the intervention program. These results provide preliminary evidence of the web-based Parenting Forest’s efficacy, although further research is needed to assess its effectiveness for prevention and in clinical populations.

1. Introduction

Parental practices have been shown to have a protective effect on family problems and their children’s psychological adjustment. Undemocratic parenting styles can lead to internalizing and externalizing problems and less prosocial behavior in childhood [1,2]. Authoritarian and controlling parental styles are risk factors for bullying [3], internalizing problems [4], anxiety, depression, and suicidal ideation [5]. Parental stress has also been shown to be a mediator of child behavioral problems [6] and leads to worse parenting practices related to psychological inflexibility [7].
Parental psychological flexibility is defined as the ability to be fully aware of the emotions and thoughts related to childcare, taking them in perspective, and redirecting behavior into parenting practices in line with values [8]. In contrast, experiential avoidance refers to attempts to reduce or eliminate events or stimuli that produce discomfort [9]. Experiential avoidance responses have been associated with problems of anxiety [9], stress [10], depression and family conflict [11] in both children and parents. Psychological inflexibility in parenting practices can lead to dysfunctional reactions or interactions as a result of cognitive fusion and avoidance of the emotional distress associated with parenting, i.e., avoiding children’s reactions, overprotective or impulsive parental reactions that occur with the function of trying to avoid confronting situations or private events (thoughts, emotions, feelings, sensations, etc.) that might cause discomfort [12,13,14,15]. Moreover, psychological flexibility has been related to emotion regulation [16] as an adaptive strategy, that is, strategies that are linked as a protective factor against psychopathology or are associated with a good health and well-being outcome for individuals [17]. Related to psychological flexibility in parenting, mindful parenting is defined as the use of mindfulness strategies, i.e., full attention in the present moment, to enable parents to redirect their attention to the needs and interactions with their children [18]. Promotion of psychological flexibility and mindful parenting are related to a better development of emotion expression [19], regulation skills [20], and mental health of children and adolescents [21]. These skills are mainly addressed by third-wave behavior therapies (or contextual therapies) that attempt to enhance skills as (a) acceptance and emotion regulation, (b) mindfulness or attention-focusing strategies, and (c) motivation to change towards values [22]. These are a heterogeneous group of therapies, that share common characteristics and psychological process, which emerged as an expansion of behavioral and cognitive–behavioral therapies [22].
The goal of parenting training programs is to improve skills that improve or facilitate a positive parenting style. Training in behavioral parenting has shown positive effects, especially for behavior problems, developmental disorders, and eating problems [23]. For example, behavioral parenting intervention has also been associated with improved family dynamics, problem solving and communication between family members [24]. Despite the strong evidence that it improves parenting and family relations, there are often problems in carrying out parenting intervention. Only about 35% to 50% of parents who enroll in interventions actually attend [25]. Lack of time, the burden of childcare, scheduling and transportation incompatibility [26], social stigma [27], or emotional barriers [13] are some of the factors that make it difficult to maintain treatment adherence. Adherence to technology-based parenting interventions increases by 83% [28].
In addition to these limitations in evidence-based intervention, the general population has little access to them [29]. Technology-based intervention is a promising method in clinical psychology, as a large number of people who seek help can attend without the limits of time or space [30,31], and with an accessible, brief and flexible format [32]. The potential of technology-delivered intervention for parents has been emphasized [33,34].
Technology-based parenting intervention has mainly been online [35], although computer programs, podcasts, and video series have also been used [36]. Web-based parenting intervention has shown parent–child outcomes with moderate effect sizes and a high level of satisfaction [35,37], and larger effect sizes for parents with children diagnosed with a medical illness [28]. As larger effect sizes have been found for treatment intervention than for prevention [28], different levels of support may be necessary to target family needs [38]. However, parents who have had access to universal prevention programs in parenting have indicated greater satisfaction and sense of competence than those who did not [39]. Online intervention for families has shown evidence of reducing children’s behavioral problems [40,41], preventing such affective problems as anxiety and depression [42], improving communication skills of children with autism spectrum disorder [43], and improving problem-solving by parents with children with cancer [44]. Web-based Parent–Child Interaction Therapy showed a larger effect size than online psychoeducation or cognitive–behavioral therapy-based parenting intervention [28].
Informed parenting intervention with third-wave therapies has mainly been implemented with Acceptance and Commitment Therapy (ACT) [45], Dialectical Behavior Therapy (DBT) [46], and Mindfulness [47]. A review of ACT for parents found that it has mainly been applied to parents of children with developmental disorders and ASD, chronic pain, physical, and mental health problems [48]. These interventions have shown positive effects on parental distress status, parental psychological flexibility, emotion regulation and improvement in children’s problems [48].
Mindful parenting intervention has shown positive effects in reducing stress, worry, improvement in parenting practices, and family functioning [49], emotional well-being and stress for parents of children with autism, for parents with children with disabilities [50], and on reactivity and stress in parents of children with ADHD [51]. The inclusion of mindfulness strategies in parenting training can be beneficial to family functioning, parent–child relationships, and parenting strategies [52].
DBT intervention for parenting is included in adaptation for children (DBT-C) and adolescents (DBT-A), with improvement in depressive symptomatology, behavioral problems and functioning [53]. Although these adaptations are child-oriented, they include a parenting module that aims to create a validating environment and influence behavior [54,55]. Other DBT-informed interventions, such as parental intervention with DBT strategies, have shown positive effects on emotion regulation, reduced affective symptoms, increased maternal acceptance, and parental practices [53].
Few studies have applied third-wave therapies to promoting parenting on the internet. A study by Sairanen et al. [56] of online ACT applied to parents of children with chronic medical conditions found large effect sizes in improving stress and mindfulness skills, as well as moderate effect size for improvement in depression and anxiety. A combination of ACT and ABA (applied behavior analysis) on the Internet for parents of children with ASD found improvement in parental knowledge and prosocial behavior and children’s hyperactivity [57]. Positive effects on mindfulness skills and depressive and anxiety symptoms have been found in pregnant mothers with online mindfulness intervention [58].
This pre-experimental study was a preliminary test of the effectiveness and satisfaction of a web-based parenting intervention protocol for promoting emotion regulation and parental psychological flexibility. It assessed the feasibility of an online clinical protocol based on an informed contextual therapy parenting program, using mainly ACT strategies, that has been tested in a face-to-face group format with positive results [59]. We hypothesized that a significant improvement would be found in the main variables, that is, parental psychological flexibility, emotion regulation, and parental stress. A resulting reduction in problems and difficulties of their children was also expected.

2. Materials and Methods

2.1. Design

This study employed a controlled, non-randomized, pre–post open trial. It received the ethical approval of the Andalusian Health Service’s Almeria Research Committee. The CONSORT 2010 Statement: extension to randomized pilot and feasibility trials [60] adapted to non-randomized studies was followed [61].

2.2. Participants

Parents were recruited by posting information on Facebook, at children’s mental health centers and parents associations in Almeria. The inclusion criteria were: (a) parent or legal guardian, (b) of a child from 3 to 16 years old (c) with difficulties in flexible parenting based on Parental Acceptance Questionnaire scores (6-PAQ) [62]; or (d) child with borderline or abnormal scores in emotional or behavioral difficulties, as measured by the Strengths and Difficulties Questionnaire (SDQ) [63]; (e) email and daily access by internet; (e) parent/guardian not diagnosed with any psychological or substance use disorder that would impede participation in the study; (f) no parent/guardian language or comprehension barriers.
The participants filled out an online form with their data. If they met the criteria for inclusion, they gave their informed consent before starting the study. Participants did not receive any financial incentive for study participation.
The pilot trial sample size was calculated in relation to the clinical trial [64]. Julious [65] recommends a group of 12 per experimental condition for pilot studies based on feasibility, precision of the mean and variance, and regulatory considerations. We would need a sample size of 12 to reliably (with probability greater than 0.7) detect an effect size of δ ≥ 0.8, assuming a two-sided criterion for detection that allows for a maximum Type I error rate of α = 0.05.
Figure 1 shows the participant flowchart. Of the 27 participants that met the study’s inclusion criteria, 15 were excluded for not accessing the platform, not completing the units, or withdrawing from treatment due to lack of time. Twelve participants who were included in this pilot study completed the intervention.
The incidental sample was comprised of 12 Spanish participants (fathers and mothers individually) aged 36 to 50 (M = 41.1; SD = 4.17), of whom 83.3% (n = 10) were women. A total of 66.7% (n = 8) of the parents were married, 1 widowed, 1 common-law couple and 2 divorced. Most parents had two children (66.7%; n = 8) aged six months to 15 years (M = 7.84; SD = 3.48). Table 1 presents the sociodemographic data and psychological characteristics of each participant.

2.3. Intervention

The online parenting intervention was carried out using the Psychology and Technology Platform [66]. This web platform has been successfully employed in previous clinical trials for emotional disorders [67]. When the participants had filled in their sociodemographic data and confirmed their informed consent, they were contacted by e-mail and given individual 24/7 access to the intervention on the Web platform with a username and password. Parents complete the program individually.
“The Parenting Forest” intervention protocol aims to promote emotion regulation skills and parental psychological flexibility. The components are based on third-wave therapies [13,15,68,69]. To achieve this goal, exercises and activities are proposed for: (a) fostering full awareness and attention to the present, both in parents’ private events and in their relationships with their children; (b) the use of verbal regulation strategies based on acceptance, as opposed to avoidance behavior and emotional suppression; (c) increasing perspective taking of self-thoughts, emotions or sensations (defusion); (d) providing an environment of emotional validation in relation to parenting and to oneself; (e) promoting behavioral activation in actions towards personal values, specifically related to their children. This protocol has previously been applied in a face-to-face group format with positive effects on parental psychological flexibility, parental stress and emotion regulation, and reduction in emotional and hyperactivity symptoms in their children [59].
The parenting intervention is organized in six sequential self-applied units. Participants are encouraged to take one unit a week and practice the homework the following week. The first four units focus on promoting personal and parenting skills. The last two units are focused on parent–child relationships. From the very beginning, the parents work on setting the clear individual values-related goals that will guide the treatment (Table 2). Unit contents are delivered as video instructions, metaphors and exercises, text and explanations about the content, and downloadable worksheets.
Table
Table 2. Descriptions of the modules, components and contents included in the clinical protocol.
Table 2. Descriptions of the modules, components and contents included in the clinical protocol.
Module TitleModule Content
1.
Welcome to the parenting forest		1. Introduction to clinical protocol: objectives and sequence.
2. The “Forest Metaphor”: introduction to work on actions towards values.
3. The mind is a lake metaphor: emphasizes that the thoughts and emotions form part of the mind, that they are there and are part of our nature.
4. Explanation of what is mindfulness and first practice of mindfulness to breathing.
5. The garden metaphor [70] values clarification and general goals.
6. Self-evaluation of contents and homework.
2.
Emotion regulation with acceptance		1. Welcome and objectives of the module.
2. The “garden exercise”: Reminder of the contents of the last session. Establishment of specific goals and steps for actions in the direction of values.
3. The “shelter”: exercise of the wise mind [71]. Identify actions involving the emotional mind vs. the rational mind. Establish cohesion with the parenting and look for alternative reactions.
4. Self-evaluation of contents and homework.
3.
Walking through the forest		1. Welcome and objectives of the module.
2. The “garden exercise”: evaluation of the achievement of actions. Identification of barriers and difficulties and alternative actions.
3. “Body scan” mindfulness exercise. Physical sensations related to emotions.
4. “The star observatory”: a defusion exercise to strengthen perspective-taking in private events.
5. Self-evaluation of contents and homework.
4.
Allow your emotions to flow		1. Welcome and objectives of the module.
2. The “garden exercise”: evaluation of the achievement of actions. Identification or barriers. Generalization of parenting skills to goals.
3. “Full attention of sound” mindfulness exercise.
4. The “cascade of emotions”: deliteralization of private events.
5. Self-evaluation of contents and homework.
5.
Analyzing your parenting		1. Welcome and objectives of the module.
2. The “garden exercise”: self-assessment of achievements of actions. Identification or barriers.
3. Functional analysis of parenting for of both parents’ and children’s behavior. Generalization of parenting skills applied to analyzed situation.
4. Thoughts are clouds exercise: deliteralization of private events.
5. The “Connect and Shape” parenting model [72].
6. Self-evaluation of contents and homework.
6.
The other side of the forest		1. Welcome and objectives of the module.
2. The “garden exercise”: evaluation of the achievement of actions. Empowerment for future goals.
3. Review of the “Connect and Shape” parenting model [72].
4. Provide behavioral strategies for managing their children’s behavior and emotional problems: modeling, identification of reinforcers, reinforcement, control of antecedent stimuli, differential reinforcement, Premack principle, extinction, overcorrection and timeout.
5. Review skills that have been most useful and relapse prevention.
The participants received an automatic email after not accessing the platform for 15 days. Every 10 days, the researcher sent an e-mail to parents who had not accessed the platform for over a week to monitor the evolution of the intervention and to promote adherence to the treatment. Participants received an automatic email reinforcing actions toward parental values if they reported a score of seven or more on the consistency in actions toward values progress question. Participants were automatically emailed encouraging them to continue working on their goals if they reported a score of six or less.
After Unit 5, participants received a report on parental practices based on their results in the Parental Education Styles Questionnaire [73], which was completed at the end of the module. Participants could see their progress in a section called “How am I?”. In this section, they could see their pre- and post-flexible parenting scores, and their mood, coping and actions towards values progress scores. They could also access the modules again to review them after completion.

2.4. Measuring Instruments

2.4.1. Instruments to Assess Primary Parental Outcomes

The Parental Acceptance Questionnaire (6-PAQ) [74] was applied to assess parental psychological flexibility and three related behavioral styles: An open response style to the experience with acceptance and a full and flexible willingness (Open). A style that is fully aware of the present moment and has a perspective on their private events (Aware). A style committed in its actions towards values that make sense to the person (Active). The Spanish version of the scale consists of 16 items on a four-point Likert scale. A higher score indicates greater parental psychological inflexibility. The Cronbach’s alpha is 0.81 [62]
The Acceptance and Action Questionnaire-II (AAQ-II) [75,76] measures experiential avoidance in 7 items on a 7-point Likert scale that reflects the tendency to avoid or escape from thoughts, sensations or emotions that produce discomfort. A higher score indicates greater experiential avoidance. The scale has a Cronbach’s alpha of 0.88.
The Difficulties in Emotion Regulation Scale (DERS) [77,78] was used to evaluate treatment effects on emotion regulation. This scale measures a total score and 5 emotion regulation processes in 28 items on a 5-point Likert scale. The instrument assesses emotional regulation difficulties related to lack of emotional awareness, poor emotional clarity, interference in goal-oriented behaviors, lack of acceptance strategies, and limited access to regulation strategies. A higher score indicates more difficulties in emotional regulation. The total score has a Cronbach’s alpha of 0.91.
The Parenting Stress Scale (PSS) [79,80] was used to asses stress related to parenting. This scale consisting of two dimensions in 12 items rated on a 5-point Likert-type scale: baby rewards, which refers to satisfaction with the parental role, and parent stressors, which refers to the level of stress related to parenting. A higher score indicates greater parental stress. The scale has a Cronbach’s alpha of 0.77.
The Satisfaction with Life Scale (SWLS) [81,82] was used to assess general satisfaction with life. It consists of 5 items on a 5-point Likert scale. The average score of the instrument is 24.16. A higher score indicates higher life satisfaction. The Cronbach’s alpha is 0.88.

2.4.2. Measures of Parent’s Process Outcomes

At the beginning and at the end of each module mood, coping, importance of values (children as value), and consistency in valued actions was assessed. Mood was measured asking “How do you feel right now?” followed by a 5-face visual scale that has been used in previous studies [83]. Coping perception was assessed using the question “How able do you feel to address your concerns about your children right now?”. At pre-module, the importance and implication in valued direction actions towards positive parenting was assessed on a 10-point Likert scale (we have transformed this score to a 5-point scale to make it comparable to the other process outcome).

2.4.3. Instrument to Assess Children Outcomes

The Strengths and Difficulties Questionnaire (SDQ) [63,84] was used to assess treatment effects in children. This is a screening scale used to detect psychological problems in childhood and adolescence (between 4 and 17 years old). It has scales for parents, teachers, and self-reporters from the age of 11. The scale consists of a total score and 5 factors: emotional symptoms, behavioral problems, hyperactivity, problems with peers and prosocial behavior. The cutoff points for borderline and clinical range are set at the 80th and 90th percentiles, respectively (except for the prosocial behavior scale which is reverse). The total scale has a Cronbach’s alpha of 0.77 and subscales vary between 0.64 and 0.85. The parent scale was used for this study and it was assessed at pre-test and post-test.

2.4.4. Measure of Intervention Satisfaction

The Client Satisfaction Questionnaire (CSQ) [85,86] was used to assess general satisfaction with the treatment of parents. The instrument was designed to assess user satisfaction with the community and health services received. It consists of 8 items in a 4-point Likert scale. A higher score indicates greater satisfaction with the intervention. The scale has a Cronbach’s alpha of 0.90. This variable was measured in the post-test.

2.5. Statistical Analysis

The statistical R software Jamovi [87] was used for data analysis. First, a descriptive analysis of the variables evaluated pre–post intervention, and the process was carried out. Given the sample size, in order to explore the effects of the intervention, a nonparametric test was applied for two Wilcoxon T-related samples. The effect size was assessed with Cohen’s d test for nonparametric tests, considering a small effect size for scores of 0.1, medium effect for scores of 0.3, and large effect for scores of 0.5 or higher [88]. To estimate the clinical efficacy of the intervention, the reliable change index (RCI) was calculated through the Jacobson and Truax [89] method for the main variables, that is, parental flexibility, emotional regulation and stress. To calculate the cut-off points, the criterion of “c” proposed by Jacobson and Truax [89] was used. The cut-off point was determined from the post-treatment scores mean and standard deviation score of the participants (dysfunctional population) and scores of the normative population of the validation studies of the instruments (functional population). The score obtained represents the weighted midpoint between the mean of the functional and dysfunctional distributions. The participants were classified as “recovered” if their score in the post-test implies a change in the value of the RCI and if it is lower than the cut-off score. They were classified as “improved” if it represents a change in the RCI value but not in the cut-off score. It was classified as “no change” when it did not meet the criteria of the RCI value, even though there was an improvement in the mean scores. It was determined as “deteriorated” if the score represents an increase in the RCI value in a dysfunctional direction.

3. Results

3.1. Descriptive Analysis

In a descriptive analysis of participant scores, we found that the parents were above the cut-off point on all the general and parental psychological inflexibility variables. In emotion regulation skills, scores were high on emotional acceptance, interference in goals, and access to emotional regulation strategies. The DERS total score was above the cut-off point, as were the average stress scores (Table 3).
Child-related clinical scores were found for emotional symptoms, behavioral problems, problems with peers, and SDQ total difficulty score. The prosocial behavior score was borderline (Table 3).
In the post-test, all psychological flexibility scores, both general and parental, were down to below the cut-off point. The average post-test scores for emotional regulation and parenting stressors were also below the cut-off point, while the life satisfaction score increased two points (Table 3).
Child-related emotional symptoms and behavioral problem scores fell to borderline. The prosocial behavior score increased to above the cut-off point, although still borderline (Table 3).

3.2. Intervention Effects

Participants completed the intervention in a mean time of 64.2 days (SD = 20.5), with a duration of between 37 and 100 days between the start and end of the intervention. Table 3 shows the results of the outcomes for parents and children. A Wilcoxon signed rank test showed a significant decrease in parental psychological flexibility associated with a large effect size. There were also significant reductions with a large effect size associated with two of the 6-PAQ subscales—open and aware parenting response style. Due to the small sample size, we also considered marginally significant results. There were significant reductions in acceptance and rewards of parenting with moderate-to-large effect sizes. Child-related scores were significantly lower on behavioral problems and prosocial behavior, both with large effect sizes.

3.3. Significant of Clinical Change

The results for significant clinical improvements in main outcomes (a: parental psychological flexibility, b: emotion regulation, and c: parental stress) are summarized in Figure 2.
A functional change in parental psychological flexibility score was achieved by 67% of participants, of whom 58% recovered. Similar percentages were achieved in open response to experience (recovered: 50%; improved: 8%). Furthermore, 33% of the parents achieved a functional improvement in committed response style (recovered: 25%), and 25% of participants achieve a clinical improvement in centered-response style.
In emotion regulation, 50% of the participants achieved a functional improvement in emotional access to strategies (recovered: 34%; improved: 16%) and total emotion regulation scores (recovered: 41%; improved: 9%), 41% had a clinically significant recovery in acceptance skills, and 41% functional recovery in attention skills (recovered: 33%; improved: 8%), and goal-directed (recovered: 33%; improved: 8%) emotional regulation skills.
In parental stress, 58% of parents achieved a functional improvement in parenting stressors, of whom 25% recovered. Similarly, 41% of participants had a functional improvement in rewards of parenting (recovered: 25%; improved: 16%).

3.4. Process Outcomes

Figure 3 shows the average scores on mood and coping before and after each unit. It shows that the scores at the end of the session were higher on both measures. There was an upward trend from 3.16 (SD = 0.9) to 4.5 (SD = 0.52) in mood and from 3.66 (SD = 0.9) to 4.66 (SD = 0.49) in coping. In Unit 1, the greatest difference between before (M = 3.16; SD = 0.9) and after (M = 3.83; SD = 0.93) scores was in mood. In coping, the largest difference between before (M = 3.41; SD = 0.9) and after (M = 3.9; SD = 0.6) scores was in Unit 2.
In actions committed to values (specifically to parenting), there was an upward trend from 4.26 (SD = 1.16) to 4.66 (SD = 0.65).

3.5. Measure of Intervention Satisfaction

The average score for satisfaction with the program was 28, which is considered average. Seventy-five percent of the participants believed that this program was very helpful in solving their problems and would strongly recommend it to someone they know. Similarly, in a descriptive analysis, 75% of the participants reported a high level of satisfaction, that is, scores between 3 and 4 for the items of the instrument. If it was necessary to seek help again, 75% of the participants said they would repeat it.

4. Discussion

The aim of this pilot study was to evaluate the preliminary effectiveness of and satisfaction with the “Parenting Forest” program, an online informed third-wave parenting training intervention to promote parental emotion regulation and psychological flexibility. This study examined the effects of the intervention on parental psychological flexibility, emotion regulation skills, parental stress, and satisfaction with life. Changes in the psychological adjustment of their children were also assessed.
The intervention achieved an improvement in parental psychological flexibility with a large effect size. Parental open to experience practices (Open) and awareness in the present moment (Aware) in particular increased with a large effect size. These changes are consistent with findings in the face-to-face group intervention in which large effect sizes were found for parental psychological flexibility, with open, aware and active response styles [59]. Web-based ACT intervention for parents of children with chronic conditions has shown improvement in mindfulness skills (related to Aware in this study) and cognitive defusion (related to Open) [56]. ACT online intervention has explored acceptance skills as a mediator of change in chronic diseases in adults [90,91]. Psychological acceptance has also been found to be a mechanism for change in the severity of mental health problems for parents of children with ASD [92].
In this pilot web-based intervention, no statistically significant changes in action and commitment to values (Commitment) were found. However, in the analysis of clinical changes, 33% of parents achieved a functional improvement. This small effect of the intervention can be explained by the lack of difficulties in this variable of parental psychological flexibility. Indeed, scores on this dimension were very close to the cut-off point at baseline. Furthermore, the self-reported progress scores on value-directed actions were also high from baseline.
No changes in experiential avoidance measures (i.e., psychological inflexibility as measured by the AAQ-II) were found in this study. These results are similar to those of previous web-based ACT parent studies [56], which suggests the importance of specific measures for more precise measurement of change in this population. Changes in parental psychological flexibility (measured with the 6-PAQ) in this study had a large effect size.
Acceptance-based emotion regulation skills and the rewards of parenting related to parental stress improved, but with a moderate effect size and lower statistical significance. In the analysis of clinical changes, the main effects were observed in access to adaptive strategies, acceptance skills, and the total score of emotion regulation skills. These changes are consistent with the face-to-face group parenting intervention, where large effect sizes were observed in acceptance skills at post-treatment, and in access to strategies and total DERS scores at follow-up [59]. Third-wave behavioral therapies for parenting have demonstrated positive effects on emotion regulation skills with DBT [93] and mindfulness strategies [94,95].
Parental stress has been related to parental psychological flexibility [7], so it is consistent to find improvements in these two variables in this intervention, although in this web-based intervention, it was found mainly in rewards of parenting. Previous studies of parenting interventions using mindfulness strategies have improved parental stress in parents of children with ADHD [96], intellectual disability [97], and ASD [98,99]. Moreover, mindfulness skills have a mediating effect on symptoms of depression, anxiety, burnout and stress in ACT online interventions for parents [100]. Improvement in parental stress was also found with intervention in mothers with emotion dysregulation using DBT strategies applied to parenting [93].
Child-related outcomes showed significant improvements in behavioral and prosocial behavior problems with a large effect size. Behavioral problems were found in a clinical range in most of the participating parents at the beginning of the intervention (eight out of the total), which could explain why large effect sizes were reported in this sample. A review of technology-based interventions for parents was more effective for problems than for their prevention [28]. Behavior problems have been improved in previous parenting programs such as the Online Triple P program [41] or parents with bipolar disorder [101]. Interventions for mindful parenting have been reported to reduce externalizing problems in children [102,103]. DBT-informed parenting interventions in multi-family settings have shown positive effects in reducing externalizing problems in adolescents [104].
Baseline prosocial behavior scores were clinical in four participants and borderline in seven. This was also one of the most common problems in the participating parents. However, the intervention was conducted in the context of the COVID-19 pandemic, when two of the main problems found in children were emotional symptoms and prosocial behavior [105]. Therefore, it could have be an incipient concern in parents, which was highlighted in this study. Mindfulness intervention in parents of teenagers with intellectual disabilities improved their children’s prosocial skills [106].
Progress showed an upward trend from the first module in mood and parental coping skills. After the modules, the post-module scores were higher than the pre-module scores. These data are consistent with those found in the face to face “Parenting Forest” group program [59]. The online intervention for positive parenting caused a progressive increase in mood from the first sessions [39]. Even though scores of actions were high from the beginning of the intervention, an ascending trend was still observed until the end, reaching a maximum of 4.66 points (on a 5-point scale). Previous studies using an ACT approach with parents of children with autism have shown progress in value-directed actions throughout the intervention [107]. From the ACT perspective, these values are differentiated from goals and increased task performance in students [108,109]. These effects might be transferable to other populations, such as parents.
Satisfaction with the program was high: 75% of the participants were very satisfied with the intervention. These data are consistent with previous studies that show a high level of satisfaction with online parenting programs [41,43,110].
One of the main problems identified in this intervention was the difficulty parents had in completing it. In this study, about 45% (12/27) completed the intervention. Only 12% explicitly left the study due to lack of time, while the rest continued with the intervention, but took longer than they should have. These attendance rates are similar to those found in face-to-face interventions [23,67], but lower than expected for online intervention [28]. We therefore want to explore usability and accessibility issues that would facilitate adherence to the treatment. Previous studies have shown that factors such as psychological distress, early help-seeking, or parental cognitions are important for adherence [111]. Future research would analyze the profiles of participants who did not complete it in comparison with those who did.
As a pilot study, it has a series of limitations that must be considered. First, the sample size is too small and located (Spanish) to allow generalization of the results or accurately detect the effects of treatment. However, this type of design provides a better ideographic analysis of the participants, which is useful for showing the preliminary results of the intervention. A second limitation is the lack of a control group, so the effects of the intervention cannot be isolated from other variables. To lessen this limitation, the effects of the intervention were compared with the preliminary group in the face-to-face intervention which used the same protocol. Thirdly, all the measures used are self-reported, which may have biased the data provided. The selection of participants has been made taking into account both parental and child factors, which may affect the outcome of the intervention, given that levels of impairment may be different at baseline. Finally, there is no medium or long-term follow-up of the effects of the intervention.
Although findings are promising and we are encouraged to continue in this line, this study is a preliminary trial. The limitations pointed out suggest future directions of research: (a) The need to conduct a larger study with a sample size large enough for a higher confidence interval in the effect sizes. Additionally, the wide age range of the children may affect the outcomes of the intervention, so future studies with larger sample sizes should consider differentiating the sample by age of the children (i.e., children vs. adolescents). (b) Assess usability and acceptability effects of the intervention to find out what variables can improve adherence to web-based intervention with this population. In addition, given that the intervention was carried out in a self-administered format, there is little control over intervention time. Therefore, in acceptability studies, it will be useful to explore how time may affect the outcomes of the intervention. (c) Comparative analyses with a control group for other intervention variables and follow-up of the results. (d) Given the differences found in the intervention effects between prevention and treatment in online interventions for parenting [59], it is important to consider cluster analyses of clinical versus community populations to check for differential effects.

5. Conclusions

To summarize, the results of the present pilot study suggest that the web-based parenting intervention is an alternative with positive effects on the main change variables. It shows the preliminary results of an intervention for parents based on third-wave therapies for improving parental psychological flexibility and emotion regulation. This protocol has already shown positive effects in face-to-face group intervention in families, so this study provides an alternative in an online format.
Parents involved in this pilot study significantly improved their parental psychological flexibility skills with a large effect size, specifically in openness to experience and greater awareness. Furthermore, improvements in emotion regulation skills with acceptance and greater perception of rewards of parenting were also achieved. This study also found reductions in problem behavior scores and increases in prosocial behavior of their children as a result of the intervention.
These promising results encourage us to further explore the effects of the intervention in a controlled clinical trial. In future research, the acceptability and usability of the online program will be explored. Although preliminary, the results of this pilot trial show high satisfaction of the participating parents.

Author Contributions

Conceptualization, J.M.F.-C., A.G.-P. and I.G.; Data curation, J.M.F.-C.; Formal analysis, J.M.F.-C.; Funding acquisition, I.G.; Investigation, J.M.F.-C., A.G.-P. and I.G.; Methodology, J.M.F.-C., A.G.-P. and I.G.; Project administration, I.G.; Resources, A.G.-P. and I.G.; Software, A.G.-P.; Supervision, A.G.-P. and I.G.; Validation, J.M.F.-C., A.G.-P. and I.G.; Visualization, J.M.F.-C.; Writing—original draft, J.M.F.-C.; Writing—review and editing, J.M.F.-C., A.G.-P. and I.G. All authors have read and agreed to the published version of the manuscript.

Funding

This work is supported by the Spanish Ministry of Science and Innovation (Project reference/AEI/10.13039/501100011033) awarded to I.G. and by the University of Almeria, the Ministry of Economy, Knowledge, Business and University of the Junta de Andalucía and the European Regional Development Fund (FEDER) (grant UAL18-HUM-D017-B1) awarded to I.G. The present study has been carried out thanks to a pre-doctoral grant of the Internal Research Plan (Plan Propio de Investigación) of the University of Almeria awarded to J.M.F.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Ethics Committee of Andalusian Health Service’s Almeria Research Committee (protocol code PI-REFLEX-ESFA-19, approved: 2470672020).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to it is an open trial.

Acknowledgments

We would like to thank all the participating families in this study. We would like to thank Diana Castilla for her support in the development of the online adaptation of this parenting program.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Kuppens, S.; Ceulemans, E. Parenting Styles: A Closer Look at a Well-Known Concept. J. Child Fam. Stud. 2019, 28, 168–181. [Google Scholar] [CrossRef] [Green Version]
  2. Rekart, K.N.; Mineka, S.; Zinbarg, R.E.; Griffith, J.W. Perceived Family Environment and Symptoms of Emotional Disorders: The Role of Perceived Control, Attributional Style, and Attachment. Cogn Ther Res 2007, 31, 419–436. [Google Scholar] [CrossRef]
  3. Martínez, I.; Murgui, S.; Garcia, O.F.; Garcia, F. Parenting in the Digital Era: Protective and Risk Parenting Styles for Traditional Bullying and Cyberbullying Victimization. Comput. Hum. Behavior 2019, 90, 84–92. [Google Scholar] [CrossRef]
  4. Zarra-Nezhad, M.; Viljaranta, J.; Sajaniemi, N.; Aunola, K.; Lerkkanen, M.-K. The Impact of Children’s Socioemotional Development on Parenting Styles: The Moderating Effect of Social Withdrawal. Early Child Dev. Care 2020, 1–13. [Google Scholar] [CrossRef]
  5. Gorostiaga, A.; Aliri, J.; Balluerka, N.; Lameirinhas, J. Parenting Styles and Internalizing Symptoms in Adolescence: A Systematic Literature Review. Int. J. Envion. Res. Public Health 2019, 16, 3192. [Google Scholar] [CrossRef] [Green Version]
  6. Mak, M.C.K.; Yin, L.; Li, M.; Cheung, R.Y.; Oon, P.-T. The Relation between Parenting Stress and Child Behavior Problems: Negative Parenting Styles as Mediator. J. Child Fam. Stud. 2020, 29, 2993–3003. [Google Scholar] [CrossRef]
  7. Fonseca, A.; Moreira, H.; Canavarro, M.C. Uncovering the Links between Parenting Stress and Parenting Styles: The Role of Psychological Flexibility within Parenting and Global Psychological Flexibility. J. Contextual Behav. Sci. 2020, 18, 59–67. [Google Scholar] [CrossRef]
  8. Burke, K.; Moore, S. Development of the Parental Psychological Flexibility Questionnaire. Child Psychiatry Hum. Dev. 2015, 46, 548–557. [Google Scholar] [CrossRef] [PubMed]
  9. Emerson, L.-M.; Ogielda, C.; Rowse, G. The Role of Experiential Avoidance and Parental Control in the Association Between Parent and Child Anxiety. Front. Psychol. 2019, 10, 262. [Google Scholar] [CrossRef] [PubMed]
  10. Moyer, D.N.; Sandoz, E.K. The Role of Psychological Flexibility in the Relationship Between Parent and Adolescent Distress. J. Child Fam. Stud. 2015, 24, 1406–1418. [Google Scholar] [CrossRef]
  11. Biglan, A.; Gau, J.M.; Jones, L.B.; Hinds, E.; Rusby, J.C.; Cody, C.; Sprague, J. The Role of Experiential Avoidance in the Relationship between Family Conflict and Depression among Early Adolescents. J. Contextual Behav. Sci. 2015, 4, 30–36. [Google Scholar] [CrossRef]
  12. Murrell, A.R.; Wilson, K.G.; LaBorde, C.T.; Drake, C.E.; Rogers, L.J. Relational Responding in Parents. Behav. Anal. Today 2008, 9, 196–214. [Google Scholar] [CrossRef] [Green Version]
  13. Coyne, L.W.; Wilson, K.G. The Role of Cognitive Fusion in Impaired Parenting: An RFT Analysis. Int. J. Psychol. Psychol. Ther. 2004, 4, 469–486. [Google Scholar]
  14. Brown, F.L.; Whittingham, K.; Sofronoff, K. Parental Experiential Avoidance as a Potential Mechanism of Change in a Parenting Intervention for Parents of Children With Pediatric Acquired Brain Injury. J. Pediatric Psychol. 2015, 40, 464–474. [Google Scholar] [CrossRef] [PubMed]
  15. Whittingham, K. Parenting in Context. J. Contextual Behav. Sci. 2014, 3, 212–215. [Google Scholar] [CrossRef]
  16. Paulus, D.J.; Vanwoerden, S.; Norton, P.J.; Sharp, C. Emotion Dysregulation, Psychological Inflexibility, and Shame as Explanatory Factors between Neuroticism and Depression. J. Affect. Disord. 2016, 190, 376–385. [Google Scholar] [CrossRef] [PubMed]
  17. Aldao, A.; Nolen-Hoeksema, S.; Schweizer, S. Emotion-Regulation Strategies across Psychopathology: A Meta-Analytic Review. Clin. Psychol. Rev. 2010, 30, 217–237. [Google Scholar] [CrossRef]
  18. Duncan, L.G.; Coatsworth, J.D.; Greenberg, M.T. A Model of Mindful Parenting: Implications for Parent-Child Relationships and Prevention Research. Clin. Child Fam. Psychol. Rev. 2009, 12, 255–270. [Google Scholar] [CrossRef] [Green Version]
  19. Eisenberg, N.; Losoya, S.; Fabes, R.A.; Guthrie, I.K.; Reiser, M.; Murphy, B.; Shepard, S.A.; Poulin, R.; Padgett, S.J. Parental Socialization of Children’s Dysregulated Expression of Emotion and Externalizing Problems. J. Fam. Psychol. 2001, 15, 183–205. [Google Scholar] [CrossRef] [PubMed]
  20. Moreira, H.; Canavarro, M.C. Mindful Parenting Is Associated with Adolescents’ Difficulties in Emotion Regulation Through Adolescents’ Psychological Inflexibility and Self-Compassion. J. Youth Adolesc. 2020, 49, 192–211. [Google Scholar] [CrossRef] [PubMed]
  21. Makriyianis, H.M.; Adams, E.A.; Lozano, L.L.; Mooney, T.A.; Morton, C.; Liss, M. Psychological Inflexibility Mediates the Relationship between Adverse Childhood Experiences and Mental Health Outcomes. J. Contextual Behav. Sci. 2019, 14, 82–89. [Google Scholar] [CrossRef]
  22. Hayes, S.C.; Villatte, M.; Levin, M.; Hildebrandt, M. Open, Aware, and Active: Contextual Approaches as an Emerging Trend in the Behavioral and Cognitive Therapies. Annu. Rev. Clin. Psychol. 2011, 7, 141–168. [Google Scholar] [CrossRef] [Green Version]
  23. Lozano-Rodríguez, I.; Valero-Aguayo, L. Una Revisión Sistemática de La Eficacia de Los Programas de Entrenamiento a Padres. Rev. Psicol Clínica Niños Adolesc. 2017, 4, 85–91. [Google Scholar]
  24. Robles, Z.; Romero, E. Programas de Entrenamiento Para Padres de Niños Con Problemas de Conducta: Una Revisión de Su Eficacia. An. Psicol. 2011, 27, 86–101. [Google Scholar]
  25. Heinrichs, N.; Bertram, H.; Kuschel, A.; Hahlweg, K. Parent Recruitment and Retention in a Universal Prevention Program for Child Behavior and Emotional Problems: Barriers to Research and Program Participation. Prev. Sci. 2005, 6, 275–286. [Google Scholar] [CrossRef] [PubMed]
  26. Nix, R.L.; Bierman, K.L.; McMahon, R.J. How Attendance and Quality of Participation Affect Treatment Response to Parent Management Training. J. Consult. Clin. Psychol. 2009, 77, 429–438. [Google Scholar] [CrossRef] [PubMed]
  27. Prinz, R.J.; Sanders, M.R. Adopting a Population-Level Approach to Parenting and Family Support Interventions. Clin. Psychol. Rev. 2007, 27, 739–749. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Flujas-Contreras, J.M.; García-Palacios, A.; Gómez, I. Technology-Based Parenting Interventions for Children’s Physical and Psychological Health: A Systematic Review and Meta-Analysis. Psychol. Med. 2019, 49, 1787–1798. [Google Scholar] [CrossRef]
  29. Kazdin, A.E. Evidence-Based Treatment Research: Advances, Limitations, and next Steps. Am. Psychol. 2011, 66, 685. [Google Scholar] [CrossRef] [Green Version]
  30. Mohr, D.C.; Burns, M.N.; Schueller, S.M.; Clarke, G.; Klinkman, M. Behavioral Intervention Technologies: Evidence Review and Recommendations for Future Research in Mental Health. Gen. Hosp. Psychiatry 2013, 35, 332–338. [Google Scholar] [CrossRef] [Green Version]
  31. Vilardaga, R.; Bricker, J.B.; McDonell, M.G. The Promise of Mobile Technologies and Single Case Designs for the Study of Individuals in Their Natural Environment. J. Contextual Behav. 2014, 3, 148–153. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  32. Holmes, E.A.; Ghaderi, A.; Harmer, C.J.; Ramchandani, P.G.; Cuijpers, P.; Morrison, A.P.; Roiser, J.P.; Bockting, C.L.H.; O’Connor, R.C.; Shafran, R.; et al. The Lancet Psychiatry Commission on Psychological Treatments Research in Tomorrow’s Science. Lancet Psychiatry 2018, 5, 237–286. [Google Scholar] [CrossRef] [Green Version]
  33. Funderburk, B.W.; Ware, L.M.; Altshuler, E.; Chaffin, M. Use and Feasibility of Telemedicine Technology in the Dissemination of Parent-Child Interaction Therapy. Child Maltreatment 2008, 13, 377–382. [Google Scholar] [CrossRef]
  34. Danebakc, C.; Platin, L. Research on Parenthood and the Internet: Themes and Trends. Cyberpsychology 2008, 2, 2. [Google Scholar]
  35. Nieuwboer, C.C.; Fukkink, R.G.; Hermanns, J.M.A. Online Programs as Tools to Improve Parenting: A Meta-Analytic Review. Child. Youth Serv. Rev. 2013, 35, 1823–1829. [Google Scholar] [CrossRef] [Green Version]
  36. Corralejo, S.M.; Domenech Rodríguez, M.M. Technology in Parenting Programs: A Systematic Review of Existing Interventions. J. Child Fam. Stud. 2018, 27, 2717–2731. [Google Scholar] [CrossRef]
  37. Breitenstein, S.M.; Gross, D.; Christophersen, R. Digital Delivery Methods of Parenting Training Interventions: A Systematic Review: Digital Delivery of Parent Training. Worldviews Evid. Based Nurs. 2014, 11, 168–176. [Google Scholar] [CrossRef]
  38. Breitenstein, S.M.; Fehrenbacher, C.; Holod, A.F.; Schoeny, M. A Randomized Trial of Digitally Delivered, Self-Administered Parent Training in Primary Care: Effects on Parenting and Child Behavior. J. Pediatrics 2020. [Google Scholar] [CrossRef] [PubMed]
  39. Suárez, A.; Byrne, S.; Rodrigo, M.J. Effectiveness of a Universal Web-Based Parenting Program to Promote Positive Parenting: Patterns and Predictors on Program Satisfaction. J. Child Fam. Stud. 2018, 27, 3345–3357. [Google Scholar] [CrossRef]
  40. Baumel, A.; Faber, K. Evaluating Triple P Online: A Digital Parent Training Program for Child Behavior Problems. Cogn. Behav. Pract. 2018, 25, 538–543. [Google Scholar] [CrossRef]
  41. Baker, S.; Sanders, M.R.; Turner, K.M.T.; Morawska, A. A Randomized Controlled Trial Evaluating a Low-Intensity Interactive Online Parenting Intervention, Triple P Online Brief, with Parents of Children with Early Onset Conduct Problems. Behav. Res. Ther. 2017, 91, 78–90. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Sim, W.H.; Fernando, L.M.N.; Jorm, A.F.; Rapee, R.M.; Lawrence, K.A.; Mackinnon, A.J.; Yap, M.B.H. A Tailored Online Intervention to Improve Parenting Risk and Protective Factors for Child Anxiety and Depression: Medium-Term Findings from a Randomized Controlled Trial. J. Affect. Disord. 2020, 277, 814–824. [Google Scholar] [CrossRef]
  43. Ibañez, L.V.; Kobak, K.; Swanson, A.; Wallace, L.; Warren, Z.; Stone, W.L. Enhancing Interactions during Daily Routines: A Randomized Controlled Trial of a Web-Based Tutorial for Parents of Young Children with ASD: A Parenting Tutorial for Daily Routines. Autism Res. 2018, 11, 667–678. [Google Scholar] [CrossRef]
  44. Phipps, S.; Fairclough, D.L.; Noll, R.B.; Devine, K.A.; Dolgin, M.J.; Schepers, S.A.; Askins, M.A.; Schneider, N.M.; Ingman, K.; Voll, M.; et al. In-Person vs. Web-Based Administration of a Problem-Solving Skills Intervention for Parents of Children with Cancer: Report of a Randomized Noninferiority Trial. EClinicalMedicine 2020, 24, 100428. [Google Scholar] [CrossRef]
  45. Hayes, S.C.; Strosahl, K.; Wilson, K.G. Acceptance and Commitment Therapy The Process and Practice of Mindful Change; Guilford Press: New York, NY, USA, 2012; ISBN 978-1-60918-962-4. [Google Scholar]
  46. Linehan, M. Cognitive-Behavioral Treatment of Borderline Personality Disorder; Guilford Press: New York, NY, USA, 1993; ISBN 978-0-89862-183-9. [Google Scholar]
  47. Segal, Z.V.; Williams, J.M.G.; Teasdale, J.D. Mindfulness-Based Cognitive Therapy for Depression: A New Approach to Preventing Relapse; Guilford Press: New York, NY, USA, 2002; ISBN 1-57230-706-4. [Google Scholar]
  48. Byrne, G.; Ghráda, Á.N.; O’Mahony, T.; Brennan, E. A Systematic Review of the Use of Acceptance and Commitment Therapy in Supporting Parents. Psychol. Psychother. Theory Res. Pract. 2020, 13, e12282. [Google Scholar] [CrossRef] [PubMed]
  49. Bögels, S.M.; Hellemans, J.; van Deursen, S.; Römer, M.; van der Meulen, R. Mindful Parenting in Mental Health Care. Mindfulness 2014, 5, 536–551. [Google Scholar] [CrossRef]
  50. Rayan, A.; Ahmad, M. Mindfulness and Parenting Distress among Parents of Children with Disabilities: A Literature Review. Perspect. Psychiatr. Care 2018, 54, 324–330. [Google Scholar] [CrossRef]
  51. Van der Oord, S.; Bögels, S.M.; Peijnenburg, D. The Effectiveness of Mindfulness Training for Children with ADHD and Mindful Parenting for Their Parents. J. Child Fam. Stud. 2012, 21, 139–147. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  52. Harnett, P.H.; Dawe, S. The Contribution of Mindfulness-Based Therapies for Children and Families and Proposed Conceptual Integration. Child Adolesc. Ment. Health 2012, 17, 195–208. [Google Scholar] [CrossRef] [PubMed]
  53. Zalewski, M.; Maliken, A.C.; Lengua, L.J.; Gamache Martin, C.; Roos, L.E.; Everett, Y. Integrating Dialectical Behavior Therapy with Child and Parent Training Interventions: A Narrative and Theoretical Review. Clin. Psychol. Sci. Pract. 2020, e12363. [Google Scholar] [CrossRef]
  54. MacPherson, H.A.; Cheavens, J.S.; Fristad, M.A. Dialectical Behavior Therapy for Adolescents: Theory, Treatment Adaptations, and Empirical Outcomes. Clin. Child Fam. Psychol. Rev. 2013, 16, 59–80. [Google Scholar] [CrossRef] [PubMed]
  55. Perepletchikova, F.; Nathanson, D.; Axelrod, S.R.; Merrill, C.; Walker, A.; Grossman, M.; Rebeta, J.; Scahill, L.; Kaufman, J.; Flye, B.; et al. Randomized Clinical Trial of Dialectical Behavior Therapy for Preadolescent Children With Disruptive Mood Dysregulation Disorder: Feasibility and Outcomes. J. Am. Acad. Child Adolesc. Psychiatry 2017, 56, 832–840. [Google Scholar] [CrossRef] [PubMed]
  56. Sairanen, E.; Lappalainen, R.; Lappalainen, P.; Kaipainen, K.; Carlstedt, F.; Anclair, M.; Hiltunen, A. Effectiveness of a Web-Based Acceptance and Commitment Therapy Intervention for Wellbeing of Parents Whose Children Have Chronic Conditions: A Randomized Controlled Trial. J. Contextual Behav. Sci. 2019, 13, 94–102. [Google Scholar] [CrossRef]
  57. Pennefather, J.; Hieneman, M.; Raulston, T.J.; Caraway, N. Evaluation of an Online Training Program to Improve Family Routines, Parental Well-Being, and the Behavior of Children with Autism. Res. Autism Spectr. Disord. 2018, 54, 21–26. [Google Scholar] [CrossRef]
  58. Yang, M.; Jia, G.; Sun, S.; Ye, C.; Zhang, R.; Yu, X. Effects of an Online Mindfulness Intervention Focusing on Attention Monitoring and Acceptance in Pregnant Women: A Randomized Controlled Trial. J. Midwifery Women’s Health 2019, 64, 68–77. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  59. Flujas-Contreras, J.M.; García-Palacios, A.; Gómez, I. Intervención en flexibilidad parental a través de una intervención grupal en familias. Behav. Psychol. 2020, 28, 25–57. [Google Scholar]
  60. Eldridge, S.M.; Chan, C.L.; Campbell, M.J.; Bond, C.M.; Hopewell, S.; Thabane, L.; Lancaster, G.A. CONSORT 2010 Statement: Extension to Randomised Pilot and Feasibility Trials. BMJ 2016, 355, i5239. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  61. Lancaster, G.A.; Thabane, L. Guidelines for Reporting Non-Randomised Pilot and Feasibility Studies. Pilot Feasibility Stud. 2019, 5, 114. [Google Scholar] [CrossRef]
  62. Flujas-Contreras, J.M.; García-Palacios, A.; Gómez, I. Spanish Validation of the Parental Acceptance Questionnaire (6-PAQ). Int. J. Clin. Health Psychol. 2020, 20, 163–172. [Google Scholar] [CrossRef]
  63. Español-Martín, G.; Pagerols, M.; Prat, R.; Rivas, C.; Sixto, L.; Valero, S.; Artigas, M.S.; Ribasés, M.; Ramos-Quiroga, J.A.; Casas, M.; et al. Strengths and Difficulties Questionnaire: Psychometric Properties and Normative Data for Spanish 5- to 17-Year-Olds. Assessment 2020. [Google Scholar] [CrossRef]
  64. Whitehead, A.L.; Julious, S.A.; Cooper, C.L.; Campbell, M.J. Estimating the Sample Size for a Pilot Randomised Trial to Minimise the Overall Trial Sample Size for the External Pilot and Main Trial for a Continuous Outcome Variable. Stat. Methods Med. Res. 2016, 25, 1057–1073. [Google Scholar] [CrossRef] [PubMed]
  65. Julious, S.A. Sample Size of 12 per Group Rule of Thumb for a Pilot Study. Pharm. Stat. 2005, 4, 287–291. [Google Scholar] [CrossRef]
  66. LabPsiTec Psicología y Tecnología. Available online: https://psicologiaytecnologia.labpsitec.es/ (accessed on 24 December 2020).
  67. González-Robles, A.; Díaz-García, A.; García-Palacios, A.; Roca, P.; Ramos-Quiroga, J.A.; Botella, C. Effectiveness of a Transdiagnostic Guided Internet-Delivered Protocol for Emotional Disorders Versus Treatment as Usual in Specialized Care: Randomized Controlled Trial. J. Med. Internet Res. 2020, 22, e18220. [Google Scholar] [CrossRef] [PubMed]
  68. Ben-Porath, D.D. Dialectical Behavior Therapy Applied to Parent Skills Training: Adjunctive Treatment for Parents with Difficulties in Affect Regulation. Cog. Behav. Pract. 2010, 17, 458–465. [Google Scholar] [CrossRef]
  69. Gómez, I.; Hódar, J.C.; Barranco, M.R.G. Intervención en valores con familias de riesgo social desde la Terapia de Aceptación y Compromiso. Análisis Modif. Conducta 2012, 38, 20. [Google Scholar] [CrossRef] [Green Version]
  70. Hayes, S.C.; Strosahl, K.D.; Wilson, K.G. Acceptance and Commitment Therapy: An Experiential Approach to Behavior Change; Guilford Press: New York, NY, USA, 1999. [Google Scholar]
  71. Linehan, M. DBT Skills Training Manual; Guilford Press: New York, NY, USA, 2014; ISBN 978-1-4625-1699-5. [Google Scholar]
  72. Whittingham, K. Connect and Shape: A Parenting Meta-Strategy. J. Contextual Behav. Sci. 2015, 4, 103–106. [Google Scholar] [CrossRef]
  73. Gómez, I.; de Salazar, A.M.; Barranco, R.G.; Cobos, L.; Filippo, C. Validación Del Cuestionario de Estilos Educativos Parentales: Estudio Piloto. In Proceedings of the II Symposium Nacional de Psicología Clínica y de la Salud con Niños y Adolescentes, Elche, Spain, 21 November 2012. [Google Scholar]
  74. Greene, R.L.; Field, C.E.; Fargo, J.D.; Twohig, M.P. Development and Validation of the Parental Acceptance Questionnaire (6-PAQ). J. Contextual Behav. Sci. 2015, 4, 170–175. [Google Scholar] [CrossRef]
  75. Bond, F.W.; Hayes, S.C.; Baer, R.A.; Carpenter, K.M. Preliminary Psychometric Properties of the Acceptance and Action Questionnaire–II: A Revised Measure of Psychological Inflexibility and Experiential Avoidance. Behav. Ther. 2001, 42, 676–688. [Google Scholar] [CrossRef] [Green Version]
  76. Ruiz, F.J.; Herrera, A.I.L.; Luciano, C.; Cangas, A.J.; Beltran, I. Measuring Experiential Avoidance and Psychological Inflexibility: The Spanish Version of the Acceptance and Action Questionnaire—II. Psicothema 2013, 25, 123–129. [Google Scholar] [CrossRef] [PubMed]
  77. Gratz, K.L.; Roemer, L. Multidimensional Assessment of Emotion Regulation and Dysregulation: Development, Factor Structure, and Initial Validation of the Difficulties in Emotion Regulation Scale. J. Psychopathol. Beh. Assess. 2004, 26, 41–54. [Google Scholar] [CrossRef]
  78. Hervás, G.; Jódar, R. Adaptación al Castellano de La Escala de Dificultades En La Regulación Emocional The Spanish Version of the Difficulties in Emotion Regulation Scale. Clínica Salud 2008, 19, 139–156. [Google Scholar] [CrossRef] [Green Version]
  79. Berry, J.O.; Jones, W.H. The Parental Stress Scale: Initial Psychometric Evidence. J. Soc. Pers. Relatsh. 1995, 12, 463–472. [Google Scholar] [CrossRef]
  80. Oronoz, B.; Alonso-Arbiol, I.; Balluerka, N. A Spanish Adaptation of the Parental Stress Scale. Psicothema 2007, 19, 687–692. [Google Scholar] [PubMed]
  81. Diener, E.; Emmons, R.A.; Larsem, R.J.; Griffin, S. The Satisfaction With Life Scale. J. Pers. Assess. 1985, 49, 71–75. [Google Scholar] [CrossRef] [PubMed]
  82. Vázquez, C.; Duque, A.; Hervás, G. Satisfaction with Life Scale in a Representative Sample of Spanish Adults: Validation and Normative Data. Span. J. Psychol. 2013, 16, 1–15. [Google Scholar] [CrossRef] [PubMed]
  83. Garcia-Palacios, A.; Herrero, R.; Belmonte, M.A.; Castilla, D.; Guixeres, J.; Molinari, G.; Baños, R.M.; Botella, C. Ecological Momentary Assessment for Chronic Pain in Fibromyalgia Using a Smartphone: A Randomized Crossover Study. Eur. J. Pain 2014, 18, 862–872. [Google Scholar] [CrossRef] [Green Version]
  84. Goodman, R. Psychometric Properties of the Strengths and Difficulties Questionnaire. J. Am. Acad. Child Adolesc. Psychiatry 2001, 40, 1337–1345. [Google Scholar] [CrossRef] [PubMed]
  85. Larsen, D.L.; Attkisson, C.C.; Hargreaves, W.A.; Nguyen, T.D. Assessment of Client/Patient Satisfaction: Development of a General Scale. Eval. Program Plan. 1979, 2, 197–207. [Google Scholar] [CrossRef]
  86. Roberts, R.E.; Attkisson, C.C. Assessing Client Satisfaction among Hispanics. Eval. Program Plan. 1983, 6, 401–413. [Google Scholar] [CrossRef]
  87. Jamovi. Available online: https://www.jamovi.org/ (accessed on 26 December 2020).
  88. Cohen, J. Statistical Power Analysis for the Behavioral Sciences; Academic Press: New York, NY, USA, 1988. [Google Scholar]
  89. Jacobson, N.S.; Truax, P. Clinical Significance: A Statistical Approach to Defining Meaningful Change in Psychotherapy Research. J. Consult. Clin. Psychol. Psychology 1991, 59, 12–19. [Google Scholar] [CrossRef]
  90. Cederberg, J.T.; Cernvall, M.; Dahl, J.; von Essen, L.; Ljungman, G. Acceptance as a Mediator for Change in Acceptance and Commitment Therapy for Persons with Chronic Pain? Int. J. Behav. Med. 2016, 23, 21–29. [Google Scholar] [CrossRef] [PubMed]
  91. Hesser, H.; Westin, V.Z.; Andersson, G. Acceptance as a Mediator in Internet-Delivered Acceptance and Commitment Therapy and Cognitive Behavior Therapy for Tinnitus. J. Behav. Med. 2014, 37, 756–767. [Google Scholar] [CrossRef]
  92. Weiss, J.A.; Cappadocia, M.C.; MacMullin, J.A.; Viecili, M.; Lunsky, Y. The Impact of Child Problem Behaviors of Children with ASD on Parent Mental Health: The Mediating Role of Acceptance and Empowerment. Autism 2012, 16, 261–274. [Google Scholar] [CrossRef] [Green Version]
  93. Martin, C.G.; Roos, L.E.; Zalewski, M.; Cummins, N. A Dialectical Behavior Therapy Skills Group Case Study on Mothers With Severe Emotion Dysregulation. Cogn. Behav. Pract. 2017, 24, 405–415. [Google Scholar] [CrossRef]
  94. Gershy, N.; Meehan, K.B.; Omer, H.; Papouchis, N.; Sapir, I.S. Randomized Clinical Trial of Mindfulness Skills Augmentation in Parent Training. Child Youth Care Forum 2017, 46, 783–803. [Google Scholar] [CrossRef]
  95. Haydicky, J.; Wiener, J.; Shecter, C. Mechanisms of Action in Concurrent Parent-Child Mindfulness Training: A Qualitative Exploration. Mindfulness 2017, 8, 1018–1035. [Google Scholar] [CrossRef]
  96. Anderson, S.; Guthery, A. Mindfulness-Based Psychoeducation for Parents of Children with Attention-Deficit/Hyperactivity Disorder: An Applied Clinical Project. J. Child Adolesc. Psychiatr. Nurs. 2015, 28, 43–49. [Google Scholar] [CrossRef] [PubMed]
  97. Bazzano, A.; Wolfe, C.; Zylowska, L.; Wang, S.; Schuster, E.; Barrett, C.; Lehrer, D. Mindfulness Based Stress Reduction (MBSR) for Parents and Caregivers of Individuals with Developmental Disabilities: A Community-Based Approach. J. Child Fam. Stud. 2015, 24, 298–308. [Google Scholar] [CrossRef]
  98. Dykens, E.M.; Fisher, M.H.; Taylor, J.L.; Lambert, W.; Miodrag, N. Reducing Distress in Mothers of Children With Autism and Other Disabilities: A Randomized Trial. Pediatrics 2014, 134, e454–e463. [Google Scholar] [CrossRef] [Green Version]
  99. Hwang, Y.S.; Kearney, P.; Klieve, H.; Lang, W.; Roberts, J. Cultivating Mind: Mindfulness Interventions for Children with Autism Spectrum Disorder and Problem Behaviours, and Their Mothers. J. Child Fam. Stud. 2015, 24, 3093–3106. [Google Scholar] [CrossRef]
  100. Sairanen, E.; Lappalainen, R.; Lappalainen, P.; Hiltunen, A. Mediators of Change in Online Acceptance and Commitment Therapy for Psychological Symptoms of Parents of Children with Chronic Conditions: An Investigation of Change Processes. J. Contextual Behav. Sci. 2020, 15, 123–130. [Google Scholar] [CrossRef]
  101. Jones, S.H.; Jovanoska, J.; Calam, R.; Wainwright, L.D.; Vincent, H.; Asar, O.; Diggle, P.J.; Parker, R.; Long, R.; Sanders, M.; et al. Web-Based Integrated Bipolar Parenting Intervention for Parents with Bipolar Disorder: A Randomised Controlled Pilot Trial. J. Child Psychol. Psychiatr. 2017, 58, 1033–1041. [Google Scholar] [CrossRef]
  102. Bögels, S.; Hoogstad, B.; van Dun, L.; de Schutter, S.; Restifo, K. Mindfulness Training for Adolescents with Externalizing Disorders and Their Parents. Behav. Cogn. Psychother. 2008, 36, 193–209. [Google Scholar] [CrossRef] [Green Version]
  103. Coatsworth, J.D.; Duncan, L.G.; Greenberg, M.T.; Nix, R.L. Changing Parent’s Mindfulness, Child Management Skills and Relationship Quality with Their Youth: Results from a Randomized Pilot Intervention Trial. J. Child Fam. Stud. 2010, 19, 203–217. [Google Scholar] [CrossRef] [Green Version]
  104. Uliaszek, A.A.; Wilson, S.; Mayberry, M.; Cox, K.; Maslar, M. A Pilot Intervention of Multifamily Dialectical Behavior Group Therapy in a Treatment-Seeking Adolescent Population: Effects on Teens and Their Family Members. Fam. J. 2014, 22, 206–215. [Google Scholar] [CrossRef]
  105. Gómez-Becerra, I.; Flujas, J.M.; Andrés, M.; Sánchez-López, P. Evolución del estado psicológico y el miedo en la infancia y adolescencia durante el confinamiento por la COVID-19. Rev. Psicol. Clínica Niños Adolesc. 2020, 7, 11–18. [Google Scholar]
  106. Heifetz, M.; Dyson, A. Mindfulness-Based Group for Teens with Developmental Disabilities and Their Parents: A Pilot Study. Mindfulness 2017, 8, 444–453. [Google Scholar] [CrossRef]
  107. Gould, E.R.; Tarbox, J.; Coyne, L. Evaluating the Effects of Acceptance and Commitment Training on the Overt Behavior of Parents of Children with Autism. J. Contextual Behav. Sci. 2018, 7, 81–88. [Google Scholar] [CrossRef]
  108. Chase, J.A.; Houmanfar, R.; Hayes, S.C.; Ward, T.A.; Vilardaga, J.P.; Follette, V. Values Are Not Just Goals: Online ACT-Based Values Training Adds to Goal Setting in Improving Undergraduate College Student Performance. J. Contextual Behav. Sci. 2013, 2, 79–84. [Google Scholar] [CrossRef]
  109. Firestone, J.; Cardaciotto, L.; Levin, M.E.; Goldbacher, E.; Vernig, P.; Gambrel, L.E. A Web-Based Self-Guided Program to Promote Valued-Living in College Students: A Pilot Study. J. Contextual Behav. Sci. 2019, 12, 29–38. [Google Scholar] [CrossRef]
  110. Sveen, J.; Andersson, G.; Ekselius, L.; Sjöberg, F.; Buhrman, B.; Willebrand, M. Internet-Based Information and Self-Help Program for Parents of Children with Burns: Study Protocol for a Randomized Controlled Trial. Internet Interv. 2015, 2, 367–371. [Google Scholar] [CrossRef] [Green Version]
  111. Gonzalez, C.; Morawska, A.; Haslam, D.M. A Model of Intention to Participate in Parenting Interventions: The Role of Parent Cognitions and Behaviours. Behav. Ther. 2020. [Google Scholar] [CrossRef]
Ijerph 18 02958 g001 550
Figure 1. Flowchart of participants.
Figure 1. Flowchart of participants.
Ijerph 18 02958 g001
Ijerph 18 02958 g002 550
Figure 2. Percentages of participants recovered, improved, did not change, and deteriorated on (a) parental psychological flexibility; (b) difficulties to emotional regulation; and (c) parental stress.
Figure 2. Percentages of participants recovered, improved, did not change, and deteriorated on (a) parental psychological flexibility; (b) difficulties to emotional regulation; and (c) parental stress.
Ijerph 18 02958 g002
Ijerph 18 02958 g003 550
Figure 3. Process outcomes of pre-module and post-module scores of (a) mood and (b) coping.
Figure 3. Process outcomes of pre-module and post-module scores of (a) mood and (b) coping.
Ijerph 18 02958 g003
Table
Table 1. Participants sociodemographic and psychological characteristics.
Table 1. Participants sociodemographic and psychological characteristics.
ParentChildren6-PAQ Parents Characteristics at Pre-TestSDQ Children Characteristics at Pre-Test
142 (f)10 (f)
7 (f)		↑ parental psychological inflexibility; ↓ acceptance; ↓ commitment ↑ SDQ score; ± emotional, ↑behavioral, ± hyperactivity, ↑peer, ↓ prosocial
237 (f)8 (f)↑ parental psychological inflexibility; ↓ acceptance; ↓commitment↑ SDQ score; ↑ behavioral, ±hyperactivity, ↑ peer, ↓ prosocial
343 (f)10 (m)
7 (f)		↑ parental psychological inflexibility; ↓ acceptance; ↓awareness↑ SDQ score; ↑ emotional, ↑behavioral, ↑ hyperactivity, ↑peer, ↓ prosocial
437 (f)3 (m)± parental psychological inflexibility; ↓ acceptance; ↓awareness↑ SDQ score; ± emotional, ↑behavioral, ± hyperactivity, ↑peer, ± prosocial
542 (f)9 (m)
6 (m)		↑ parental psychological inflexibility; ↓ acceptance; ↓commitment↑ SDQ score; ± emotional, ±peer./ADHD diagnosis < 6 months
636 (f)9 (f)
13 (m)		↑ parental psychological inflexibility; ↓ acceptance; ↓awareness↑ SDQ score; ↑ emotional, ↑behavioral, ↑ hyperactivity ADHD diagnosis > 1 year
740 (f)7 (m)
3 (m)		-↑ SDQ score; ± behavioral, ↑ peer
845 (m)10 (m)
7 (m)		↑ parental psychological inflexibility↑ SDQ score; ↑ emotional, ↑behavioral, ± prosocial
943 (m)8 (m)
5 (m)/5 (f)		-↑ SDQ score; ↑ emotional, ↑behavioral, ↑ peer
1050 (f)15 (m)
13 (m)		-↑ SDQ score; ↑ emotional
1142 (f)10 (f)-↑ SDQ score; ↑ emotional, ±behavioral, ± peer, ± prosocial
1236 (f)8 (m)
4 (f)		↑ parental psychological inflexibility; ↓ awareness↑ SDQ score; ↑ emotional, ↑behavioral, ± hyperactivity, ↓prosocial
Notes: (f): female; (m): male; ↑: high score; ↓: low score; ±: borderline score; <6 months: less than 6 months of diagnosis; >1 year: more than 1 year of diagnosis. The 6-PAQ range scores were calculated using the reliable change index criterion “c” described in Section 2.5. The SDQ range scores is obtained from the instrument’s validation study scales.
Table
Table 3. Means (M), standard deviations (SD), Wilcoxon test and effect sizes (Cohen’s d) for outcomes of parents and children in the total score and factors of variables.
Table 3. Means (M), standard deviations (SD), Wilcoxon test and effect sizes (Cohen’s d) for outcomes of parents and children in the total score and factors of variables.
Pre-Treatment
(n = 12)		Post-Treatment
(n = 12)		Wilcoxon ZpCohen’s d
MSDMSD
Primary parental outcomes
6-PAQ32.506.6926.925.3278.00.002 *1.43
Open 11.173.138.671.9250.50.021 *0.91
Aware12.423.0010.002.7345.00.009 **1.39
Commitment8.921.838.252.6733.50.2080.29
AAQ-II23.177.8719.338.7649.50.1540.49
DERS66.0024.7357.3318.3353.50.2720.38
Attention8.583.928.582.5024.00.9050.00
Clarity7.502.247.422.4738.00.6840.03
Acceptance18.588.4614.256.5962.00.076 †0.57
Goals11.174.329.924.6845.00.3040.29
Strategies20.179.0117.176.1636.00.4140.33
PSS32.757.2130.678.0750.00.1410.27
Rewards8.923.407.832.8244.50.089 †0.56
Stressors23.835.5722.835.7734.00.5400.15
SWL23.257.9425.085.5827.00.3650.31
Children outcomes (reported by parents)
SDQ23.756.9723.256.8223.50.4780.16
Emotional sx.4.002.413.582.2728.00.5450.24
Behavior prob.4.832.723.752.8028.00.018 *0.71
Hyperactivity4.753.145.253.4915.50.4330.28
Peers prob.3.503.293.083.3416.00.2810.35
Prosocial6.672.937.582.68.0030.034 *0.84
Notes: ** p < 0.01; * p < 0.05; † p <0.10. 6-PAQ: Parental Acceptance Questionnaire; AAQ-II: Acceptance and Action Questionnaire; DERS: Difficulties for Emotion Regulation Scale; PSS: Parental Stress Scale; SWL: Satisfaction with Life Scale; SDQ: Strengths and Difficulties Scale; Sx.: Symptoms; Prob.: problems.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Flujas-Contreras, J.M.; García-Palacios, A.; Gómez, I. Effectiveness of a Web-Based Intervention on Parental Psychological Flexibility and Emotion Regulation: A Pilot Open Trial. Int. J. Environ. Res. Public Health 2021, 18, 2958. https://doi.org/10.3390/ijerph18062958

AMA Style

Flujas-Contreras JM, García-Palacios A, Gómez I. Effectiveness of a Web-Based Intervention on Parental Psychological Flexibility and Emotion Regulation: A Pilot Open Trial. International Journal of Environmental Research and Public Health. 2021; 18(6):2958. https://doi.org/10.3390/ijerph18062958

Chicago/Turabian Style

Flujas-Contreras, Juan M., Azucena García-Palacios, and Inmaculada Gómez. 2021. "Effectiveness of a Web-Based Intervention on Parental Psychological Flexibility and Emotion Regulation: A Pilot Open Trial" International Journal of Environmental Research and Public Health 18, no. 6: 2958. https://doi.org/10.3390/ijerph18062958

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop