Next Article in Journal
Understanding Environmental Pollutions of Informal E-Waste Clustering in Global South via Multi-Scalar Regulatory Frameworks: A Case Study of Guiyu Town, China
Next Article in Special Issue
Teaching-Learning Processes: Application of Educational Psychodrama in the University Setting
Previous Article in Journal
Relationship between Health-Related Quality of Life and Physical Activity in Children with Hyperactivity
Previous Article in Special Issue
Passion or Perseverance? The Effect of Perceived Autonomy Support and Grit on Academic Performance in College Students
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 17
Issue 8
10.3390/ijerph17082805
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Effectiveness of Video-Feedback with Cognitive Preparation in Improving Social Performance and Anxiety through Super Skills for Life Programme Implemented in a School Setting

by
Mireia Orgilés
*,
Silvia Melero
,
Iván Fernández-Martínez
,
José Pedro Espada
and
Alexandra Morales
Department of Health Psychology, Miguel Hernández University, 03202 Elche, Spain
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2020, 17(8), 2805; https://doi.org/10.3390/ijerph17082805
Submission received: 8 March 2020 / Revised: 6 April 2020 / Accepted: 16 April 2020 / Published: 18 April 2020
(This article belongs to the Special Issue Advances in Understanding Educational Context: Promoting Adaptative Outcomes)
Browse Figure
Versions Notes

Abstract

:
Effectiveness of video-feedback with cognitive preparation to treat anxiety problems (especially social anxiety) has been scarcely explored on children. Super Skills for Life (SSL) is a CBT-based intervention to reduce anxiety and comorbid problems that, apart from social skills training and behavioural activation, integrates video-feedback with cognitive preparation. This study aimed to evaluate SSL effects, implemented in a school setting, on social performance and to test self-concept and social skills as potential mediators of pre- and post-test changes in social anxiety and generalized anxiety. Sample comprised 57 children aged 8–11 years with emotional symptoms. Children were video recorded in the first and last session to assess social performance. Anxiety and self-concept measures were completed by children pre-test and post-test. Participants reduced anxiety behaviours and improved social and communication skills after treatment. In general, girls showed better social performance than boys, but SSL impact was greater in males. Social self-concept was the only mediator of change in pre- to post-treatment social anxiety. This study provides evidence of SSL to improve children’s social performance and reduce anxiety through video-feedback with cognitive preparation. Improving social concept seems essential to reduce social anxiety. An SSL programme is an ideal prevention protocol for anxious children.

1. Introduction

Anxiety disorders are among the most common psychological problems in children and adolescents. Recent studies claim that approximately 9–46.7% of minors suffer from anxiety symptoms, which implies a great assistance demand in the mental health centres [1,2,3]. If childhood-onset anxiety disorders are untreated, they tend to become chronic [4] and the risk of developing other anxiety disorders, depression, and substance abuse in adulthood increase considerably [5,6]. In addition, these disorders cause impairment in the academic, social, and family settings and entail a high cost in health, educational, and social care services [1,7,8].
Among anxiety disorders in childhood and adolescence, generalised anxiety and social anxiety are the most prevalent and persistent [3]. Generalised anxiety is characterized by a feeling of permanent stress and worry, which can impair social functioning [9]. Fear of negative evaluation in social situations (social anxiety) is highly common in populations with generalised anxiety. Children who suffer social anxiety show a deficit of social skills [10] and have a negative perception of their social performance [11]. Other studies suggest that, when examining social interaction behaviours separately, socially anxious children do not differ from others in their social skills (e.g., gaze, smile, voice, etc), but by showing more nervous behaviours (e.g., trembling, stumbling over words, stuttering, self-manipulations, etc) [12]. When anxious children enter a social situation, negative rumination activates a public self-image which threatens their social self-concept [12,13]. Spence, Donovan, and Brechman-Toussaint [14] found that socially anxious children tended to anticipate more negative outcomes and reported poorer expected performance on social tasks. Social anxiety reduces the quality of social interactions and children’s social, academic, and emotional self-concept [15]. Given that anxious children show impairments in peer relations [16], early detection and treatment are key to improve their social performance and prevent social rejection.
Regarding gender differences, the literature indicates that anxiety disorders are more prevalent in girls [2,3]. Most of the research on social anxiety have found that girls report higher levels of socially anxious symptoms than boys [3,17,18]. Furthermore, girls’ friendships are more impaired by social anxiety than are boys’ [17,19]. Despite this, studies indicate that males show greater signs of nervousness in their social performance, and females have better social skills and non-verbal behaviour (e.g., eye contact, nodding, and smiling), even if they have social anxiety [20,21]. The effectiveness of social anxiety interventions has also been examined in terms of gender, finding a higher improvement in girls’ anxious symptoms, social skills, and other associated problems [18,20].
Cognitive-behavioural therapy (CBT) is the first-choice treatment for childhood anxiety [22,23]. Most studies of CBT-based programmes have been conducted in research settings, such as university clinics, with more highly qualified and experienced professionals than the average clinicians [24]. Thus, concern about generalising results to real-world settings leads researchers to suggest applying interventions in school contexts, which are more familiar and natural to children [25,26]. This would also reduce the stigma of receiving treatment in mental health clinics [25]. Combining effective cognitive-behavioural techniques, behavioural activation, social skills training, and video-feedback with cognitive preparation, Essau and Ollendick [27] developed Super Skills for Life (SSL), a transdiagnostic prevention protocol for children with anxiety and comorbid problems (e.g., depression, low self-esteem, and lack of social skills) that can be implemented in different contexts.
Generalised and social anxiety symptoms are particularly addressed in SSL, because video-feedback with cognitive preparation is an effective CBT-based technique to contrast children’s negative thoughts [11,28,29,30,31], increasing their confidence and reducing their anxiety in the following speech tasks [32]. In the SSL programme, children must perform in front of peers, facing a video camera, and assess their social performance. Cognitive preparation before viewing videos encourages children to create a mental image of their performance and to watch it objectively, which allows changing and improving their social performance appraisal [33]. The SSL programme includes video-taped role-play activities with which children learn social interaction skills with peers and problem solving, which they should practise as homework. Furthermore, to increase positive social experiences and enhance their social self-perception, children’s participation in rewarding activities is encouraged through different tasks based on behavioural activation.
SSL has been recently translated and culturally adapted for Spanish children and adolescents [34,35,36]. The Spanish version of SSL is effective in decreasing anxiety, children’s life anxiety interference, depression, negative self-esteem, and behavioural problems (e.g., peer and conduct problems). Despite the promising results of SSL, the evaluation of its effectiveness to improve social performance and communication skills through video-feedback with cognitive preparation is pending. Effectiveness of video-feedback with cognitive preparation to treat anxiety problems (especially social anxiety) has been widely explored in adolescents and adults, but few studies have focused on the child population. According to Essau et al. [20], more evidence is needed about the mechanisms underlying the effectiveness of video-feedback in SSL to reduce anxiety symptoms, including self-concept and social performance.
This study aimed to evaluate the effectiveness of video-feedback with cognitive preparation of the SSL programme, implemented in the school setting, through three objectives: (1) to analyse SSL effects through video-feedback with cognitive preparation on children’s social performance, and separately by gender; (2) to compare SSL effects on social performance behaviours (gaze, vocal quality, length, discomfort, conversation flow, micro-behaviours, nervous behaviours and global impression) between boys and girls; and (3) to study the dimensions of self-concept (social, academic, emotional, family, and physical self-concept) and social performance as mediators of change in social anxiety and generalised anxiety. Based on previous studies, it is hypothesised that: (1) children’s social performance will improve after treatment; (2) SSL’s effect on social performance will differ by gender; girls will present higher social performance than boys after treatment; and (3) social performance and self-concept areas (specifically, the social area) will mediate the change in pre- to post-treatment of social anxiety and generalised anxiety.

2. Materials and Methods

2.1. Participants

In the original study, sample was made up of 112 children. Of these participants, a subsample of 50.89% (n = 57) children were assessed for improvements in social performance using video-feedback with cognitive preparation (Figure 1). The final sample in the present study comprised 57 children (68.4% were males) aged between 8 and 11 years (M = 9.35, SD = 1.15). The age distribution was as follows: 8 (n = 20; 35.1%), 9 (n = 8; 14%), 10 (n = 18; 31.6%), and 11 (n = 11; 19.3%). Participants were recruited from 9 schools located in the southeast of Spain. Practically all the participants were Spanish-born (98.2%), except for one (1.8%) who was born in the United States, but all of them were Spanish-speaking. The mean number of siblings was 1.16 (SD = 1.17).
Given that parents tend to report more global information about their children’s emotional state [37], they completed the Strengths and Difficulties Questionnaire-Parent version (SDQ-P) [38] to screen their children’s emotional symptoms. Children who showed emotional symptoms based on scores equal to or greater than 4 on the Emotional symptoms subscale of the SDQ-P were selected to participate in the study. Another inclusion criterion was that the child had not received previous psychological or pharmacological treatment for emotional problems. Sample was evaluated at baseline (Session 1) and immediately after receiving the intervention (Session 8).

2.2. Measures

Sociodemographic variables were obtained by a set of items that evaluated age, gender, school year, number of siblings, and birthplace of participants.
The AF-5: Self-Concept Form 5 [39] measures five dimensions of self-concept: Social (performance in social relationships); Academic/Professional (role as a student/worker); Emotional (perception of emotional state in general and in specific situations); Family (participation and integration in the family unit); and Physical self-concept (appearance and physical state). It consists of 30 statements that are rated from 1 to 99 according to the degree of agreement with the content of each statement. Cronbach’s alpha was 0.85 in the current study.
The Screen for Child Anxiety Related Emotional Disorders [40] assesses anxiety disorder symptoms in children and adolescents. This instrument consists of 41 items grouped into five subscales: Somatic/Panic; Generalised Anxiety; Separation Anxiety; Social Anxiety; and School Phobia. Respondents rate on a 3-point scale (0 = almost never; 1 = sometimes; 2 = often) the frequency with which they experience each symptom. The total score is obtained by summing the relevant items. In this study, the Social Anxiety and Generalised Anxiety subscales were used for mediation analyses. The Cronbach’s alpha for this sample was 0.89 for the total score, 0.66 for Social Anxiety, and 0.73 for Generalised Anxiety.
The Social Performance Rating Scale [41] evaluates behavioural indicators of anxiety in a videotaped social performance. The scale consists of five dimensions: Gaze; Voice Quality; Length; Discomfort; and Conversation Flow. The total score is obtained by combining the 5 dimensions. In the current study, items were adapted to the 2-min speech task (there was no conversation partner). Observers were trained according to the guidelines provided by Fydrich et al. [41]. The internal consistency was adequate (α = 0.70) for this sample.
The Objective Performance Questionnaire [12] requires an observer to rate the child’s performance during the 2-min speech task. This scale contains eight items grouped into three dimensions: Micro-Behaviours (displayed during the social situation), Nervous Behaviours (how comfortably the child performs in front of the camera) and Global Impression (overall opinion of the performance). Each item is scored on a four-point Likert scale ranging from 1 (not very much) to 4 (very much). The scale had adequate reliability (α = 0.75) in this study.
The Strengths and Difficulties Questionnaire—Parent version (SDQ-P) [38] was only used for study participant selection purposes. It is a brief screening questionnaire for assessing children’s and adolescents’ psychological adjustment. The 25 items in the SDQ comprise 5 scales of 5 items each: Emotional Symptoms; Conduct Problems; Hyperactivity/Inattention; Peer Problems; and Prosocial Behaviour. Items are rated on a 3-point scale ranging from 0 (Not true) to 2 (Certainly true). In this study, only the Emotional Symptoms subscale score was analysed for participant selection. Following the original 3-band categorization, a cut-off score ≥4 was chosen as a selection criterion, which includes the categories borderline and abnormal. Cronbach’s alpha of the SDQ-P was 0.76 in this study.

2.3. Procedure

This study was approved by the Institutional Review Board (IRB) at the Miguel Hernández University in Spain (DPS.MO.02.14). Nine primary schools in southeast Spain were randomly selected, and their headmasters were invited to participate in the study. The headmasters agreed to collaborate by sending a letter to the students’ families with information about the study. Interested parents completed an online form that served as a screening tool. Meetings with parents of the selected children were held to provide information about the programme, confidentiality, and voluntary participation, and also to complete the informed consent.
As noted, SSL was delivered at the children’s schools in the afternoons. Facilitators were therapists specialized in child and adolescent psychology and with at least 2 years’ experience. All of them received intensive training in the SSL programme before its implementation and were given a leader’s manual with a detailed description of each session. Weekly meetings with facilitators were held to discuss possible problems during the implementation and to collect data. Besides, each facilitator recorded in writing the degree of application of each session, which allowed assessing the implementation fidelity.
The videos’ rating through the social performance scales (see measures section) was carried out by two doctoral students trained as observers. Each observer independently rated the 2-min speech tasks of each participant at pre- and post-intervention. The discrepancies between them were discussed until an agreement on the scores was reached. Raters were blind to other children’s scores in other measures and did not take part in the programme implementation. The final rating scores were used to evaluate the effectiveness of the intervention to change social performance outcomes in the speech task.

2.4. Super Skills for Life (SSL) Programme

The SSL is a transdiagnostic CBT-based programme aimed to treat emotional problems and their comorbid symptoms [27]. Through the programme, children learn to identify and manage their own and others’ emotions, cognitive restructuring skills, behavioural activation, relaxation techniques, social and communication skills, and problem-solving strategies. The SSL intervention consisted of eight weekly 45-min sessions in small groups comprising 6–8 children. Sessions 1 and 2 involve teaching children skills to enhance their social performance and making a 2-min speech facing a video camera. In the first session, they had to imitate a television presenter giving information about themselves (name, favourite food, animal, and hobby). In the eighth session, as an exercise of the generalization of the skills learned, they had to talk like a TV presenter in front of the camera about which SSL activities they liked the most and which skills they considered most useful for their life. Before watching their recordings, the children were instructed to attend to their social behaviours and not focus on how they felt during the speech. Subsequently, the children watched their recordings and were requested to assess their real social performance. Orgilés et al. [34] provide a detailed description of the Spanish version of SSL.

2.5. Statistical Analyses

The sample was described using frequencies (percentages) and means (standard deviation) of the sociodemographic variables. Gender differences in main outcomes and sociodemographic variables were studied and controlled in the analyses. Cohen’s d was calculated to report effect size for statistically significant differences. Because the measurement scales were ordinal, Spearman’s rank order coefficient was used to test inter-rater reliability (IRC). These analyses provided evidence of the extent to which individual evaluators’ rating pointed in the same direction. An intent-to-treat perspective was applied; therefore, the number of sessions attended by participants was not a criterion to be included in the analyses. Attrition analyses were included to identify possible differences between children who were selected for the current study and the rest (n = 55). As recommended by Rosenthal and Rosnow [42], pre-specified trial hypotheses were planned and were tested as follows: (1) a contrast compared post-test with pre-test in the entire sample, (2) a contrast compared post-test with baseline scores only for girls, (3) a contrast compared post-test with baseline scores only for boys, and (4) a contrast compared post-intervention scores by gender to detect whether SSL shows effects differentially for boys and girls. Short-term effects of SSL to improve social performance outcomes were evaluated using generalised estimating equations (GEE), adjusting for gender, age, baseline differences, and clustering in participating schools [43,44]. All analyses were controlled for participating school, age, gender (boys were coded as 1 and girls were coded as 2), and baseline scores. Estimated marginal means, adjusted odds ratio (AOR) and their 95% confidence intervals (CI) are reported. All analyses were performed using SPSS v.25 (IBM Corp., Armonk, NY, USA). Following Essau et al. [20], we studied whether the incremental scores of social performance and self-concept (including academic, social, emotional, familial, and physical areas) were potential mediators of change between pre- and post-treatment social anxiety and generalised anxiety symptoms. Mediation analyses using SPSS PROCESS v3.0 macro [45] were conducted. We calculated 95% bias-corrected bootstrap confidence intervals (5000 samples) for indirect effects. The significant criterion was p ≤ 0.05.

3. Results

3.1. Attrition

Regarding attrition analysis, differences between selected children for this study and those who were not selected were only found in gender (AOR = 0.40, CI [0.19, 0.85], p = 0.01). A higher percentage of males completed the post-intervention survey compared to females (68.4% vs. 31.6%); however, most of the participants were males. No differences were found in age (p > 0.05), number of siblings (p > 0.05), anxiety (SCARED) (p > 0.05), generalised anxiety (p > 0.05), social anxiety (SCARED subscale) (p > 0.05), or social self-concept (AF-5 subscale) (p > 0.05) between the two groups (selected and non-selected children for the study). The mean number of sessions that children attended was 7.52 (SD = 0.78: range: 1–8). For the current sample, there were no gender differences in the sociodemographic variables.

3.2. Inter-Judge Reliability

Table 1 shows the correlation coefficients of the two independent observers’ ratings for all the SPRS (Social Performance Rating Scale) and OPQ-C (The Objective Performance Questionnaire) subscales. The concordance index for these calculations was above 0.90 pre-test and 0.95 post-test, which indicates excellent inter-rater reliability on all subscales. These results show high agreement between the raters in the recording assessments.

3.3. Behavioural Changes in the Speech Task

Children’s social performance was evaluated through the 2-min speech task pre- and post-treatment. As shown in Table 2, all the participants improved their scores after the intervention in all the SPRS subscales, and in the total score; as well as in all the OPQ-C subscales. As shown in Table 3, this within-subject improvement in the different subscales of social performance was statistically significant, with p-values ranging from 0.005 to <0.001, except for the subscale Length (p > 0.05). Therefore, children reduced their anxious behaviours and increased their overall social and communication skills from pre- to post-treatment.

3.4. Gender Differences

Before the intervention, a significant between-subject effect of gender was obtained. At baseline, girls presented higher scores in Conversation Flow (p < 0.05, d = 0.73) and Nervous Behaviours compared to boys (p < 0.05, d = 0.63). Furthermore, Discomfort was marginally significant (p > 0.05), which meant that girls felt more comfortable than boys during their social performance.
In girls, significant effects were obtained after treatment in four of the five SPRS dimensions (except for Conversation Flow), in the total SPRS score, and in all three OPQ-C variables (Table 3). Boys also improved significantly in four of the five SPRS dimensions (except for Length), in the total SPRS score and all three OPQ-R variables (p < 0.05 to p < 0.001) (Table 3).
Gender differences in social performance were observed post-test. Boys reduced their Nervous behaviours more than girls (p < 0.01) and showed higher scores in Discomfort (p < 0.05) (Table 3). Given that higher scores on the Discomfort scale indicate greater comfort, these results show that, after treatment, boys felt more comfortable during their social performance than did girls.

3.5. Mediators of Change

Table 4 shows the 95% confidence intervals for the mediating effects. Social self-concept was the only significant mediator of change from the pre- to post-test Social Anxiety subscale (M = −0.36, 95% CI [−0.65, −0.08]). Mediating effects were not found for the rest of the variables. None of the potential mediators was significant for change from the pre- to post-test Generalised Anxiety subscale.

4. Discussion

The main objective of the study was to examine the effects of the SSL video-feedback with cognitive preparation on children’s social performance and its relationship with social anxiety and generalised anxiety in a Spanish sample of schoolchildren with subclinical anxiety symptoms. The SSL has proven to be an adequate programme for decreasing anxiety symptoms and improving mood, self-esteem, and social performance in primary school children aged 8–12 in the United Kingdom [20] and in Spain [34].
In this study, the use of video-feedback with cognitive preparation has provided benefits in children’s social performance, enhancing their prosocial behaviours and communication skills, and reducing signs of discomfort in these social situations. These findings are consistent with previous studies, in which children and adolescents presented fewer anxiety symptoms, improved their performance in the speech task, and increased their self-confidence [11,26,32]. Moreover, video-feedback is a useful component to modify children’s negative appraisal of their social performance, because children who suffer from social anxiety show a tendency to self-rate themselves as socially unskilled [12,14].
The ratings of the 2-min speech task by observers at pre- and post-treatment indicated that the children’s social performance improved significantly after the SSL intervention at school. In the current study, children showed fewer behavioural indicators of anxiety (such as inadequate movements, stiffness, self-manipulation, stuttering, stumbling over words etc.), and greater self-assurance when speaking, their voice volume and tone were more appropriate, their speech fluency and coherence had improved, and they showed adequate gaze and friendliness in their second speech. In the study of Essau et al. [20], similar results were found at 6-month follow-up on behavioural indicators of anxiety. However, our results suggest that the programme effects are noticeable from the end of the intervention, as in other studies conducted in the school setting [26].
Although social performance improved in the entire sample in the post-intervention evaluation, SSL impact was greater in males. Compared to girls, boys showed significant improvements in behavioural measures associated with displaying discomfort and nervousness. These results contradict previous literature, as it has been found that interventions in social anxiety showed a greater impact on girls [18]. Despite this, according to the study of Essau et al. [20], girls obtained better ratings in almost all the variables of social performance, specifically in gaze, length, conversation flow, micro-behaviours, and global impression. This may be because girls had better social performance at baseline and also due to a ceiling effect. Although girls improved their performance after the programme, it was difficult to find significant differences between the two time points. Therefore, our working hypothesis is partially fulfilled because, after treatment, girls showed better social performance, although they also continued to manifest greater anxious behaviours than boys. These findings agree with previous literature, which states that girls are socially more skilled, but also suffer more social anxiety than boys, and this is associated with social functioning problems [2,17,19,26].
Social self-concept mediated the effect of SSL on social anxiety symptoms. According to previous studies, social self-concept of individuals with social anxiety is negatively affected by a highly distorted public self-image [14,15,28]. Video-feedback with cognitive preparation, as a component of the SSL, contributed to children’s disconfirming their negative beliefs about their social performance and adjusting their self-image during the speech [29,31]. As suggested in the literature, [11,29,33], during the programme, participants were instructed to view the videotape neutrally, as if they were watching a real television presenter. Their own evaluation as external observers in the first speech increased their self-confidence and made their predictions for the second speech more positive [32]. Therefore, it is not surprising that the improvement of social self-concept through the SSL programme has a beneficial impact on reducing social anxiety symptomatology. In line with the study of Essau et al. [20], social performance was not a mediator for pre- to post-test change in the Social Anxiety and Generalised Anxiety subscales. Thus, the third hypothesis of this study was partially confirmed. Results agree with the findings of Cartwright-Hatton et al. [12], who argued that children maintain social anxiety through their negative beliefs about their social performance, regardless of their social skills. Future studies should explore other potential mediators of SSL effectiveness, for example, the children’s cognitive emotion-regulation strategies or the presence of other comorbid problems, such as depression or behavioural problems.

Limitations

The results of the current study need to be interpreted considering several limitations. First, sample size was relatively small, which makes it difficult to generalize the results. Future studies including a larger sample should be carried out to confirm the results and analyse differences by age and gender. Second, there was no control group with which to compare the obtained findings. Randomized controlled trials are needed in future research to incorporate untreated children’s outcomes to provide more evidence of the programme’s effect [26]. Third, there are no data on the self-assessment of the children’s social performance, but only on the objective rater’s evaluation. An inclusion of the children’s perspective in future research would be interesting to analyse changes in the appraisal of their own social performance [20,33].

5. Conclusions

In summary, despite the above limitations, this study provides evidence of the positive effects of video-feedback with cognitive preparation in the SSL programme to improve children’s social performance and reduce anxious symptomatology. In addition to clinical improvements, at an educational level, enhancing social performance can contribute to reducing peer rejection and increase the quality of social interactions [16]. Improvement of social self-concept seems a key to reduce social anxiety. Thus, these findings suggest that interventions aimed at treating social anxiety should include components that enhance children’s social self-concept. The SSL methodology makes the programme an ideal prevention protocol for children with anxiety problems, especially those with social anxiety, which can be applied in both clinical and school contexts. These results support the use of transdiagnostic approaches, which include different strategies based on CBT for the treatment of anxiety and social performance problems in children and contribute to understanding the transportability of a clinical intervention into real-world school settings. Therefore, this research provides strategies to promote children’s mental health that are useful for the development of public health policies.

Author Contributions

Conceptualization, methodology, funding acquisition, supervision and writing—review and editing, M.O.; investigation, resources, writing—original draft preparation, S.M. and I.F.-M.; software, formal analysis, A.M.; validation, visualization, J.P.E. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Ministry of Economy and Competitiveness (MINECO) of Spain, grant number (PSI2014-56446-P).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Essau, C.A.; Gabbidon, J. Epidemiology, comorbidity and mental health services utilization. In The Wiley-Blackwell Handbook of the Treatment of Childhood and Adolescent Anxiety; Essau, C.A., Ollendick, T.H., Eds.; Wiley Blackwell: Chichester, UK, 2013; pp. 23–42. [Google Scholar] [CrossRef]
  2. Orgilés, M.; Méndez, X.; Espada, J.P.; Carballo, J.L.; Piqueras, J.A. Síntomas de trastornos de ansiedad en niños y adolescentes: Diferencias en función de la edad y el sexo en una muestra comunitaria. Revista de Psiquiatría y Salud Mental 2012, 5, 115–120. [Google Scholar] [CrossRef]
  3. Voltas, N.; Hernández-Martínez, C.; Arija, V.; Canals, J. The natural course of anxiety symptoms in early adolescence: Factors related to persistence. Anxiety Stress Coping 2017, 30, 671–686. [Google Scholar] [CrossRef] [PubMed]
  4. Broeren, S.; Muris, P.; Diamantopoulou, S.; Baker, J. The course of childhood anxiety symptoms: Developmental trajectories and child-related factors in normal children. J. Abnorm. Child Psychol. 2013, 41, 81–95. [Google Scholar] [CrossRef] [PubMed]
  5. Roza, S.J.; Hofstra, M.B.; van der Ende, J.; Verhulst, F.C. Stable prediction of mood and anxiety disorders based on behavioral and emotional problems in childhood: A 14-year follow-up during childhood, adolescence, and young adulthood. Am. J. Psychiatry 2003, 160, 2116–2121. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Benjamin, C.L.; Harrison, J.P.; Settipani, C.A.; Brodman, D.M.; Kendall, P.C. Anxiety and related outcomes in young adults 7 to 19 years after receiving treatment for child anxiety. J. Consult. Clin. Psychol. 2013, 81, 865–876. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Essau, C.A. Frequency and patterns of mental health services utilization among adolescents with anxiety and depressive disorders. Depress Anxiety 2005, 22, 130–137. [Google Scholar] [CrossRef]
  8. Snell, T.; Knapp, M.; Healey, A.; Guglani, S.; Evans-Lacko, S.; Fernandez, J.; Meltzer, H.; Ford, T. Economic impact of childhood psychiatric disorder on public sector services in Britain: Estimates from national survey data. J. Child Psychol. Psychiatry 2013, 54, 977–985. [Google Scholar] [CrossRef] [Green Version]
  9. Huberty, T.J. Test and performance anxiety. Princ. Leadersh. 2009, 10, 12–16. [Google Scholar]
  10. Albano, A.M.; Hayward, C. Social anxiety disorder. In Phobic and Anxiety Disorders in Children and Adolescents: A clinician’s Guide to Effective Psychosocial and Pharmacological Interventions; Ollendick, T.H., March, J.S., Eds.; Oxford University Press: New York, NY, USA, 2004; pp. 198–235. [Google Scholar]
  11. Ramos, V.; Piqueras, J.A.; García-López, L. Efficacy and efficiency of the use of video-feedback in the cognitive-behavioral treatment of young people with social anxiety disorder. Clin. Salud 2008, 19, 249–264. [Google Scholar]
  12. Cartwright-Hatton, S.; Hodges, L.; Porter, J. Social anxiety in childhood: The relationship with self and observer rated social skills. J. Child Psychol. Psychiatry 2003, 44, 737–742. [Google Scholar] [CrossRef]
  13. Beidel, D.C.; Turner, S.M.; Morris, T.L. Behavioral treatment of childhood social phobia. J. Consult. Clin. Psychol. 2000, 68, 1072–1080. [Google Scholar] [CrossRef] [PubMed]
  14. Spence, S.H.; Donovan, C.; Brechman-Toussaint, M. Social skills, social outcomes, and cognitive features of childhood social phobia. J. Abnorm. Psychol. 1999, 108, 211–221. [Google Scholar] [CrossRef] [PubMed]
  15. Beidel, D.C.; Ferrell, C.; Alfano, C.A.; Yeganeh, R. The treatment of childhood social anxiety disorder. Psychiatr. Clin. N. Am. 2001, 24, 831–846. [Google Scholar] [CrossRef]
  16. Coplan, R.J.; Prakash, K.; O’Neil, K.; Armer, M. Do You “Want” to Play? Distinguishing Between Conflicted Shyness and Social Disinterest in Early Childhood. Dev. Psychol. 2004, 40, 244–258. [Google Scholar] [CrossRef] [PubMed]
  17. La Greca, A.M.; Harrison, H.M. Adolescent peer relations, friendships, and romantic relationships: Do they predict social anxiety and depression? J. Clin. Child Adolesc. Psychol. 2005, 34, 49–61. [Google Scholar] [CrossRef] [PubMed]
  18. Rodríguez, J.O.; Rosa-Alcázar, A.I.; Caballo, V.E.; García-López, L.J.; Orgilés, M.; López-Gollonet, C. Treatment of social phobia in children and adolescents: A meta-analytic review. Behav. Psychol. 2003, 11, 599–622. [Google Scholar]
  19. Borowski, S.K.; Zeman, J.; Braunstein, K. Social Anxiety and Socioemotional Functioning During Early Adolescence: The Mediating Role of Best Friend Emotion Socialization. J. Early Adolesc. 2018, 38, 238–260. [Google Scholar] [CrossRef]
  20. Essau, C.A.; Olaya, B.; Sasagawa, S.; Pithia, J.; Bray, D.; Ollendick, T.H. Integrating video-feedback and cognitive preparation, social skills training and behavioural activation in a cognitive behavioural therapy in the treatment of childhood anxiety. J. Affect. Disord. 2014, 167, 261–267. [Google Scholar] [CrossRef]
  21. Pilkonis, P.A. The behavioral consequences of shyness 1. J. Personal. 1977, 45, 596–611. [Google Scholar] [CrossRef]
  22. Essau, C.A.; Conradt, J.; Sasagawa, S.; Ollendick, T.H. Prevention of Anxiety Symptoms in Children: Results from a Universal School-Based Trial. Behav. Ther. 2012, 43, 450–464. [Google Scholar] [CrossRef]
  23. James, A.C.; James, G.; Cowdrey, F.A.; Soler, A.; Choke, A. Cognitive behavioural therapy for anxiety disorders in children and adolescents. In Cochrane Database of Systematic Reviews; John Wiley & Sons, Ltd.: Hoboken, NJ, USA, 2015. [Google Scholar] [CrossRef]
  24. Weisz, J.R.; Weiss, B.; Han, S.S.; Granger, D.A.; Morton, T. Effects of psychotherapy with children and adolescents revisited: A meta-analysis of treatment outcome studies. Psychol. Bull. 1995, 117, 450–468. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Masia-Warner, C.; Nangle, D.W.; Hansen, D.J. Bringing evidence-based child mental health services to the schools: General issues and specific populations. Educ. Treat. Children 2006, 29, 165–172. [Google Scholar]
  26. Essau, C.A.; Sasagawa, S.; Jones, G.; Fernandes, B.; Ollendick, T.H. Evaluating the real-world effectiveness of a cognitive behavior therapy-based transdiagnostic programme for emotional problems in children in a regular school setting. J. Affect. Disord. 2019, 253, 357–365. [Google Scholar] [CrossRef] [PubMed]
  27. Essau, C.; Ollendick, T.H. The Super Skills for Life programme; University of Roehampton: London, UK, 2013. [Google Scholar]
  28. Clark, D.M.; Wells, A. A cognitive model of social phobia. In Social Phobia: Diagnosis, Assessment, and Treatment; Heimberg, R.G., Liebowitz, M.R., Hope, D.A., Schneier, F.R., Eds.; Guilford Press: New York, NY, USA, 1995; pp. 69–93. [Google Scholar]
  29. Harvey, A.G.; Clark, D.M.; Ehlers, A.; Rapee, R.M. Social anxiety and self-impression: Cognitive preparation enhances the beneficial effects of video feedback following a stressful social task. Behav. Res. Ther. 2000, 38, 1183–1192. [Google Scholar] [CrossRef]
  30. Rapee, R.M.; Hayman, K. The effects of video feedback on the self-evaluation of performance in socially anxious subjects. Behav. Res. Ther. 1996, 34, 315–322. [Google Scholar] [CrossRef]
  31. Rodebaugh, T.L. I might look OK, but I’m still doubtful, anxious, and avoidant: The mixed effects of enhanced video feedback on social anxiety symptoms. Behav. Res. Ther. 2004, 42, 1435–1451. [Google Scholar] [CrossRef]
  32. Parr, C.J.; Cartwright-Hatton, S. Social anxiety in adolescents: The effect of video feedback on anxiety and the self-evaluation of performance. Clin. Psychol. Psychother. 2009, 16, 46–54. [Google Scholar] [CrossRef]
  33. Morgan, J.; Banerjee, R. Social Anxiety and Self-Evaluation of Social Performance in a Nonclinical Sample of Children. J. Clin. Child Adolesc. Psychol. 2006, 35, 292–301. [Google Scholar] [CrossRef]
  34. Orgilés, M.; Fernández-Martínez, I.; Espada, J.P.; Morales, A. Spanish version of Super Skills for Life: The short- and long-term effectiveness of a transdiagnostic prevention protocol for Spanish children. Anxiety Stress Coping 2019, 36, 694–710. [Google Scholar] [CrossRef]
  35. De la Torre-Luque, A.; Fiol-Veny, A.; Essau, C.A.; Balle, M.; Bornas, X. Effects of a transdiagnostic cognitive behaviour therapy-based programme on the natural course of anxiety symptoms in adolescence. J. Affect. Disord. 2020, 264, 474–482. [Google Scholar] [CrossRef]
  36. Fernández-Martínez, I.; Morales, A.; Espada, J.P.; Essau, C.A.; Orgilés, M. Effectiveness of the program Super Skills For Life in reducing symptoms of anxiety and depression in young Spanish children. Psicothema 2019, 31, 298–304. [Google Scholar] [CrossRef] [PubMed]
  37. Michels, N.; Vanaelst, B.; Stoppie, E.; Huybrechts, I.; Bammann, K.; De Henauw, S.; Sioen, I. Parental and children’s report of emotional problems: Agreement, explanatory factors and event-emotion correlation. Child Adolesc. Ment. Health 2013, 18, 180–186. [Google Scholar] [CrossRef]
  38. Goodman, R. Psychometric Properties of the Strengths and Difficulties Questionnaire. J. Am. Acad. Child Adolesc. Psychiatry 2001, 40, 1337–1345. [Google Scholar] [CrossRef] [PubMed]
  39. García, F.; Musitu, G. (Eds.) In AF-5. Autoconcepto forma 5. Versión revisada y ampliada; TEA Ediciones: Madrid, Spain, 2014. [Google Scholar]
  40. Birmaher, B.; Brent, D.A.; Chiappetta, L.; Bridge, J.; Monga, S.; Baugher, M. Psychometric properties of the screen for child anxiety related emotional disorders (SCARED): A replication study. J. Am. Acad. Child Adolesc. Psychiatry 1999, 38, 1230–1236. [Google Scholar] [CrossRef] [PubMed]
  41. Fydrich, T.; Chambless, D.L.; Perry, K.J.; Buergener, F.; Beazley, M.B. Behavioral assessment of social performance: A rating system for social phobia. Behav. Res. Ther. 1998, 36, 995–1010. [Google Scholar] [CrossRef]
  42. Rosenthal, R.; Rosnow, R.L. Contrast Analysis: Focused Comparisons in the Analysis of Variance; Cambridge University Press: Cambridge, UK, 1985. [Google Scholar]
  43. Fitzmaurice, G.M.; Laird, N.M.; Ware, J.H. Applied Longitudinal Analysis; John Wiley & Sons: Boston, MA, USA, 2012. [Google Scholar]
  44. Liang, K.; Zeger, S.L. Longitudinal data analysis using generalized linear models. Biometrika 1986, 73, 13–22. [Google Scholar] [CrossRef]
  45. Hayes, A.F. Introduction to Mediation, Moderation, and Conditional Process. Analysis: A Regression-Based Approach; Guilford Press: New York, NY, USA, 2013. [Google Scholar]
Ijerph 17 02805 g001 550
Figure 1. Progress of children participating in the trial.
Figure 1. Progress of children participating in the trial.
Ijerph 17 02805 g001
Table
Table 1. Spearman correlations for inter-judge reliability.
Table 1. Spearman correlations for inter-judge reliability.
OutcomesPre-TestPost-Test
SPRS 1
 Gaze0.990.99
 Vocal quality0.960.95
 Length0.980.97
 Discomfort0.950.95
 Conversation flow0.901
 Total0.950.99
OPQ-C 2
  Micro-behaviours0.980.97
 1. How loud and clear was the child’s voice?0.990.96
 2. How much did the child look at the camera?0.970.97
 3. How much did the child smile0.970.98
  Nervous0.980.99
 4. How nervous did the child look?0.980.97
 5. Did the child stumble over the child’s words?11
  Global0.970.98
 6. How clever did the child look?0.930.98
 7. How friendly did the child look?0.990.97
 8. How good was the child’s speech?0.950.98
1 Social Performance Rating Scale; 2 The Objective Performance Questionnaire.
Table
Table 2. Estimated marginal means (95% confident interval) of the outcomes pre-test and post-test by gender.
Table 2. Estimated marginal means (95% confident interval) of the outcomes pre-test and post-test by gender.
OutcomesSamplePre-TreatmentPost-Treatment
SPRS 1
 GazeGirls3.48 (3.39, 3.57)4.04 (3.69, 4.39)
Boys3.54 (3.47, 3.61)3.87 (3.61, 4.13)
Total3.51 (3.45, 3.57)3.96 (3.74, 4.17)
 Vocal qualityGirls3.67 (3.53, 3.80)4.05 (3.77, 4.34)
Boys3.61 (3.51, 3.71)4.25 (4.06, 4.44)
Total3.64 (3.55, 3.72)4.15 (3.98, 4.32)
 LengthGirls3.78 (3.69, 3.87)4.11 (3.88, 4.34)
Boys3.71 (3.64, 3.78)3.81 (3.55, 4.08)
Total3.74 (3.69, 3.80)3.96 (3.79, 4.14)
 DiscomfortGirls3.50 (3.37, 3.63)4 (3.70, 4.31)
Boys3.32 (3.24, 3.41)4.30 (4.03, 4.56)
Total3.41 (3.33, 3.49)4.15 (3.96, 4.34)
 Conversation flowGirls3.83 (3.73, 3.94)4 (3.68, 4.33)
Boys3.66 (3.58, 3.74)3.99 (3.75, 4.23)
Total3.75 (3.68, 3.81)4 (3.80, 4.20)
 TotalGirls18.21 (17.90, 18.51)20.15 (19.20, 21.10)
Boys17.90 (17.67, 18.13)20.28 (19.42, 21.14)
Total18.05 (17.87, 18.24)20.22 (19.60, 20.83)
OPQ-C 2
  Micro-behavioursGirls8.18 (7.96, 8.39)9.84 (9.14, 10.55)
Boys8.22 (8.11, 8.33)9.32 (8.93, 9.72)
Total8.21 (8.10, 8.31)9.49 (9.14, 9.38)
 1. How loud and clear was the child’s voice?Girls2.88 (2.78, 2.98)3.61 (3.33, 3.88)
Boys2.87 (2.79, 2.95)3.59 (3.43, 3.74)
Total2.88 (2.81, 2.94)3.60 (3.44, 3.75)
 2. How much did the child look at the camera?Girls2.68 (2.59, 2.77)3.12 (2.81, 3.43)
Boys2.71 (2.65, 2.76)2.94 (2.70, 3.17)
Total2.69 (2.64, 2.74)3.03 (2.83, 3.22)
 3. How much did the child smile?Girls2.61 (2.50, 2.71)3.11 (2.74, 3.48)
Boys2.64 (2.56, 2.71)2.79 (2.58, 3)
Total2.62 (2.56, 2.68)2.95 (2.74, 3.16)
  Nervous behavioursGirls2.95 (2.83, 3.08)2.62 (2.38, 2.86)
Boys3.01 (2.91, 3.11)2.55 (2.34, 2.76)
Total2.98 (2.90, 3.06)2.59 (2.43, 2.75)
 4. How nervous did the child look?Girls2.02 (1.92, 2.13)1.63 (1.39, 1.88)
Boys2.16 (2.10, 2.23)1.29 (1.13, 1.46)
Total2.12 (2.06, 2.18)1.40 (1.26, 1.54)
 5. Did the child stumble over the child’s words?Girls1.18 (1.11, 1.26)1.02 (0.92, 1.11)
Boys1.22 (1.15, 1.28)1.01 (0.94, 1.08)
Total1.21 (1.16, 1.25)1.01 (0.95, 1.07)
  Global impressionGirls8.62 (8.42, 8.81)10.06 (9.58,10.55)
Boys8.45 (8.28, 8.62)9.58 (9.04, 10.12)
Total8.54 (8.41, 8.66)9.82 (9.46, 10.18)
 6. How clever did the child look?Girls2.94 (2.85, 3.03)3.33 (3.11, 3.54)
Boys2.84 (2.77, 2.91)3.23 (3.02, 3.43)
Total2.89 (2.83, 2.95)3.28 (3.13, 3.43)
 7. How friendly did the child look?Girls2.88 (2.77, 2.99)3.20 (2.99, 3.42)
Boys2.87 (2.78, 2.96)3.43 (3.22, 3.65)
Total2.87 (2.80, 2.95)3.32 (3.17, 3.47)
 8. How good was the child’s speech?Girls2.80 (2.73, 2.87)3.30 (3.07, 3.53)
Boys2.73 (2.66, 2.80)3.14 (2.93, 3.35)
Total2.76 (2.72, 2.81)3.22 (3.06, 3.37)
1 Social Performance Rating Scale; 2 The Objective Performance Questionnaire; Higher scores denote better social performance except for “Nervous behaviours” where higher scores indicate greater anxiety.
Table
Table 3. Generalised linear models and effect size estimates for the intervention effect on the speech-task outcomes in the post-test (compared to the baseline) by gender.
Table 3. Generalised linear models and effect size estimates for the intervention effect on the speech-task outcomes in the post-test (compared to the baseline) by gender.
OutcomesSamplePost-Treatment
AOR 1 (95% CI 2)p Value
SPRS 3
GazeGirls1.74 (1.18, 2.55)0.005
Boys1.39 (1.04, 1.86)0.02
Total1.49 (1.18, 1.88)0.001
Vocal qualityGirls1.47 (1.01, 2.16)0.04
Boys1.89 (1.47, 2.44)<0.001
Total1.75 (1.41, 2.16)<0.001
LengthGirls1.39 (1.06, 1.82)0.01
Boys1.10 (8.82, 1.48)0.49
Total1.19 (0.95, 1.48)0.11
DiscomfortGirls1.64 (1.15, 2.34)0.005
Boys2.64 (1.98, 3.56)<0.001
Total2.28 (1.79, 2.89)<0.001
Conversation flowGirls1.18 (0.80, 1.73)0.39
Boys1.39 (1.06, 1.82)0.01
Total1.32 (1.06, 1.65)0.01
TotalGirls6.99 (2.53, 19.29)<0.001
Boys10.85 (4.30, 27.39)<0.001
Total9.44 (4.63, 19.24)<0.001
OPQ-C 4
Micro-behavioursGirls5.29 (2.37, 11.78)<0.001
Boys3.01 (1.96, 4.60)<0.001
Total3.59 (2.43, 5.32)<0.001
 1. How loud and clear was the child’s voice?Girls2.05 (1.46, 2.88)<0.001
Boys2.05 (1.67, 2.50)<0.001
Total2.05 (1.72, 2.44)<0.001
 2. How much did the child look at the camera?Girls1.56 (1.09, 2.21)0.01
Boys1.26 (0.98, 1.61)0.07
Total1.34 (1.09, 1.65)0.005
 3. How much did the child smile?Girls1.64 (1.08, 2.49)0.01
Boys1.16 (0.91, 1.48)0.21
Total1.30 (1.04, 1.61)0.01
Nervous behavioursGirls0.71 (0.52, 0.97)0.03
Boys0.63 (0.47, 0.82)0.001
Total0.65 (0.53, 0.81)<0.001
 4. How nervous did the child look?Girls0.67 (0.51, 0.89)0.005
Boys0.41 (0.34, 0.51)<0.001
Total0.48 (0.40, 0.57)<0.001
 5. Did the child stumble over the child’s words?Girls1.18 (1, 1.40)0.05
Boys0.81 (0.71, 0.92)0.002
Total0.82 (0.74, 0.91)<0.001
Global impressionGirls4.23 (2.80, 6.41)<0.001
Boys3.09 (1.62, 5.86)0.001
Total3.41 (2.15, 5.40)<0.001
 6. How clever did the child look?Girls1.47 (1.17, 1.84)0.001
Boys1.46 (1.14, 1.89)0.003
Total1.47 (1.22, 1.77)<0.001
 7. How friendly did the child look?Girls1.74 (1.38, 2.19)<0.001
Boys1.39 (1.05, 1.84)0.01
Total1.49 (1.21, 1.83)<0.001
 8. How good was the child’s speech?Girls1.64 (1.30, 2.07)<0.001
Boys1.50 (1.18, 1.92)0.001
Total1.55 (1.29, 1.86)<0.001
1 Adjusted Odds Ratio; 2 Confidence Interval; 3 Social Performance Rating Scale; 4 The Objective Performance Questionnaire. Higher scores denote better social performance, except for “Nervous behaviours”, where higher scores indicate greater anxiety. Each analysis was adjusted for the baseline measure, gender, age, and school level.
Table
Table 4. Results of confidence intervals for mediating effects.
Table 4. Results of confidence intervals for mediating effects.
M1SE2Lower LimitHigher Limit
Social performance (OPQ-C)
 (change in social anxiety scores)0.010.04−0.040.12
 (change in generalised anxiety scores)−0.00070.01−0.030.02
Academic self-concept
 (change in social anxiety scores)−0.090.08−0.290.03
 (change in generalised anxiety scores)0.0020.01−0.040.02
Social self-concept
 (change in social anxiety scores)−0.360.15−0.65−0.08
 (change in generalised anxiety scores)0.020.02−0.010.07
Emotional self-concept
 (change in social anxiety scores)0.0040.02−0.010.01
 (change in generalised anxiety scores)0.0030.01−0.020.03
Familiar self-concept
 (change in social anxiety scores)−0.050.08−0.230.07
 (change in generalised anxiety scores)0.0020.01−0.020.03
Physical self-concept
 (change in social anxiety scores)−0.100.11−0.340.10
 (change in generalised anxiety scores)−0.0060.02−0.040.04
1 Mean; 2 Standard Error

Share and Cite

MDPI and ACS Style

Orgilés, M.; Melero, S.; Fernández-Martínez, I.; Espada, J.P.; Morales, A. Effectiveness of Video-Feedback with Cognitive Preparation in Improving Social Performance and Anxiety through Super Skills for Life Programme Implemented in a School Setting. Int. J. Environ. Res. Public Health 2020, 17, 2805. https://doi.org/10.3390/ijerph17082805

AMA Style

Orgilés M, Melero S, Fernández-Martínez I, Espada JP, Morales A. Effectiveness of Video-Feedback with Cognitive Preparation in Improving Social Performance and Anxiety through Super Skills for Life Programme Implemented in a School Setting. International Journal of Environmental Research and Public Health. 2020; 17(8):2805. https://doi.org/10.3390/ijerph17082805

Chicago/Turabian Style

Orgilés, Mireia, Silvia Melero, Iván Fernández-Martínez, José Pedro Espada, and Alexandra Morales. 2020. "Effectiveness of Video-Feedback with Cognitive Preparation in Improving Social Performance and Anxiety through Super Skills for Life Programme Implemented in a School Setting" International Journal of Environmental Research and Public Health 17, no. 8: 2805. https://doi.org/10.3390/ijerph17082805

APA Style

Orgilés, M., Melero, S., Fernández-Martínez, I., Espada, J. P., & Morales, A. (2020). Effectiveness of Video-Feedback with Cognitive Preparation in Improving Social Performance and Anxiety through Super Skills for Life Programme Implemented in a School Setting. International Journal of Environmental Research and Public Health, 17(8), 2805. https://doi.org/10.3390/ijerph17082805

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop