The Emotional Competence Assessment Questionnaire (ECAQ) for Children Aged from 3 to 5 Years: Validity and Reliability Evidence

Abstract

In order to assess emotional competence in children, it is necessary to have psychometrically sound measures. To the best of our knowledge, there is no available tool to assess emotional competence in children from 3 to 5 years old that assesses the five emotional competences of the Bisquerra model and can be easily and quickly answered in the school environment. The objective of this study is to develop a measure, the Emotional Competence Assessment Questionnaire (ECAQ), and to provide evidence of its psychometric quality. Qualitative evidence was obtained from a systematic review, from two expert committees and from five discussion groups. On the other hand, quantitative validity and reliability evidence was obtained from a sample of 1088 students and other smaller subsamples. The results suggest that the ECAQ is a short and easy-to-use tool, easily understood by administrators. The quantitative results confirm a general factor of emotional competence adjusted for three specific factors. This factor has excellent internal consistency and test-retest reliability. The ECAQ has therefore been shown to be a promising tool for assessing emotional competence in children between 3 and 5 years of age.

1. Introduction

According to the World Health Organization, health, and particularly mental health, can be promoted by boosting emotional competence in children [1]. In addition, some studies point to the key role of emotional competence in influencing the severity of long-term mental health problems [2]. Emotional competence can be enhanced in the school environment through structured programs of emotional and social education [3]. The school is not only the place where children spend a good part of their day but also the place where equitable access to programs to increase different competences, including emotional competence, can be guaranteed. The implementation of these programs has proven to be useful for improving academic performance [4], relationships with peers and adults, problem solving, self-control, perceived self-efficacy, school involvement and well-being [2,5,6]. Emotional competence has also been related to executive function, the latter being below average in children with low emotional competence [7]. Different school-based universal interventions and programs for children aim to enhance their emotional competence [3], but their outcomes should always be evaluated to ensure resources are allocated to interventions with proven positive results [8]. Therefore, we need tools that help us to assess emotional competence in a population-based way.

In an educational context, there are different methods to measure emotional competence in children. When children can read and write properly, we can use self-administered tests. In children aged from 3 to 5 years, although there are studies that suggest the possibility of self-assessment using photographs or drawings, other forms are required, especially when many children are to be assessed [9]. Some procedures are qualitative and include direct observation of the children when they are asked to do a specific activity [10,11,12]. Other procedures are quantitative, and the diversity of them is higher. Moreover, some tests are administered to children by trained researchers and others are answered by teachers [13,14,15] or parents [16,17]. In public health, it is easier to access teachers than families, as teachers are responsible for implementing and evaluating different school-based universal interventions and programs. Most methods assess just one emotional competence, usually emotional regulation, as it occurs for the Emotional Regulation Checklist (ERC), a 24-item scale used in different investigations [18,19]. If they evaluate more than one competence, the number of items is usually high, as for the Behaviors Underpinning Skills for Social-Emotional School Readiness (BUSSE-SR), a social-emotional questionnaire that assesses self-awareness, self-regulation, social communication, empathy and adaptive behavior [20], or the ACER Well-being Survey (teacher form) that assesses social-emotional well-being and social-emotional competence, each containing 50 items [21].

After an extensive literature review, Bisquerra and his research group in emotional education from the University of Barcelona (Grup de Recerca i Orientació Psicopedagògica—GROP), who have trained teachers in emotional education since 2000, established that emotional competence includes five emotional competences [22]. Three of them are of an intrapersonal nature: (i) emotional awareness—the ability to know what you feel and what others feel and why—(ii) emotional regulation—the ability to regulate unpleasant emotions and enhance pleasant ones—and (iii) emotional autonomy—an ability related to self-esteem and the strength to be self-motivated and maintain a positive attitude in spite of adversity. Moreover, the other two are of an interpersonal nature: (iv) social competence—the ability to establish and maintain positive relationships with others—and (v) life skills—related to the ability to adopt appropriate problem-solving. The last three are associated with the capacity to generate positive life experiences and well-being. Each of the competences included in this model overlap to some extent, forming part of the same construct of emotional competence. This model is inspired on Goleman’s emotional intelligence model, which includes five competences—emotional self-awareness, emotion management, self-motivation, empathy and social skills, which in turn include the existence of twenty-five competences [23]. The model proposed by Bisquerra includes problem solving as a main skill and its structure facilitates the understanding of the broad construct of emotional competence defined as “the set of knowledge, abilities, skills, and attitudes necessary to understand, express, and appropriately regulate emotional phenomena” [22]. It is also the model under which most of the teachers in Barcelona are trained.

To the best of our knowledge, no test evaluates the five competences mentioned above and at the same time can be answered easily and quickly. The latter aspect is important from a public health perspective, as there is the need to evaluate a large number of children to design population interventions. We propose to fill this gap by developing a new test, the Emotional Competence Assessment Questionnaire (ECAQ), which is answered by teachers to evaluate emotional competence considering the five emotional competences in children aged from 3 to 5 years. In this process, we follow the Standards for Educational and Psychological Testing [24], one of the most well-known guidelines related to the development and interpretation of tests, which are used to provide evidence of their psychometric quality. The Standards state that “validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” [24]. To provide evidence of validity, these guidelines suggest five sources of validity evidence: content, response processes, the consequences of testing, internal structure and relations to other variables. To analyze the reliability of a questionnaire, its internal consistency can be assessed as well as its replicability by test-retesting.

The hypotheses of our study are: (a) the ECAQ will show good evidence of validity and reliability; (b) when examining the relationship with other variables, we will find a high correlation between the ECAQ total score and other scales or subscales measuring emotional regulation or social competence (i.e., the Student Rating Scale [25], the Social Competence Scale—Teacher [26] and the subscale the Emotional Regulation Checklist [27]) and a low correlation between the ECAQ and other scales or subscales measuring emotional lability (i.e., the subscale the Lability and Negativity of the Emotional Regulation Checklist) and (c) in the assessment of test dimensionality, we will find a general factor including all five competences.

Therefore, the general objective of this study is to develop and provide evidence of the psychometric quality of the ECAQ. To do so, we first propose to provide qualitative evidence by gathering validity evidence based on test content, response processes and test outcomes. We also propose to provide quantitative evidence by gathering validity evidence based on internal structure and relationships with other variables and by assessing the internal consistency reliability of the ECAQ and its test-retest reliability.

2. Materials and Methods

2.1. Participants

2.1.1. Qualitative Evidence Stage

A multidisciplinary committee of seven experts in emotional education were invited to assess the first generated questionnaire’s items. Selected experts were professors (86% female) of postgraduate emotional education in Spanish universities. Another committee of four teachers (100% female) were invited to assess the items proposed by the experts committee. All agreed to participate when invited. Finally, we organized five discussion groups with a maximum of 10 teachers. To be eligible, teachers had to be actively teaching children three to five years old.

2.1.2. Quantitative Evidence Stage

Eleven schools in Barcelona were eligible to collaborate and all of them agreed to take part in the study. A total of 1088 children (48.1% female) aged three to five years old were included in this part of the study once their teachers (n = 52) had answered the ECAQ. To assess validity evidence based on relations to other variables, these same teachers were also asked to respond to other questionnaires for the first 8 students on their class rosters. We obtained responses from 51 teachers. One teacher did not respond because he left school due to illness. Thus, 134 responses were obtained from 21 teachers for the ERC (46.2% girls), 119 from 17 teachers for the SRS (56.3% girls) and 89 from 13 teachers for the SCS-T (44.9% girls).

To assess test-retest reliability, we asked six teachers from two different schools to answer the ECAQ twice for each child in their class. A total of 121 responses were included in this part of the study.

2.2. Measures

The Emotional Competence Assessment Questionnaire (ECAQ) is a questionnaire assessing the emotional competence of children from 3 to 5 years of age. The ECAQ has a total of 30 items and uses a Likert-type scale with six response categories (1 = "never” to 6 = “always”). The ECAQ was developed in Catalan and Spanish and was only adapted to English for publication purposes. It is based on the emotional competence model of Bisquerra [22]. Respondents are the children’s teachers.

The Emotion Regulation Checklist (ERC) [27] is a scale assessing the emotion regulation of children up to 12 years of age with 24 items in two subscales (Lability and Negativity and Emotional Regulation). It can also be used in a unidimensional scale using a 4-point Likert scale (1 = “never” to 4 = “almost always”). Our study used the Spanish [28] and Catalan (Cicchetti D, personal communication) versions of the ERC. It has not been necessary to adapt the instrument to the application context. We have obtained for the total score a Cronbach’s alpha of 0.79. This instrument includes questions such as the following: “Its easily frustrated” (Lability and Negativity Subscale) and “Can say when s/he is feeling sad, angry or mad, fearful or afraid” (Emotional Regulation Subscale).

The Student Rating Scale (SRS) [25] is 33-item scale in a 7-point Likert scale (1 = “not at all like this child” to 7 = “very much like this child”) assessing 11 different domains related to social-emotional learning and health promotion in children from 3 years of age. The domains are: (1) understanding of positive action, (2) self-concept, (3) physical health, (4) intellectual health, (5) self-management, (6) self-control, (7) respect, (8) considerate, (9) social bonding, (10) honesty and (11) self-improvement. In this research, we used a total score. This scale was completed by adult observers familiar with the child. Two researchers not involved in this research translated the questionnaire into Spanish and Catalan, separately. A third researcher resolved differences in translation. In this sample, the internal consistency was excellent (Cronbach’s alpha = 0.97, omega = 0.92). This instrument includes questions such as the following: “Is generally happy, outgoing, optimistic, confident, feels good about him/herself” (self-concept domain), “Shows self-control” (self-control domain) and “Persist in tasks, turns problems into challenges, receives suggestions” (self-improvement domain).

The Social Competence Scale—Teacher (SCS-T) [26] is a 25-item scale in a 5-point Likert scale (0 = “not at” to 4 = "very well”) assessing prosocial behaviors, emotional self-regulation and academic skills in children from 3 years of age. Each item states a behavior that a child may display at school. Respondents are the children’s teachers. The scale contains three subscales: prosocial/communication skills, emotional regulation skills and academic skills. The SCS-T can also be used on a one-dimensional scale assessing social competence and a total score can be calculated. Two researchers not involved in this research translated the questionnaire into Spanish and Catalan, separately. A third researcher resolved differences in translation. Inverse items were recoded so that a high score would indicate high possession in that dimension. We obtained for the total score a Cronbach’s alpha of 0.97. This instrument includes questions such as the following: “Understands others” (prosocial/Communication skills dimension), “Can calm down” (emotional Regulation Skills dimension) and “Is a self-starter” (academic skills dimension).

2.3. Procedure and Data Analysis

2.3.1. Qualitative Evidence Stage

The procedure for obtaining evidence of validity based on test content involved three key steps: the development of the items, their assessment by a committee of emotional education experts and the gathering of teachers’ opinions. In the first two steps, three elements were considered: the definition, representation and relevance of the domain [29].

The conceptual definition of the assessment questionnaire and the initial preparation of the items were derived from scientific literature (theoretical basis) and involved three professional researchers with expertise in this area. In January 2017, we queried online databases, such as PubMed, Embase, PsycINFO and the Cochrane Database of Systematic Reviews, for methods used to assess at least one of the five emotional competences in 3- to 5-year-old children. The search strategy can be seen in

Table 1. Search strategy for methods used to assess at least one of the five emotional competences in children aged from 3 to 5 years. January 2017.

Database	Search Strategy
PubMed	(“emotional intelligence” [All Fields] OR “emotional education” [All Fields] OR “emotion regulation” [All Fields] OR “emotion recognition” [All Fields] OR “emotion competence” [All Fields] OR “emotional awareness” [All Fields] OR “emotional regulation” [All Fields] OR “emotional learning” [All Fields] OR “emotional literacy” [All Fields] OR “socioemotional skills” [All Fields] OR “life skills” [All Fields] OR “emotional skills” [All Fields]) AND ((“evaluation studies” [Publication Type] OR “evaluation studies as topic” [MeSH Terms] OR “evaluation” [All Fields]) OR (“Assessment” [Journal] OR “assessment” [All Fields]) OR program[All Fields] OR (“Intervention (Amstelveen)” [Journal] OR “intervention” [All Fields] OR “Interv Sch Clin” [Journal] OR “intervention” [All Fields]) OR (“research design” [MeSH Terms] OR (“research” [All Fields] AND “design” [All Fields]) OR “research design” [All Fields] OR “test” [All Fields]) OR (“weights and measures” [MeSH Terms] OR (“weights” [All Fields] AND “measures” [All Fields]) OR “weights and measures” [All Fields] OR “scale” [All Fields]) OR tool[All Fields] OR (“Measurement (Lond)” [Journal] OR “measurement” [All Fields] OR “Measurement ( Mahwah N J)” [Journal] OR “measurement” [All Fields]) OR (“weights and measures” [MeSH Terms] OR (“weights” [All Fields] AND “measures” [All Fields]) OR “weights and measures” [All Fields] OR “measure” [All Fields]) OR instrument[All Fields] OR (“indicators and reagents” [Pharmacological Action] OR “indicators and reagents” [MeSH Terms] OR (“indicators” [All Fields] AND “reagents” [All Fields]) OR “indicators and reagents” [All Fields] OR “indicator” [All Fields])) AND ((“child” [MeSH Terms] OR “child” [All Fields] OR “children” [All Fields]) OR kids[All Fields] OR preschool[All Fields] OR (“child, preschool” [MeSH Terms] OR (“child” [All Fields] AND “preschool” [All Fields]) OR “preschool child” [All Fields] OR “preschooler” [All Fields]))
Embase	(“emotional intelligence” OR “emotional education” OR “emotion regulation” OR “emotion recognition” OR “emotion learning” OR “emotion competence” OR “emotional awareness” OR “emotional regulation” OR “emotional learning” OR “emotional literacy” OR “socioemotional skills” OR “life skills” OR “emotional skills”) AND (evaluation OR assessment OR program OR intervention OR test OR scale OR tool OR measurement OR measure OR instrument OR indicator) AND (children OR kids OR preschool OR preschooler)
PsycINFO:	(“emotional intelligence” OR “emotional education” OR “emotion regulation” OR “emotion recognition” OR “emotion learning” OR “emotion competence” OR “emotional awareness” OR “emotional regulation” OR “emotional learning” OR “emotional literacy” OR “socioemotional skills” OR “life skills” OR “emotional skills”) AND (evaluation OR assessment OR program OR intervention OR test OR scale OR tool OR measurement OR measure OR instrument OR indicator) AND (children OR kids OR preschool OR preschooler)
Cochrane Database of Systematic Reviews	http://www.cochranelibrary.com/cochrane-database-of-systematic-reviews/ (accessed on 15 January 2017)

. To be eligible, tests needed to meet the 5 criteria listed in

Table 2. Required criteria for tests to be eligible for selection. January 2017.

Criteria	Description of the Criteria
Criterion 1	At least one emotional competence was assessed.
Criterion 2	Free access to the test was authorized by the author/s.
Criterion 3	The respondents were teachers.
Criterion 4	Evidence of reliability and validity was provided.
Criterion 5	The test was used in at least one research project.

. The researchers reviewed the items of the selected tests and chose the ones that best fit the conceptual definition of the ECAQ questionnaire. They also brainstormed on missing content and created new items.

After the three researchers had selected and created the items, the multidisciplinary committee of seven experts evaluated them. We used the conceptual framework of the Delphi method [30].

In the first round, we sent the experts an initial list of items and a definition of each emotional competence we intended to evaluate with the questionnaire [31]. We asked them to classify each item into one of the five emotional competences. They also evaluated whether the items were appropriate (“Please specify whether each item is appropriate. An appropriate item should be well described and relevant to the competence where you have previously classified it”). In the case of inappropriate items, we asked the experts whether they would delete them or suggest modifications to improve them. We selected the items that all experts, or all but one, classified in the same emotional competence and redesigned the ones which most experts had said were not well built. This way, a second version of the questionnaire was obtained.

In the second round, we sent the seven experts the new version of the questionnaire. For each item, we asked if it was classified in the correct emotional skill and, if not, to indicate the reason. Then, we revised the given reasons. The experts also evaluated whether there were any similarities between items. In this case, we asked them to choose which items should be part of the questionnaire. Finally, we asked which ones the experts would eliminate, as the final version of the questionnaire should have had no more than 30 items. This limit was set considering that the teachers could spend a maximum of 10 min per child by answering 3 questions per minute. The experts also had the opportunity to suggest new items. Moreover, if at least half of them considered that two or more items were similar, we kept the one selected by the majority. Lastly, we deleted those that all or all but one expert said that should be eliminated. Then, we sent this version to four teachers to gather their opinions before sending it again to the experts. Therefore, a third version of the questionnaire was obtained.

In the third and final round, we sent the experts the third version of the questionnaire. We asked them what percentage of items each competence should have. From the responses, we calculated the average of the percentages of items that each expert indicated for a competence. We also asked the experts to give a score of 1 to 4 to each item to indicate its relevance for the skill to which it was attributed (1 = “not truly relevant”; 4 = “truly relevant”). Finally, we selected the items that had the highest average score and obtained a fourth version of the questionnaire.

At the end of the second round with experts, the resulting items were sent via e-mail to the teachers’ committee. We encouraged these teachers to suggest possible modifications or propose relevant aspects that should be considered but had not been included in the test. For each item, the teachers answered the question “what do you think is evaluated with this item?” They also indicated if they could answer referring to their children. In the case of a negative answer, they were encouraged to make a new proposal. Finally, they also had the opportunity to recommend new items or suggest modifications to the existing ones. If at least two out of four did not correctly explain what an item was intended to evaluate, or affirmed that they could not evaluate it, the item was removed.

To gather validity evidence based on the response processes, we tried to make relevant adjustments according to the performance and responses of the teachers’ committee. Interviews were sent to them by e-mail. First, teachers were asked to answer the questionnaire (the fourth version) about five children in their class. Once this phase was completed, the fifth version of the test was obtained.

To obtain evidence of validity based on the consequences of testing, we asked 52 teachers from 11 different schools in the city of Barcelona to answer the ECAQ for each child in their class (25 on average), at the beginning and at the end of the school year. To evaluate if there were any consequences that could bias the results obtained, we conducted five discussion groups with a maximum of 10 teachers in each one of them. Two questions were formulated in these groups: “Did you feel that your work was under evaluation and that you should change the score of a child if they got a low score in the test? In case you respond yes, in what way did you feel that your work was under evaluation?” and “Do you think that the fact of having to answer the test twice somehow influenced you in scoring the children? If yes, how do you think it influenced you?”

2.3.2. Quantitative Evidence Stage

The teachers (n = 52) did not receive any training before answering the ECAQ, but they received an official accreditation for the hours spent answering questionnaires. We then invited the teachers also to answer other questionnaires for the first 8 students on their class rosters because the task was very demanding and required a considerable effort in time. A sample of 342 children was finally obtained.

To study the internal structure of the questionnaire, we assessed four confirmatory factor analyses models that were compatible with theory with the R package lavaan [32]: a five-factor correlated model; a three-factor correlated model; a unidimensional model and a general factor with all items and three specific theoretical components (i.e., awareness, internal/external regulation and wellness). Considering that items had six response options and skewness and kurtosis values were above 1 point (see Supplementary Material; Figure S1), we used the unweighted least squares estimator and the Pearson’s correlation as input matrixes. To assess the goodness of fit indexes (GOFIs), we considered as excellent fit a value with a Tucker–Lewis index (TLI) and comparative fit index (CFI) above 0.95 and with a root mean square error of approximation (RMSEA) below 0.05 [33]. Alternatively, we considered as acceptable an RMSEA value below 0.08 [34].

To examine the relationship with other variables, we performed Pearson correlations: the ECAQ total score was correlated with other tools that measure similar constructs (i.e., SCS-T and SRS) or different constructs (i.e., ERC subscale Lability and Negativity).

To assess the ECAQ’s reliability, we evaluated internal consistency using the coefficient omega for the general factor with all items and three specific theoretical components (i.e., awareness, internal/external regulation and wellness) [35,36]. We also obtained Cronbach’s alpha to compare it with previous results. Furthermore, as recommended by Rodriguez et al. (2016) [35], we also computed the Explained Common Variance for the general factor. We considered values above 0.70 as acceptable [37]. We estimated the intraclass correlation coefficient (ICC) with the best values closest to 1.00. We chose absolute agreement ICC because it takes into account systematic differences as part of the measurement error, as there is no clear consensus regarding the cutoff point for the ICC.

To assess the test-retest reliability, the teachers answered the ECAQ for each child on two separate occasions (time lapse of one month). Neither the teachers nor the children who participated in this assessment had participated in the validity evidence part of the study. During the second ECAQ round, the respondents did not have access to the responses given in the first round to prevent any form of recall bias. The schools that participated in the test-retest reliability evaluation received a collection of books on emotional education in appreciation for their collaboration in the study.

3. Results

3.1. Evidence of Validity Based on Test Content

From our database search, we found 164 instruments used to assess at least one of the five emotional competences established by the GROP. After checking eligibility against our five criteria (Table 2), we selected various instruments from which we extracted 126 items that we considered best fit the model of the five emotional competences. We also created 12 new items to cover the aspects that were not addressed by the selected pre-existing ones. This way, we obtained an initial version of the questionnaire of 138 items.

The development of the ECAQ can be found in Figure 1. In the first round of seven experts, 67 items were selected, as they were classified in the same emotional competence by all experts, or by all but one. Of these items, we reformulated the ones that were considered not to be well developed by all experts (n = 7).

In the second round, 59 items were considered to be well classified by all the experts. On the contrary, the other eight items were considered not to be properly classified by at least one expert. Nevertheless, most of the experts had valued these eight items as well classified and the arguments offered by those who disagreed were not congruent with our theoretical model. For example, related to the item classified in the emotional awareness competence “The child is aware that emotions are related to how we behave”, an expert said it should be classified in emotion regulation competence although in no case does it refer to regulation strategies, and so we considered it was focused on awareness of the impact of emotion on behavior. Therefore, we decided to maintain these items in the competence in which they were initially assigned. Finally, we deleted 22 items that the experts considered similar to others that were chosen to be included in the test. Moreover, an expert proposed 11 new items that were included in the questionnaire. Finally, 56 items were obtained at the end of this round.

This version of the questionnaire was examined by four teachers. We removed nine items because at least two of the teachers did not correctly explain what the items were evaluating and/or they were not able to evaluate them. None of the teachers suggested new items; therefore, a test version of 47 was obtained.

In the third round, we assigned a percentage of 22.5% to emotional awareness (seven items), 22.5% to emotional regulation (seven items), 18% to emotional autonomy (5 items), 21% to social competence (six items) and 16% to life skills (five items). We finally selected the items with the highest average scores and 17 were removed to obtain a maximum of 30 in a preliminary version of the test.

3.2. Evidence of Validity Based on Response Processes

The average time teachers took to answer the test was 7 min per child (an average of 3 h for each class). Following a suggestion from teachers, we added the question “Does the child have any special educational needs?” We also changed the response categories to “never”, “rarely”, “sometimes”, “often”, “very often” and “always”, as they said that the suggested ones (“totally disagree”, “disagree”, “agree” and “totally agree”) made it difficult to answer the questionnaire.

3.3. Evidence of Validity Based on the Consequences of Testing

A total of 36 teachers agreed to participate in the five discussion groups. Eighty-nine percent of them (32 out of 36) did not feel that their work was under evaluation and that they should change it in case a child got a low score. A total of 94% (34 out of 36) said that the fact of having to answer the test twice did not influence them in the task.

3.4. Evidence of Validity Based on the Internal Structure

The GOFIs for all tested models are shown in

Table 3. Comparison of Fit Indexes in Models Fitted to Confirmatory Factor Analysis.

Model	X2	df	CFI	TLI	RMSEA	RMSEA 90%CI
5 factors	5990.54	395	0.98	0.98	0.11	[0.11–0.12]
3 factors	4296.12	402	0.99	0.99	0.09	[0.09–0.10]
1 factor	8737.37	405	0.98	0.97	0.14	[0.14–0.14]
Bifactor (3 specific)	2357.29	375	0.99	0.99	0.07	[0.07–0.07]

X²: chi-square; df: degrees of freedom; CFI: comparative fit index; TLI: Tucker–Lewis Index; RMSEA: root mean square error of approximation.

. First, consistently with the preceding literature and the development of the tool, we tested two models: (1) a five-factor correlated model and (2) a three-factor correlated model. The GOFIs were above the acceptable limits for the RMSEA values (0.11 and 0.09, respectively). A detailed analysis of the results of the two models (for the complete output, see Supplementary Material) indicated a lack of discrimination between the questionnaire factors. The correlation between the factors ranged between 0.77 and 0.98 for the five-factor model and between 0.78 and 0.87 for the three-factor model. Subsequently, a one-dimensional solution was evaluated, also obtaining high RMSEA values. Considering the previous results and the fact that the questionnaire was developed by experts who contemplated evaluating three to five dimensions, we evaluated a bifactor model with a general factor and with factors specific to emotional competences. The model tested with five specific and one general factors did not converge (results not shown). On the other hand, the model with three specific and one general factor yielded suitable GOFIs.

Figure 2 shows the results for the standardized solution of the bifactor model. Factor loadings were high for the general factor (range 0.60–0.95) but not for the three specific factors. The factor loadings of the awareness factor ranged between 0.00 and 0.52. The correspondence between the factors of the bifactor model and the item content can be found in

Table 4. Item content for the Emotional Competence Assessment Questionnaire (ECAQ) and factor loadings for general factors and for specific factors.

Factor	Item	Factor Loadings for General Factor	Factor Loadings for Specific Factors
awa1	The child can express why he/she or another person can feel a certain emotion	0.73	0.51
awa2	The child is able to communicate aspects of himself/herself of which to feel proud about, both when asked and spontaneously	0.73	0.48
awa3	The child identifies body signs (facial expression, posture, and tone of voice) reflecting every emotion	0.74	0.32
awa4	The child can explain what emotion he/she is feeling	0.75	0.52
awa5	The child knows that different people can feel different emotions when facing the same situation	0.82	0.13
awa6	The child is aware that what he/she does can affect how others feel	0.83	0.05
awa7	The child recognizes that emotions can be preceded by a specific situation	0.95	0.07
awa8	The child is able to make coherent decisions considering how he/she feels, what he/she needs, and the impact that the decision will have	0.88	0.00
awa9	The child is aware that his/her emotions influence subsequent behaviors	0.89	0.02
reg1	The child controls his rage when there is a disagreement	0.65	0.57
reg2	The child waits his/her turn	0.49	0.62
reg3	The child can autonomously control his/her excitement not to bother the others	0.60	0.65
reg4	The child manages his/her negative emotions independently, not lingering on them	0.75	0.36
reg5	The child uses appropriate strategies to control anxiety	0.77	0.39
reg6	When the child has a problem, he/she strives to see the positive side of things	0.79	0.24
reg7	The child demonstrates self-control	0.68	0.59
reg8	The child treats the others fairly and tolerates differences	0.71	0.42
reg9	The child is able to share things	0.63	0.50
reg10	The child can soothe himself/herself when excited or angry	0.67	0.57
reg11	The child is able to accept and tolerate that things do not turn out as he/she expected	0.68	0.53
reg12	The child can do an activity for a while, enjoying it and concentrating while being oblivious to what happens elsewhere	0.74	0.19
wel1	The child usually has a positive attitude	0.76	0.21
wel2	The child helps the others	0.76	0.09
wel3	The child is able to initiate social contacts and interactions	0.71	0.40
wel4	The child seems to be emotionally well	0.72	0.42
wel5	The child is able to enjoy	0.74	0.50
wel6	The child has a positive self-image	0.71	0.54
wel7	The child has reasonable self-expectations considering his/her age and the period of life	0.83	0.20
wel8	The child feels confident and self-confident	0.72	0.42
wel9	The child shows good social skills with friends	0.76	0.38

Awa: awareness, reg: regulation, wel: wellness.

. Similarly, the factor loadings of the well-being factor ranged between 0.19 and 0.65. Finally, the factor loadings for the well-being factor ranged between 0.09 and 0.54. This analysis gives favorable evidence for the unidimensionality of the ECAQ if specific factors are considered for modeling the common variance of theoretical variance.

3.5. Evidence of Validity Based on Relations to Other Variables

Table 5. Descriptive Statistics and Convergent Validity Evidence.

	M	SD	Min	Max	ECAQ	1	2	3	4	5	6	7
	ERC Subsample (n = 134)
ECAQ	117.69	27.44	49	177	1.00
Total ERC (1)	50.49	8.13	27	78	−0.29	1.00
Lability/Negativity (2)	23.28	7.30	14	55	−0.62	0.84	1.00
Regulation (3)	23.27	4.46	11	32	0.63	0.22	−0.33	1.00
	SRS Subsample (n = 119)
ECAQ	126.73	27.87	39	178	1.00
Total SRS	172.18	31.29	62	219	0.79
	SCS-T Subsample (n = 89)
ECAQ	123.75	29.08	63	179	1.00
Total SCS-T (4)	82.16	18.58	43	119	0.83				1.00
Prosocial (5)	25.39	6.00	13	37	0.83				0.95	1.00
Regulation(6)	32.75	7.73	19	48	0.80				0.95	0.88	1.00
Academic (7)	24.01	5.96	10	34	0.70				0.92	0.83	0.80	1.00

ECAQ: Emotional Competences Assessment Questionnaire; ERC: Emotion Regulation Checklist; SRS: Student Rating Scale; SCS-T: Social Competence Scale—Teacher; M: mean; Min: minimum value; Max: maximum value; SD: standard deviation.

shows descriptive statistics and correlations between the main variables. As can be seen, the children obtain scores above the mid-point on the ECAQ. The same occurs with the scores of the ERC, SRS and SCS-T. Therefore, the children in the sample show good emotional competence scores in both the ECAQ questionnaire and the other questionnaires used.

To obtain evidence of the convergent and discriminant validity of the ECAQ, the correlation of the total scale score was evaluated against that of other questionnaires (ERC, SRS and SCS-T). As expected, the total score of the ECAQ showed a high negative correlation with the subscale Lability and Negativity (−0.62). This shows that the higher the emotional competence (measured with the ECAQ) which includes the ability to regulate negative moods and enhance positive ones, the lower the emotional instability and tendency for negativity. We also found a high positive correlation with the subscale Emotional Regulation (r = 0.63), showing that greater emotional competence (measured with the ECAQ), which includes the ability to manage emotions, correlates positively with the subscale measuring emotional regulation. We found high correlations with the SRS (r = 0.79) and the SCS-T (r= 0.83) and with the SCS-T subscales (range: 0.70–0.83), showing that when we obtain high scores in emotional competence measured with the ECAQ, we also obtain them when measuring socioemotional learning and social competence measured with the SRS and the SCS-T. Therefore, we can say that we have positive evidence of convergent and discriminant validity.

3.6. Reliability

Regarding internal consistency reliability, a hierarchical omega of 0.90 for the general factor and a Cronbach’s alpha of 0.95 were obtained, which were considered excellent. Furthermore, the explained common variance results showed that 77% of the variance was accountable to the general factor.

Figure 3 shows the test and retest scores and the linear regression line (y = 40.74 + 0.62 × test). To assess the absolute agreement between both scores, the test-retest reliability was evaluated with the ICC. The correlation obtained was 0.78 and the ICC was 0.76 (95% Cl = 0.67–0.83), which can be considered good.

4. Discussion

This study develops and provides evidence of the validity and reliability of the ECAQ. The ECAQ is based on Bisquerra’s theoretical model of emotional competence, which distinguishes, as recommended in the literature, between the intrapersonal and interpersonal domains in emotional competence. The process of development has shown positive evidence of the test content, response process and consequences of testing. On the other hand, the provided quantitative evidence of psychometric quality supports using a general score to measure the emotional competence of children three to five years old. Moreover, results show positive evidence of convergent and discriminant validity and excellent reliability indicators.

The development of this test has not only been a result of a bibliographic review from Bisquerra’s theoretical model but also of the participation of experts and the opinions of the target population. Moreover, this is aligned with the recommendation for stakeholder collaboration in the measurement of socioemotional learning to be consistent, developmentally appropriate and culturally sensitive [38]. Furthermore, we evaluated the response processes and consequences of administering the test to verify that the scores were not biased in any way. Finally, this study suggests that ECAQ could be a quicker and easier questionnaire to administer, in comparison to others that are widely used in research. For instance, administering the Affect Knowledge Test (AKT) [39,40]) or the Emotion Matching Task (EMT) [41,42] requires 15–20 min per child, whereas the average time for the ECAQ is 7 min. Additionally, for the AKT or the EMT, the researcher administers the test to the children; on the contrary, for the ECAQ, the teachers are the respondents. This has several advantages: the criteria are the same for all the children in the same class, the teachers know them better than an outside researcher, and the method is more economical. Our study of the internal structure of the ECAQ yielded positive evidence for only one general factor of competence. Although, a priori, our approach started from the development of a questionnaire based on the five competences proposed by Bisquerra [22], this result could be considered positive. The questionnaire contemplates all the domains of the necessary construct, and therefore, finding a unidimensional structure could be considered a success. We should bear in mind that the author himself indicates that the five competences included in their model overlap in some way with each other, forming as a whole the construct of emotional competence. It should be noted that discerning between emotional competences remains challenging at such early ages. In fact, the concepts of emotional intelligence and emotional competence and the theory surrounding both concepts have been evolving over the last three decades, generating extensive knowledge that allows for the improvement of emotional education [43]. However, both constructs are constantly under debate [23]. This implies that the different instruments that are emerging to measure them are likely to differ, even slightly, in what they are measuring and that it is difficult to know whether we are measuring emotional competence partially or completely. Different scales are used to measure emotional competence, often developed ad hoc to measure the outcome of a program or an intervention [44,45,46]. This makes it difficult to compare the results of such interventions as well as the benefits of one or another instrument. In spite of this, the predictive value of measuring social competence in the preschool stage is relevant [47] and it is therefore necessary to continue advancing in this field despite the complexity of comparison between instruments and constructs [48] and to try to arrive at a consensual conceptual framework [49].

As has been said, the ECAQ is a simple, short and easy-to-answer questionnaire comprising 30 questions. According to our study, the questionnaire is capable of yielding valid and reliable scores. Therefore, the ECAQ could be a good candidate for large-scale studies and health surveys to map where children’s emotional competences are the weakest. This would allow us to prioritize where to intervene to boost those competences related to health, especially mental health [50,51], in a way that contributes to reducing social inequalities in health [52,53]. In addition, the high-reliability values obtained would even allow for taking individual decisions following the criterion of Nunnally [37].

Regarding validity, as expected, the total score of the ECAQ correlated positively with the Emotional Regulation subscale of the ERC and the total scores of the SRS and SCS-T. The observed negative correlation with the Lability and Negativity subscale of the ERC was also in line with expectations. This is because emotional lability and negativity involve the difficulty to regulate oneself emotionally and to enhance positive moods, and emotional competence includes the ability to regulate unpleasant emotions and to enhance pleasant ones [22]. The ECAQ focuses exclusively on emotional competence, whereas the SRS incorporates items related to health promotion, and the SCS-T includes items related to academic skills. Given that the aim is to assess emotional competence, the fact that aspects other than emotional competence are not assessed is a positive aspect. In this way, the results also respond to the construct to be measured without being contaminated by other aspects that, although related to emotional competence [3], are not part of it. At the same time, the ECAQ incorporates several dimensions of emotional competence, which differentiates it from questionnaires such as the ERC, which focuses solely on emotional regulation, leaving aside key aspects of emotional competence (e.g., emotional regulation).

This study has some limitations: (1) When selecting the items in the development process, we did not review any questionnaire that required payment to access it. Notwithstanding, more than 150 tests, scales and questionnaires were reviewed and experts in emotional education and teachers were asked if there was any key aspect missing. (2) We gathered the opinions of just four teachers for the evidence of validity based on test content and of just three for the evidence based on response processes. However, most items were adapted from other validated tools, and we assured the participation of a teacher from each age group (3- to 5-year-old children). In addition, from the 56 items that these teachers valued, 47 items (84%) were clear to all of them and so self-explanatory. (3) This questionnaire would not be appropriate for children with special educational needs; however, we will study it further to see how it could be adapted for them. (4) We focused on studying the consequences of administering the questionnaire, but the consequences of testing should also include the effect of communicating the results and making decisions based on them. (5) We only have data on children in the city of Barcelona. This implies that our results may not be representative of other realities. (6) For the study of the relations to other variables, we have a relatively small sample if we compare it with the sample used to study the internal structure of the ECAQ. Despite this, we have enough power to perform the aforementioned analysis. (7) The fact that families have not been able to participate could be a limitation for not having a complete vision of the child. However, some studies suggest that early childhood teachers, with appropriate training and support, can make reliable judgments about children’s competence [54]. (8) Related to the limitations of the questionnaire itself, it must be said that it was designed for epidemiological studies and therefore the time commitment in such a case may be high overall if teachers have to answer the questionnaire for 20–25 students in their class.

In conclusion, given the importance of emotional competence in mental health [1] and the need to know its distribution over the population, and taking into account the creation of school interventions to increase it [3] and the need to evaluate whether they are effective [8] in improving emotional competence, the ECAQ is postulated as a good instrument. This instrument, the development and validation of which is presented in this study, covers the unmet public health need for a quick questionnaire answered by teachers to assess the five emotional competences proposed by Bisquerra [22] in children aged 3 to 5 years. Obtaining evidence of validity based on the test content, response processes and consequences was essential to develop the ECAQ. As has been seen, the ECAQ has shown favorable evidence of psychometric quality. Therefore, the ECAQ could be considered a psychometrically sound questionnaire to evaluate the emotional competence of three- to five-year-old children based on their teachers’ records. From a public health perspective, having a quick and easy-to-apply instrument that is validated with good psychometric properties in a big sample could allow future research to evaluate the population of 3- to 5-year-olds in cities of Spain. As the ECAQ can be applied on a large scale, this could allow researchers and policymakers to identify which areas of a city require more effort to increase the emotional competence of the children. This questionnaire can also be used to evaluate the interventions and programs implemented at these ages, which is essential in public health [8], allowing us to know if they improve the emotional competence of the target population as well as to understand the challenges and limitations of implementation efforts in order to sustain and improve them [55].