When Video Improves Learning in Higher Education

Abstract

The use of video in education has become ubiquitous as technological developments have markedly improved the ability and facility to create, deliver, and view videos. The concomitant pedagogical transformation has created a sense of urgency regarding how video may be used to advance learning. Initial reviews have suggested only limited potential for the use of video in higher education. More recently, a systematic review of studies on the effect of video use on learning in higher education, published in the journal Review of Educational Research, found, overall, effects to be positive. In the present paper, we critique this study. We reveal significant gaps in the study methodology and write-up and use a cognitive processing lens to critically assess and re-analyse study data. We found the results of this study to be only applicable to learning requiring lower-level cognitive processing and conclude, consistent with prior research, that claims of a universal benefit are not yet warranted.

1. Introduction

In recent years, the use of video in education has rapidly expanded. As an increasingly important pedagogical tool, the pace of its penetration into educational processes has outstripped the ability of researchers to evaluate its effectiveness [1]. In one recent and significant effort to evaluate the effect of video usage in education, Noetel et al. [2] conducted a meta-analysis titled, ‘Video improves learning in higher education: A systematic review’, published in the journal Review of Educational Research and directed at the use of video as either a replacement or supplement to classroom learning.

Given the methodological approach of the study, it was reasonable to expect that greater clarity would be achieved with video, at least in higher education. Yet a close inspection of the study reveals significant gaps that call into question the overall study findings. As accurate research findings are urgently needed to inform policy decisions, this paper sets out to critically assess the aforementioned study and conduct a re-analysis of associated study data. In our critique and re-analysis we use a cognitive processing lens to build on Noetel et al.’s work and provide some needed clarity to better inform future educational research and policy development.

Initial Critical Assessment and Review of the Literature

In early media reports, Noetel et al. [3] characterized their findings by stating the use of video was ‘consistently good for learning’ and later, in the published study, stated the effect was ‘unlikely to be detrimental and usually improve student learning’ ([2]: p. 204). Though such a disparity in characterizations may raise concern, of further interest was how their overall study conclusions appeared to contradict prior research, suggesting that the video medium has only limited potential for advancing student learning in higher education.

For example, from a stronger methodological basis, prior research regarding the use of video, claiming to be the first controlled experiment of its kind, yet not covered by Noetel et al., has shown detrimental learning effects for some minority and other student demographic groups when relying on videos for instruction [4]. These findings mirror a recent meta-analysis for a related medium: low-SES school students were found to be disadvantaged by the use of screens as compared to paper-based books [5]. As an important issue regarding generalizability, Noetel et al. did not account for any participant demographics in their meta-analysis.

Moreover, related prior reviews on the use of video in education also suggest limited potential for advancing learning. For example, an early review by Hansch et al. [6] presented critical reflections on the state of the field. Based upon a review of the literature, personal observations, and 12 semi-structured interviews, largely focused in the context of higher education, it was concluded there was ‘little conclusive research to show that video is indeed an effective method for learning’, recommending consideration of a variety of pedagogical resources rather than a simple reliance on video (p. 10). Their conclusions were later confirmed in a systematic review undertaken by Poquet et al. [7]. In this review, 178 papers published between 2007 and 2017 were selected using strict inclusion criteria, all experimental and case studies conducted in the context of higher education and professional learning. Their detailed descriptive analysis summarized the effects of a variety of interventions on a variety of learning outcomes, with results highlighting some of the complexities involved in undertaking this research. In particular, among other variables, they suggest that the effect of video-based teaching is dependent on the nature of the knowledge to be learned, noting the effectiveness of video-based learning may vary depending, for example, on whether learning objectives involve simple recall vs. comprehension.

Indeed, an emerging body of research has found the efficacy of different instructional media varies depending on the nature of the associated learning task [8,9,10]. For example, in early research, ChanLin [8] investigated (n = 135 undergraduate students) the use of three visual treatments (no graphics, still graphics, and animated graphics) using co-variates of learner prior knowledge (high vs. low) and the nature of knowledge being learned (procedural vs. descriptive facts), with results supporting early claims that visual treatment effects vary according to the nature of knowledge being learned. Particularly of note, the use of visuals did not always guarantee successful learning. In related research, Garrett [9] used a novel data-mining approach to analyse PowerPoint files (n = 30,263) and understand differences in slide presentations relative to academic discipline. Though focused on the teaching approach rather than the learning effect, the nature of the discipline was found to significantly predict how text and graphics were used. Finally, Hong, Pi, and Yang [10], in a randomized controlled experiment, examined the learning effectiveness of video lectures (n = 60 undergraduate students) using co-variates of knowledge type (declarative vs. procedural) and instructor presence (with vs. without). The results suggested that ‘the learning effectiveness of video lectures varies depending on the type of knowledge being taught and the presence or absence of an instructor’ (p. 74). Taken as a whole, prior research suggests the nature of learning in each study context is an important moderating variable needed to properly understand the effects of video on learning.

Despite such efforts, the cognitive processes and mechanisms underlying the use of video remain poorly understood. Some insight, however, may be gained from research directed at related media. For example, almost a century after its invention, television viewing continues to be associated with a weakened cognitive investment [11,12,13,14]. Indeed, some argue television viewing is conditioning people to use ‘poorer executive functioning alongside automatic processes that may be erroneous and even difficult to undo’ ([15], p. 3019). Such claims and findings appear consistent with cognitive research on the use of images, found to be processed much faster and automatically compared to the slower and more controlled processing of printed text [16,17]. All this may suggest that the video medium is priming students to rely on quicker intuitive ‘feelings of rightness’ ([18], p. 236) rather than engaging in slower, deliberative reflection [15]. This dynamic may explain why the use of video is effective for teaching in some knowledge areas but not others.

Reflecting on the state of current research in relation to Noetel et al.’s review, we recognize two important shortcomings. First, Noetel et al. considered learning tasks as a relatively simplistic ‘skill’ vs. ‘knowledge’ dichotomy (later characterized in their review as ‘teaching skills’ vs. ‘transmitting knowledge’; p. 222). Second, complicating the ability to interpret their findings, they provide very limited information about the educational contexts represented in their meta-analysis, which they refer to as ‘learning domains’, providing no definition for what is meant by a learning domain and, perhaps most surprising, no descriptive analysis or clear summary of domains included in the study. In sum, their review employed an analytical framework that was not based in theory but one reflecting a relatively simplistic view of the nature of knowledge and learning. Additionally, in what may have helped interpret their findings, little information was provided concerning the learning contexts represented in their meta-analysis.

Given this assessment, we executed a close investigation of the Noetel et al.tudy data (available on bit.ly/betteronyoutube), asking two research questions:

RQ1. What is the nature of the learning contexts covered in the study as suggested by (i) a basic descriptive analysis of the Noetel et al. data and (ii) a descriptive analysis by way of using a relevant established theoretical framework?

RQ2. What does a re-analysis of the data tell us about how the use of video affects learning in higher education when the aforementioned theoretical framework is employed?

We first present the methodology, results, and some discussion for each research question. Following this, we present a summary discussion where we conclude, consistent with prior research, that the use of video has limited potential for advancing learning in higher education.

2. Methodology and Results

Alongside the associated methodology, we present the results of our re-analysis in the following subsections.

2.1. Research Question 1

As previously discussed, the original study write-up provides limited information about the learning contexts represented by the included studies. This led us to seek a clearer understanding of the contexts covered by the review.

We first made use of the original source data and learning domain categorizations (as categorized by Noetel et al.) to present a simple descriptive analysis of the contexts, the results of which may be seen below in

Table 1. Learning domains as classified by Noetel et al. ^a.

Learning Domain (as Categorized by Noetel et al.)	Tally	Percent (1 d.p.)
biology	3	2.8
computer science	2	1.9
dentistry	8	5
engineering	1	0.9
English as a foreign language	5	4.7
medicine	52	49.1
nursing	13	12.3
nursing, paramedicine	1	0.9
nutrition	1	0.9
pharmacy	1	0.9
physical education	1	0.9
physical therapy	4	3.8
physics	1	0.9
physiotherapy	1	0.9
psychology	4	3.8
psychology, education	1	0.9
sign language	1	0.9
sport science	2	1.9
teaching	4	3.8
Total	106	100

^a See bit.ly/betteronyoutube > Supplementary File 3: Characteristics of Included Studies, Consensus Extraction and Risk of Bias Spreadsheets, supplementary file > column titled ‘learning_domain’.

.

From this basic analysis, it is clear, consistent with expectations from the literature (e.g., [19]), that more than 80% of included studies were in health science contexts (e.g., medicine, nursing, dentistry). As a broad context, learning in the health sciences has been found to focus mostly on lower-level cognitive processes, such as learning facts and procedures (e.g., Medicine: [20,21,22]; Nursing: [23]; Dentistry: [24,25]), as typically revealed using the lens of Bloom’s [26] taxonomy of cognitive learning objectives (Bloom’s objectives have been categorized as remember, understand, apply, analyze, evaluate, and create, in order of cognitive complexity from those requiring lower to higher levels of cognitive processing [27,28]). For example, approximately half (49%) of included studies were classified in the learning domain of ‘medicine’, an area of study long known for its ‘persistent focus’ on learning ‘factual minutiae’ ([20], p.1343). In other words, a simple descriptive analysis quickly made clear that the vast majority of included studies focused on learning contexts targeting lower-level cognitive processing. Indeed, virtually all (102 of 106 or 96.2%) of the learning domains relate to what we would categorize as professional degree programs. This skewed representation raised some concern regarding generalizability across higher education, which prompted us to take a closer look at the nature of learning represented in the meta-analysis.

To undertake this investigation, several potential theoretical frameworks were considered, including the seminal works of Bloom [26], Biggs [29] and Biglan [30]. The latter, a taxonomy for classifying academic disciplines in higher education, was identified as a clear choice given the nature of the available data. Indeed, strengthening this selection, Biglan’s [30] framework is perhaps the most well-known system for classifying academic disciplines in higher education [31]. Moreover, the taxonomy was originally developed to provide a ‘framework exploring the role of cognitive processes in academic fields’ ([30], p. 202) and has repeatedly demonstrated its validity in subsequent research [31,32,33]. Importantly, as it relates to our research questions, and notwithstanding further complexities [34], prior research using this framework has found generalities and differences regarding the nature of learning within and between disciplinary contexts [19,35,36,37].

We first make use of this framework to categorize each of the 106 studies included in this review and demonstrate how the included studies represent a relatively limited learning focus. Our results, displayed in

Table 2. Learning domains as classified using Biglan’s taxonomy ^a.

	Hard		Soft
	Life	Nonlife	Life	Nonlife	Totals
Pure	3	2	2	5	12
Applied	63	11	20	0	94
Totals	66	13	22	5	106 b
	79		27

^a Stoecker’s [33] revision was used to classify previously unclassified domains of dentistry and nursing. ^b Number disparity due to Study Number 17, representing two study contexts, being counted twice (see bit.ly/betteronyoutube).

below, indicate that almost all (94 of 106 or 88.8%) learning domains were confined to teaching and learning contexts in the applied sciences where, consistent with our previous findings, learning has been associated with lower-level cognitive processes [38]; see also, for example, [39]. Moreover, a closer look at the 12 remaining studies, all in pure disciplines, similarly suggests a focus on lower-level learning. This includes, for example, learning facts about biology (for an introductory microbiology course) or the correct procedures for using statistical software (for a psychology course). We conclude, based on the use of Biglan’s framework, that lower-level learning was targeted by the vast majority, if not all, of the learning contexts represented in this review.

2.2. Research Question 2

We next make use of Biglan’s framework by undertaking a meta-regression re-analysis of the data sets behind figures 2 and 3 in the review investigating, respectively, the effect of using video as a replacement for and supplement to live instruction (see [2], p. 214 and 218, respectively; the original study R code and files related to these two data sets are found in the repository linked to the original paper (bit.ly/betteronyoutube)). Related data sets are termed ‘swap’ (i.e., video as a replacement) and ‘sup’ (video as a supplement). However, in our re-analysis, we include the three levels from Biglan’s classification shown in Table 1 above as an additional moderator variable (i.e., hard vs. soft, pure vs. applied, and life vs. nonlife).

2.2.1. Statistical Methodology

Our re-analysis was conducted via meta-regression (MR). MR is a regression model applied to data obtained from a meta-analytic study in which, most likely, the dependent variable is numeric and corresponds to effect sizes. The regression model is usually the ordinary least squares linear model, but other alternatives exist when parametric assumptions such as normality and homoscedasticity are not met (mainly, the distribution of the residuals is not normal). In such cases, a linear (mixed) model would give biased results; thus, non-parametric or semi-parametric approaches are recommended.

In this re-analysis, a generalized additive model for a location, scale, and shape (GAMLSS; [40]) approach was first used. The GAMLSS approach allows examining the effects of covariates on the dependent variable’s location, scale, skewness, and kurtosis parameters. GAMLSS is a form of supervised machine learning that allows for flexible regression and smoothing models to fit the data [41].

For the sake of simplicity, we focused on the effects of the covariates on the location parameter of the dependent variable, assuming this is best described by the four-parameter Skew Power exponential type 2 (SEP2) distribution [42]. Although we found the SEP2 distribution fit the data well, the results of the model fit were not convincing due to the shape of the residuals, thus suggesting the adoption of a non-parametric approach. We thus chose a robust linear mixed model (RLMM; [43]) as a second analytical approach. RLMMs were consequently used for all our analyses. As the current R implementation of the RLMM does not allow ANOVA-type outputs, pairwise differences were examined via multiple comparisons [44] and boxplots (for details in relation to the modelling, see the Supplementary Files at https://cutt.ly/rUbeOPa, accessed on 26 March 2024).

The model we investigated had the following structure:

$$DV\sim v1+v2+\dots +\left(1|rv\right)$$

That is, the model was additive (i.e., no interactions are included), the dependent variable ($DV$) was numeric, and there was a random (intercept) variable ($rv$). In order to investigate parsimonious models (i.e., models with few covariates), the number of covariates included variables that seem essential to the model according to the results by Noetel et al. (see their Table 1 on p.). That is, no variable-selection method was pursued.

Thus, as applied to the original data sets, the final model investigated was

$$\begin{gathered} smd\sim hard\_vs\_soft+applied\_vs\_non.applied+life\_vs\_nonlife+Setting+Comparison+Outcome \\ +Which\_is\_more\_interactive+Topic\_or\_course+\left(1|studynumber\right) \end{gathered}$$

2.2.2. Results

We summarize our major results in this section (for more detailed results, see https://cutt.ly/rUbeOPa, accessed on 26 March 2024). Overall, as may be expected given the learning contexts uncovered in RQ1, the effect sizes remained positive. However, despite the relative data homogeneity, our results demonstrate much greater complexity associated with learning via video. In particular, as we conclude in this section, we found significant differences emerging between major disciplinary groups.

To begin, when video was used as a replacement, we found several significant differences emerge. First, differences in effect sizes were found between ‘educational settings’; particularly between ‘tutorial’ and ‘homework’, where the use of video was found to be more useful with homework than with tutorials (Mdn_homework = 0.59, 95% CI [0.52, 0.65]; Mdn_tutorial = 0.40, 95% CI [0.35, 0.46]; t_permutation = −3.02, p = 0.018). Second, in contrast to the original study findings where the use video was found more effective when ‘skill acquisition’ was assessed (vs. knowledge), no difference in effect was found between the two types of outcome assessments (Mdn_knowledge test = 0.16, 95% CI [0.06, 0.26]; Mdn_{skills assessment} = 0.27, 95% CI [0.13, 0.42]; t_permutation = 1.26, p = 0.238).

Next, when investigating the use of video as a supplement to existing content, other new findings emerged. First, significant differences were found in effect sizes between educational settings: mixed (‘Mixed’ is a term used in the original data set, though not explained in the main manuscript) and homework (Mdn_mixed = 0.30, 95% CI [0.22, 0.39]; Mdn_homework = 0.56, 95% CI [0.46, 0.65]; t_permutation = −2.80, p = 0.046) and between mixed and tutorial (Mdn_mixed = as above; Mdn_tutorial = 0.68, % CI [0.60, 0.75]; t_permutation = 4.85, p < 0.001). Second, there was an effect of comparison such that there was a difference between ‘human’ (or teacher) and ‘static media’, with static media found to be more effective as a supplement than human input (Mdn_human = 0.30, 95% CI [0.08, 0.52]; Mdn_{static media} = 1.07, 95% CI [0.80, 1.34]; t_permutation = 5.76, p < 0.0010). Third, the difference between the type of outcome was borderline at the 0.05 level, with video supplements found to be more helpful for skill assessments than knowledge tests (Mdn_{knowledge test} = 0.54, 95% CI [0.28, 0.80]; Mdn_{skills assessment} = 1.05, 95% CI [0.86, 1.23]; t_permutation = 2.16, p = 0.048).

Finally, despite the relatively homogeneous nature of the original study data, we found important differences in effect sizes emerge between major disciplinary subgroups. In particular, when swapping video for any other learning opportunity, the results indicate that soft learning domains had significantly larger effect sizes than hard learning domains (Mdn_{soft domain} = 0.37, 95% CI [0.19, 0.55]; Mdn_{hard domain} = 0.15, 95% CI [0.07, 0.23]; t_permutation = 2.69, p = 0.008). However, somewhat in contrast, when videos are provided in addition to existing content, hard learning domains tended to have larger effect sizes than soft learning domains (t_permutation = −1.76, p = 0.08). We now turn to discussing these results in light of Noetel et al.’s study findings.

3. Discussion

The original study concluded that the effect of video on learning in higher education was generally positive. Noetel et al. concluded this effect based on an analysis which employed no theoretical framework for categorizing their data while providing little contextual information concerning the source of that data. As a methodological issue, this approach was surprising given that the use of theory and the importance of contextualizing findings are considered basic research practices. Given these issues, we set out to examine the study data more closely and conduct a re-analysis using a relevant theoretical framework.

The results of our re-analysis were at variance with Noetel et al.’s findings. First, in our descriptive analysis, we found almost all included studies were in contexts where the associated learning may be characterized as involving lower-level cognitive processes, such as learning facts and procedures. Second, in our meta-analysis using an established theoretical framework, though effect sizes remained positive, we found greater complexity around how video was used and its effects relative to the learning contexts. Taken together, from a cognitive processing perspective, we did not consider the rediscovery of positive effect sizes as surprising given the relative homogeneity of the original study data, but an affirmation of the suggestion that, overall, the use of video in higher education benefits learning requiring lower-level cognitive processing.

Indeed, we suggest significant negative effect sizes would emerge if learning requiring higher-level cognitive processing was adequately represented in the original review. These are, for example, disciplines typically associated with abstract reasoning, such as pure mathematics [45], where associated cognitive demands are known to be high [46,47]. For example, related meta-analytic research comparing distance education to live classroom instruction has found mathematics instruction ‘best suited to the classroom’ ([48], p. 400). Indeed, regarding the specific use of video, recent consecutive systematic reviews have found, overall, student use of recorded lecture videos (RLVs; RLVs are experiencing rapid growth [49]. For both systematic reviews, included studies permit individual students to use RLVs as a supplement to and/or replacement for attending live lectures) in undergraduate mathematics negatively correlated with academic performance [50,51] (Note: pure mathematics learning contexts were not represented in Noetel et al.’s review. For comparison, the review included only two studies in a pure discipline (i.e., both biology) where the comparison involved student performance when using recorded-only vs. live-only lectures [52,53]. Both reported negative effects), with some early research supporting causality [15]. In particular, RLVs appear to enable students to engage in surface learning—such as rote memorization—of course content, which leads to poorer academic performance [54,55]. As early research suggests, mathematics students approach the use of RLVs in similar ways as they approach viewing television [15], weakening their cognitive investment [11,12,13,14]. In sum, though such approaches may be sufficient to undertake tasks involving lower-level cognitive processing, such as learning facts or acquiring procedural knowledge, they may be detrimental when tasks require higher-level processes, such as acquiring richly connected conceptual knowledge [56].

4. Future Research

The effects of a screen-based video medium on the learning process remain poorly understood. Further exploration of factors influencing the video-based learning process is needed. As the results of our re-analysis point out, this includes distinguishing between whether videos are used as a replacement for or supplement to live instruction. Further to this exploration, a variety of theoretical lenses may be explored. In our view, self-regulation theories show particular promise for future research. This is because much of the current video-based teaching, such as RLVs, is delivered asynchronously with students mostly responsible for monitoring, judging, and controlling their learning (see, for example, [57]; to be clear, we do not suggest the synchronous delivery of video-based teaching to be free from issues related to those we discuss in this section. For simplicity, we focus on asynchronous delivery not least because of its prevalence in video-based teaching in higher education).

Moreover, when a video is delivered asynchronously, the experience is obviously one-way: the teacher presents material but does not interact with students in real time. This, for example, denies teachers the ability to ‘read their audience’ and adjust pacing or the method of how new concepts are scaffolded. Furthermore, if, for example, only the teacher’s head is shown in the video, students may be denied additional resources, such as hand gestures, considered a support to the learning process [58,59].

Notably, as a proxy for learning involving higher-level cognitive processing, those researching learning in mathematics have highlighted the nature of interactivity as vital to the learning process (e.g., [60,61,62]). Indeed, deeper learning in mathematics has been theorized as a form of interactivity involving iterative cycles of discussion, feedback, and reflection [63,64,65]. When this interactivity is almost entirely regulated by the student, this presents one plausible reason for diminished learning outcomes in knowledge areas requiring high-level cognitive processing, as student objectives (e.g., time efficiency) and their regulation of resources may be at odds with the teacher’s target outcomes (e.g., depth of understanding). In consideration of all these factors, the use of self-regulation theory may yield important new insights.

We further hypothesize, as framed by self-regulated learning, that the video medium may cue learners to a weakened cognitive investment, inhibiting learners from undertaking higher-level cognitive processing and thus achieving deeper learning. By contrast, consistent with work on self-regulated learning, these learners may be under the illusion of achieving the goal of understanding, even though their thinking is actually poor or even incorrect. The resulting dynamic is thought to lead to cycles of ‘poor self-regulation and lower levels of achievement’ ([57], p.427). Testing such a hypothesis presents an important avenue for future research

Finally, some may envisage addressing current issues by leveraging AI technologies to direct the presentation of video-based teaching. As this potential remains unclear (e.g., [66]), more research is needed to understand how AI may be used to assist teaching via video.

5. Conclusions

In sum, understanding the effects of any pedagogical innovation involves the unravelling of a complex web of influences related to the learning process in varied contexts. In relation to this review, we highlight crucial yet missing complexities. We conduct simple descriptive analyses as well as a re-analysis providing evidence demonstrating, consistent with prior reviews, that current findings do not support broad generalizations across higher education. Moreover, while we have no doubt that the use of video has some beneficial effects in higher education, as demonstrated by Noetel et al.’s review and our re-analysis, we remain concerned about the potentially adverse effects a reliance on this innovation may have on students from, for example, differing demographic backgrounds studying in varied disciplinary contexts. While more research is needed to reveal how video may be an optimal or suboptimal instructional medium for instruction, it is clear this research must employ robust, rigorous, and well-grounded methodological approaches which will provide clear and, ultimately, accurate findings.