Quantitative Analysis of the Usage of a Pedagogical Tool Combining Questions Listed as Learning Objectives and Answers Provided as Online Videos

Abstract

To improve the learning of basic concepts in molecular biology of an undergraduate science class, a pedagogical tool was developed, consisting of learning objectives listed at the end of each lecture and answers to those objectives made available as videos online. The aim of this study was to determine if the pedagogical tool was used by students as instructed, and to explore students’ perception of its usefulness. A combination of quantitative survey data and measures of online viewing was used to evaluate the usage of the pedagogical practice. A total of 77 short videos linked to 11 lectures were made available to 71 students, and 64 completed the survey. Using online tracking tools, a total of 7046 views were recorded. Survey data indicated that most students (73.4%) accessed all videos, and the majority (98.4%) found the videos to be useful in assisting their learning. Interestingly, approximately half of the students (53.1%) always or most of the time used the pedagogical tool as recommended, and consistently answered the learning objectives before watching the videos. While the proposed pedagogical tool was used by the majority of students outside the classroom, only half used it as recommended limiting the impact on students’ involvement in the learning of the material presented in class.

1. Introduction

The transformation of undergraduate science education started over 20 years ago in the USA with the objective to improve competitiveness in technology development and shortage of qualified candidates for high-tech jobs. Less than 40% of students enter universities and colleges with an interest in science, technology, engineering and mathematics (STEM); this combined with the low percentage who graduate with a degree in those fields motivated decision makers to target research in higher education in science [1,2,3,4]. The matter was reviewed extensively by the National Science Foundation and a major criticism was that STEM subjects are often taught in isolation instead of as an integrated curriculum [5]. Further analysis of the teaching methods practiced in science courses pointed at the observation that traditional lectures often promote the acquisition of facts rather than understanding of scientific concepts [6,7,8,9].

To address the challenge of teaching with the objective to promote understanding and application of concepts in science, government sponsored programs as well as institutions funded research-based initiatives aimed at developing and implementing new teaching interventions. Popular interventions are designed to increase students’ active participation in their learning, and are offered either as a complement to, or in replacement of, traditional lectures [10,11]. Interventions characterized as active learning are defined as “anything course-related that all students in a class session are called upon to do other than simply watching, listening and taking notes” [12]. The impact of active learning on examination performance and failure rates for undergraduate STEM education was recently quantified by applying statistical analysis to the findings of combined studies [13]. This meta-analysis found that the average examination scores improved by 6% in active learning sections, and students taught with traditional lectures were 1.5 times more likely to fail, providing strong support for the application of pedagogical interventions as a teaching practice in undergraduate science classes.

The design of pedagogical interventions intended to promote students’ participation in learning concepts in science remains a challenge [14]. Due to the rapid development of specialized fields such as molecular biology, academics teaching undergraduate science courses at the more advanced level are confronted with the pressure to cover recently discovered facts in addition to the main established concepts and with no additional class time allocated. Furthermore, the reiteration of the importance of understanding and application of scientific concepts previously acquired in the context of the specialized information presented add to the challenge of integrating students’ centered activities to promote learning. The design and implementation of interventions aimed at increasing students’ participation in their learning is compounded by the reality of class sizes and often large groups of students, which makes the application of traditional “active learning” approaches very challenging. For researchers in the field of education, a universal definition of “active learning” is lacking and often attributed to the different interpretation of some terms used by various authors. Perhaps core elements of active learning, including student activity and engagement in the learning process, provide a more accepted definition.

Taking advantage of internet distribution of learning material, another approach was developed to increase students’ participation in their learning, and referred to as “learner control” [15,16,17]. This later gives trainees control over when and where their training occurs, and to some extent allows them to make decisions on the content. While “active learning” takes place in the classroom, “learner control” is designed for learning outside the classroom, and provide students with more freedom “in becoming, or not, mindfully engaged” [18]. Clearly, traditional lectures and learner control approaches in education are at opposite ends of the spectrum of students’ involvement in learning; from passive to active, and neither benefiting all students [19].

Taking into consideration the benefits of active learning and the reality of large class size, a pedagogical tool was previously developed to increase learning of scientific concepts relevant to a Molecular Biology course offered to a large class of undergraduate students registered in the third year of a science program [20]. This tool has two components: a list of learning objectives listed at the end of each lecture, and a series of videos posted online providing answers to each learning objective. Students were advised to first answer the learning objectives and then to watch the videos to evaluate the content of their answers. In our analysis of the impact of the tool on students’ learning, we measured students’ exam scores, and reported a 2% increase of students’ average scores when videos were made available [20]. The next step in the evaluation of this intervention is to measure the use of the pedagogical tool by students and to explore students’ perception of the tool. The aims of the current study were to measure the use of the pedagogical tool; to determine if the tool was used as recommended; and to explore the students’ perception of its usefulness.

2. Results

In the current study, a pedagogical tool was used to complement the lecture component of a 3rd year undergraduate Molecular Biology course. The pedagogical tool consisted of learning objectives (questions) listed at the end of each lecture, and answers to those objectives in video format were made available to students online [20]. Using online tracking tools, a total of 7046 views were recorded for all 77 videos. The number of daily views of the videos during the complete academic term is illustrated in Figure 1. The two peaks in the number of views correspond exactly to the dates of examinations; first exam was held on 1 October 2013 (831 views), the second was held on 5 November 2013 (459 views). Expectedly, more views were recorded at the time of the first examination, since more videos were posted before this examination (47 vs. 30; Table 1). The pedagogical tool was not used by the faculty member teaching the third section of the course; hence, no viewing activity was detected during that period.

A survey was designed to assess students’ usage and perceptions of the learning tool; 64 students or 90.1% completed the survey. Based on the survey, the majority of the students (98.4%) found the videos to be useful and to promote understanding of the material presented in class. Students' usage patterns of the online videos are summarized in Table 2. Most students (73.4%) accessed all the videos, and about 80% watched more than half the videos. Very few students (3%) admitted to having watched the videos instead of attending class on at least one occasion. The majority of students watched several videos at a time, with 67.7% watching six or more videos in one sitting. All participants agreed that they liked having the option to control the pace and/or repeat sections of the videos.

Figure 1. Daily numbers of views of the videos during the complete academic term.

Figure 1. Daily numbers of views of the videos during the complete academic term.

Table 1. Structure of the molecular biology course.

Lecture title (number of lectures)	Faculty member	Number of videos
DNA structure and chemistry	A	8
Genome organization	A	9
DNA replication	A	11
DNA repair	A	8
DNA recombination (2)	A	10
First examination
Control gene expression in prokaryotes	A	11
Control gene expression in phage lambda	A	9
Protein synthesis and the genetic code (2)	A	10
Transcription in prokaryotes	B	0
Transcription in eukaryotes	B	0
Control of gene expression in eukaryotes: regulatory proteins	B	0
Control of gene expression in eukaryotes: Combinatorial control	B	0
Second examination
Control of gene expression in eukaryotes: DNA methylation	B	0
Control of gene expression in eukaryotes: translational control	B	0
Maturation and transport of RNA	C	0
Regulatory RNA	C	0
Mobile genetic elements	C	0
Genetic control of the immunoglobulin locus (2)	C	0
Methods to measure gene expression	C	0
Final examination

Table 1. Structure of the molecular biology course.

Lecture title (number of lectures)	Faculty member	Number of videos
DNA structure and chemistry	A	8
Genome organization	A	9
DNA replication	A	11
DNA repair	A	8
DNA recombination (2)	A	10
First examination
Control gene expression in prokaryotes	A	11
Control gene expression in phage lambda	A	9
Protein synthesis and the genetic code (2)	A	10
Transcription in prokaryotes	B	0
Transcription in eukaryotes	B	0
Control of gene expression in eukaryotes: regulatory proteins	B	0
Control of gene expression in eukaryotes: Combinatorial control	B	0
Second examination
Control of gene expression in eukaryotes: DNA methylation	B	0
Control of gene expression in eukaryotes: translational control	B	0
Maturation and transport of RNA	C	0
Regulatory RNA	C	0
Mobile genetic elements	C	0
Genetic control of the immunoglobulin locus (2)	C	0
Methods to measure gene expression	C	0
Final examination

Table 2. Students’ usage patterns.

Usage pattern	Percentage of students (n = 64)
Watched all videos	73.4%
Watched more than half	79.7%
Watched instead of attending class	3.1%
Watched at least two videos per sitting	96.7%
Watched six or more in one sitting	67.7%
Liked having the option to repeat/control pace	100%

Table 2. Students’ usage patterns.

Usage pattern	Percentage of students (n = 64)
Watched all videos	73.4%
Watched more than half	79.7%
Watched instead of attending class	3.1%
Watched at least two videos per sitting	96.7%
Watched six or more in one sitting	67.7%
Liked having the option to repeat/control pace	100%

An important question we sought to address was whether the students were using the pedagogical tool as it was intended. Students were advised to first answer the learning objectives and then watch the videos to evaluate the content of their answers. Interestingly, approximately half of the students (53.1%) always or most of the time used the pedagogical tool as recommended, and consistently answered the learning objectives before watching the videos (Figure 2). A substantial proportion of students (37%) followed the instructions only sometimes, while about 10% never attempted to solve the learning objectives before accessing the videos.

Figure 2. Percentage of students who attempted to solve the learning objectives before accessing the videos.

Figure 2. Percentage of students who attempted to solve the learning objectives before accessing the videos.

Next, we wanted to assess whether or not using the professor’s voice in the videos was important to students. The majority of participants (93.8%) found it important that the narrator of the videos was the professor (Figure 3). Since the frame of each video was limited to a white page on which the narrator was drawing a diagram to illustrate the answer to a learning objective, we asked whether seeing the narrator’s face would improve the student’s learning experience. The majority of students (68.7%) indicated that seeing the face of the narrator rather than just the hands was not important, while 28% were unsure whether or not seeing the speaker’s face would improve their learning experience (Figure 4).

Figure 3. Importance of using the professor’s voice in the videos.

Figure 3. Importance of using the professor’s voice in the videos.

Figure 4. Percentage of students who agreed that seeing the speaker’s face in the video would improve their learning experience.

Figure 4. Percentage of students who agreed that seeing the speaker’s face in the video would improve their learning experience.

3. Discussion

The quantitative evaluation of the effectiveness of a new pedagogical intervention designed to increase students’ participation in the learning of concepts outside the classroom was studied. In the current study, we measured the use of the pedagogical tool and perception of students of its usefulness. Data collected using either online tracking tools or a survey indicated that the students used the pedagogical tool. The total number of views of online videos was 7046 for the 77 videos available to 71 students. Viewing activities of individual students were not tracked but rather represented viewing regardless of user identity, and represented a limitation in the method of quantification used. The possibility of multiple viewings made by the same student would lead to an overestimation of the number of views. Similarly, potential viewing by the 10 students who dropped the course within the first few weeks could have led to an overestimation of the number of views. Downloading the videos for later viewing without accessing the institution Blackboard Learn, although requiring more advanced computer skills, would have led to an underestimation of the number of views. The measure of daily views was also indicative of video usage by students, and two sharp peaks of activity were recorded on the days of the first and second examination. Peak heights were proportional to the number of videos made available to students: 831 views of 47 videos for the first examination, 459 views of 30 videos for the second examination, and no views on the date of the final examination, consistent with the lack of videos made available for the final examination.

The high viewing activity measured with online tools was consistent with the data collected from the survey, and indicated that 73.4% of the students who answered the survey watched all the videos. The survey provided additional information regarding the usefulness of the pedagogical tool, particularly about the recommended use of the videos and of the learning objectives. Instructions provided aimed at stimulating engagement in the learning of the concepts by addressing each learning objective independently, and knowing that students could verify the content of their answers by watching the videos and obtaining immediate feedback on their performance. While 73.4% of the students watched all the videos, only 53.1% used the tool as instructed—first answered the learning objectives before watching the video. For the other half of the students, limiting the use of the tool to viewing the videos reduced their participation in the learning of the material. Clearly, the video component of the tool was more popular than the learning objectives. For those students, benefits of the tool were limited to the viewing of videos, which represents a passive activity.

The recording of daily views was also indicative of limiting the use of the tool to viewing the videos. The two sharp peaks of activity measured on both days of the examinations limited the time available for answering the learning objectives and using the tool as instructed. In addition, 67.7% of students reported having watched 6 or more videos in one sitting, restricting the time for answering the learning objectives, and reducing the benefits of engagement in this activity. Reasons for not using the tool as instructed and limiting the participation for approximately half the students were not explored. The lack of engagement and reluctance of some students to engage in active learning activities was previously reported [19,21,22,23]. Previous reports indicated a lack of understanding of the benefits students would draw even without trying, a resistance to participation in class activities in front of classmates, and simply the lack of evidence of the benefits, including scores on tests [23]. The most important factor in promoting students’ participation in new learning activities is the demonstration of improved academic achievement as a result of using the proposed pedagogical tool. Similar to previous studies using standard measures of academic achievement, we measured a small increase in scores of 2% in association with the tool [20,24]. Increasing students’ use of the tool appropriately rather than with a limitation to video viewing remains a challenge.

The survey included questions designed to measure the perception students have of the usefulness of the intervention. Previous studies have identified the perception students have of interventions designed to increase learning as a significant contributor to the decision to engage in the suggested activity [23]. Similarly, the technical advantages offered by online videos, and previously reported to increase students’ use of the tools, were reported in the current study and included the option to repeat/control the pace of the videos [25,26,27]. The characteristics of the videos students might find important were investigated. Interestingly, the majority (93.8%) found it important that the narrator of the videos was the Professor, while only a small percentage (3.1%) would have preferred to see the face of the narrator rather than just the hands. Clearly, students attributed high importance to having the professor involved in the learning activity, albeit outside the classroom. This observation is significant for the development of videos used in complement to traditional lectures, and provides evidence for how students relate to Professors both in the classroom and when accessing videos featuring the Professor outside the classroom. Further research is required to identify the elements of an effective pedagogical intervention aimed at increasing the participation of students in learning.

The interpretation of the survey results has limitations including the previously described Hawthorne effect [28], whereby students knowingly react positively to any novel intervention regardless of its merit. This is particularly relevant for the question enquiring about the usefulness of the videos and 98.4% found the videos to be useful in assisting their learning. Although the contribution of the Hawthorne effect was not measured in the current survey, the positive impact of the pedagogical tool on students could have been attributed in part to the Hawthorne effect.

The current study provides an initial evaluation of the usefulness of the pedagogical tool and students’ reporting a high rate of usefulness motivates additional studies. Assessing how students benefited from the intervention and if learning increased requires looking at a broad range of learning outcomes including but not limited to: identifying what constitute an improvement in learning, quantifying the magnitude of the reported improvements, the impact of the tool on learning outcomes, and interpreting data. Our previous study measuring a modest 2% increase of scores on exams in association with the introduction of the tool in combination with the measured use of the tool and perception of students of its usefulness in the current study, justifies the use of the pedagogical tool in the future.

4. Methods

4.1. Course Format and Evaluation

The Molecular Biology course is offered once a year during the fall term, and taught in either English or French language. The course is a requirement for the Bachelor of Science (B.Sc.) degrees in Biology, Biochemistry, Biomedical Sciences, and Biopharmaceutical Sciences, and is worth 3 of the 90 credits required for the completion of the B.Sc. degree. For the current study, data were collected from the course taught in French, and lectures were presented twice a week as 80-min sessions for a total of 33 h of lectures during the academic term. Attendance to lectures was not recorded and was not mandatory. The material was presented by three faculty members all using the traditional lecture format consisting of PowerPoint Presentation guided lectures delivered in a classroom equipped with multimedia technology. All PowerPoint Presentations were made available to students between 24 and 48 h before each lecture through the Blackboard Learn system managed by the institution. The pedagogical tool was used only for the first half of the course and by only one of the faculty members’ who designed both the learning objectives and the videos (Table 1).

Students’ evaluation was conducted through three written examinations all taken in the classroom. The three exams had the same format, and consisted of three sections. Marks for each section were attributed as followed: 15 multiple choice questions for 30 marks, 10 short-answer questions for 50 marks and 2 long-answer questions for 20 marks. The students were given 80 min to complete the first two exams and 160 min to complete the final exam. None of the exams were cumulative. The final mark for the course was calculated from all three written exams with the following contribution from each exam: 30% from the first mid-term, 30% from the second mid-term and 40% from the final exam.

4.2. Student Population

The Molecular Biology course was offered to undergraduate students who had completed the first two years of a program offered in the Faculty of Science with the majority of students registered in the Biochemistry and Biology programs. Courses completed by students during the first two years were at the introductory level on the topics of biology, chemistry, mathematics and physics, and were completed at the same institution. The prerequisite for the Molecular Biology course was an introductory genetics course taken during the second year of the undergraduate degree. The number of students who completed the course and the three written examinations was 71 and the initial number of registered students was 81.

4.3. Pedagogical Tool—Learning Objectives and Online Videos

The pedagogical tool consisted of two components: a series of 8 to 15 learning objectives, formatted as questions and listed at the end of each lecture; and online videos presenting answers to each learning objective. Details of the content of the learning objectives and of the videos were described in our previous publication [20]. Table 1 includes a description of the subject covered in class, and the number of learning objectives associated with each lecture. Further details, including the subject of each learning objective and duration of each video, are listed in Table S1. Essentially, the pedagogical tool was designed to promote the learning and application of concepts presented in class. Concepts were either previously learned or presented in class and their application to the field of molecular biology was emphasized during the lectures. The learning objectives were posted online at the end of the lecture material for each individual lecture in the format of a PowerPoint Presentation, and provided 24 to 48 h prior to each lecture. The videos were made available online using the institution Blackboard Learn to students immediately after class, and all material remained available until the date of the final examination. At the end of each lecture, students were invited to address each of the learning objectives before watching the videos, and it was reiterated that videos could be used to verify the completeness and exactitude of answers. With the intent to promote students’ engagement in the learning of the concepts, the Professor strongly recommended to use the pedagogical tool early on in the preparation for the examination rather than only watching the videos the night before exams. The content and preparation of the pedagogical tool including the filming was done by faculty member A, who also posted of all the material online. The online delivery tool was managed by the educational institution and described as Blackboard Learn, and access was limited to students registered in the course at all times. Videos describing the learning objectives for each lecture were deposited on the online management platform Bits On The Run (purchased by LongTail Video, New York, NY, USA), accessible to students via a link posted on Blackboard Learn. Built-in tools to track viewing were used to measure the number of views on a daily basis. At the end of the term, viewing activities including the number of views and the duration of views for individual videos were extracted from the BitsOnTheRun platform as “.csv” files and analyzed using Microsoft Excel software (Mircosoft Corporation, Redmond, WA, USA). Ethics approval for the two components of the pedagogical tool, namely the learning objectives as the videos, was not required.

4.4. Survey Design and Implementation

The survey consisted of 10 questions (Figure S1) printed on a single page, and was distributed to all students at the end of the last lecture presented by Faculty member A, who developed the learning objectives. All questions of the survey were of the multiple choice type and did not require students to provide any written comments. Students were invited to answer all the questions directly on the survey, anonymously, and to return the questionnaire during the next lecture to Faculty member B, who did not use the pedagogical tool. To maximize the response rate, students were informed that approximately 5 to 10 min would be required to answer all the questions, and outcomes would provide valuable information for improving all aspects of the tool including video quality and content. Answers from filled surveys were compiled and analyzed by the authors. An ethics approval for the survey was submitted to the University of Ottawa Office of Research Ethics and Integrity, and classified under the category “Research Based on Secondary Use of Data”.