A Case Study in the Application of the Systematic Approach to Training in the Logging Industry

Abstract

The purpose of this research was to develop and assess a targeted emergency first-aid and safety training program for professional loggers in Montana. There were two key objectives for the program: (1) participant demonstration of recall and retention of key concepts and (2) improved participant reception in comparison to the previous year’s training program. The Systematic Approach to Training provided the overall model for the development and conduct of the training program. Qualitative and quantitative analyses were used to assess the effectiveness of the training program. The training program was administered to 873 loggers. Pre-, post-, and follow-up examinations were used to assess recall and retention of key learning objectives, while surveys were used to assess learner reception of the updated training program. Post-training survey data indicated increases in training applicability, understanding of learning objectives, and overall course enjoyment of the updated program in comparison to the previous year’s training program. Participants scored significantly higher on the post-training exams, which demonstrated recall of key training objectives. The results obtained by the training evaluation will guide future research and the continued development of the training program to align with ongoing analysis activities and participant suggestions.

1. Introduction

Commercial logging continues to be one of the most dangerous occupations in the United States [1]. Despite advances in harvesting techniques, which have decreased occupational risk of injury and fatality from chainsaw injuries and environmental hazards, the occupational fatality rate in the logging industry (100.1 per 100,000 full-time equivalents) was 27.8 times greater than the rate reported for all other occupations combined (3.6 per 100,000 full-time equivalents) [2].

Researchers in the logging industry have focused on the annual rate and characteristics of injuries [3,4,5,6,7,8,9], the effects of mechanization [10,11,12,13,14], workers’ perceptions of the occupational risk [8,11], as well as the development of training strategies [15,16,17]. Currently, the Occupational Safety and Health Administration (OSHA) defines training requirements for the logging industry including “the minimal acceptable first-aid and Cardio Pulmonary Resuscitation (CPR) training program for employees engaged in logging activities” [18]. The standard includes a list of required topics, as well as acceptable training methods [18]. Research has been, and continues to be, conducted to determine the needs and best methods of training new, as well as experienced workers, on safe work practices [8,11,16,19]. While prior research has measured training effectiveness in the logging industry, the authors are not aware of an intervention-based study that assesses the effectiveness and reception of a training program for the logging industry based on end-user (professional loggers) perspectives.

Other high-risk occupations like construction and agriculture have been investigated in terms of the effectiveness of various safety training strategies [20,21,22,23,24]. With regard to a study published using a population of Australian construction workers, Loosemore and Malouf [22] determined that current safety training methods were largely ineffective in changing workers’ safety attitude. The authors pointed out that following training, workers had better intentions to behave safely but no longer cared about safety as an issue. Alternative methods of training have been more effective at reaching and influencing workers in high-risk occupations. In an evaluation of safety training in the US construction sector, Eggerth et al. [23] reported that including a narrative and discussion questions in tool box talks increased the effectiveness of training. Alternatively, a study of workers in the dairy industry found that using technology and mobile learning techniques was an effective means to deliver safety awareness content [22]. While the method of delivery is important, the content of training programs is important as well. It has been known for some time that effective safety training materials must be appropriate in terms of ethnicity, culture, literacy level and language [23].

The Systematic Approach to Training (SAT) is a five-step training development process that aims to provide training programs so workers can “do their jobs safely, efficiently, and effectively, and to protect the workforce, the public and the environment” [25]. The five steps of the SAT include: analysis, design, development, implementation, and evaluation. Key features of ensuring the development of a successful training program include ensuring frequent communication with subject matter experts, and management commitment to the development and ongoing implementation and improvement of the program [25,26].

Previous researchers from various industries have demonstrated success in using the SAT model for the adaptation or creation of training programs. For example, Heikka applied the SAT to solve a problem related to information security breaches in the telecommunications industry [27]. These researchers found that by using the SAT to develop and implement an information security program, there was a positive change in the security behaviors, as well as attitudes, after completion of the program [27]. Similarly, the SAT has been used worldwide to develop training programs in the fields of emergency preparedness [28], medicine [29], and even civil service [30].

The logging industry may benefit from employing a systematic and highly structured approach to emergency first-aid training because regimented procedural knowledge and skills are required for effective performance during emergency situations. In addition, by creating clear and concise learning objectives for the training program, a unified view of the desired training outcomes can be formed by both instructors and training participants. Using a systematic approach to create a video-based training program and corresponding learning objectives may improve the consistency of the information presented during different sessions of the training program. Thorough documentation of the design and development process used to create the program may prove essential if there were a need to verify completion of an adequate and approved emergency first-aid training program for the purposes of a training audit.

The purpose of this research was to develop a targeted emergency first-aid and safety training program that will reduce the injury rate and severity of injuries among professional loggers in the Intermountain region (Montana and Idaho). In the logging industry, population demographics, risks, and logging systems vary by region. Due to this variation, training attitudes, needs, and preferences also vary by region. To develop a training program to best meet the needs of the end-user, research must be conducted to understand the population of interest. To our knowledge, this is the first study to use the SAT to develop, implement, and evaluate a refined training program that specifically addresses the challenges and hazards of the logging industry.

The objectives of this study were to demonstrate recall and retention of key concepts included in the training program and to determine if the program was received more positively by the intended audience than the previous year’s training program. Through iterative evaluation and continuous improvement, the training program can continue to improve in applicability, reception, while meeting OSHA-specified training requirements. Major study hypotheses were that participants would have increased test scores from pre-training to post-training (demonstrating recall of key training objectives), increased test scores from pre-training to follow-up (demonstrating retention of key training objectives), but decreased test scores from post-training to follow-up as was also demonstrated in a previous CPR training study [31]. In addition, participants would respond more favorably to Likert scale questions regarding the reception and applicability of the updated training program in comparison to the prior-year’s program.

2. Materials and Methods

2.1. Data Acquisition

The SAT provides the overall model for this research. A mix of qualitative and quantitative data were used to complete the five phases of training program development (Figure 1). The methodology contained in this study was reviewed and approved by the Research Integrity and Compliance Review Office at Colorado State University.

2.1.1. Analysis Phase (Spring 2016)

In spring 2016, a survey was administered by the safety staff of the Montana Logging Association (MLA) to all attendees of existing OSHA-required emergency first-aid training workshops that were held in various locations throughout Montana. The survey contained a combination of demographic questions, Likert scale questions to assess reception of the existing training program, and short answer questions. Likert scale questions were designed to assess the degree to which participants agreed with various statements such as: “I understood the objectives of the course,” “The course was enjoyable,” “The course provided me with new information,” and “The length of the course was appropriate to cover the content.” The purpose of the short answer section was to provide an open response section to gather information about suggested course topics, areas of improvement, and circumstances or scenarios where the participant had need to use their training or skills in the past.

Participation in the survey was voluntary and anonymous, and all participants in the training program were eligible to participate. Compensation was not provided to survey participants.

2.1.2. Design and Development Phase (Spring 2016)

An emergency first-aid and safety training program was developed in response to a training needs analysis. The training needs analysis was completed in accordance with requirements identified during the analysis phase of The SAT. Topics covered during the training program were derived from three sources: OSHA Standards, an analysis of workers’ compensation data [32], and the results of the surveys administered during the prior year (spring 2016).

OSHA requires that loggers be trained in CPR and emergency first-aid covering a range of specific medical emergencies and traumatic injuries. According to OSHA requirements, training should consist of lectures, demonstrations, practical exercises and examination [18]. In addition to guidance on the delivery of training materials, OSHA standards also dictate required topics to be covered in the training program [18].

Following completion of the training needs analysis, learning objectives for the emergency first-aid training course were developed. Learning objectives were written to clearly describe the trainee’s desired performance of a specific task under a set of pre-defined conditions [33]. After developing learning objectives for the overall course, as well as for individual topics areas, the course was organized and outlined into a series of training modules. The overall structure of the training program consists of 16 different modules. Each module contained a real-world video scenario followed by a didactic training session. In total, the videos covered approximately two hours of material. After each module was presented, the course instructor/moderator provided time for discussion and practice of the skill presented in the video. For example, during the cardiac emergency module, a video scenario was presented with a logging worker going into cardiac arrest in a maintenance facility. After the real-world scenario was presented, a didactic training message was presented on types of cardiac injuries, signs and symptoms, as well as treatment. Finally, after the module concludes, there was time to practice CPR and rescue breathing on mannequins in the classroom.

The script for the didactic instruction was based on learning objectives created during the analysis and design phases of the SAT. To create a script that was factually correct and relevant, an interdisciplinary team of healthcare providers was assembled. Two occupational medicine physicians with military experience and an Emergency Medical Technician (corresponding author) contributed to the didactic material covered in the script. The script was developed to provide the necessary information to adequately and appropriately respond to, and provide treatment for injuries that occur in the logging industry. The material presented during the didactic portion of the training modules was designed to be concise, with treatment recommendations achievable with the skill level and materials available to the Montana logging population.

While both courses were designed to provide loggers with the necessary safety, first-aid, and CPR training, there were many updates to the 2017 program. The training course administered in 2016 did not meet the minimum requirements identified by OSHA standards. In addition, the program contained generic video-based examples. Most importantly, the 2017 program was specifically designed to include all the required topics in the OSHA standard. In addition, survey data from 2016 guided the development of new curricula and realistic scenarios for new training materials (instructor’s manual and trainee videos). The new training videos contained documentary type characterizations utilizing regional landscapes and actual logging practices specific to Montana forestry.

2.1.3. Implementation

The updated emergency first-aid and safety training program was administered to all attendees of the MLA training sessions in the spring 2017 (n = 873). A total of 14 training sessions were completed at 8 different locations throughout Montana.

2.1.4. Evaluation

As a part of the training development process, a pre- and post-training examination was developed. The purpose of the examination was to evaluate participant learning of key points from the first aid and safety topics recommended by OSHA. In addition, the post-training examination was designed to fulfill OSHA requirements for written evaluation of the materials presented during the emergency first-aid training course. The test questions were designed to test the participants on achievement of the key learning objectives for each module. The examination contained 44 questions, approximately three questions per module topic. All responses were either multiple choice or true/false.

After completion of the training session and associated examination, participants were asked to complete a survey to assess the reception, relevance, and perceived gaps in the updated training program. The spring 2017 survey included the same questions as the survey administered during the analysis phase of the project in the spring 2016. Participation in the survey was voluntary and anonymous and all participants (n = 873) in the training program were eligible to participate. Compensation was not provided to survey participants. All study participants provided consent for participation and all study methods were reviewed and approved by the IRB at the authors’ university.

As a part of the evaluation strategy, a follow-up training evaluation was conducted at approximately seven months post-training (fall 2017). A total of six follow-up sessions were conducted over a two-week period in various locations throughout Montana as a part of the Montana Logging Association’s fall chapter meetings. Participants were recruited for participation by the MLA through emails to their list of annual members. Participants electing to participate in the follow-up activities received a $50 incentive and signed consent was obtained for the follow-up study once the participants arrived at the location of the follow-up program. All participants who attended the chapter meeting were eligible for participation in the follow-up activities regardless of training status (attended training in spring 2017 versus did not attend training in spring 2017). The follow-up activities consisted of an examination and focus group session. The follow-up examination was identical to the pre- and post-training examination. If the follow-up participant had attended the updated (2017) emergency first-aid and safety training session, their tests were matched with their pre- and immediate post-training examination scores.

2.2. Data Analysis

2.2.1. Survey Data

Survey data obtained during the analysis phase (spring 2016) was compared to the survey data obtained during the evaluation phase (spring 2017) of the project. Response rates were calculated by dividing the total number of survey responses by the total number of people in attendance at the spring training sessions hosted by the MLA. Descriptive statistics were calculated for the demographic variables collected as a part of the survey. Means and standard deviations were calculated for the age of the survey participants. A t-test was performed to determine if there was a significant difference in the age of the survey respondents from 2016 to 2017. Frequency statistics were calculated for the categorical demographic variables gender and education level. A chi-square test of independence was used to determine if the distribution of gender or education level was statically different from 2016 to 2017.

Likert-scale responses to the 12 questions, which assessed the reception (i.e., participants’ opinion regarding reception, applicability, pace, content, etc.) of the training program, were compared between 2016 and 2017. Means and standard deviations were calculated for each question by year. The difference between the 2017 and the 2016 mean survey response score was calculated and a Wilcoxon–Mann–Whitney test was used as a non-parametric alternative to a two-sample t-test to determine if there was a significant difference in survey responses from 2017 to 2016.

During the short-answer section of the surveys, participants were asked for suggestions regarding improvements to future years’ programs and additional topics that they believe should be included or emphasized. Qualitative responses from the short-answer section of the surveys were categorized according to key words and/or general theme. The number of participant responses in each response category was then tallied and the most frequently occurring themes were reported.

2.2.2. Examination Data

Descriptive statistics were calculated for each examination period: pre-training, post-training, and follow-up. Paired sample t-tests were performed to determine the change in examination scores for three time periods: pre-training to post-training, post-training to follow-up, and pre-training to follow-up to determine participants’ recall and retention of key learning objectives. Changes in examination responses from pre-training to post-training were assessed using McNemar’s Test [34].

Follow-up participants were categorized as either having attended the updated training program during spring 2017 (trained), or not in attendance at the spring training session (untrained). Descriptive statistics were calculated for the age and gender of follow-up participants to determine if there was a significant difference in the demographic composition of these groups and to ensure any changes in test scores were due to the effect of training rather than demographic composition. A t-test was performed to determine if there was a significant difference in the age of trained versus untrained follow-up participants. Fisher’s exact test was used to determine if the distribution of gender was statistically similar between trained and untrained follow-up participants. Follow-up examination scores were calculated for all participants. A t-test was performed to determine if there was a significant difference between the mean score for trained versus untrained follow-up participants.

3. Results

A total of 742 (70% response rate) and 568 (65% response rate) surveys were returned in 2016 and 2017, respectively (

Table 1. Participant demographics in training survey: baseline (2016) vs. update (2017).

Participant Demographics	2016 (15 Sessions, N = 742)	2017 (14 Sessions, N = 568)
	Mean (SD)	Mean (SD)
Response Rate (Number Responses/Total Number in Attendance)	742/1059 = 70.07%	568/873 = 65.06%
Age (p > 0.05)	45.88 (13.67)	45.85 (13.99)
	(n = 688)	(n = 507)
	Frequency (Percentage)	Frequency (Percentage)
Gender (Χ2 = 0.20, p > 0.05)	Female—21 (2.91%)	Female—18 (3.35%)
	Male—701 (97.09%)	Male—519 (96.65%)
	(n = 722)	(n = 537)
Education Level
NHS—Did not finish High School	NHS—55 (7.82%)	NHS—52 (9.44%)
HS—High School Diploma	HS—410 (58.32%)	HS—297 (53.90%)
SC—Some College/Associates Degree	SC—179 (25.46%)	SC—135 (24.50%)
BS—Bachelor’s Degree or Higher	BS—59 (8.39%)	BS—67 (12.16%)
(Χ2 = 6.49, p > 0.05)	(n = 703)	(n = 551)

). There were no significant differences in the age (p > 0.05), gender distribution (Χ² = 0.20, p > 0.05), or education level (Χ² = 6.49, p > 0.05) between the two years.

The scale used to assess reception and relevance of the training program was based on a 1–5 Likert scale, with 1 corresponding to strongly disagreeing with the statement, 3, having a neutral response, and 5, strongly agreeing with the statement. Across all survey questions, participants at the 2017 training session responded more positively than respondents at the 2016 training sessions (

Table 2. Training survey responses: baseline (2016) vs. update (2017).

Statement	2016 Mean (SD)	2017 Mean (SD)	Difference between Means (p-Value)
1. I understood the objectives of the course	4.38 (0.58)	4.52 (0.58)	0.13 (p < 0.01)
2. The length of the course was appropriate to cover the content	4.12 (0.70)	4.35 (0.62)	0.23 (p < 0.01)
3. The course provided me with new information	3.75 (0.81)	4.19 (0.65)	0.44 (p < 0.01)
4. On-the-job application of each objective was discussed during the course	4.10 (0.59)	4.28 (0.59)	0.18 (p < 0.01)
5. The examples presented helped me to understand the content	4.18 (0.55)	4.34 (0.55)	0.16 (p < 0.01)
6. The course was enjoyable	3.68 (0.86)	4.08 (0.70)	0.40 (p < 0.01)
7. The materials covered will be useful on the job	4.23 (0.59)	4.38 (0.56)	0.15 (p < 0.01)
8. The instruction materials were clearly presented	4.27 (0.54)	4.36 (0.60)	0.10 (p < 0.01)
9. The pace of the course was appropriate to cover the content	4.13 (0.64)	4.30 (0.59)	0.17 (p < 0.01)
10. The time given by the instructor to complete practice activities was appropriate	4.17 (0.61)	4.33 (0.58)	0.17 (p < 0.01)
11. I think that being trained on CPR and First-Aid is worthwhile	4.51 (0.62)	4.56 (0.58)	0.05 (p > 0.05)
12. I would attend this training even if it was not required.	3.79 (0.97)	3.98 (0.85)	0.19 (p < 0.01)

). This difference was significant in every question except question 11, “I think that being trained on CPR and First-Aid is worthwhile”. The survey questions that had the largest change between the two years were question three (the course provided me with new information) and question six (the course was enjoyable).

The categorical distribution of responses for statement three “the course provided me with new information” changed significantly (Χ² = 89.33, p < 0.001); in 2016, 68% of respondents either agreed or strongly agreed with the statement, while in 2017, 89% or respondents agreed or strongly agreed with the statement. This pattern was repeated for question six “the course was enjoyable” (Χ² = 69.12, p < 0.001). In 2016, 63% of respondents either agreed or strongly agreed with the statement, whereas in 2017, 83% of respondents either agreed or strongly agreed with the statement.

Qualitative responses from the short-answer section of the surveys were categorized according to key words and/or general theme. The number of participant responses in each response category was then tallied and the most frequently occurring themes were reported. Two categories included on the suggested topic list for both 2016 and 2017 (

Table 3. Top five suggestions for improvement from training survey: baseline (2016) vs. update (2017).

2016	2017
More/New Videos	More information on Radio Communication
Practical Applications/Realistic Scenarios	Fire Safety Training
New Material	Shorten the course
Include Instruction on Radio Communication	Hands-on/ Practical Experience
Include CPR/Choking for Children	Lengthen the course/provide more information/slow down

) were more information on radio communication procedures and more time for hands on/practical experience. In 2017, two of the most common suggestions for improvement were in regard to the length of the course, with some respondents commenting that the course should be shortened, and others suggesting that the course should be lengthened to provide more information and to allow the instructor to slow down.

A total of 826 pre-training examinations and 802 post-training examinations were completed during spring 2017 for response rates of 95% and 92%, respectively (

Table 4. Participant examination scores (pre-training, post-training, and follow-up).

Exams Compared	Percent Difference (mean, SD)
Mean Pre-Training Examination Score (n = 826)	80.45% (mean = 35.40, SD = 4.29)
Mean Post-Training Examination Score (n= 802)	90.57% (mean = 39.85, SD = 3.14)
Mean Follow-Up Examination Score (n = 44)	82.02% (mean = 36.09, SD = 2.79)
Mean Difference Pre-Training to Post-Training (n = 799)	10.25% (mean = 4.51, SD = 3.88) (paired t-test p < 0.0001)
Mean Difference Post-Training to Follow-Up (n = 36)	8.95% (mean = −3.50, SD = 3.43) (paired t-test p < 0.0001)
Mean Difference Pre-Training to Follow-Up (n = 36)	0.75% (mean = 0.33, SD = 4.06) (paired t-test, p = 0.6251)

). Seven hundred ninety-nine pre- and post-training examinations were paired. A paired-sample t-test was performed to provide a measure of immediate recall of learned knowledge from pre-training to post-training (Table 4). On average, a participant’s score increased significantly by 4.51 (SD = 3.88) points (out of 44 questions), indicating an approximate 10% improvement from pre-training to post-training.

Of the 44 trained follow-up participants, 35 were matched to both their pre-training and post-training examinations. On average, participants in the follow-up scored significantly lower on their follow-up examination than they did on the post-training examination (mean = −3.50, SD = 3.45) (Table 4). There was no significant difference between pre-training examination scores and follow-up examination scores (p > 0.05) (Table 4).

McNemar’s test was used to determine if training influenced examination question response. In 32 of the 44 examination questions, McNemar’s test was significant, indicating a change in question response from pre-training to post-training. For example, three multiple choice questions that showed significant improvement from pre-training to post-training were: (1) which is a sign of poor circulation, (2) where should you place your hand for abdominal thrusts, and (3) what is the ratio of compressions to breaths when performing CPR. In 12 of the 44 questions, McNemar’s test was non-significant, indicating no change in response from pre-training to post-training. In nine of these of these questions, over 90% of respondents answered the question correctly on the pre-training examination, indicating that they knew that material, which left little room for a significant change in the correct proportion of responses.

In total, 69 participants attended follow-up sessions. Of the 69 in attendance, 44 attended the updated training program while 25 did not attend the updated training program. There was no significant difference in mean age (p > 0.05), or gender composition (p > 0.05) between trained and untrained groups (

Table 5. Follow-up/focus group participant demographics and examination scores.

Training Status	n (Total = 69)	Age Mean (SD)	Gender	Follow-Up Score Mean (SD)
Trained this year	44	51.19 (13.60)	Male (42) Female (2)	36.09 (2.79)
Not trained this year	25	53.27 (12.97)	Male (21) Female (4)	32.56 (2.99)
		p = 0.55	(Fishers Exact Test: p = 0.18)	p < 0.01

). There was a significant difference in mean follow-up examination scores between trained and untrained participants, with trained participants scoring nearly four points higher than untrained participants (p < 0.01).

4. Discussion

Based on a thorough analysis of the specific needs and challenges facing professional loggers, the investigators were able to develop training content that had greater applicability and specificity to the target audience. The loggers’ perceptions of increased applicability, understanding of learning objectives, and overall course enjoyment of the updated program were confirmed based upon post-training survey responses. While participants demonstrated recall of key training objectives, the investigators were unable to verify retention using the differences between pre-training examination scores and follow-up examination scores. However, the mean examination score at follow-up was 82%. There was a 10.12% mean exam score increase from pre-training to post-training and an 8.55% mean exam score decrease over the follow-up period, for a mean score difference of 1.57%. Similarly, Einspruch et al. found a decrease in overall performance in CPR training at follow-up of 12% to 16% depending on the training format [31]. Qualitative data from the surveys provided ideas and suggestions that can be used as part of a continuous improvement process in the development of updated and new training programs. As recommended in survey responses, additional sections could be developed and added to the video program to cover the requested topics. While the examination questions must assess key learning objectives, the questions should be designed at high knowledge levels of the construct being taught. In this regard, test items will be at a level of difficulty in which participants are unlikely to answer correctly until after the material is taught. This will help researchers to better determine if the training program was effective at teaching the learning objectives.

There were several benefits of using the SAT to develop an industry- and population-specific training program. The updated program was perceived by participants to be more effective at conveying useful and new knowledge required when performing emergency first-aid. In addition, by creating clear and concise learning objectives for the training program, participants responded more favorably to survey questions regarding understanding the learning objectives of the program. Using the SAT was useful in creating a unified view of the desired training outcomes for both instructors and training participants and ensured the consistency of the information presented during different sessions.

Although there is a limited amount of research related to the SAT development and effectiveness among logging cohorts, the results of the present study are consistent with the work of previous research conducted on training in the logging industry. Many of the suggestions from previous researchers on training preferences in the logging industry were used in the design and development of the updated training program for the present study.

Helmkamp et al. conducted a study in which the researchers assessed safety behaviors in the West Virginia logging industry and reported that “many of the loggers said they related to the real life victim stories portrayed in the (training) video” [19]. Researchers surveying loggers in the Southeastern United States had the following suggestions for training programs: have experienced logging workers participate in the development of more effective training program, have the training programs be sensitive to the economic cost of training and safe work practice and emphasize the positive economic impact of safe work, and create training programs on proper safety procedures [11]. A third research study in which the authors surveyed loggers from northern New England, made the following suggestions: “training should be tailored to the needs and preferences of the logger audience, or run the risk of becoming irrelevant,” “most loggers prefer less formal, on-the-job training, especially training that involves members of the logging community in its delivery”, and efforts should be made to “illustrate how specific training may benefit loggers” [16]. The authors of the New England study concluded that “logger training programs require contact with the target audience—the logging community—in order to maintain the relevance and credibility of the programs’ content and delivery” [16].

As per the suggestions of the aforementioned research studies, the authors of the present study ensured that the updated training program provided relevant and specific training that included the hazards and environmental conditions that loggers in the targeted population experience. As suggested by Bordas et al. [11], experienced loggers involved in the present study contributed to the design and script of the updated training program, and were portrayed in video examples of scenarios, which were filmed on logging sites throughout Montana and Idaho.

Following suggestions to include the target population in the development of the training program may have been one of the reasons the authors found that participants responded more positively to all survey questions after receiving the updated training in comparison to training participants at baseline. The survey questions with the greatest mean score improvement from baseline to the updated training were: “the course provided me with new information” and “the course was enjoyable.” The survey questions with the most positive responses were: “I think that being trained on CPR and first-aid is worthwhile,” “I understood the objectives of the course,” and “the materials covered will be useful on the job.”

In addition to the positive responses observed in the survey, on average, participants experienced an increase in their examination score from pre-training to post-training by over four questions (over a 10% increase), demonstrating recall of key training objectives. While there was significant improvement from pre-training to post-training, there was no significant difference in training scores from pre-training to follow-up. The authors were unable to determine if participants retained new information from the updated training program and missed questions they had correctly responded to on the pre-training exam, or if they had reverted to their baseline (pre-training) responses. Therefore, the authors were not able to verify retention of key training objectives from the difference between pre-training and follow-up examination scores. However, at follow-up, participants answered 82% of the questions correctly and there was a significant difference in the examination scores of trained versus untrained participants; participants who had attended the 2017 training program had significantly higher follow-up examination scores than follow-up participants who did not attend the 2017 training.

While the authors were not able to verify long-term retention of key learning objectives through the examination, the training program created as a result of this project had several strengths. Survey respondents appreciated the applicability and usefulness of the training program. One of the challenges facing the logging industry is developing safe work practices and safety training that are cost-effective. In the southern U.S., a survey of logging training preferences indicated that although safety training was considered useful, it was believed to be impractical and detrimental to productivity [11]. Similarly, researchers in Ireland reported that despite mixed findings on the practicality and ability for training to decrease injury and fatality rates, loggers held the opinion that training was useful and should be more readily available [8]. One of the advantages of a video-based training program is the increased accessibility of the training. While the program created as a result of this work was not designed to be a stand-alone resource, much of the information provided in the program is now available upon request at any location or time, rather than waiting for the annual training program to occur. While using the videos from the training program alone would lack the practical experience on the trainers and discussion segments of the course, they could provide valuable didactic safety information to newly hired workers prior to the formal scheduled training period.

Limiations

This study had several limitations. The overall design of the study was limited to a specific population: loggers in the Intermountain region of Idaho and Montana. While this allowed the updated training program to be tailored to a specific group, the authors are unable to determine if the same program would have similar reception and applicability in different regions within or outside of the United States. An additional limitation related to the research design included the selection of participants. Nearly all members of the MLA cohort received the updated training program as opposed to randomly selected experimental and control groups. Due to this limitation, the authors were unable to determine causation. It was unknown what contribution the training or other events or programs may have had on the changes that were evident in surveys and examinations from baseline to post-training and follow-up. The baselines to update comparisons of participant responses generally improved. Although these improvements were statistically significant (possible result of sample size), the magnitude of the improvements was relatively small and may have limited practical significance.

5. Conclusions

The strength of the training program developed in the current study was the customization of content and delivery specific to the target population of professional loggers in Montana. Not only was the content specific to the environmental and occupational hazards encountered in Montana logging, but the new program was also developed in partnership with end-user opinions regarding delivery of the training objectives. Prior research often isolates the development of content, or the method in which it was delivered; this study combined both aspects of training program development. Conducting extensive background research for optimal program design and development was essential, as training effectiveness is tied to the appropriateness of the materials in terms of ethnicity, culture, literacy level and language [23]. In terms of delivery, this study utilized the survey feedback obtained from the target population and empirical evidence [21], which highlights the importance of including numerous opportunities for group discussion and expansion of the learning objectives within the training program.

By conducting an analysis of the specific needs and challenges facing professional loggers in the target population, applicability and specificity of the updated training program was optimized. The results of surveys obtained during this investigation demonstrated increased participant reception scores from the safety training program that was developed using the SAT. Analysis of pre-training, post-training, and follow-up examination scores revealed participant recall of key learning concepts in the updated training program. The results obtained through the training evaluation strategy employed in the present study will help guide training research targeted to professional loggers.