A Contemporary Analysis of Aircraft Maintenance-Related Accidents and Serious Incidents

Abstract

Aircraft maintenance has been identified as a key point of concern within many high-risk areas of aviation; still being a casual/contributory factor in a number of accidents and serious incidents in commercial air transport industry. The purpose of this study is to review and analyse the aircraft maintenance-related accidents and serious incidents which occurred between 2003 and 2017, to provide a better understanding of the causal and contributory factors. To achieve this, a dataset of maintenance-related accidents and serious incidents was compiled and then qualitatively analysed by thematic analysis method. Coding these events by using NVivo software enabled the development of a taxonomy, MxFACS. The coded output was then evaluated by subject matter experts, and an inter-rater concordance value determined to demonstrate the rigour of the research process. Subsequently, the events were evaluated in terms of their relationship to known accident categories such as loss of control, runway excursions. The most frequent maintenance event consequences were found to be runway excursions and air turnbacks, with the second level categories being related to failures in engine and landing gear systems. The greatest maintenance factor issues were ‘inadequate maintenance procedures’ and ‘inspections not identifying defects’. In terms of fatalities, ‘collision events’ were the most prominent consequence, ‘engine-related events’ were the most significant event, and ‘inadequate maintenance procedures’ were the most concerning maintenance factor. The study’s findings may be used in conjunction with existing risk analysis methodologies and enable the stakeholders to develop generic or customised bowties. This may identify the existing barriers in the system as well as weaknesses which will enable the development of mitigation strategies on both organisational and industry-wide levels.

1. Introduction

Managing safety risks in the Commercial Air Transport System in Europe is achieved by a 5-step ‘safety risk management process’ (as shown in Figure 1), which requires collaborative efforts by European Aviation Safety Agency (EASA), National Aviation Authorities of EU member states and—most importantly—all the other stakeholders in the industry.

In order to address the industry-wide risks, EASA annually publishes the following two key documents:

• Annual Safety Review.
• European Plan for Aviation Safety (four year rolling plan) which now also includes Rulemaking and Safety Promotion Programme.

Annual safety reviews include multiple, domain specific ‘safety risk portfolios’, which are developed based on the analysis of accidents, serious incidents, and other reportable occurrence data. This analysis is further reviewed and assessed by the domain focused ‘collaborative analysis groups’ (CAGs), which include representatives from the industry. CAGs function is to contribute to the first and second steps in the ‘safety risk management process’, which ultimately aims for the development of the ‘European Plan for Aviation Safety’ [1] (pp. 15, 16).

1.1. Study Rationale

The most recent in-depth study into the nature of aircraft maintenance error, by the UK CAA [2], was published in 2015, yet only made use of data up to 2011. This shows that there is an underlying need to provide an up-to-date analysis of maintenance error types in order to understand the trends as well as emerging issues. Additionally, a majority of the most recent available studies are only from the 1990s, warranting the scope of this study to look at maintenance accidents and serious incidents from the early millennium onwards.

There is also suggestion that the most popular aircraft maintenance taxonomies presently in use can be difficult to apply to retrospective analyses. Therefore, exploration of a new taxonomy to further aid the process may assist in the categorisation of aircraft maintenance-related occurrences. Further to this, it may be of benefit to review how the discerned maintenance-specific issues correlate with key risks identified for CAT as a whole, as discussed by the UK CAA [2,3] and EASA [4].

Problem statement: since 2015, the European Aviation Safety Agency consistently identified aircraft maintenance as one of the safety issues and included it in the safety risk portfolios for “Commercial Air Transport (CAT)–Large Aeroplanes” [4,5,6,7]. However, there has been no further analysis conducted to enable the stakeholders to develop any mitigation strategies to address risks associated with aircraft maintenance and continuing airworthiness. Therefore, further analysis of accidents, serious incidents, and occurrence data was essential to better understand the causal and contributory factors in this area.

While the analysis of occurrence data extracted from European Central Repository was subject to another study [8], this paper focuses on the accidents and serious incidents where maintenance actions or continuing airworthiness processes played a causal or contributory role. The ICAO Annex 13 definitions of ‘accident’, and ‘serious incident’ apply, which can be found in Appendix B. Also, the details of the nature of flights are shown in

Table 1. Breakdown of Events by Nature of the Flights (the full list of categories for the ‘nature of flights’ used by the Aviation Safety Network can be accessed @ https://aviation-safety.net/about/ASN-standards.doc).

Nature of Flights (Analysed by the Researchers)	Number of Events
Domestic Scheduled Passenger	31
Cargo	23
International Scheduled Passenger	17
Passenger	12
Scheduled Passenger	12
Domestic Non-Scheduled Passenger	10
Executive	2
Ferry/Positioning	2
International Non-Scheduled Passenger	2
Non-Scheduled Passenger	1

below and the aircraft types can be found in Appendix A.

1.2. Aim and Objectives

The aim of this research is to explore the nature of aircraft maintenance-related accidents and serious incidents between 2003 and 2017, in order to better understand this safety-critical function.

In order to achieve this aim, the following objectives have been developed:

• Identify the maintenance-related accidents and serious incidents for CAT category aeroplanes, between 2003–2017;
• Qualitatively analyse the accident/serious incident data using thematic analysis;
• Develop and validate a taxonomy which stems from this qualitative analysis; and
• Propose next steps for how such data analysis output may be used to aid in identifying high-risk areas and mitigation strategies.

Achieving the above aim and objectives will enable the development of some specific safety issues related to aircraft maintenance with clear focus. They can be further scrutinised by industry experts in CAG and then included in the Safety Risk Portfolio for Commercial Air Transport. Subsequently, mitigation strategies can be developed and included in the next ‘European Plan for Aviation Safety’.

1.3. Scope

The accident rate in aviation is very low. The EU member states’ accident statistics for each domain can be seen in

Table 2. Cross-Domain Comparison of EASA MS Aircraft Fatal Accidents and Fatalities, 2008-2018 (Annual Safety Review 2019 [1] (p. 27)).

Aircraft Domain	Fatal Accidents 2018	Fatal Accidents 2008–2017 Mean	Fatalities 2018	Fatalities 2008–2017 Mean	Fatalities 2008–2017 Median
CAT Airlines	0	0.8	0	66.1	4.0
NCC Business	1	0.4	1	0.9	0
Specialised Operations	6	6.8	7	13.8	13.0
Non-commercial Operations	49	47.1	95	86.0	82.0

below. Global statistics also show similar trends [9]. It is clear that accident rate in commercial air transport is much lower than non-commercial operations. The analysis of all accidents and serious incidents (including non-commercial and military events) related to aircraft maintenance could have produced a larger data set and potentially revealed interesting and beneficial results; however, there are considerable differences between how large commercial aeroplanes and small airplanes involved in non-commercial operations or military aircraft are maintained.

Considering the problem statement described in the introduction, and the aim and objectives defined above, the scope of this study was determined to be limited to the global commercial air transport accidents and serious incidents related to aircraft maintenance and continuing airworthiness.

2. Methodology

Firstly, based on the problem statement, a conscious decision was made to analyse secondary data related to accidents and serious incidents related to aircraft maintenance.

Secondly, an ‘Interpretivist’ philosophical approach formed the basis of this research.

“…….. interpretative and constructionist research does not only focus on the content of empirical data, but also on how the content is produced through language practices. Furthermore, research done from these philosophical positions does not predefine dependent and independent variables, but focuses on the full complexity of human sense making as the situations emerge. It is also assumed that there are many possible interpretations of the same data, all of which are potentially meaningful.”

Eriksson and Kovalainen [10] (p. 21)

Thirdly, an inductive approach was used during the design of this study. According to Eriksson and Kovalainen [10] (p. 24], “the research process develops, starting from empirical materials, not from theoretical propositions”.

Considering these chosen philosophical positions and methods in relation to research, the data was not analysed by using existing taxonomies such as ICAO ADREP, ECCAIRS, HFACS, etc., but coded by using thematic analysis. In order to apply rigour to the research process, primary data was also collected from subject matter experts (SME’s) to receive feedback not only about the outcome of the analysis but also about the methods used and the taxonomy developed. The overall research design and key steps followed, can be seen in Figure 2 below.

2.1. Secondary Data: Accident and Serious Incidents

In order to discern the maintenance-related accidents and serious incidents, two sources were consulted: Aviation Safety Network’s (ASN) Accident Database supported by Flight Safety Foundation; and SKYbrary’s Accidents and Incidents database. Generating the data set for the analysis involved the review of all accidents and serious incidents to identify the aircraft maintenance-related events for CAT category aeroplanes occurred between 2003 and 2017. These events, once identified, were then compiled within a singular dataset, which can be provided on request.

The ASN database contains data on worldwide accidents and hijackings involving airliners (of 12 or more passengers), military transport aircraft and corporate jets since 1919. In order to refine this data to appropriately match the scope of the study, it was first filtered for the date range of relevance and then further filtered for accidents only relating to CAT aeroplanes. The remaining data was then reviewed to ensure that only maintenance-related accidents were contained within the dataset.

The second source, SKYbrary’s Accidents and Incidents Database, was also refined for the time period of interest, and then filtered for airworthiness events related to maintenance. Where accidents were identified in both the ASN and SKYbrary databases, the relevant information was merged within the dataset.

The official investigation reports for these events were then sourced and consulted to ensure the validity of the data provided for each of the events within the dataset. Some events listed on the ASN database did not have traceable official investigation reports. The majority of these events were omitted from the dataset as it was not possible to assure their validity. However, a small selection of these events were allowed into the dataset, when there was significant indication within the narrative of maintenance-related contributory factors.

2.2. Primary Data Collection: Subject-Matter Experts (SME)

In order to scrutinise the results of the data analysis, SME feedback was sought. The method for this data collection was an online questionnaire, which was delivered through Qualtrics survey software.

The questionnaire had four open-ended questions and was distributed to five participants, who are from International Federation of Airworthiness and had extensive experience in design, production, operation, and maintenance domains including regulatory oversight and safety data analysis. The topics covered by the questions were:

• Experience of and opinion on existing taxonomies;
• Feedback on the study’s methodology and taxonomy coding;
• How they would approach classifying risk from coded events; and
• Suggesting use for the research findings in helping regulators to plan better mitigation action or increased/targeted oversight.

The questionnaire was accompanied by a PowerPoint presentation which detailed the aim and objectives of the study alongside the project methodology. The participants were also provided with an Excel spreadsheet which gave an overview of the accidents/serious incident data, as coded by the developed taxonomy at the time.

2.3. Data Analysis

The decision to develop and validate a taxonomy suitable to code the events dataset was made due to other commonly utilised taxonomies being identified as lacking applicability for retrospective analyses. Consequently, initial qualitative analysis was required in order to create a basis for the development of this taxonomy.

Thematic analysis, which Braun and Clarke [11] explain, is a method for identifying, analysing and reporting patterns (or themes) within data, was chosen as the primary qualitative analysis method for this study. A specific type of thematic analysis, known as template analysis, was utilised for the purpose of taxonomy creation and development.

Brooks et al. [12] describe template analysis as:

“a form of thematic analysis which emphasises the use of hierarchical coding but balances a relatively high degree of structure in the process of analysing textual data with the flexibility to adapt it to the needs of a particular study.”

The coded themes which emerge from the data during this analysis are known as the “template” [13] and it is this template which forms the impetus for the taxonomy creation. The template analysis structure tends to be hierarchical with sub-themes emerging within themes [13], ideal for the development of a taxonomy.

NVivo 12 Plus qualitative analysis software was selected as the main tool for the template analysis. The events contained within the dataset were uploaded into the software as individual “cases”, where each event was analysed for key themes, known as “nodes”. This inductive thematic analysis is described by Braun and Clarke [11] as a “process of coding data whereby no attempt is made to fit it into a pre-existing coding frame, or the researcher’s analytic preconceptions”.

During the generation of the baseline themes, the language of other taxonomies was consciously not utilised, in order to encourage the development of a new taxonomy which would classify the event categories purely from the narrative of the official accident/serious incident reports, and without interpretations or assumptions.

Once the baseline themes were identified within NVivo, as the template, it was then possible to begin creation of the taxonomy, named ‘MxFACS’ (Maintenance Factors Analysis and Classification System). This process initially involved separation of the node coding into a three-level hierarchy:

• Level 1—Event outcome;
• Level 2—System/component failure causing the accident/serious incident; and
• Level 3—The maintenance contributing factor(s) which led to the system/component failure and the ultimate accident/serious incident

The selection of this hierarchical structure was developed from the Bowtie Model (illustrated in Figure 3) to complement risk analysis and assessment processes. While using this model, inevitably there is considerable level of subjectivity and interpretation involved. For example, Level 1 Event Outcomes can be considered as ‘Top Events’ or alternatively if Level 2 ‘System/Component Failures’ are considered as top events, then the Level 1 ‘Event Outcomes’ will be considered at ‘Consequences’ on the right-hand side of the bowtie. Subsequently, Level 3 factors can be considered as weaknesses in the barriers or escalation factors. In this paper, it is not our aim to define all of the specific components of bowtie model with a rigid approach but ultimately for each event that a bowtie analysis can be conducted, the ‘hazard’ would be the work(s) undertaken by the maintenance personnel which resulted in the accident or serious incident. We aim to make the strong link between the maintenance factors (high risk areas within maintenance environment) and the key risk areas identified in ‘Safety Risk Portfolios’ published by EASA. Without presenting this information to industry representatives contributing to the CAT-CAG, it is extremely challenging to influence the decision making for taking risk mitigation actions and include them in the European Plan for Aviation Safety.

Once distinguished into three levels, each event was then re-evaluated and coded in accordance with the template. This process allowed for evolution of the taxonomy as more appropriate themes became apparent throughout the coding. Once the MxFACS taxonomy had been fully established, each event was assessed and classified in accordance with MxFACS in order to allow for further analysis of the output, and associated high risk areas, to be undertaken.

Once the taxonomy and resultant output had been finalised, it was then possible to utilise this data to produce a sample Bowtie model. This was achieved using BowTieXP software and allowed for a demonstration of the applicability of MxFACS to existing risk assessment methodologies.

Following the collation of SME responses, their data was entered into NVivo. This allowed for the identification of the themes within each question answer in order to reflect upon the study’s methodological framework and to provide guidance on future utilisation of the coded data.

Whilst inferential statistical analysis of other variables were considered (for example: country or operator type), it was concluded that the combination of the number of events within the dataset over an extended period of time would not allow for statistically significant results to subsequent from such analyses.

2.4. Assessing the Rigour of the Research Process

Whilst a study’s validity and reliability are key concerns for any piece of research, Liamputtong and Ezzy [15] argue that such terms are problematic in their application to qualitative research, suggesting that they are more suited to quantitative methods. The term ‘rigour’ is therefore used as a more appropriate, conceptualised measure of the underlying themes addressed by ‘validity’ and ‘reliability’ [15].

Inter-rater reliability, or inter-rater concordance, is a tool used within qualitative analysis to assess the level of agreement amongst two or more ‘raters’ [16]. However, as highlighted by Liamputtong and Ezzy [15], it does not guarantee the reliability or validity of interpretations but is a useful tool in assessing the rigour of qualitative research.

Cohen’s Kappa [17] is a popular statistic for inter-rater concordance; it shows the proportion of agreement, corrected for chance. Equation (1) demonstrates how Cohen’s Kappa is derived, while Equations (2) and (3) detail how the components of the formula are determined.

$$\kappa = \frac{P_o-P_e}{1-P_e}$$

(1)

where κ = Cohen’s Kappa; $P_o$ = joint probability of agreement; and $P_e$ = chance agreement.

$$P_o= \frac{{{\displaystyle \sum }}_{i=1}^{n}R}{n}$$

(2)

where $P_o$ = joint probability of agreement; $R$ = rater agreements; and $n=$ total number of ratings.

$$P_e= \frac{{{\displaystyle \sum }}_{i=1}^n\left(\frac{c_i\times r_i }{n}\right)}{n}$$

(3)

where $P_e$ = chance agreement; $c=$ column marginal; $r=$ row marginal; and $n=$ total number of ratings.

In order to assess the rigour of this study, a SME coded a sample of 10 events using the MxFACS taxonomy. The SME’s responses were then compared with the researcher’s so that Cohen’s Kappa could ultimately be determined. IBM SPSS statistics software was used to aid in the determination of a Cohen’s Kappa value for the taxonomy as a whole.

2.5. Ethical Considerations

Whilst ethically low-risk, the study does contain survey elements, so it is therefore essential that the confidentiality of participants is maintained throughout. In complement of this, it was necessary to acquire informed consent prior to conducting the questionnaire. Research ethical approval was sought and granted by the university (Reference: CURES/6042/2018).

3. Results and Discussion

The results of the study, alongside a critical discussion of their implications, are presented herein.

3.1. Taxonomy Inter-Rater Concordance

Altman [18] categorises levels of agreement for Cohen’s Kappa as shown in

Table 3. Levels of agreement, adapted from Altman [18].

Value of Kappa	Strength of Agreement
<0.20	Poor
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Good
0.81–1.00	Very good

.

The Kappa value for this study’s inter-rater concordance, derived using IBM SPSS Statistical Software, is given in

Table 4. Determined inter-rater concordance.

	κ	Level of Agreement
Researcher and SME	0.90	Very good

. The derived agreement statistics are provided in

Table 5. Derived agreement statistics across all levels.

Researcher and SME	κ	${\mathit{P}}_{\mathit{o}}$	${\mathit{P}}_{\mathit{e}}$
Overall	0.90	0.90	0.03
Level 1	0.80	0.80	0.002
Level 2	0.70	0.70	0.001
Level 3	1	1	0.0001

.

This shows a good to very good strength of agreement between the researcher and SME when utilising the MxFACS taxonomy and consequently indicates a high level of rigour for this qualitative research.

3.2. Distribution of Serious Incidents and Accidents for 2003–2017

From the data collection process, 112 accidents and serious incidents were identified as maintenance-related for 2003–2017. The global distribution for these events can be seen in Figure 4a,b.

The results shown in Figure 4a vary quite significantly from the maintenance threat levels given by IATA [19]. IATA [20], do however highlight that air accident investigations require greater cooperation on global standards; of around 1000 accidents over the last ten years, accident reports are only available for approximately 300. They state that of those reports, many contain insufficient information or lack rigorous analysis [20]. Consequently, this may provide some justification for the disparity.

A comparison of the number of EASA Member State events identified for the same time period as EASA’s 2014–2017 Annual Safety Reviews (ASR) is shown in

Table 6. Comparison of study data with EASA ASRs.

Time Period	ASR Date	n (Study)	n (ASR)	Difference
2012–2016	2017	13	14	−1
2011–2015	2016	13	8	+5
2009–2013	2014	23	3	+20

.

A plausible explanation for this disagreement may be due to interpretation of “CAT” used within the studies. EASA’s ASR focus on CAT aeroplane airline operations for aircraft greater than 5700 Kg maximum take-off weight (MTOW), which they describe as covering “the bulk of the commercial air transport activity” [7]. Comparatively, this study analysed events for any flight which could be acknowledged as CAT under Commission Regulation (EU) no. 965/2012, regardless of number of passengers, aircraft type, or MTOW.

3.3. Level 1—Event Outcomes

112 top-level event outcomes were coded with the MxFACS taxonomy. Figure 5 illustrates the top-level event outcomes. Once the initial outcome category for each event was identified, they were coded in further depth to better detail the nature of the event.

Trends within this data are in complement of the UK CAA’s Significant Seven [21], and the Key Risk Areas (KRA) identified by EASA [4]. The KRA’s present the outcome of accidents in the Safety Risk Portfolios published by EASA. Of the applicable top safety risks the UK CAA [21] present, three can be seen as of relevance from this study: runway overrun or excursion; airborne and post-crash fire; and loss of control (discussed further in Section 3.4). Whilst possible for maintenance actions to result in a runway incursion or ground collision, none of these types of events were identified within the study’s time period.

Further exploring these Level 1 results alongside the five maintenance KRAs, runway excursion can be seen as the highest-ranking event outcomes in Figure 5. Terrain collision is fifth overall, and aircraft environment demonstrates some relevance within the study also.

3.4. Level 2—System/Component Failure

When coding the dataset at Level 2, 112 events were coded at the system/component level, as demonstrated by Figure 6.

Inevitably, due to differing taxonomies, it is not possible to directly compare the study’s findings with previous research. However, when looking at similarities between the results of the UK CAA [2,3], considerable homogeneity exists. In both UK CAA [2,3] studies, powerplant, landing gear, and flight controls are respectively ranked as the second to fourth most populous areas for maintenance error. These three systems are ranked in the same order, but as the first to third most coded, within the top-level coding for MxFACS Level 2 (the ‘engine’ category can be used interchangeably with ‘powerplant’).

Given that the CAA studies were of low-level occurrences identified in MORs, and this comparison evidences the trend continuing to permeate through to the higher-level serious incidents and accidents, these three systems can be said to be of continuing significance from 1996 through to present day. Without further research into the targeted actions of maintenance organisations, it is not possible to postulate whether or not this trend continuation is due to inattention to this area of aircraft maintenance. It does however highlight key areas for regulators to target in their risk identification and assessment processes and when proposing oversight measures.

A plausible explanation for equipment and furnishings not attracting a higher frequency of coding within the findings of this study is the severity of the events analysed. In the analysis of low-level, low-severity MORs, it is understandable that frequent occurrences involving the equipment and furnishings listed in ATA 25 will arise. It would, however, be rare for such events to propagate into a serious enough outcome for a serious incident, or even accident, to transpire. Thus, one can expect to see a significantly lower number of occurrences related to ATA 25 within this study.

After identifying the initial event type for each occurrence, more detailed coding of the nature of these events was undertaken. The loss of control events affiliated with ‘flight controls’ can be seen as the third-highest ranking event type. This area is identified as of high risk by the UK CAA [2] and EASA [4], so this prevalence within the dataset therefore indicates a high frequency of event for a high-risk event.

3.5. Level 3—Maintenance Factors

Coding at the third level of the MxFACS taxonomy revealed 221 maintenance factors which were identified as contributory to the events in the dataset. The high-level maintenance factors are shown in Figure 7.

As with previous studies [2,3,22] cited in [23,24,25] omission errors, particularly incorrect installations, remain prevalent. However, the number of commission errors (procedures undertaken but not to an appropriate level) are also of note. Particularly inadequate maintenance procedures, which are not only the largest commission error but also the leading maintenance factor overall. Inspections, incorrect procedures, and operator oversight are further areas of significance within the dataset.

An initially surprising result is the low presence of human factors (HF) analysis within the published accident/serious incident official reports. The data was coded based only on the information available; the researcher did not make any assumptions about the HF nature of maintenance factors. Whilst many of the reports detailed HF information for how the flight crew responded to the situation created as a result of the maintenance factors, little or—in many cases—no attention was given to the human performance issues from the maintenance perspective. This raises the question as to why in-depth HF analysis related to maintenance personnel is not conducted during the investigations. This is one of the most significant findings of this study. Just like the flight crew, maintenance personnel performance relies on a wide range of factors from a personal and organisational perspective, to an industry-wide sociotechnical system level.

There is no doubt the investigators are constrained by many factors and finding out facts about the issues impacting on performance of the individuals involved during an investigation can be very challenging and—in some cases—impossible but as Burban [26] suggests there is a reluctance amongst some investigation bodies to fully embrace HF and address potentially important HF issues in detail in their investigations.

Many of the more-detailed maintenance factors have interdependencies where the combination of two or more of these factors led to the event. Identifying these interdependencies allows for an understanding of the barrier failings which lead to an event. One example of this is the 30 July 2008 event where incorrect maintenance procedures lead to a Boeing 777 fuel supply hose O-ring being damaged, thus causing an engine fire from the associated fuel leak. Following the aircraft maintenance manual (AMM) procedure should normally have prevented this from taking place, however the AMM did not contain the appropriate information for this procedure and thus contributed to the event [27].

3.6. Sample Bowtie Application

A sample event, Engine Fire, was selected and analysed by developing a Bowtie shown in Figure 8 in order to demonstrate the applicability of MxFACS output to existing risk methodologies, currently utilised by regulators. This allows users of the MxFACS data to better understand what further analysis may be required or can be done as well as the interdependencies between maintenance factors. Consequently, this aids the risk analysis and assessment processes through qualitative analysis of the control barriers which are in place.

It should be noted that the sample shown in Figure 8 shows only the threats, consequences, and escalating factors derived directly from the MxFACS qualitative analysis process. Such bowties would require analysts to further evaluate the barriers which failed in order for the top event to occur, as part of their risk analysis processes.

By developing and maintaining such Bowties, it would be possible for users to continually monitor the effectiveness of barriers as part of their risk management process, as well as aiding in identifying the particularly high-risk areas which require further attention. This process is of course reliant upon an appropriate level of detail being captured following an event so as to be able to accurately depict the barriers which were in place and failed. Otherwise, the process becomes an exercise of interpretation as opposed to use of factual knowledge.

3.7. SME Survey

A multitude of information was collected within the responses to the SME questionnaire. The most noteworthy elements from the survey were extracted to be further discussed.

3.7.1. Challenges with Taxonomies

A particularly insightful description of the challenges faced when using taxonomies to code aircraft maintenance-related events was provided by this SME:

“All taxonomies suffer the problem that categorisation can condition results… [Classifying] events after the event often requires a lot of imagination. There’s an inverse relationship. Rare fatal accidents provide much detail whereas frequent occurrences can be one line in a log book.”

The challenge of attaining detailed information from low level occurrences was certainly faced in the initial data collection process for this study. It was behind the reasoning to scope the research to focus on the lower quantity, but higher quality, serious incidents and accidents as opposed to high volume, minimal detail, low-level occurrences. That does not mean to say that these low-level occurrences should be ignored, to the contrary, but rather that further action is required to ensure the associated reporting processes capture adequate, actionable information.

Regarding the statement on categorisation, it can be argued that using peer review to determine inter-rater concordance could certainly aid in reducing the subjectivity of taxonomies. It was also highlighted by another SME that this use of peer review to assess the inter-rater concordance of the taxonomy categorisation, as was undertaken for the purpose of this study, matched the methodology of the UK CAA [3] which had a peer review to try and validate the initial categorisation.”

One SME argued that the importance in learning from occurrences does not lie in coding the existing data for interpretation, but rather in comparing theory with reality: “Taxonomy is unimportant. What is important is comparing practice with prediction.”

These SME opinions suggest that the utility of pre-existing aircraft maintenance taxonomies may perhaps be a source of some contention.

3.7.2. Feedback on the Study’s Methodology

Feedback on the study’s methodological framework was largely positive, as this extract demonstrates: “The study has done a great job in collecting the data and classifying it into useful information.”

Another SMEs detected the traits of Bowtie within the taxonomy, referencing the “threats” and “escalating factors” as: “causal factors (remove them and the accident is avoided) and circumstantial factors (increased the probability of the event).”

Whilst not directly referring to bowtie, this statement does reflect the thinking that there are different types of factors which can contribute to accidents, serious incidents and occurrences. The decision to name MxFACS Level 3 “maintenance factors” was done so for precisely this reason, so it is therefore encouraging to hear a SME mirror this sentiment.

3.7.3. Assigning Risk and Identifying High Risk Areas from Coded Event Data

One SME proposed an assessment of the effectiveness of remaining safety barriers for non-accident level occurrences as a means of assigning risk, acknowledging existing EASA methodology:

“The risk should be based on the effectiveness of the remaining barriers left before it ended up as a credible accident. See ARMS methodology.”

[28]

This statement brings about the consideration of the use of MxFACS output in conjunction with the ARMS Event Risk Classification (ERC). The UK CAA [2] highlight that Bowtie is often used within the ARMS methodology regarding the ERC barrier effectiveness assessment. This shows a possible pathway for the integration of MxFACS with the ARMS ERC methodology as both have strong applicability to bowtie.

Another SME proposed the use of an expert panel to effectively assess and allocate risk:

“Use a team of experts. [It is] hard work to get consensus but expert challenge is a good way of getting a realistic classification. Use a problem statement to get everyone on the same page.”

The use of MxFACS output, alongside the associated Bowties, could aid this approach by providing the experts with an outline of the risks and barriers involved in the events to be analysed.

3.7.4. Using the Findings of the Study

It was suggested by one SME that the key to better targeted action may be to address near misses rather than accidents: “Focus on the near miss events rather than the accidents to try and determine how close to an accident we are.”

In contrast, another SME, who highlighted a regulatory resource shortage as being challenging to acting on occurrences, suggested a different approach:

“[Regulators] all struggle with limited resources-being driven by events provides only the basics. Continuous improvement means being proactive. Uncover the trends, pick off 5 ‘low hanging’ fruit and work them through. A Pareto analysis was used by US CAST to good effect… Target priorities on the higher risk items that are easiest fixed first.”

This complements the thinking that prevalence of high frequency occurrences does not necessarily indicate a propensity for accident propagation from their associated risks, particularly if sufficient barriers are in place. By instead focusing on identifying high risk areas within maintenance, regardless of the frequency of actual catastrophic outcome, it could be argued that the resultant targeted prevention strategies may be more effective.

3.8. Identifying High Risk Areas

Maurino [29] proposes that the findings of investigations should encourage error tolerance and error recovery, as opposed to error suppression. By identifying the high-risk areas in aircraft maintenance, it is possible to understand what factors shaped particular human errors.

Upon reviewing the fatalities and damage count for the 112 analysed events, it was found that 16 flights had a fatal outcome and 77 lead to aircraft damage. EASA [4] identify damage to be of a medium level risk within their key risk areas. As 69% of the events identified within the dataset resulted in some level of damage, it can be said that the likelihood of event for this KRA was substantial over the past 15 years.

In order to better relate the frequency of events data from Figure 5, Figure 6 and Figure 7 to risk, the events within the dataset were evaluated at each level of the taxonomy for number of fatal accidents and instances of aircraft damage. The fatal accident figures were then plotted alongside number of events and number of fatal events (represented as the size of the bubble), to replicate the same risk visualisation approach used by EASA [4] in Figure 9.

Figure 10 shows this chart for the Level 1 results, with more detail about the relationship between fatalities, the number of fatal events (represented by the size of the bubble), and the total number of Level 1 outcomes.

Three of the four event outcomes identified as having fatal outcomes are congruent with three of the maintenance KRAs listed in EASA ASR [4]. One particular point of significance is the positioning of collision: this area has a large proportion of fatalities, damage and frequency; it could be a key area of focus for further risk analysis processes. The coded MxFACS data may be used in conjunction with analysis methodologies such as bowtie to examine the particular barrier failings which lead to these kinds of accidents. Further information about aircraft damage can be found in

Table 7. Level 1 fatal accident and aircraft damage relationships.

Event Outcomes	Fatal Accident		Aircraft Damage
	n (Fatal Accidents)	% of Total Fatalities	n (Aircraft Damage)	% of Total Damage
Runway-related events	1	6.3%	19	24.7%
Collision	11	68.8%	17	22.1%
Diversion or air turnback			10	13.0%
Landing-related events	3	18.8%	10	13.0%
Fire	1	6.3%	6	7.8%
LG-related events			6	7.8%
Structural damage			6	7.8%
Depressurisation			2	2.6%

.

The ranked orders for greatest number of fatal accidents and aircraft damage for Level 2 of the dataset are given in

Table 8. Level 2 fatal accident and aircraft damage relationships.

System/Component Failures	Fatal Accident		Aircraft Damage
	n (Fatal Accidents)	% of Total Fatalities	n (Aircraft Damage)	% of Total Damage
Engine	8	50.0%	29	37.7%
Landing gear	1	6.3%	29	37.7%
Flight controls	2	12.5%	5	6.5%
Steering			3	3.9%
Electrical power	2	12.5%	2	2.6%
Fuel			2	2.6%
Instrumentation and indication	1	6.3%	2	2.6%
Structure	1	6.3%	2	2.6%
Windscreen			1	1.3%
Workload	1	6.3%	1	1.3%

.

Figure 11 shows the relationship between fatalities, number of fatal events (represented by the size of the bubble) and the total number of Level 2 events.

Engine-related events can be seen to have the largest propensity for both fatalities and aircraft damage. As shown in Figure 11, these events were also the most frequently occurring across the dataset. This would suggest that maintenance related to aircraft powerplants should be placed high on the agenda for regulators when proposing better-targeted action and oversight, as well as being a key focus for maintenance organisations. Similar comment can be made in relation to landing gear and flight controls, which also rank highly across all three areas.

It is more difficult to directly compare the Level 3 events with the fatality and damage figures as many of the events have multiple maintenance factor categorisations assigned to them. Therefore, the events at this level were analysed as a percentage of the total number of instances where a maintenance factor was attributed to a fatal accident or aircraft damage. The maintenance factors related to fatal accidents and aircraft damage are shown in Figure 12, with a full breakdown given in

Table 9. Breakdown of maintenance factors related to fatalities and aircraft damage.

Maintenance Factor	% Related to Fatalities	% Related to Damage
Inadequate maintenance procedures	14.6%	14.4%
Inspection does not identify defect	7.3%	9.2%
Incorrect procedure	2.4%	7.8%
Incorrect installation	4.9%	7.2%
Operator’s inadequate maintenance oversight	14.6%	6.5%
Incorrect component installed	4.9%	4.6%
Insufficient inspection	2.4%	4.6%
AMM Not followed	2.4%	3.9%
Non-airworthy component released into service	7.3%	3.9%
Inadequate maintenance documentation	2.4%	3.3%
Overhaul not undertaken	7.3%	3.3%
Check not undertaken	2.4%	2.6%
Inadequate instructions		2.6%
Part or latch not secured	2.4%	2.6%
Incorrect adjustment		2.6%
Incorrect assembly	2.4%	2.6%
Part missing		2.0%
Inspection not undertaken	4.9%	2.0%
Regulator’s inadequate maintenance oversight	7.3%	2.0%
Not followed		1.3%
AMM procedures difficult to follow		1.3%
Incorrect rigging	2.4%	1.3%
No fault found		1.3%
Inadequate reporting	2.4%	1.3%
Part used past expiry		1.3%
AMM incorrect information	2.4%	0.7%
AMM missing information		0.7%
Inadequate check	2.4%	0.7%
Tool left in aircraft		0.7%
Inadequate training		0.7%
Deliberate exceedance of service lifetime		0.7%

.

Inadequate maintenance procedures and operator oversight can be seen as the two predominating areas within the top nine maintenance factors for fatalities. This may suggest that organisational safety management requires particular attention and would perhaps warrant further risk analysis to identify the interdependencies which interact with these maintenance factors.

4. Conclusions

In order to conclude this study, it is of relevance to first evaluate the challenges and limitations before drawing together a series of final concluding remarks.

4.1. Challenges and Limitations

A number of challenges and limitations were encountered within the study. These were in relation to data collection and analysis, SME survey, and the MxFACS taxonomy as a whole.

4.1.1. Using the Selected Data Sources

One limitation of the data collection for this study was the sourcing of credible aircraft maintenance events. The data was collected from English language databases, and relied on English translations of investigation reports being available where the investigating body was not English-speaking. This means that the dataset, although very much international, may not be 100% globally representative.

The broad nature of the databases used to collect the data was a further challenge to the data collection process. Manual extraction of maintenance-related accidents and serious incidents was the only means of identifying these events, amongst a vast quantity of events which were irrelevant to the study.

4.1.2. Data Analysis

It was challenging to analyse the data in a way that would provide meaningful and insightful output as the focus of the study was somewhat narrow, and there is little existing literature for means of comparison. By evaluating the most recent and relevant studies [2,3,4,5] it was subsequently possible to identify key areas to initially focus upon.

4.1.3. Taxonomy Development and Inter-Rater Reliability

As mentioned in Section 2.4, only one subject matter expert coded a sample of 10 events out of 121 using the MxFACS taxonomy. Despite the Cohen’s Kappa was calculated and revealed a certain level of confidence, this is still one of the limitations of this study. Nevertheless, subsequent to this study, MxFACS taxonomy was used during another research project analysing the accidents and serious incidents only in Nigeria and it was found to be beneficial to analyse all events in the dataset.

4.1.4. SME Survey

Whilst the SME survey provided a number of insightful and helpful comments, it proved difficult to extract the desired level of information through open-ended questions than would have been available through more-structured interviews.

It was initially the intention to conduct semi-structured interviews with these SMEs, however the logistics of such interviews proved to be challenging to arrange, particularly due to differing time zones. Consequently, a questionnaire was decided as a more efficient means of collecting this data.

4.1.5. MxFACS

The MxFACS taxonomy aids in identifying three out of four of the basic elements of error identified by Reason [30]; the action, the outcome, and the context. However, it is not always possible to determine from investigative reports the intention of those who produced the errors.

As such, interview (perhaps in the MEDA format) with the maintenance personnel involved with particular events, shortly after the event, would complement a more representative coding of the accident or serious incident. This is not without its challenges, particularly for lower level occurrences where investigation may be undertaken sometime after the fact, therefore making it difficult for the personnel to accurately recall the intentions which lead to the error.

4.2. Study Conclusions

The study provides a modern-day and maintenance-specific viewpoint on CAT accidents and serious incidents. The development of the MxFACS taxonomy brings about a contemporary approach for exploring the nature of these maintenance events, by looking at a combination of maintenance-specific causations, system/component failures, and event outcomes. Such taxonomy output is demonstrated as being applicable to existing risk analysis processes and methods and could be used to complement existing taxonomy and methodologies.

The results of the study show that the three most frequent maintenance events for 2003–2017 are runway excursions, air turnbacks, and on-ground landing gear-related events. The most common system/component failures were related to engine, landing gear, and flight controls. At the causation level, the largest maintenance factor issues were inadequate maintenance procedures, inspections not identifying defects, incorrect installation, and incorrect procedures.

By combining the frequency of event data with number of fatalities, it was possible to begin to create a picture of the higher risk areas within maintenance for this time period. Collision events were the most prominent consequence, engine-related events were the largest event type, and inadequate maintenance procedures were the most concerning maintenance factor.

The study’s findings could be used to aid in a Safety-II approach to understanding where barrier weaknesses lie within maintenance safety management systems, particularly when integrated with bowtie analysis. Such an approach may allow regulators and maintenance organisations to develop means of ensuring as much as possible goes right within the maintenance system, as opposed to looking solely at just what went wrong.

5. Recommendations

In acknowledgement of SME feedback, ARMS and ERC methodologies may be explored for means of better identifying the high-risk areas within the MxFACS output. By combining the MxFACS output and fatality data with bowtie models to understand the effectiveness of the barriers in place, it would then be possible to develop an ERC score and consequently substantiate the higher risk areas for this data. This would then allow for a maintenance-specific depiction of key risk areas akin to the work of EASA in Figure 9 [4].

Further to this, it would be advisable to create a number of bowtie models for the high-risk areas and continually maintain and update these models as time progresses. This would allow for continual monitoring of the barriers in place for higher risk areas. Additionally, the MxFACS taxonomy and database should also be continually maintained and updated to ensure relevance for new accidents and serious incidents as they evolve.