What Makes a Good Critical Literature Review Paper?

1. Introduction

Literature review papers are highly influential in directing academic discourse, not only because they map the state of the art in a subject area, but also because they show future research prospects by identifying weaknesses, limitations, and gaps in the extant body of knowledge on a particular subject. They hence serve as important catalysts for future research [1]. There are, however, a number of different types of literature review [2], and many journals prefer some formats over others. Sometimes this is explicitly stated in the journal’s guide to authors. Tourism and Hospitality, for instance, welcomes both narrative and systematic reviews, but whichever of these two formats it chosen, it encourages prospective authors to submit literature reviews that take a critical approach. As such, any paper that is submitted should comprise more than a simple bibliometric approach to surveying the literature [3]. Tourism Management does not publish bibliometric studies or systematic reviews as a matter of policy [4]. Current Issues in Tourism encourages systematic studies and the use of bibliometrics, but cautions that a purely bibliometric study is unlikely to make the academic contribution required [5]. In other instances, the preference is more implicit, exerted by editors or reviewers who may or not approve of the format adopted in the article.

The key to writing a good critical literature review paper is to be sure that is not simply a literature review [2]. Both a literature review and a critical literature review should attempt to make a systematic summary of existing knowledge on the subject or theme that has been chosen for them. The vital difference is that although the former does not necessarily involve the use of a critical approach, the latter always must. Using a critical approach enables a critical literature review to stand alone as a piece of writing, unlike a literature review, which is usually just a component of a larger piece of work.

A critical literature review paper is, therefore, a self-standing academic piece of work that seeks to review the literature on a particular subject or theme using a critical-thinking approach. A good critical review paper therefore requires the use of critical thinking, which the National Council for Excellence in Critical Thinking [6], defines as:

“The intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information […] In its exemplary form, it is based on universal intellectual values that transcend subject matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.”

2. Use of a Critical Thinking Approach

2.1. Conceptualisation

All critical thinking is based on the formulation and expression of intellectual concepts and ideas. The most important aim of a critical literature review paper should therefore be to identify the concepts and ideas that define the subject or theme under discussion. What, for example, are the key concepts that have been used to underpin the subject theoretically? What key ideas that have been used to explore the subject? What methodologies and methods have been used? What new ideas have been generated as a result?

These key concepts should be identified and explained to the reader. If they are organised and expressed in a clear and logical way, it will make it easier for the reader to digest and understand them. Readers will then be in a better position to appreciate how researchers have approached the subject in the past, what the current state of the art looks like, and how researchers might more effectively address the subject in the future.

2.2. Application

The purpose of a critical literature review should be to investigate a particular research problem and set out some proposed solutions to it. This problem needs to be clearly elaborated from the outset. Frequently, the research problem is to identify what has been said about the subject, in certain respects, and to identify gaps in the literature. This is not the only possibility, however: for example, a paper on destination marketing might take the form of a critical review of literature on destination marketing organisations to determine what their format has been in the past and how their role has shifted over time. This could allow the researcher to comment on how the format of DMOs may need to change in order to allow them to undertake their destination marketing role effectively in the future.

The scope and significance of the problem also need to be stated and explained clearly from the start. Unless a critical literature review addresses a clear and significant problem; there is very little to be gained by adopting a critical-thinking approach. It also makes the paper more accessible, as it gives readers a good reason for wanting to read the paper.

2.3. Analysis

A good critical literature review paper will perform a rigorous analysis of the evidence base, i.e., the body of literature on the subject: what specific topics have been studied?; using what methodologies?; drawing what conclusions?; and facing what limitations? Individual studies can be summarised in as much detail as desired, remembering that the number of words used, and the amount of depth, will often be assumed by readers to reflect the significance of that particular study.

Like any piece of effective writing, a good critical literature review will be well ‘signposted’ to enable readers to follow the arguments being employed. This is often achieved by dividing the review paper into numbered sections and sub-sections, giving each a title that will help to orientate the reader [7].

2.4. Synthesis

A good critical literature review paper needs to synthesise the literature in order to draw clear conclusions. The analysis may be quantitative, focusing on the amount (or proportion) of papers reviewed that adopt a particular theoretical approach taken, the empirical method used, or the conclusions they draw. The analysis may examine a paper’s keywords or the frequency of particular words in the paper. Alternatively, a more qualitative synthesis can be achieved simply by grouping similar studies together. An effective critical literature review might even blend these two broad approaches. There is no correct approach: the best solution will depend on context and purpose.

It should be borne in mind, however, that purely bibliometric literature reviews are often deemed unlikely make a significant or useful contribution to knowledge. Such studies set out principally to measure the number of papers being published according to different characteristics, such as the year of publication, journal name, or country of author. They can thus identify the growth of the subject area and how it has changed in various measurable respects over time [8]. This may be useful to identify gaps but not to determine their importance; nor do such reviews assist the reader to appreciate the subtleties of research quality [9]. Indeed, a subject area may be replete with papers but none of sufficient research quality to enable the reader to have confidence that this is not actually a research gap.

2.5. Evaluation

Finally, a good critical literature review paper will perform an overall evaluation of the problem being analysed. In doing so, it will ensure that any conclusions it is drawing are based on the available evidence and consistent with one another. It will also be careful to identify and clarify the assumptions upon which these conclusions are based.

3. Use of Robust Critical Statements

To be effective as a critical review of the topic, a critical literature review will need to be constructed from critical statements, as depicted in Figure 1.

3.1. Clear

Clarity is a ‘gateway standard’ in critical analysis, i.e., it is a necessary condition for critical thinking. The reader can only judge the accuracy and precision of a critical statement if it has first been clearly expressed.

However, it is important to note that clarity is not a sufficient condition for critical thinking. Critical thinking involves more than just clear argumentation: it also requires accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.

3.2. Accurate

A statement can be clear but not accurate. For example, ‘the daily per capita expenditure of domestic tourists in the UK in 2013 was GBP 515.18′ is very clear but it is totally inaccurate (even though it gives the impression of accuracy by citing the value to two decimal places). This is because daily per capita expenditure of domestic tourists in the UK in 2013 was simply not that figure.

In the same way, the statement “Urry’s writing on the tourist gaze, which was first published in exactly the same year as Butler’s paper on the tourist area lifecycle” is clear and unequivocal, but it is also unequivocally wrong. Urry’s book [10] and Butler’s paper [11] were not even first published in the same decade.

3.3. Precise

A statement can be both clear and accurate, but not precise. For example, “many governments have adopted such a policy” does not tell the reader how many governments have adopted it or, indeed, which ones (for example, are they mainly the governments of developed or developing countries?). Precision is a feature of the arguments used in a good critical literature review paper. In particular, it can differentiate between the reader appreciating the significance of the findings of the paper and misunderstanding it entirely. In the case above, the reader might conclude that the policy is commonly adopted in developing countries when, for various vital reasons, it is generally not.

3.4. Consistent

A statement can be clear, accurate, and precise, but not consistent with other statements made in the course of the critical literature review. The author of a good critical literature review paper will check to see whether all the statements it uses are consistent with the other statements it makes. Inconsistencies, if there are any, should be drawn to the readers’ attention. Doing so will help readers to appreciate the limitations of any conclusions that are drawn from the review with respect to the issue at hand (in that they may not be entirely conclusive).

3.5. Relevant

A statement can be clear, accurate, precise, and consistent, but not relevant to the subject of the critical literature review. Relevance should itself be determined by the nature of the problem being addressed, which should be established before proceeding to write the critical literature review paper. Irrelevant statements should be identified and eliminated from the arguments being made, otherwise they will serve to obfuscate rather than illuminate the issues being discussed.

3.6. Evidence-Based

A statement can be clear, accurate, precise, consistent, and relevant, but not based on sound evidence. For the conclusions drawn from the critical review to be worthwhile, they must be based on robust evidence. A good critical review will restrict the inferences and interpretations it makes strictly to those that can be supported by the evidence.

A good critical literature review paper will use a recognised academic referencing system (the choice of which is often determined by the editorial policy journal to which the review is to be submitted) and this system will be applied thoroughly and consistently. Doing so will enable readers to establish the legitimacy and robustness of the critical statements being made in the paper, and thereby judge the extent to which the conclusions that are being drawn stand up to academic scrutiny.

In doing so, it is important to use each reference to its maximum potential. It is poor practice, for example, to include long, undifferentiated lists of citations to back up a critical statement. It is much better to consider how the references being cited vary in terms of how they relate to the statement, and to break the statement down accordingly.

As in all academic writing, it is entirely legitimate for the author to include their personal opinion; however, unless these opinions are based on solid evidence, they are not academically credible and should therefore not be included in a critical review paper.

3.7. Well-Reasoned

A statement can be clear, accurate, precise, consistent, relevant, and evidence-based, but not well-reasoned. A good critical literature review will develop a logical line of argument that makes sense to the reader [7]. This means selecting and using concepts that are mutually supporting. The order in which the concepts are addressed should also be well-reasoned, in that each concept should logically lead on to the next.

3.8. Deep

A statement can be clear, accurate, precise, consistent, relevant, evidence-based, and well-reasoned, but nevertheless lack intellectual depth. A critical review is of little value if it only superficially deals with the issues at hand. A shallow line of argument may, for example, effectively trivialise an issue by failing to recognise its full complexity.

3.9. Broad

A statement can be clear, accurate, precise, consistent, relevant, evidence-based, well-reasoned, and deep, but nevertheless be narrow. For example, it might be possible to present a statement from the perspective of either conservative or liberal political views, but if only one of these is presented in the critical review paper, it will lack breadth. A critical literature review needs to be ‘even’, insofar as it recognises the full breadth of perspectives possible.

3.10. Fair

A statement can be clear, accurate, precise, consistent, relevant, evidence-based, well-reasoned, deep, and broad, but not fair. All reasoning is, of course, performed from a particular point of view. A good critical literature review will, however, seek to incorporate a range of points of view, other than that of the author. The conclusions of a critical literature review are likely to be biased if only data that support one viewpoint are used. It is important, therefore, to use a fair range of data.

4. Putting Together a Critical Argument

Critical statements need to be woven together effectively into a critical argument. Figure 2 presents a flow chart of how this can be undertaken at the paragraph level.

Each paragraph in a critical literature review should develop a complete self-contained argument addressing the purpose set for the paper. In order to construct one, the researcher first needs to marshal the relevant sources (i.e., references), which will have been identified through a literature search process and an initial reading to establish their relevance. The literature search could be systematic, in that it is based on pre-set criteria [12], or narrative, in that it is based on reviewer judgement [7]. Whichever method is used, however, should be clearly stated at the outset of the paper. In the case of a systematic review, a PRISMA statement is required [13].

Having identified the relevant set of sources, the next step is to make connections between them. This involves a more in-depth, critical reading of the sources deemed to be relevant to the specific issue being considered in each particular paragraph. The reviewer should try to identify touch points in terms of the approach, methods, theories, arguments, results, conclusions, etc.

The next step is to identify which of these touch points are in agreement and which are contradictory. In many cases, of course, it would be a little of both. This is, in itself, a valuable finding, introducing important nuance to the critical literature review. It is unlikely that the review will produce a clear-cut conclusion, and be unrealistic to anticipate one.

Having compared and contrasted the main bones of contention found in the sources, the next step is to accept or reject them as propositions. This must, of course, be performed on the basis of evidence, and the authors’ reasons for accepting or rejecting propositions made in the sources about the topic at hand must be backed up with references. These references may come from within or without the sources collected in the literature search.

The researcher will now be in a position to establish the limitations of the current literature base in respect of the stated purpose of the critical literature review. There may, for example, be an absence (or near-absence) of research focusing on the application of a particular method of analysis. Alternatively, the gap may be in terms of a lack of research examining a particular phenomenon within the scope of the research problem.

The final step will be to indicate the conclusions for this particular paragraph of the critical literature review. These should relate explicitly to the research purpose. The paper can then move on to consider the next topic of interest in the subsequent paragraph.

5. Conclusions

There is, of course, no ‘magic formula’ for writing a critical literature review: each will be a one-off that would be impossible to replicate through further attempts. Indeed, any new attempt to write a critical literature review on the subject will need to include new sources, which are being published at an unprecedented rate. Future critical literature reviews would need to include these into the analyses and would doubtless find that some gaps in the literature have been (at least partially) filled, while others may have opened up. Similarly, the relevance of particular topics may have changed in relation to inevitable shifts in the business, political, social, economic, technological, or natural environment. As such, no critical literature review can ever claim to be decisive, conclusive, or irrefutable: it can only represent the situation at the time it is written. This means that there is always scope for another critical literature review.