Ecological Diversity: Measuring the Unmeasurable

Abstract

Diversity is a concept central to ecology, and its measurement is essential for any study of ecosystem health. But summarizing this complex and multidimensional concept in a single measure is problematic. Dozens of mathematical indices have been proposed for this purpose, but these can provide contradictory results leading to misleading or incorrect conclusions about a community’s diversity. In this review, we summarize the key conceptual issues underlying the measurement of ecological diversity, survey the indices most commonly used in ecology, and discuss their relative suitability. We advocate for indices that: (i) satisfy key mathematical axioms; (ii) can be expressed as so-called effective numbers; (iii) can be extended to account for disparity between types; (iv) can be parameterized to obtain diversity profiles; and (v) for which an estimator (preferably unbiased) can be found so that the index is useful for practical applications.

1. Introduction

A central aim of ecology is to understand the processes that sustain biodiversity, which is critically important for the viability of ecosystems [1]. Biodiversity loss degrades ecosystem functionality, and in drastic cases can even lead to mass extinctions and total ecosystem collapse [2]. In recent years, this vulnerability has worsened to critical levels as a significant portion of the Earth’s species are driven to extinction largely due to human actions [3]. Unprecedentedly high extinction rates have caused extensive and damaging changes in ecosystem structure and functionality and have brought particular urgency to the issue of biodiversity loss [4].

Biodiversity is a rich concept that admits a wide range of possible definitions. In its broadest sense, it has been defined as the variety of life forms at all organizational levels of an ecosystem, ranging from molecules over individuals to species [5]. Examples of different definitions of biodiversity thus include species (taxonomic), functional, genetic, phylogenetic, and chemical diversity, among very many others.

Any meaningful study of biodiversity, no matter which aspect is in question, must involve its quantitative measurement. This is a complex task both conceptually and practically. Biodiversity is quantified by constructing mathematical functions generally known as diversity indices. The use of such indices permits comparisons between different spatial regions, temporal periods, taxa (species), functional groups, or trophic levels. These measures are therefore essential tools for ecological monitoring and conservation, as well as for any efforts to study and address the biodiversity crisis [6].

However, there is no consensus on which indices are more suitable and informative than others. The available indices are so numerous and disparate in their ecological interpretation and mathematical behaviour that researchers must start by asking themselves the most basic question of all: what is even meant by ‘diversity’?

This review aims to provide guidance to those seeking to use diversity indices in ecological settings. Due to the multitude of available indices, such reviews arise each year in an attempt to bring some order and insight to this overwhelming body of work. This review focuses on the mathematical foundations of diversity indices, which have consequences for their use and, perhaps more importantly, their misuse.

What is driving this proliferation of diversity indices? In recent years, the era of “Big Data” has spread to ecology, in particular microbial ecology. This has driven a reassessment of which diversity indices to use to summarize and analyze these data [7]. This has led to increased focus on the mathematical behaviour of different measures, since there has been an increasing recognition that ecologists (often non-experts in mathematics) are using measures of diversity without proper consideration of their behaviour and limitations. This has led to difficulties in interpreting studies, and especially in comparing them. As we shall see, the choice of index is highly dependent on the particular study context. But there are a number of important points to consider.

We have settled on five recommendations, each of which we motivate in detail in subsequent sections. The choice of a diversity index should ensure that:

(1)

it satisfies key mathematical axioms;

(2)

it can be converted to an effective number;

(3)

it can be extended to account for disparity between types;

(4)

it can be parameterized to obtain diversity profiles; and

(5)

an estimator (preferably unbiased) can be obtained to allow the index to be used in practical applications.

We first address in Section 2 the fundamental question of how to conceptualize diversity, and the difficulties in settling on a single definition. We then outline briefly the three components of diversity in Section 3. In Section 4, we survey the axioms that characterize diversity indices, and discuss which are necessary for indices to have meaningful mathematical properties and ecological interpretations. The diversity indices most used in ecological applications are described in Section 5, along with their mathematical properties—in particular, their conversion to effective numbers (which endows a common set of mathematical properties and ecological behaviours) and the calculation of diversity profiles. In Section 6, we discuss the partitioning of diversity, which allows diversity indices to be used across different landscape scales, and is a mathematical and conceptual problem with both theoretical and practical implications. We then address in Section 7 the different techniques that can be used to estimate diversity indices in practical settings. Finally, in Section 8 we summarize the conclusions and recommendations that may be drawn from this review of diversity indices and their use in ecology.

2. Defining Diversity

Dozens upon dozens of different diversity indices can be found in scientific literature. Such an abundance of indices and their often discrepant behaviour has led to such confusion that some authors have concluded that the very concept of diversity is meaningless. Even as far back as 1971, Hurlbert was moved to declare that “the term ‘species diversity’ […] now conveys no information other than ‘something to do with community structure’; species diversity has become a nonconcept” [8], a reproval often repeated in the intervening years. Although there is an important nuance in this criticism, as we shall shortly discuss, it is also true that since then the picture has only become busier, with the plethora of diversity indices already to be found in the literature being joined by new indices proposed each and every year. We may identify four broad reasons why this is so.

First and most importantly, ‘biodiversity’ is such a broad concept that it can and has been defined in many different ways, depending on researchers’ specific needs and interests. These definitions can range from species or morphological diversity to functional or chemical diversity, and any number of others in between. One review—now already more than 20 years old—unearthed no fewer than 85 different definitions [5]. Thus biodiversity, as a general ecological concept, has been described (rather mildly) as “extremely confusing” [9]. It is also clear that different diversity indices can therefore measure patently different aspects of diversity. But even once researchers have decided which form of diversity they wish to measure, quantifying biodiversity is still problematic because there is no single index that can provide a suitable summary.

A second problem is the fact that the concept of diversity is often confounded with the indices that measure it. Jost illustrates this issue with the example of a sphere: its radius can be used as an index of its volume, but these two quantities are obviously not equivalent [10]. Analogously, the most commonly used diversity measure, the Shannon index, is actually a measure of entropy. Entropy refers to uncertainty in information: it is more difficult to predict the identity of an individual (in terms of its species, functional group, or whichever biodiversity aspect is in question) in a very diverse system, whereas this prediction is less uncertain in a system with only a few types. Hence the former system has a higher entropy than the latter. Entropy therefore shares important conceptual similarities with diversity, and hence entropy and diversity measures also share many (but not all) of the characterizing axioms that we will examine in Section 4. Unsurprisingly, entropic and other information-theoretic indices have a long history of use in ecology [11,12], and have therefore contributed significantly to the explosion in number of ecological diversity indices. But although entropy measures are reasonable and frequently used indices of diversity, this of course does not imply that entropy is equivalent to diversity. Similar arguments can be made regarding many other diversity indices.

Third, indices typically condense all relevant information about an ecosystem’s diversity into a single real number, so that there are unfathomably many ways of calculating such an index from the complex and extensive data on the ecosystem in question. Different indices can weigh different components of these data more strongly than others, and can even entirely neglect some [13]. Needless to say, this freedom of formulation has permitted the emergence of dozens of different indices, which often provide wildly different estimates of ostensibly the same quantity.

A final confounding issue is the relatedness but non-equivalence of the concept of diversity across different scientific disciplines. The obvious similarities between diversity measurement in, for example, sociology or economics compared to ecology, have encouraged the adoption of indices by one field from another, thus swelling their number without proper consideration of their conceptual differences. As an example, numerous indices have been developed in economics to measure the diversity or inequality of income, corporate productivity, or racial representation [14,15,16]. But a key conceptual difference often underlies economic and ecological studies of diversity: the latter take into account the actual abundances of the different species present in the ecosystem, while the former are instead often concerned with an abstract allocation of commodities (species), and so take no account of actual abundances [17].

Before we delve further into ecological diversity and its mathematical representations, it is important to emphasize that none of the numerous diversity indices are wrong per se. On the contrary, each index has its own unique properties that are useful for specific applications. The key point of Hurlbert’s criticism relates to diversity as a unified concept: since raw diversity indices exhibit such a wide variety of mathematical behaviours, they cannot all give reasonable results when directly inserted into any general equation or formula of diversity [8]. Thus it is critically important to consider the purpose for which an index will be used, when addressing how it is constructed mathematically and how it should be interpreted ecologically.

The purpose of an index’s use will of course depend strongly on the context of the study. Examples might include: classifying shifts in diversity after a disturbance; ranking ecosystems in terms of their biodiversity (to determine which are more in need of protection); detecting the effects of external (typically anthropogenic) factors on diversity; understanding interactions between diversity at different trophic levels (i.e., how changes in one can affect another); or searching for mechanistic explanations of diversity changes [6]. For each of these goals and many others, some indices will be more suitable than others, and a poor choice may lead to misleading or even false conclusions.

In this review, we will focus on species diversity (hereafter referred to merely as ‘diversity’ for ease of reference). But while species diversity is by far the most commonly considered form of diversity in ecology, it should be noted that other forms of diversity may in some contexts be more important. In particular, intra-species diversity (for example in individuals’ age or body size) has been recognized as often playing a more significant role in ecosystem functioning and dynamics than species diversity [18]. We focus this review on species diversity since it is more often the focus of study, and can be more daunting to newcomers to the field, due to the large and disparate amount of theory and practice that has already been established.

3. Components of Diversity

It is generally understood that diversity can be divided into three components: richness, evenness, and disparity [10]. Most indices (including almost all the classical and best established indices) do not account for disparity, and hence this component of diversity is often neglected in reviews of the topic. In this section, we provide definitions of these three components (illustrated in Figure 1), as well as a brief summary of their ecological significance.

3.1. Richness

The absolute number of species present in the population of interest is referred to as its richness. Two assumptions underlie the definition of richness [19]. First, that a classification of types exists and is known. If such a classification would not exist, any richness calculation would become difficult since it might not be clear to which class or taxon any particular individual belongs. This issue is especially relevant to microbial ecology, where the definition of species is particularly shaky due to distinct life history features like frequent genetic exchanges and the lack of sexual reproduction [20]. The second assumption is that each class is equally distinct, so that no two classes are more or less similar than any others.

In ecological terms, increased richness has been shown to both enhance community functionality (i.e., increase productivity) and to stabilize it in the face of disturbances [21,22]. Two mechanisms are believed to drive this effect [23]. The complementarity mechanism assumes that different species use different resources, even if only slightly. Then communities with higher richness will use more of the available resources, thus increasing their productivity. On the other hand, the selection mechanism assumes that different species contribute differently to the community’s overall functionality, namely that some species are more productive than others. Then richer communities are more likely to contain more productive species, enhancing the community productivity.

From a mathematical and quantitative perspective, richness is the most straightforward component of diversity, being a simple enumeration of the different types present in a community. Of course, in practice ecologists can only count those species that they can find: rarely do they encounter communities that can be fully censused, for obvious practical reasons. Instead, the richness of a community must often be estimated from a sample or subset of the community, a topic that will be discussed in Section 7.

3.2. Evenness

Aside from the total number of species in a community, the distribution of their abundances is also an important component of diversity. If a species is represented by only a few individuals, it should be clear that it contributes less to the community’s diversity than a species represented by several thousand individuals. The equitability of a community’s species abundance distribution is referred to as its evenness. A community is perfectly even if every species is present in equal proportions, and uneven if one species dominates the abundance distribution.

The important role of species evenness in ecological diversity has received less attention than that of species richness, but is also inarguable [24]. Evenness has been shown to be a key factor in preserving the functional stability of ecosystems [25], as well as in enhancing productivity by increasing the representation of each species’ functional traits [26]. In contrast, uneven communities are often more susceptible to invasion and less resilient to stresses and disturbances [27].

Unlike richness, there is no consensus on how to measure evenness. The literature relating to the formulation and choice of an evenness index is a self-contained and well-developed sub-field of the diversity index literature. This topic is more narrowly focused, since it only considers one component of diversity, and thus we restrict ourselves here to a short summary of the most important considerations relating to evenness, and mention several extensive reviews of this topic for the interested reader.

Many researchers have attempted to address the problem of choosing one evenness index from the available legion by listing desirable criteria for an index, and then assessing how well these are satisfied by candidate indices. In their influential review paper on this subject, Smith and Wilson [28] list 14 criteria for evenness indices, and subdivide these into four essential requirements and 10 desirable features. The latter have not been as widely accepted as the former, in part because some of the desirable features conflict with each other. The list has also been modified and/or expanded by others, see for example [29,30,31,32].

Thus the question of an index’s desirable or appropriate mathematical behaviour is much less settled. Most reviews on the topic conclude by stating that there is no universal way to measure evenness, and thus researchers must choose the index most suited for their particular needs [28,30,33,34,35]. While this seriously limits the comparability of different studies, it also reflects the reality that studies are generally interested in different aspects of evenness, which consequently are optimally described by different indices.

3.3. Disparity

As we will shortly see, most classical diversity indices account for two components of diversity—richness and evenness—and thus implicitly assume that distinct species have nothing in common. That is to say, they do not account for any disparity between species. This conception of diversity is referred to as species-neutral diversity [19]. It implies, for instance, that a community of six dramatically different species is considered to be no more diverse than a community of six species of butterflies.

Hence there are relatively fewer diversity indices that reflect the more realistic and nuanced situation of varying dissimilarities between species. Such indices are termed similarity-sensitive. The reason for the dearth of similarity-sensitive indices may well be the fact that “theoretical ecologists have been hesitant to introduce new diversity indices when the profusion of similarity insensitive indices is already perceived to form an impenetrable jungle” [36].

Although wading into this “jungle” may be a forbidding prospect for many users of diversity indices, it is worthy of the effort. Indices of neutral diversity depend too much on the notion of species, a concept based upon the division of organisms into different classes that is known to be problematic [37]. Classical indices (such as species richness) depend entirely on this division, and thus behave badly in the face of taxonomic reclassification [36].

To account for species disparity, similarity-sensitive indices incorporate in their formulation some measure of similarity. Many scientific fields make use of similarity measures, for example in biology for taxonomic classification, in economics for analyzing investment policy, in information theory for document filtering, or in genetics for sequence analysis [38]. A similarity measure is related in a complementary way to the mathematical concept of a distance metric, since dissimilarity can often be represented as a distance. However, many ecological dissimilarity measures do not fulfill all the properties necessary to be considered a distance metric [19], and hence cannot make use of the many theorems and results developed by mathematicians relating to distance metrics.

Within ecology, various methods for measuring similarity between species have already been developed. Some are genetic [39], others are functional [40], taxonomic [41], morphological [42], or phylogenetic [43]. They generally associate with each focal species some data concerning the characteristics deemed to be important, such as a list of functional traits, a DNA sequence, a location on a phylogenetic tree, and so on. The similarity values are then computed in terms of some notion of difference between the associated data, depending on its particular characteristics.

4. Axioms Characterizing a Diversity Index

Due to the fluid definition of diversity, the mathematical formulations of diversity indices are also extremely varied. This is due to the high-dimensional nature of ecological data, which a diversity index must synthesize and summarize, typically by mapping it to a scalar. The number of ways in which this can be achieved is virtually unlimited.

While this might provide a pithy summary statistic of diversity for easy comprehension and comparison, it can also result in a significant loss of information. Thus the precise mathematical formulation of a diversity index will affect how a community’s diversity can be connected to its structure, functionality and other significant features, and therefore have important implications for the index’s use in drawing ecological conclusions.

For these reasons, it is essential to understand the mathematics underlying a particular index. In an effort to bring rigour to this endeavour, researchers turned to the question of whether an axiomatic basis could be identified for any of the numerous proposed diversity indices [9]. Such a set of axioms would identify the most important properties of diversity indices, as well as allow for their differentiation based on which axioms they satisfy or fail to satisfy. The axiomatic foundations of entropy measures were addressed in this way quite some time ago [44], while the axioms for non-entropic families of indices were outlined relatively later [45].

Let us now consider a community of S species, where $p_i$ represents the proportional abundance of species i, so that $0\le p_i\le 1$ for $i=1,\dots ,S$ and ${\sum }_{i=1}^S{p}_i=1$. If the abundances are measured by counting individuals, then the relative abundance $p_i$ represents the probability of randomly selecting an individual of species i from amongst all individuals in the community. If the abundances are measured in terms of biomass, then the relative abundance $p_i$ represents the relative share of the community’s biomass that is stored in individuals of species i.

The richness of the community is simply S, while its evenness is calculated from the vector of relative abundances $\mathbf{p}=(p_1,\dots ,p_S)$. Calculating the disparity additionally requires a similarity measure, which we discuss later in Section 5.

The most commonly assumed axioms that a diversity index $H(\mathbf{p})$ should satisfy are also common to entropy measures, and are as follows [46].

(1)

Symmetry.$H(p_1,\dots ,p_S)$ must be a symmetric function.

This ensures that species are interchangeable and their order in the abundance vector has no effect on diversity.

(2)

Continuity.$H(p_1,\dots ,p_S)$ must be a continuous function.

(3)

Evenness. The diversity measure is maximal for a fixed number of species S when all species abundances are equal, i.e., $H(p_1,\dots ,p_S)\le H(1/S,\dots ,1/S)$, for any $(p_1,\dots ,p_S)$.

Additional axioms that are also satisfied by most indices of diversity were adopted later, and require additional definitions [19]. First, we define a transfer of abundance following the terminology of Patil and Taillie [47]. This involves taking two species i and j with relative abundances $p_i>p_j>0$, and modifying these to increase $p_j$ by $ϵ$ and decrease $p_i$ by $ϵ$, where $p_i-p_j>ϵ>0$. Hence we have made the more common species a bit less common, and the rarer species a bit less rare.

More precisely, if we have two communities C and $C^{\prime }$ with respective relative abundance vectors $\mathbf{p}$ and $\mathbf{q}$ with $m:=\dim (\mathbf{p})=\dim (\mathbf{q})$, where $\dim (·)$ denotes the dimension of a vector, we say that C leads to $C^{\prime }$ by a transfer of abundance if $\exists i,j\in \{1,\dots ,m\}$ such that

$$q_k=\left\{\begin{array}{cc}{p}_k\hfill & \mathrm{if} k\notin \left\{i,j\right\}\hfill \\ p_i-ϵ\hfill & \mathrm{if} k=i\hfill \\ p_j+ϵ\hfill & \mathrm{if} k=j\hfill \end{array}\right.,$$

(1)

where $p_i-p_j>ϵ>0$.

Then we may state the following axiom.

(4)

Principle of transfers. A transfer of abundance must increase diversity.

This axiom is clearly a stronger version of Axiom 3: if every transfer of abundance must increase diversity, then the index must be maximized when no more transfers between common and rare species are possible. This occurs when all species have the same abundance, i.e., for $(p_1,\dots ,p_S)=(1/S,\dots ,1/S)$.

Indices that satisfy Axiom 4 must have a property known as Schur-concavity, which earned the right to a specific name due to its importance in econometrics for measuring income inequality [48]. The property is defined as follows [49]. For indices of the form $H(\mathbf{p})={\sum }_{i}V(p_i)$, it holds that H is Schur-concave if

$$V(p_j+ϵ)-V(p_j)\ge V(p_i)-V(p_i-ϵ),$$

(2)

where $p_i>p_i-ϵ\ge p_j+ϵ>p_j\ge 0$. If V is differentiable, then this condition can be replaced by

$$V^{\prime }(p_i)\le V^{\prime }(p_j),$$

(3)

where $p_i>p_j\ge 0$ and $p_i+p_j\le 1$. The proof that Schur-concave indices satisfy Axiom 4 is considered standard and is thus generally omitted in reviews of the topic (for details, see e.g., Solomon [50]).

For ecological purposes, Axiom 4 is important because it permits a partial ordering of communities based on their diversities (discussed later in Section 5.5).

For the next axiom, we first rigorously define the term introducing a species.

For two communities C and $C^{\prime }$ with respective relative abundance vectors $\mathbf{p}$ and $\mathbf{q}$ with $\dim (\mathbf{p})=m$ and $\dim (\mathbf{q})=m+1$, we say that C leads to $C^{\prime }$ by introducing a new species if $\exists i\in \{1,\dots .m\}$ such that

$$q_k=\left\{\begin{array}{cc}{p}_k\hfill & \mathrm{if} k\ne i\hfill \\ p_i-ϵ\hfill & \mathrm{if} k=i\hfill \\ ϵ\hfill & \mathrm{if} k=m+1\hfill \end{array}\right.,$$

(4)

where $0<ϵ<p_i$. Note that this definition differs only slightly from that of transferring abundance given in Equation (1). In Equation (4), $p_j=0$ which implies in this notation that we have transferred abundance from one species to another that previously had zero abundance.

Now we may state the following axiom:

(5)

Monotonicity in number of species. The introduction of a new species must increase diversity.

The monotonicity axiom holds for indices of the form $H\left(\mathbf{p}\right)={\sum }_i{p}_{i}I(p_i)$, where $I(p_i)$ is a decreasing function of $p_i$ over $\left[0,1\right]$ (this proof is straightforward).

A final axiom was proposed still later [51] as a necessary condition for any ecological diversity index.

(6)

Replication principle. The diversity of a pooled sample of n maximally distinct (i.e., no shared species) and equally diverse sub-communities is n times the diversity of a single sub-community. That is, if for $k=1,\dots ,n$ the communities ${\mathbf{p}}^k=(p_1^k,\dots ,p_S^k)$ do not have any common species and $H({\mathbf{p}}^k)=\delta$ for every k, then $H(p_1^1,\dots ,p_S^1,p_1^2,\dots ,p_S^2,\dots ,p_1^n,\dots ,p_S^n)=n\delta$.

This axiom is important because, as we shall see in later sections, ecologists often measure diversity by considering the ratio of within-group diversity (‘alpha’ diversity) to the total pooled diversity (‘gamma’ diversity). But using this ratio, which reflects compositional similarity, to make inferences about diversity is not always justified: some indices, when used in this way, do not accurately reflect changes in diversity. This issue will be discussed in detail in Section 6, but the key point is that for these problematic indices, the ratio of within-group diversity to total pooled diversity will always approach unity when within-group diversity is high, often leading to nonsensical results. Such misbehaviour is prevented by the replication principle, which ensures that this ratio does indeed reflect the compositional similarity of the different communities. More specifically, this principle ensures that when a region is partitioned into distinct sub-communities of equal diversity, then the region’s total diversity is equal to the diversity of any subset of sub-communities, plus the diversity of the sub-communities not contained in the subset. Though this axiom would seem intuitive, we shall see that it is not satisfied by all indices. Indices that do not satisfy Axiom 6 can hence provide misleading results.

The axioms in this section also allow us to distinguish between indices of diversity and entropy, since the latter have a more restrictive definition [19]. Entropy measures are concave functions of the relative abundances (typically referred to as probabilities in that setting), whereas this is not necessarily the case for diversity indices [10]. Additionally, while entropy measures are linear functions of these abundances/probabilities, diversity indices are often non-linear, and so the average diversity of a set of communities is generally not the average of their diversities.

In general, the axioms listed in this section are agreed to be desirable for ecological diversity indices, since they ensure consistent and meaningful behaviour. However, if an index fails one of these axioms, this should be considered as a warning sign rather than a fatal flaw, since the index may still be useful in contexts where that axiom is less important. For example, if an index fails Axiom 4 it should not be used to rank communities by their diversities, which is frequently but not always the goal of ecological studies.

In any case, enumerating the axioms as we have done here, and ascertaining which of them are satisfied by a particular index (as we will address in the subsequent section) at least ensures that the researcher is aware of the underlying mathematical behaviour of their chosen index, and can thus take into account the effects this may have on subsequent ecological interpretations—forewarned is forearmed.

5. Diversity Indices

In this section we provide an overview of the diversity indices that are most widely used in ecology. We classify these in several groups: classical indices, effective number indices, similarity-sensitive indices, and finally parametric families of indices. This is not a strict classification since overlap between these groups is possible. We begin with the oldest, simplest, and therefore most commonly used indices: the classical indices of ecological diversity.

5.1. Classical Indices

5.1.1. Definitions

The most widely used diversity indices can account for richness and evenness, but do not account for species disparity. Instead they take as input the vector of relative abundances $\mathbf{p}=(p_1,\dots ,p_S)$, where S is the total number of species in the community. Thus these indices measure species-neutral diversity (see Section 3.3) and are known as classical diversity indices.

The simplest index, which is still used surprisingly often by ecologists as a measure of diversity despite its obvious shortcomings for this purpose [52], is simply species richness itself:

$$H_{\mathrm{SR}}\left(\mathbf{p}\right)=S.$$

(5)

The reason underlying species richness’ poor performance as an estimate of diversity should be clear: it takes no account of the species abundance distribution (i.e., evenness). In ignoring completely one key component of diversity, species richness gives exceptionally rare species equal weight as exceptionally common species, an entirely unintuitive estimate of diversity. Hence of the fundamental axioms in Section 4, species richness satisfies all but Axiom 4 since it ignores the relative abundances of species.

At the opposite extreme, the Berger–Parker diversity index entirely ignores rare species in the community [53]. It is defined as the reciprocal of the relative abundance of the most common species:

$$H_{\mathrm{BP}}\left(\mathbf{p}\right)={\displaystyle \frac{1}{{\max }_i{p}_i}},$$

(6)

and thus estimates the relative dominance of this species as a proxy for the entire community’s diversity. This index does not satisfy Axioms 4, 5, and 6 since it ignores all species abundances other than that of the most common species.

A much more balanced estimate of diversity is provided by the Shannon diversity index, also known as the Shannon–Wiener index, the Shannon–Weaver index and the Shannon entropy. It measures the uncertainty in the outcome of a sampling process [54], and is given by:

$$H_{\mathrm{Sh}}\left(\mathbf{p}\right)=-\sum _{i=1}^S{p}_{i}ln\left(p_i\right).$$

(7)

In Shannon’s original information-theoretic formulation, the logarithm was given in base 2, so that the Shannon entropy represents the number of yes/no questions necessary to determine an object’s classification. Since then, it has become more common to use either the natural logarithm (particularly in ecology) or the base 10 logarithm [55]. This index satisfies all the axioms in Section 4 aside from the replication principle (Axiom 6).

The Shannon index is also the basis of Pielou’s evenness index, which is given by $J=H_{\mathrm{Sh}}/H_{\mathrm{Sh}}^{\ast }$, where $H_{\mathrm{Sh}}^{\ast }$ is the maximum value of $H_{\mathrm{Sh}}$ (a function of S). However, this index is in fact a very poor estimate of evenness, since it depends strongly on species richness [8]. Although this weakness is widely known, Pielou’s evenness is still the most widely used evenness index in the ecological literature [55].

The Simpson diversity index represents the probability that two individuals taken at random from the community of interest (with replacement) represent the same species, and thus takes values in the unit interval [56]. Like the Shannon index, it satisfies all axioms in Section 4 aside from the replication principle (Axiom 6). As originally proposed by Simpson [57], it is given by:

$$H_{\mathrm{Si}}\left(\mathbf{p}\right)=\sum _{i=1}^S{p}_i^2.$$

(8)

However, this index is not a very intuitive measure of diversity since higher values indicate lower diversity (note that it still satisfies Axiom 3 since diversity is high for low values of the index, which occurs when the species proportions are equal). For this reason, two other formulations of this index are more often used. Most common is the Gini-Simpson diversity index, also called the probability of interspecific encounter (PIE), which represents the probability that the two individuals represent different species, and is thus the complement of Simpson’s original formulation [58]:

$$H_{\mathrm{GS}}\left(\mathbf{p}\right)=1-H_{\mathrm{Si}}\left(\mathbf{p}\right).$$

(9)

In contrast, the Simpson dominance index is the reciprocal of Simpson’s original formulation. The term ‘dominance’ has been attached to this formulation since it gives more weight to common species than to rare species. It is given by:

$$H_{\mathrm{SD}}\left(\mathbf{p}\right)=\frac{1}{H_{\mathrm{Si}}\left(\mathbf{p}\right)}.$$

(10)

The name ‘Simpson index’ has been used interchangeably for all three of the formulations shown here. Hence, when encountering this index, researchers should take care to note which form of the measure has been employed.

The Simpson index is also occasionally used as a measure of evenness, but this approach is not appropriate since the index also varies with richness. To be safely used as an index of evenness, the richness effect should first be eliminated by dividing the index by its maximum value, which depends on S.

5.1.2. Issues with Classical Indices

The most widely used measures of diversity in ecology are species richness, the Shannon index and the Simpson index [59]. Together, they provide a perfect example of the problems discussed in Section 2 and Section 3: each index may be used to compare different communities to each other (although we shall discuss shortly why even this can be problematic), but diversity values of the different indices cannot be directly compared to each other. This is because each index has a different fundamental meaning in terms of the diversity it measures.

Richness is simply the number of species in a community, the Simpson index represents the probability that two randomly selected individuals belong to the same species, and the Shannon index is a measure of the entropy or disorder of the community. These three indices measure conceptually different features of a community, and hence have different units: richness is in units of species, the Simpson index is a probability, and the Shannon index is an entropy measure, and therefore has units of bits of information.

Thus, diversity measurements calculated using different classical indices are not immediately comparable, which is a serious drawback for any scientific study. But even measurements calculated using the same index can present important issues of comparison.

To illustrate this problem, let us consider the simplest possible case: a community composed of S equally-common species. In virtually any ecological context, it seems reasonable to say that a community $C_1$ with ten equally-common species is twice as diverse as a community $C_2$ with five equally-common species. But calculating for example the Shannon entropy, we find $H_{\mathrm{Sh}}({\mathbf{p}}_1)=2.30$ for the first community and $H_{\mathrm{Sh}}({\mathbf{p}}_2)=1.61$ for the second. How should we understand the difference in diversity between these two communities? The diversity of the first community is not twice that of the second, although our intuition tells us otherwise. Furthermore, it is also unclear what these values mean in absolute terms: should we consider a diversity of 2.3 to be high or low? Without intuitive units, such judgements are not immediately clear.

Diversity analysis using classical indices also suffers from their occasionally extreme non-linearity. As an example, if a perfectly even community of one million species is confronted with some disaster that wipes out all but 100 species, the Gini-Simpson index of this community will drop from 0.999999 to 0.99. So despite the fact that more than 99% of the species of the pre-catastrophe community have disappeared, the Gini-Simpson diversity index only drops by 1%. Cursory study of the catastrophe using this index would probably conclude that the community’s diversity was not greatly affected, while it is clear that the opposite is true. The Shannon index demonstrates the same non-linearity problem, but to a lesser degree.

Unfortunately, in practice many ecologists have not been too concerned that classical diversity indices may give results that are misleading or difficult to interpret [59]. In their view, the actual values of the indices are unimportant, as long as they can be used to calculate the statistical significance of the change in diversity following a disturbance [46]. But in many cases this is not a reasonable basis for study conclusions, since the statistical significance of a change in the diversity index often has little to do with the actual magnitude or ecological significance of the change [52]. Furthermore, one index may indicate a statistically significant change in diversity while another index does not.

Other researchers were not content with this state of affairs, and proposed a solution: the use of so-called effective numbers.

5.2. Effective Numbers

Converting classical indices to effective numbers of species endows them with a set of common mathematical properties and ecological behaviours. After conversion, diversity is always measured in units of number of species, allowing for easy comparison and interpretation. This also means that the serious misinterpretations spawned by the non-linearity of most diversity indices can be avoided. By unifying various indices in this way, the use of effective numbers enhances our understanding and interpretation of diversity measurement.

Converting the diversity of a community to its effective number equivalent reduces to finding an equivalent community (having the same value of the index as the initial community in question) that is perfectly even [10]. For example, if a community has a diversity of 18.2 effective species, this means that it is slightly more diverse than a community of 18 totally dissimilar and equally abundant species: there are “effectively” 18.2 species.

This concept is hence related to the better known “effective population size”, since both involve a measure that summarizes the nature of a community through a comparison to an abstract ‘ideal’ community. Effective diversity makes a comparison to a perfectly even community, while effective population size makes a comparison to a community with certain key properties such as constant size, an even sex ratio, and consisting only of breeding adults [60].

Conversion to effective numbers of species is straightforward: after calculating the diversity index for D equally-common species (each species therefore having a relative abundance of $1/D$), this is set equal to the community’s ‘raw’ diversity value, and the equation is solved for D. MacArthur named D as the community’s effective number of species [11].

Other fields have recognized the importance of effective number indices many years ago, though the concept goes by different names depending on the discipline. In physics it is known as the number of states associated with a given entropy, and in economics it is called the numbers equivalent of a diversity measure [47].

As an example of the conversion procedure, consider a community whose relative species abundance distribution is given by $\mathbf{p}=\left(0.41,0.21,0.08,0.25,0.04,0.01\right)$. The Simpson diversity of this community is $H_{\mathrm{Si}}\left(\mathbf{p}\right)=0.2828$. To convert this diversity to its effective number equivalent, we need to find a community of D equally abundant species that also has a Simpson diversity of 0.2828. We therefore have that $p_i=\frac{1}{D}$ for $i=1,\dots ,6$ and that $H_{\mathrm{Si}}\left(\mathbf{p}\right)=0.2828={\sum }_{i=1}^6{p}_i^2={\sum }_{i=1}^6\frac{1}{D^2}=\frac{6}{D^2}$. It only remains to solve for D. We obtain $D=4.61$, implying that our six-species community is effectively as diverse as a community of 4.61 equally abundant species.

Thus any diversity index can easily be converted into an effective number index. The use of effective number indices to assess changes in community diversity has become more established in ecological literature in recent years [61,62,63,64,65], as researchers recognize the advantages they bring compared to classical indices, most particularly the well-founded comparisons that they permit between different studies and even different indices.

5.3. Similarity-Sensitive Indices

Conversion to effective numbers allows researchers to compare indices whose mathematical formulations imply that they would otherwise account differently for richness and evenness. To address the third component of diversity, disparity (see Section 3.3), a further extension of classical indices is needed: the inclusion of a similarity measure.

Similarity-sensitive indices incorporate a similarity matrix encoding pairwise species similarities, which becomes an additional input of the index along with the species abundance vector. For a community of S species, an $S\times S$ matrix $Z=\left(Z_{ij}\right)$ is constructed, where $Z_{ij}$ is a measure of the similarity between species i and j. We assume that $0\le Z_{ij}\le 1$, with 0 indicating total dissimilarity and 1 indicating identical species; therefore $Z_{ii}=1$. Although we might assume that similarity matrices are always symmetric (i.e., $Z_{ij}=Z_{ji}$), this is not necessarily the case. The definition of a similarity matrix does not require symmetry, and indeed useful non-symmetric similarity matrices do exist. These are useful in cases where the direction of comparison is important.

Similarity matrices are often constructed based on distance metrics, which encode the distances between every pair of species in a multi-dimensional trait space. This is a well-established approach since it ensures that the distance matrix is Euclidean and the similarity matrix is positive definite [66]. This allows for the use of various techniques from linear algebra such as matrix decomposition or transformation.

While it is generally assumed that a similarity measure takes values in the unit interval, other formulations are possible. Genetic measures of similarity, frequently used in microbial ecology and microbiology, are often expressed as percentages so that their values lie in the interval $\left[0,100\right]$ [67]. Other typical measures of inter-species distance $d_{ij}$ range instead between zero and infinity, but these can be scaled to the unit interval through various transformations. One of the simplest uses the formula $Z_{ij}=e^{-u{d}_{ij}}$ , where u is a constant. Although in comparison to other transformations this brings an additional degree of freedom to the diversity index (through the parameter u), it has nevertheless been recommended as a good choice of transformation [36], since varying u allows the user to control the distribution of similarity values. When $u=0$, the similarity is equal to one irrespective of the value of $d_{ij}$. As $u\to \infty$, the similarity matrix tends to the identity matrix and thus the index tends to species-neutral diversity.

A seminal example of a similarity-sensitive diversity index is Rao’s quadratic diversity index, which is often called Rao’s quadratic entropy although it is in fact not an entropy measure. It is defined as the expected dissimilarity between two individuals selected at random from the community (with replacement) [68], and is given by:

$$H_{\mathrm{R}}\left(\mathbf{p}\right)=\sum _{i,j=1}^S{d}_{ij}{p}_i{p}_j,$$

(11)

where $d_{ij}$ is the dissimilarity between species i and j. Here, $d_{ij}$ is not necessarily obtained from a distance metric. Note that $H_{\mathrm{R}}$ reduces to the Gini-Simpson diversity index in the case where $d_{ij}=1$ for all $i\ne j$, and $d_{ii}=0$ for all i.

However, $H_{\mathrm{R}}$ (along with other similarity-sensitive indices) violates Axioms 1 and 3 from Section 4, which imply respectively that diversity is invariant to permutation of the species abundance vector, and that diversity is maximal for a perfectly even community.

5.4. Parametric Families of Indices

Conversion to effective numbers and accounting for species disparity are two important improvements to classical indices, which address several of their weaknesses with respect to well-reasoned diversity measurement. Yet there is another important issue with classical indices: they make a priori assumptions about how the species present in the community contribute to its diversity. Thus, for example, species richness assumes that all species contribute equally to diversity. However, it is very arguable that this leads to a good estimate of diversity. Intuitively, we would not wish for a species representing less than 0.1% of a community to contribute just as much to its diversity as a species representing 99% of the community.

While other classical indices do weigh some species more than others, this is often done in an implicit and inflexible way as a consequence of their mathematical formulation. This key fact is not always accounted for, or even recognized. For example, the Shannon index is biased more towards evenness than richness. Since richness weighs rare species just the same as abundant species, this implies that the Shannon index gives more significance to common species. In some study settings, this may not be a problem and can even be an asset, for example if the focus of the study is dominance in a community. But in other settings this is far from ideal, and there is no way to account for it if only the Shannon index is used without any independent analyses of richness and evenness.

These deficiencies can be addressed by using parametric families of indices, also referred to in the literature as multivariate or compound indices, which include an additional parameter that tunes the relative contribution of different species in the community to its diversity [6,69]. Each member of an index family is then defined by a specific value of the parameter.

If we wish to use such indices to avoid the implicit bias inherent in other diversity indices, how should they be formulated? Two schools of thought have developed [13]. The first, due largely to economists, proposes that different species should be given different weights in the index based on their characteristic features. The second approach, found mainly in ecology, weighs different species according to their relative abundances. This has been justified by observing that species’ functional contributions typically vary with their abundance in the community [70].

Ideally, we would like to vary this balance or weighing of species. Then we are not wedded to a single a priori judgement of which species contribute more to the community’s diversity, but can investigate various scenarios. This would mean introducing an additional parameter that tunes this weighing. Thus the index becomes a function of both the species abundance vector and the sensitivity parameter.

Parametric families of diversity indices introduce exactly such a parameter, which controls the index’s sensitivity to species abundances and therefore avoids the issue of bias towards one component of diversity compared to another. This parameter can be varied according to the user’s interests and needs. In this way, additional insights can be gained into the community’s composition under different assumptions of whether rare or common species are thought to play more important roles.

Although the use of these indices for ecological diversity studies is only recently becoming more widespread, this type of index is not new. Parametric families of diversity indices first emerged decades ago, initially as information-theoretic measures that were first introduced to ecology by MacArthur [71].

The Rényi entropy was formulated to generalize several other entropy measures, notably the Shannon entropy. The Rényi entropy of order $\alpha \in \left[0,\infty \right]\backslash \left\{1\right\}$ is given by:

$$H_{\alpha }\left(\mathbf{p}\right)=\frac{1}{1-\alpha }ln\sum _{i=1}^S{p}_i^{\alpha },$$

(12)

where $\alpha$ is a parameter that modulates the index’s sensitivity to species abundances; the case $\alpha =1$ is excluded since the index is not defined for this value. Rather, the Shannon entropy is the limiting case of this entropy as $\alpha \to 1$. Note that the Rényi entropy was originally defined using the base 2 logarithm (so as to measure information content in bits), but in ecology the natural logarithm is much more frequently used.

A similar index was proposed by Tsallis [72] and is known as the HCDT entropy or the Tsallis entropy. It is given by:

$${}^{q}H\left(\mathbf{p}\right)=\frac{1}{1-q}\left(1-\sum _{i=1}^S{p}_i^q\right),$$

(13)

where q is the index’s sensitivity parameter. For $q=0$, the index is equivalent to species richness minus one, for $q=1$ it is equivalent to the Shannon entropy, and for $q=2$ it is equivalent to the Gini-Simpson index.

Most of the diversity indices, including all generalized entropy measures used in ecology and those mentioned above, are monotonic functions of the sum ${\sum }_{i=1}^S{p}_i^q$, or limits of such functions as q approaches unity [73]. Such indices include: species richness, Shannon entropy, all Simpson indices, all Rényi entropies, and many others. All of these measures result in the same expression for diversity when converted to their effective number equivalent, which are known as the Hill numbers after their originator [51]. This family of indices is given by:

$${}^{q}D\left(\mathbf{p}\right)={\left(\sum _{i=1}^S{p}_i^q\right)}^{1/(1-q)},$$

(14)

where $q\in \left[0,\infty \right]\backslash \left\{1\right\}$ is known as the order of the diversity. Again, the case $q=1$ is excluded since the index is undefined for this value; however, the limit exists and will be discussed shortly. The Hill numbers are the only family of ecological diversity indices that are known to satisfy all six axioms in Section 4 [46].

For all indices that are a function of ${\sum }_{i=1}^S{p}_i^q$, their effective number diversity depends only on the value of q and the relative species abundances, and not on the index’s mathematical formulation. For example, the Simpson diversity index, the Simpson dominance index, and the Gini-Simpson index give the same effective number diversity, namely the Hill number of order 2:

$${}^{2}D\left(\mathbf{p}\right)=1/\left(\sum _{i=1}^S{p}_i^2\right).$$

(15)

We also note the link to another family of diversity indices that were proposed by Patil and Taillie [47], and which reintroduced the Hill numbers from a different mathematical viewpoint, namely the use of a sensitivity parameter $\beta$ that is linked to Hill’s q by $q=\beta +1$ [74].

The order q of the diversity index indicates its sensitivity to common and rare species. The diversity index of order zero ($q=0$) is species richness: it is completely insensitive to relative species abundances. All values of q less than one result in diversity indices that disproportionately favour rare species, while all values of q greater than one lead to diversity indices that disproportionately favour the most common species.

The critical point, which weighs all species by their frequency without favouring any, occurs when $q=1$. Note that ${}^{q}D$ is undefined at $q=1$, but the limit exists and is equal to:

$${}^{1}D\left(\mathbf{p}\right)=exp\left(-\sum _{i=1}^S{p}_{i}ln{p}_i\right)=exp\left(H_{\mathrm{Sh}}(\mathbf{p})\right).$$

(16)

This is the exponential of the Shannon entropy, a quantity that has deep connections to biology, information theory, physics, and mathematics, since it is a measure that is uniquely able to weigh elements of a system exactly by their frequency [75]. These elements might be species in a community, spin configurations in a particle system, or DNA sequences in a genome [76,77]. While the Shannon entropy (Equation (7)) provides a measure of how disordered these systems are (and hence is an index for how diverse they are), it is biased towards common elements rather than rare ones. The transformation to its exponential ${}^{1}D$ (Equation (16)) resolves this problem by not favouring any elements above others, and thus avoiding bias in its measurement. It is for these reasons that ${}^{1}D$ finds a role in various scientific disciplines.

Leinster and Cobbold [36] proposed a similarity-sensitive extension of the Hill numbers: as well as the relative abundance vector $\mathbf{p}$ and a sensitivity parameter $q\in \left[0,\infty \right]$, their index includes a similarity matrix Z. Then for $q\in \left[0,\infty \right]\backslash \left\{1\right\}$, the Leinster-Cobbold diversity of order q is given by:

$${}^q{D}^Z\left(\mathbf{p}\right)={\left(\sum _{i=1}^S{p}_i{\left({\left(Z\mathbf{p}\right)}_i\right)}^{q-1}\right)}^{1/(1-q)}.$$

(17)

The Leinster-Cobbold family includes—either directly or upon simple transformation—species richness (Equation (5)), Rao’s quadratic index (Equation (11)), the Shannon index (Equation (7)), the Gini-Simpson index (Equation (9)), the Berger–Parker index (Equation (6)), the Hill numbers (Equation (14)), and the Tsallis entropies (Equation (13)). Thus almost all of the measures discussed in Section 5, and others still, can be subsumed in one family of measures that are both effective numbers and similarity-sensitive.

Most of these indices are retrieved by setting the similarity matrix Z equal to the identity matrix, so that $Z_{ij}=0$ (total dissimilarity) if $i\ne j$, and $Z_{ij}=1$ (total similarity) if $i=j$. This is referred to as the ‘naive’ Leinster-Cobbold diversity, in the sense that it ignores species disparity. Then for example at $q=0$ the naive Leinster-Cobbold index is equivalent to species richness. At the other extreme, for $q=\infty$ the naive index corresponds to the Berger–Parker index (Equation (6)), which depends only on the most abundant species and ignores all others.

The cases $q=1$ and $q=\infty$ are excluded from the definition in Equation (17) because ${}^q{D}^Z\left(\mathbf{p}\right)$ is not a valid expression for these values. However, the index does converge at these values, to

$${}^1{D}^Z\left(\mathbf{p}\right)=\frac{1}{{\left((Z\mathbf{p}\right)}_1{)}^{p_1}{\left((Z\mathbf{p}\right)}_2{{)}^{p_2}\cdots {\left((Z\mathbf{p}\right)}_S)}^{p_S}},$$

(18)

as $q\to 1$, and to

$${}^{\infty }{D}^Z\left(\mathbf{p}\right)=\frac{1}{{max}_{i\in \{1,\dots ,S|p_i\ne 0\}}{\left(Z\mathbf{p}\right)}_i},$$

(19)

as $q\to \infty$.

To close this section, we note that all of these parametric families are deeply linked. Most obviously, the naive Leinster-Cobbold indices (Equation (17)) are by definition equivalent to the Hill numbers (Equation (14)): ${}^{q}D={}^q{D}^I$. Deeper links exist between the other indices due to the previously noted fact that all of them are monotonic functions of the sum ${\sum }_{i=1}^S{p}_i^q$, or limits of such functions as $q\to 1$ [73]. Thus, the Tsallis entropy (Equation (13)) is the base q logarithm of the q-th Hill number (Equation (14)),

$${}^{q}H={log}_q{}^{q}D,$$

(20)

while the Rényi entropy (Equation (12)) is the natural logarithm of the Hill number of the same order,

$$H_{\alpha }=ln{}^{\alpha }D.$$

(21)

These transformations are useful for the practical estimation of the Hill numbers (Section 7), since conversion to an entropy measure provides an easier approach to bias correction. Conversely, the Hill numbers can permit a more intuitive conception of diversity (in terms of effective number of species) which may be more useful for theoretical works.

5.5. Diversity Profiles

Different indices give different estimates of diversity, so it may happen that when calculating how a community’s diversity has shifted after a disturbance, one index indicates that the diversity has increased, whereas another indicates that it has decreased [74]. Therefore, restricting the analysis to a single index can lead to a biased or distorted estimate of diversity, and consequently conclusions based on this measurement may be misleading or incorrect.

This can be avoided through the use of parametric families of diversity indices: as discussed in Section 5.4, these indices include a sensitivity parameter that controls the weighing of common and rare species in the community. Varying this parameter allows for a wider view of a community’s diversity and provides more information than any single statistic.

A diversity profile is a graph that does exactly this: it shows at once multiple values of a community’s diversity, so that they may easily be compared. For any parametric family $H(\mathbf{p},q)$ where q is the index’s sensitivity parameter, its diversity profile is a graph of $H(\mathbf{p},q)$ against q. Changes in community diversity can then be studied by comparing profiles. These may be two different communities, or the same community at different points in time, typically before and after some ecological disturbance.

Communities’ diversity profiles define a partial order on their diversities [78]. If the two communities are comparable, then one profile will lie pointwisely above the other, and the former community can be said to be more diverse than the latter. However, the profiles of the communities may intersect, in which case the communities are not directly comparable (and hence the ordering is only partial). This is again an unavoidable consequence of reducing the complex concept of diversity to a relatively simple numerical analysis, which diversity profiles are not able to entirely overcome. However, they are much more meaningful than univariate indices, and even when two communities are not directly comparable, studying where their profiles intersect can reveal which sections of the community (common or rare species) have shifted [74].

The region of a diversity profile where q is small gives information about species richness and rare species, since here $H(\mathbf{p},q)$ is affected almost as much by rare species as common ones. The tail where q is large gives information about dominance and common species, since here $H(\mathbf{p},q)$ is almost entirely unaffected by rare species. As the sensitivity parameter q increases, the diversity $H(\mathbf{p},q)$ drops. More precisely, the diversity profile is always decreasing and continuous.

As an illustration, we plot in Figure 2 three classical (univariate) indices, and two parametric families of indices. For the same species abundance vector, we compute the value for univariate indices and the diversity profile for the parametric families of indices. Since univariate indices do not account for differential sensitivity to rare or common species, they do not vary with q. One of the parametric families is similarity-sensitive (the Leinster-Cobbold index), so that some pairs of species are more similar than others, while the other parametric family is insensitive to disparity (the Hill numbers). For the Leinster-Cobbold index, we use a random sensitivity matrix satisfying the conditions outlined in Section 5.4, i.e., it is symmetric and all its elements lie in the unit interval.

We first note that the univariate indices give different estimates of diversity, since they are not effective number indices. Hence they do not have a common unit, and the diversity values they give for the same community are significantly different (as discussed in Section 5.2).

For the parametric families, we notice the effect of including a similarity measure to account for disparity. For all values of q, the similarity-sensitive index gives a lower value of diversity than the naive index. This is to be expected since the naive index treats all species as equally different, whereas the similarity-sensitive index considers some species to be less distinct than others.

The steepness of the left-hand side of the profiles, where q is small, gives us information about the rare species in the community. As q increases, these rare species are given less weight by the index, and therefore the steeper the drop of the profile, the more rare species there are in the community. Again we notice that the naive index considers there to be more rare species than the similarity-sensitive index, since the slope of the former is steeper. In fact, the slope of the similarity-sensitive profile is so low that we can surmise that the similarity measure considers the rare species to be very similar.

Figure 2 also illustrates graphically the mathematical relationships between the indices that were discussed in Section 5.4: the Hill number of order 0 is equal to species richness, the Hill number of order 1 is equal to the exponential of the Shannon index, and the Hill number of order 2 is equal to the Gini-Simpson index.

To illustrate the useful application of diversity profiles in ecological studies, we provide several examples. First, we show in Figure 3 the diversity profiles calculated by Leinster and Cobbold [36] relating to an experiment comparing the microbial communities in the guts of lean and overweight humans [79].

The diversity profiles for two particular test subjects from that study were compared, one subject being overweight and the other not. Since only a fraction of microbial species has been isolated and given taxonomic classifications, it was not possible for the authors to partition the microbes into species (this issue was discussed in Section 3.3). Instead, the authors turned to DNA sequencing data, which they used to construct a similarity matrix. Using the naive similarity matrix (Figure 3a), the diversity profiles cross at $q\approx 1$. This suggests that the community in the lean subject’s gut is richer but less evenly distributed than that of the overweight subject. However, using a similarity matrix based on genetic measures (Figure 3b), the diversity in the lean subject is seen to be greater for all values of q, a conclusion supported by other analyses conducted by the authors [79].

Another study, again using the Leinster-Cobbold index, conducted a meta-analysis of microbial diversity studies by calculating the diversity profiles of the various microbial communities, in order to investigate whether the use of these profiles altered the interpretation or conclusions of the initial studies [61]. Aside from insights into community structure that were not detectable with classical (univariate) diversity measures, the authors also found that similarity-based and naive diversity profiles only agreed on which community was the most diverse in approximately 50% of cases, another strong argument for incorporating similarity information into diversity quantification [61].

In a conservation study of reptiles in Madagascar, Nopper et al. [80] used the Hill numbers to calculate diversity profiles. Their goal was to detect changes in the reptile communities’ composition in order to assess the success of different forest management strategies. Their results indeed demonstrate that such differences existed in the Madagascar forests: significant differences were detected in community compositions between used and less-used forests when management rules were respected, whereas when these rules were ignored, the diversity profiles showed no significant differences between the reptile communities in used and less-used forests, implying homogenization of reptile community composition under this scenario [80]. Diversity profiles allow for the identification of such patterns in community composition, which are stronger conclusions than merely calculating similar values of diversity, as would be the only possibility with a classical index.

Genetic diversity profiles were used in another study to assess the effect of sample size for a genetic analysis—the authors’ goal was to study how genetic diversity calculations were affected by minimum sample size and the effect of relatives (i.e., high genetic similarity) in the sample [81]. To do so, they compared the results of two classical (univariate) indices with diversity profiles calculated using the Hill numbers, for sampled populations of amphibians. The comparison indicated good agreement between the classical indices and the profiles at the relevant q values, but differences were found in profiles of rare alleles, which require higher sample sizes to calculate univariate measures. The authors took these results as evidence for the usefulness of their method for calculating minimum sample sizes, and for pinpointing possible underestimation of diversity due to rare alleles, all in service of accurate quantification of genetic diversity.

Other practical applications, particularly conservation studies, are often more interested in how diversity varies temporally rather than spatially. As an example, Iacchei et al. [82] undertook a study of temporal variations in the genetic composition of populations of two planktonic copepod species. Conventional population genetic analyses were able to characterize the structures of the populations, but found no significant differences in these over time. In contrast, the use of diversity profiles allowed for the identification of seasonal turnover patterns, particularly in rare species, adding important nuance to the researchers’ understanding of these communities. The authors therefore concluded that their results “highlight the complementary insights” that can be obtained from these two techniques, which are especially useful for examining subtle temporal shifts in community composition.

Such practical applications of diversity indices often involve considering diversity at different scales across an ecosystem or landscape. Parametric families of indices are well suited for this purpose, since they can be meaningfully decomposed into components representing these different scales, a technique known as the partitioning of diversity.

6. Partitioning of Diversity

In Section 3, the three key components of diversity were discussed: richness, evenness, and disparity. Diversity can be partitioned according to these components. For example, the Simpson evenness index is derived from the Simpson diversity index by removing the effect of richness [83] (the Simpson index already neglects disparity). However, if we still wish to consider diversity as a unified concept of at least richness and evenness (and possibly also disparity), then diversity can also be partitioned according to the scales over which it is considered.

If we think of an ecological landscape, we will typically find one large community of species that aggregates smaller sub-communities, in terms of their spatial organization or other features. So when we consider the diversity of this landscape, to what are we referring?

There is the total diversity of the landscape: this is known as its gamma diversity. There is also the diversity of the sub-communities, which is known as the landscape’s alpha diversity, or within-community diversity. More specifically, alpha diversity refers to the diversity of a uniform habitat of a fixed size [19]. The link between alpha and gamma is the diversity representing the differences between the sub-communities: its beta diversity, or between-community diversity. These concepts are illustrated in Figure 4.

Alpha and gamma diversity are straightforward to conceive and measure, while beta diversity is trickier. It is agreed that beta diversity links alpha and gamma diversity, but how exactly should the former be derived from the latter?

The first partitioning to be proposed was multiplicative [84], so that $D_{\alpha }{D}_{\beta }=D_{\gamma }$. Later, others insisted that the independence of alpha and beta diversity is not necessary, since their dependence can be accounted for in other ways, and thus proposed an additive formulation [67], so that $D_{\alpha }+D_{\beta }=D_{\gamma }$.

In practice, ecologists generally measure alpha and gamma diversity, and then derive beta diversity from these. This has led to the emergence and use of a wide variety of measures of beta diversity, similar to the proliferation of biodiversity measures, despite beta diversity at least benefiting from an agreed definition, unlike biodiversity: beta diversity refers to the average dissimilarity in composition among sub-communities. The multitude of beta diversity indices can be attributed to similar causes as those underlying the vastness of the set of biodiversity indices. Different ecological studies wish to interrogate different research questions relating to beta diversity, and thus focus on different aspects of it, while the various beta diversity measures can have quite different statistical properties [85]. This has led to the further issue of devising measures that capture diversity at these different scales while also respecting the axioms in Section 4.

Why, then, are ecologists interested in wading into this complex topic? The multiplicative and additive partitions of diversity are appreciated and widely used because they provide a single set of values of alpha and beta diversity for a given landscape (sampling) scale, and are thus very suitable techniques for analyzing patterns or changes in diversity across multiple scales. Hence, they are able to provide ecologists with an intuitive and accessible measure of species diversity, and its spatial and temporal variations [85]. They are thus particularly appreciated by conservation ecologists, who are interested in large-scale ecological processes affecting biodiversity.

The disadvantage of the simplicity of additive or multiplicative diversity partitioning is that these methods also result in a significant loss of information, as the multidimensional features of a community are projected to a much lower-dimensional space. For example, the beta diversity obtained by diversity partitioning does not directly measure the differences in composition among the sub-communities, but is instead an average of this diversity, which is not found in any particular sub-community [86]. Unsurprisingly, as with diversity in the broader sense, there is no single index that provides a perfect measurement of beta diversity (for an in-depth discussion of beta diversity measurement, we direct the interested reader to the review in [87]).

Yet, just as with biodiversity indices, users of diversity partitions have in recent years fought their way through an analogous ‘jungle’ of indices, measures and metrics to arrive at the beginnings of a consensus. There is increasing agreement that the Hill numbers (Equation (14)) should be partitioned to measure beta diversity [88,89,90]. This is because the Hill numbers, as effective numbers and in contrast to most other indices, obey the replication principle (Axiom 6, discussed in Section 4).

The replication principle was first proposed by Hill [51] merely as a desirable characteristic of his eponymous family of indices, but it was later shown by Jost [91] that indices satisfying this principle permit the partitioning of gamma diversity into independent alpha and beta components, through a multiplicative partitioning. Hence any change in the magnitude of their partitioned components has a simple interpretation [86].

To illustrate the importance of the replication principle to a sensible partitioning of diversity, we use a simple example following Jost [46]. We consider a landscape made up of twenty island communities of equal size and with the same diversity, that do not share any species; hence the islands have identical relative abundance distributions, but no common species. Take as an example the relative abundance distribution given by $\mathbf{p}$ = (0.057, 0.012, 0.052, 0.064, 0.026, 0.060, 0.034, 0.012, 0.043, 0.034, 0.052, 0.049, 0.054, 0.060, 0.037, 0.052, 0.089, 0.040, 0.078, 0.044, 0.030, 0.021). If this landscape was now under threat, for example from rising sea levels, and finite resources dictated that only some of its diversity could be protected, how would a suitable fraction be identified?

Using the Gini-Simpson index (Equation (9)), we would find that the alpha diversity of each island is 0.95, while the gamma diversity of the total landscape is 0.998 (note that to calculate $D_{\gamma }$ we must rescale $\mathbf{p}$ since the pooled community has twenty times as many species as each island). So if in the worst case only one island can be protected, this would imply that we have saved $0.95/0.998=95\%$ of the landscape’s diversity—a fantastic result for relatively little effort. But then how much of the total diversity has been lost? Together, the nineteen doomed islands have a diversity of $0.997$, so we have lost $0.997/0.998=99.9\%$ of the total diversity. Somehow, we have simultaneously saved $95\%$ of the landscape’s diversity, and lost $99.9\%$ of it.

This nonsensical result is due to the Gini-Simpson index failing the replication principle, and hence its alpha and beta components do not scale sensibly when communities are pooled. Similar (though less egregious) results would be obtained with other indices that fail Axiom 6. In contrast, if we used an index that satisfies Axiom 6 (like the effective number equivalent of the Gini-Simpson index, or the Hill numbers), this would sensibly tell us that each of the twenty islands represents $1/20=5\%$ of the total landscape diversity, while the other nineteen collectively represent $19/20=95\%$ of it. Intuitive and consistent behaviour of this kind underlies the consensus forming around the choice of the Hill numbers for diversity partitioning.

A recent review of the topic provided the following guidelines [85]. First, that beta diversity should be measured using species richness (i.e., using $q=0$), but the authors also acknowledged that depending on the research context, it may be more enlightening to also vary q so as to measure other Hill numbers as well, thereby obtaining a diversity profile for each community (see Section 5.5). Second, not to use entropy measures in diversity partitioning. Entropy measures are often not effective number indices, and also generally do not satisfy the replication principle (Axiom 6 in Section 4), leading to misleading results (a deeper discussion of this point can be found in [46]). Third, to make the necessary statistical adjustments to remove alpha/beta dependence when using an additive partitioning (see e.g., [92]).

As a consequence of this last point, an even more recent consensus is emerging around the use of the multiplicative Hill number partitioning for measuring beta diversity, and thus avoiding the additive partitioning [93]. This is due to the recognition that the independence of alpha and beta diversity is a desirable feature of diversity partitioning; among other reasons, it simplifies the statistical approaches used to detect patterns in beta diversity changes [87].

Finally, we note that the Hill family is not unique in its decomposition into independent alpha and beta components. This is also possible with other families of indices, but the analyses are more complicated than those required for the Hill numbers, and these families generally do not share the advantageous characteristics of the Hill family indices, most notably that they are effective numbers and they satisfy the replication principle, the usefulness of which we have justified in previous sections [91].

7. Practical Estimation of Diversity Indices

7.1. Background

When it comes to assessing the diversity of a community in practice, the species proportions $\mathbf{p}=\left(p_1,\dots ,p_S\right)$ are generally unknown. Without a full census of the community, calculation of its diversity using an index is not possible. Instead, this must be estimated from sampled data.

For the purpose of this discussion, we assume that we have a sample of n individuals from the community in question. Most estimators of diversity assume that this sample is random, namely that its observations are independent and identically distributed [94]. But this requires either an infinite population, or sampling with replacement. Generally, neither of these requirements are met: populations are finite and ecological studies typically sample populations without replacement [95]. This issue is avoided by considering a sample size much smaller than the population size, so that the effect of sampling without replacement from a finite population can be neglected [96]. Thus the problem reduces to estimating $\mathbf{p}$ using an appropriate sampling design, and then estimating the index as a function of the estimated species abundance vector.

A good estimator of a diversity index is one that is unbiased and with minimal variance [19]. Therefore, an appropriate measure of an estimator’s error is the mean squared error (MSE), which is the sum of squares of the bias and the variance [97]. Bias generally arises for two reasons: unobserved species (those present in the community but not in the sample), and the non-linearity of indices with respect to the species abundances. The former source of bias is exacerbated the more strongly the index depends on S, the richness of the community. Hence, for example, the bias of the Simpson index (Equation (8)) is less than that of the Shannon entropy (Equation (7)) since the former assigns less weight to rare species, which are less likely to be sampled.

As with the indices themselves, their estimators should be selected carefully to ensure that their formulation is appropriate for the particular application. Interrogation of this point is key, particularly since non-experts often resort to dedicated statistical software or packages whose underlying and disparate estimation techniques can result in different calculations of the same quantity [7].

For most diversity indices, numerous estimators exist. The statistical details of their derivation and bias reduction are beyond the scope of this review, so in this section we do not aim for an in-depth discussion of these various estimators, but we instead focus on whether an unbiased estimator exists for a particular index, and point the interested reader towards more exhaustive reviews. Unbiased estimators are of course preferable to biased estimators, but in practice many techniques exist for reducing bias, often to minimal levels, so that good estimates of diversity can still be obtained with a biased index estimator.

7.2. Estimators of Diversity Indices

7.2.1. Richness

Estimates of richness are obviously strongly correlated with the size of the sample used to obtain them: the total number of species in a sampling unit increases with its area and the number of detected individuals, and therefore comparing two different sampling units will be subject to bias. To address this, methods have been proposed that reduce samples of different sizes to a single standard size, so that they may be comparable in terms of their richness, a procedure known as rarefaction [55].

The two most frequently used richness estimators are the Chao estimators (Chao1 and Chao2) [98,99], and the jackknife method [95]. The Chao estimators estimate the number of unobserved species based on the number of species that are observed either once or twice. The jackknife method is based on resampling, and aims to reduce the estimator’s bias by considering subsets of the sample, which are obtained by removing a certain number of observations (this number is known as the order of the jackknife method).

Brose et al. [100] provide a decision tree for choosing between these estimators, based either on the estimated range of sample coverage or community evenness. Both these estimators can be efficiently implemented and calculated, hence their popularity. For an overview of other richness estimation techniques, see for example [101].

7.2.2. Simpson Index

Simpson [57] provided an unbiased estimator of his own index (Equation (8)), which is based on the number of pairs of individuals drawn without replacement:

$${\widehat{H}}_{\mathrm{Si}}=\frac{1}{n(n-1)}\sum _{i=1}^S{n}_i(n_i-1)$$

(22)

where $n_i$ is the number of sampled individuals of species i.

7.2.3. Shannon Index

Due to its importance in numerous scientific fields, the Shannon entropy (Equation (7)) has seen the development of many different estimators (Marcon [19] provides an overview). The details of these estimators are beyond the scope of this review, but for our purposes we note that the estimation of the Shannon index is subject to more bias than the estimation of Simpson’s index, since the former is more sensitive to richness, i.e., to the number of species that are sampled.

An estimator frequently used in ecology is due to Chao and Shen [102], and is based on the steepness of the rarefaction curve—recall that the rarefaction procedure reduces bias in estimating richness, and thus reduces the same bias in the Shannon estimator.

7.2.4. Hill Numbers

Chao and Jost [103] formulated a low-bias estimator of the Hill numbers (Equation (14)), and is based on the discovery rates of new species with respect to the sample size n. This estimator therefore also allows for the estimation of a community’s diversity profile, which can significantly improve the analysis of diversity shifts in sampled communities.

Haegeman et al. [104] used a different approach, by proposing two different estimators of the Hill family through the generalization of Chao’s richness estimators. These estimators provide a lower and an upper bound to delimit the range in which the true Hill diversity values are expected to lie. For larger q, including the indices of Simpson and Shannon, this range is relatively narrow and thus these estimators yield a good estimate of these Hill numbers. But as q decreases to approach zero, this unsurprisingly diverges as the Hill diversities approach species richness, whose estimate is inherently biased. Thus in general, Hill numbers of larger order can be estimated with less uncertainty. This is again unsurprising since the larger the order, the less weight is given to rare species, which are less likely to be sampled.

7.2.5. Leinster-Cobbold index

The Leinster-Cobbold indices provide a similarity-sensitive extension of the Hill numbers, and so their estimation is related but more complex. An additional complication in this similarity-sensitive case is that we have no information about the similarity between unobserved and observed species. Either the estimator can be corrected for these unobserved species, or their contribution to diversity can be explicitly estimated by assuming that the average similarity between unobserved and observed species is equal to the average similarity between observed species [105].

Using the latter approach, Marcon et al. [105] provide low-bias estimators of the Leinster-Cobbold index family (Equation (17)), along with a multiplicative partitioning that can be used to estimate alpha and beta diversity (see Section 6). They make use of a bias-corrected estimator originally developed for the Tsallis entropy (Equation (13)), through the transformation in Equation (20). The bias-corrected estimator is first applied to a similarity-sensitive extension of the Tsallis entropy, which is then transformed into the Leinster-Cobbold index.

8. Conclusions

In this review, we have provided an overview of the key conceptual issues with which researchers are confronted when they seek to measure ecological diversity, and we have surveyed and compared the most common diversity indices used in ecology. In general, we recommend that a diversity index is selected that:

(1)

satisfies the key axioms in Section 4;

(2)

can be expressed as an effective number (Section 5.2);

(3)

can be extended to account for species disparity if necessary (Section 5.3);

(4)

can be parameterized to obtain diversity profiles (Section 5.5); and

(5)

an estimator (preferably unbiased) can be obtained so that the index can be used in practical applications (Section 6 and Section 7).

In the relevant sections of this paper, we have elaborated on each recommendation by motivating its importance for reasonable and realistic diversity measurement.

Overall, the Hill numbers have been recommended for quantifying species diversity for several decades [106], and despite the significant progress in measuring and estimating diversity that has been achieved in the intervening years, this consensus has not changed [86] and we do not deviate from it. This family of indices has recently been extended to measure similarity-sensitive diversity [36], phylogenetic diversity [107] and functional diversity [108]. However, these types of diversity do not yet dispose of a knowledge base as well-developed as that of species diversity, due to their more complex nature (for a review of these issues in functional diversity measurement, see e.g., [109]; for phylogenetic diversity, see e.g., [63]). The ability of the Hill numbers to be extended to measure these different types of diversity hints at their power as a unifying framework for measuring biodiversity.

However, this general recommendation is of course not always the best choice. Depending on the particular context of an ecological study, other indices may be more appropriate. As discussed in Section 2, the vast number and variety of available diversity indices allows researchers to be flexible in their choice of index, with the key stipulation that the underlying definition of the index should first be considered carefully to ensure that it is appropriate for the particular application, and will not lead to misinterpretations.

Thus more specific study contexts may benefit from other recommendations. For example, in conservation ecology, the goal of a study is oftentimes to rank different communities (or sites) by their diversities, since only certain sites can be selected for protection or management [6]. In these cases, parametric families of indices are generally preferred since they provide information about both richness and evenness, and so outperform classical indices at discriminating between such sites. On the other hand, a ranking of sites by their diversity cannot always be obtained using a parametric family of indices, since their diversity profiles impose only a partial order on diversity. Hence a well-selected univariate index may be more appropriate in this particular setting.

In sum, when one is faced with the task of selecting a diversity index, it is enlightening to consider Baumgärtner’s observation that all diversity indices require, to a greater or lesser extent, that the user makes a priori value judgements about which aspects of biodiversity are more important [13]. This includes, for example, the choice of the sensitivity parameter q in parametric families of indices. In choosing the value of q, the user specifies whether rare or common species are more important for the type of diversity they wish to study. Hints about the most appropriate value(s) to study might be gleaned from other insights into the community, but q cannot itself be measured or inferred from the community. Hence, there is no correct value of q in any application, and it must instead be chosen by the user based on the goals of the study.

This advice applies to all indices, not just those involving a sensitivity parameter. Diversity indices merely provide a summary of an inherently complex and multidimensional concept: a community’s structure. Indices achieve this summarization in different ways by emphasizing different aspects of diversity. To avoid confusion and misinterpretation, users should first define their objectives and then choose the appropriate measure for the specific problem.