Rare Event Analysis for Minimum Hellinger Distance Estimators via Large Deviation Theory

Abstract

Hellinger distance has been widely used to derive objective functions that are alternatives to maximum likelihood methods. While the asymptotic distributions of these estimators have been well investigated, the probabilities of rare events induced by them are largely unknown. In this article, we analyze these rare event probabilities using large deviation theory under a potential model misspecification, in both one and higher dimensions. We show that these probabilities decay exponentially, characterizing their decay via a “rate function” which is expressed as a convex conjugate of a limiting cumulant generating function. In the analysis of the lower bound, in particular, certain geometric considerations arise that facilitate an explicit representation, also in the case when the limiting generating function is nondifferentiable. Our analysis involves the modulus of continuity properties of the affinity, which may be of independent interest.

1. Introduction

In a variety of applications, the use of divergence-based inferential methods is gaining momentum, as these methods provide robust alternatives to traditional maximum likelihood-based procedures. Since the work of [1,2], divergence-based methods have been developed for various classes of statistical models. A comprehensive treatment of these ideas is available, for instance, in [3,4]. The objective of this paper is to study the large deviation tail behavior of the minimum divergence estimators and, more specifically, the minimum Hellinger distance estimators (MHDE).

To describe the general problem, suppose $\mathsf{\Theta }\subset {\mathbb{R}}^d$, and let $\mathfrak{F}=\{f_{\mathit{\theta }}(·):\mathit{\theta }\in \mathsf{\Theta }\}$ denote a family of densities indexed by $\mathit{\theta }$. Let $\{X_n:n\ge 1\}$ denote a class of i.i.d. random variables, postulated to have a continuous density with respect to Lebesgue measure and belonging to the family $\mathfrak{F}$, and let X be a generic element of this class. We denote by $g(·)$ the true density of X.

Before providing an informal description of our results, we begin by recalling that the square of the Hellinger distance (SHD) between two densities $h_1(·)$ and $h_2(·)$ on $\mathbb{R}$ is given by

$$\begin{array}{c}\hfill {\mathrm{HD}}^2(h_1,h_2)={∥h_1^{\frac{1}{2}}-h_2^{\frac{1}{2}}∥}_2^2=2-2{\int }_{\mathbb{R}}{\left(h_1\left(x\right)h_2\left(x\right)\right)}^{\frac{1}{2}}dx.\end{array}$$

The quantity ${\int }_{\mathbb{R}}{\left(h_1\left(x\right)h_2\left(x\right)\right)}^{\frac{1}{2}}dx$ is referred to as the affinity between $h_1(·)$ and $h_2(·)$ and denoted by $\mathcal{A}(h_1,h_2)$. Hence, the SHD between the postulated density and the true density is given by $\mathrm{SHD}\left(\mathit{\theta }\right)={\mathrm{HD}}^2(f_{\mathit{\theta }},g).$ When $\mathsf{\Theta }$ is compact, it is known that there exists a unique ${\mathit{\theta }}_g\in \mathsf{\Theta }$ minimizing the $\mathrm{SHD}\left(\mathit{\theta }\right)$. Furthermore, when $g(·)=f_{{\mathit{\theta }}_0}(·)$ and $\mathfrak{F}$ satisfies an identifiability condition, it is well known that ${\mathit{\theta }}_g$ coincides with ${\mathit{\theta }}_0$; cf. [1]. Turning to the sample version, we replace $g(·)$ by $g_n(·)$ in the definition of SHD, obtaining the objective function ${\mathrm{SHD}}_n\left(\mathit{\theta }\right)={\mathrm{HD}}^2(f_{\mathit{\theta }},g_n)$ and

$$\begin{array}{c}\hfill g_n\left(x\right)=\frac{1}{n{b}_n}\sum _{i=1}^{n}K\left(\frac{x-X_i}{b_n}\right),\end{array}$$

(1)

where the kernel $K(·)$ is a probability density function and $b_n\searrow 0$ and $n{b}_n\nearrow \infty$ as $n\to \infty$.

It is known that when the parameter space $\mathsf{\Theta }$ is compact, there exists a unique ${\widehat{\mathit{\theta }}}_n\in \mathsf{\Theta }$ minimizing ${\mathrm{SHD}}_n\left(\mathit{\theta }\right)$, and that ${\widehat{\mathit{\theta }}}_n$ converges almost surely to ${\mathit{\theta }}_g$ as $n\to \infty$; cf. [1]. Furthermore, under some natural assumptions,

$$\begin{array}{c}\hfill n^{\frac{1}{2}}({\widehat{\mathit{\theta }}}_n-{\mathit{\theta }}_g)\stackrel{d}{\to }G,\end{array}$$

(2)

where, under the probability measure associated with $g(·)$, G is a Gaussian random vector with mean vector $\mathbf{0}$ and covariance matrix ${\mathsf{\Sigma }}_g$. If $g(·)=f_{{\mathit{\theta }}_0}(·)$, then the variance of G coincides with the inverse of the Fisher information matrix $\mathfrak{I}\left({\mathit{\theta }}_0\right)$, yielding statistical efficiency. When the true distribution $g(·)$ does not belong to $\mathfrak{F}$, we will call this the “model misspecifed case,” while when $g\in \mathfrak{F}$, we will say that the “postulated model” holds.

In this paper, we focus on the large deviation behavior of $\{{\widehat{\mathit{\theta }}}_n:n\ge 1\}$; namely, the asymptotic probability that the estimate ${\widehat{\mathit{\theta }}}_n$ will achieve values within a set away from the central tendency described in (2). We establish results of the form

$$\begin{array}{c}\hfill log{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in B)\approx -n\underset{\mathit{\theta }\in B}{inf}I\left(\mathit{\theta }\right),\end{array}$$

(3)

for some “rate function” I and given Borel subset $B\subset \mathsf{\Theta }$. Similar large deviation estimates for maximum likelihood estimators (MLE) have been investigated in [5,6,7], and for general M-estimators in [8,9]. These results allow for a precise description of the probabilities of Type I and Type II error in both the Neymann–Pearson and likelihood ratio test frameworks. Furthermore, large deviation bounds allow one to identify the best exponential rate of decrease of Type II error amongst all tests that satisfy a bound on the Type I error, as in Stein’s lemma (cf. [10]). Additional evidence of the importance of large deviation results for statistical inference has been described in [11] and in the book [12].

One of our initial goals was to derive sharp probability bounds for Type I and Type II error in the context of robust hypothesis testing using Hellinger deviance tests. This article is a first step towards this endeavor. A key issue that distinguishes our work from earlier works is that, in our case, the objective function is a nonlinear function of the smoothed empirical measure, and the analysis of this case requires more involved methods compared with those currently existing in the statistical literature on large deviations. Consistent with large deviation analysis more generally, we identify the rate function I as the convex conjugate of a certain limiting cumulant generating function, although in our problem, we uncover a subtle asymmetry between the upper and lower bounds when our limiting generating function is nondifferentiable. In the classical large deviation literature, similar asymmetries have been studied in other one-dimensional contexts (e.g. [13]), although the statistical problem is still quite different, as the dependence on the parameter $\mathit{\theta }$ arises explicitly—inhibiting the use of convexity methods typically exploited in the large deviation literature—and hence requiring novel techniques.

1.1. Large Deviations

In this subsection we provide relevant definitions and properties from large deviation theory required in the sequel. In the following, ${\mathbb{R}}_{+}$ will denote the set of non-negative real numbers.

Definition 1.

A collection of probability distributions $\{P_n:n\ge 1\}$ on a topological space $(\mathcal{X},\mathcal{B})$ is said to satisfy the weak large deviation principle if

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{P}_n\left(F\right)\le -\underset{x\in F}{inf}I\left(x\right),\mathrm{for}\mathrm{all}\mathrm{closed}F\in \mathcal{B},\end{array}$$

and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{P}_n\left(G\right)\ge -\underset{x\in G}{inf}I\left(x\right)\mathrm{for}\mathrm{all}\mathrm{open}\mathrm{sets}F\in \mathcal{B}\end{array}$$

for some lower semicontinuous function $I:\mathcal{X}\to [0,\infty ]$. The function I is called the rate function. If the level sets of I are compact, we call I a good rate function and we say that $\left\{P_n\right\}$ satisfies the large deviation principle (LDP).

We begin with a brief review of large deviation results for i.i.d. random variables and empirical measures. Let $\left\{X_n\right\}\subset \mathbb{R}$ be an i.i.d. sequence of real-valued random variables, and let $P_n$ denote the distribution of the sample mean ${\overline{X}}_n$. If the moment generating function of $X_1$ is finite in a neighborhood of the origin, then Cramér’s theorem states that $\left\{P_n\right\}$ satisfies the LDP with good rate function ${\mathsf{\Lambda }}^{*}$, where ${\mathsf{\Lambda }}^{*}$ is the convex conjugate (or Legendre–Fenchel transform) of $\mathsf{\Lambda }$, and where $\mathsf{\Lambda }\left(\alpha \right)=log\mathit{E}\left[e^{\alpha X_1}\right]$ is the cumulant generating function of $X_1$ (cf. [10], Section 2.2).

Next, consider the empirical measures $\left\{{\mu }_n\right\}$, defined by

$$\begin{array}{c}\hfill {\mu }_n\left(B\right)=\frac{1}{n}\sum _{i=1}^n{I}_{\{X_i\in B\}},B\in \mathcal{B},\end{array}$$

(4)

where $\mathcal{B}$ denotes the collection of Borel subsets of $\mathbb{R}$. It is well known (cf. [14]) that $\left\{{\mu }_n\right\}$ converges weakly to P, namely to the distribution of $X_1$. Then Sanov’s theorem asserts that $\left\{{\mu }_n\right\}$ satisfies a large deviation principle with rate function $I_P$ given by

$$\begin{array}{c}\hfill I_P\left(\nu \right)=\left\{\begin{array}{cc}\mathrm{KL}(\nu ,P)\hfill & \mathrm{if}\nu \ll P,\hfill \\ \infty \hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(5)

where $\mathrm{KL}(\nu ,P)$ is the Kullback–Leibler information between the probability measures $\nu$ and P. When $\nu$ and P each possesses a density with respect to Lebesgue measure (say p and g, respectively), the above expression becomes

$$\begin{array}{c}\hfill \mathrm{KL}(p,g):=\left\{\begin{array}{cc}{\int }_{S}p\left(x\right)log\left(\frac{p\left(x\right)}{g\left(x\right)}\right)d\mu \left(x\right)\hfill & \mathrm{if}p\ll g,\hfill \\ \infty \hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(6)

In Sanov’s theorem, the rate function $I_P$ is defined on the space of probability measures, which is a metric space with the open sets induced by weak convergence. Extensions of Sanov’s theorem to strong topologies have been investigated in the literature; cf., e.g., [15].

We now turn to a general result, which will play a central role in this paper, namely Varadhan’s integral lemma (cf. [10], Theorem 4.3.1). This result will allow us to infer the scaled limit of a sequence of generating functions from the existence of the large deviation principle.

Lemma 1

(Varadhan). Let $\left\{Y_n\right\}$ be a sequence of random variables taking values in a regular topological space $(\mathcal{X},\mathcal{B})$, and assume that the probability law of $\left\{Y_n\right\}$ satisfies the LDP with good rate function I. Then for any bounded continuous function $F:\mathcal{X}\to \mathbb{R}$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log\mathit{E}[exp\left(nF\left(Y_n\right)\right)]=\underset{x\in \mathcal{X}}{sup}\left\{F\left(x\right)-I\left(x\right)\right\}.\end{array}$$

(7)

1.2. Minimum Hellinger Distance Estimator and Large Deviations

We first observe that the MHDE is obtained by maximizing

$$\begin{array}{c}\hfill {\mathcal{A}}_n\left(\mathit{\theta }\right)\equiv {\mathcal{A}}_n(\mathit{\theta },g_n):={\int }_{\mathbb{R}}{f}_{\mathit{\theta }}^{\frac{1}{2}}\left(x\right)g_n^{\frac{1}{2}}\left(x\right)d\mu \left(x\right),\end{array}$$

(8)

which involves solving the equation $\nabla {\mathcal{A}}_n\left(\mathit{\theta }\right)=0.$ The idea behind the large deviation analysis is to observe that the large deviation behavior of the maximizer can be extracted from that of the objective function $\nabla {\mathcal{A}}_n\left(\mathit{\theta }\right)$ near $\mathbf{0}$. By the Gärtner–Ellis theorem (cf. [10], Section 2.3), this amounts to investigating the asymptotic behavior as $n\to \infty$ of

$$\begin{array}{c}\hfill \frac{1}{a_n}log{\mathit{E}}_g[exp\left\{a_n\langle \mathbf{\alpha },{\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)\rangle \right\}],\mathbf{\alpha }\in {\mathbb{R}}^d,\end{array}$$

(9)

where $a_n\nearrow \infty$ as $n\to \infty$. In the case of maximum likelihood estimation (MLE) or minimum contrast estimation (MCE), the objective function can be expressed as

$$\begin{array}{c}\hfill \sum _{i=1}^n{h}_{\mathit{\theta }}\left(X_i\right)=n{\int }_{\mathbb{R}}{h}_{\mathit{\theta }}\left(x\right)d{\mu }_n\left(x\right),\end{array}$$

(10)

where $\{{\mu }_n:n\ge 1\}$ is the empirical measure associated with $\{X_k:1\le k\le n\}$. Thus, while the objective functions associated with the MLE and MCE are linear functions of the empirical measure, the affinity is a nonlinear function of the empirical measure. This creates certain complications in identifying the rate function $I(·)$ alluded to in (3). Of course, in the case of likelihood and minimum contrast estimator analysis, an explicit formula for $I(·)$ ensues as the Legendre–Fenchel transform of the cumulant generating function of $h_{\mathit{\theta }}\left(X_1\right)$, viz. $log{\mathit{E}}_{{\mathit{\theta }}_0}\left[exp\left(\alpha h_{\mathit{\theta }}\left(X_1\right)\right)\right]$. One approach to evaluating the limiting generating function is to apply Varadhan’s lemma as given above in (7). In the context of our problem, that requires an investigation into the large deviation principle for the density estimators $g_n(·)$ viewed as elements of $L_1\left(S\right)$, viz. the space of integrable functions on S. Equivalently, we require a version of Sanov’s theorem in $L_1$-space, which leads to certain topological considerations. The main issue here is that, when $L_1$ is equipped with a norm topology, the sequence of kernel density estimates $\left\{g_n(·)\right\}$ possesses large deviation bounds, but the associated rate function may not have compact level sets, as is required for a typical application of Varadhan’s lemma. Nonetheless, one obtains a full LDP when $L_1\left(S\right)$ is equipped with the weak topology.

The asymptotic properties of MHDE, such as consistency and asymptotic normality, are established using the norm convergence of $g_n(·)$ to $g(·)$. For this reason, we focus on a subclass of densities $\mathcal{G}$ (see Proposition 1 below) possessing certain equicontinuity properties where norm convergence prevails. These issues are handled in Section 2, where the precise statements of our main results can also be found. Section 3 is devoted to the proofs of the main results. Section 4 contains some concluding remarks.

2. Notation, Assumptions, and Main Results

Let $f_{\mathit{\theta }}(·)$ denote the postulated density of $\left\{X_n\right\}$, defined on a measure space $(\mathsf{\Omega },\mathcal{F})$. Let $S\subset \mathbb{R}$ denote the support of X and $s_{\mathit{\theta }}(·)=f_{\mathit{\theta }}^{\frac{1}{2}}(·)$. Let the true density of $\left\{X_n\right\}$ be given by $g(·)$. Throughout the paper, we assume that the following regularity conditions hold.

Hypothesis 1.

$\mathsf{\Theta }$ is a compact and convex subset of ${\mathbb{R}}^d$.

Hypothesis 2.

The family $\mathfrak{F}$ is identifiable; namely, if ${\mathit{\theta }}_1\ne {\mathit{\theta }}_2$, $f_{{\mathit{\theta }}_1}(·)\ne f_{{\mathit{\theta }}_2}(·)$ on a set of positive Lebesgue measure.

Hypothesis 3.

For every $\mathit{\theta }\in \mathsf{\Theta }$, $s_{\mathit{\theta }}$ is three times continuously differentiable with respect to all components of $\mathit{\theta }$. Denote by $\nabla s_{\mathit{\theta }}$ the gradient of $s_{\mathit{\theta }}$ and its components by ${\dot{s}}_{\mathit{\theta }}^i(·)$. Let ${\mathcal{H}}_{\mathit{\theta }}$ denote the matrix of second partial derivatives of $s_{\mathit{\theta }}(·)$ with respect to $\mathit{\theta }$ and ${\ddot{s}}_{\mathit{\theta }}^{ij}$ the ${(i,j)}^{th}$ element of ${\mathcal{H}}_{\mathit{\theta }}$.

Hypothesis 4.

Let the matrix of second partial derivatives of ${\mathcal{A}}_n\left(\mathit{\theta }\right)$ and $\mathcal{A}\left(\mathit{\theta }\right)$ be denoted by $H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right)$ and $H_{\mathcal{A}}\left(\mathit{\theta }\right)$, respectively. Assume that $H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right)$ and $H_{\mathcal{A}}\left(\mathit{\theta }\right)$ are continuous in $\mathit{\theta }$ and that $H_{\mathcal{A}}\left(\mathit{\theta }\right)$ is positive definite for every $\mathit{\theta }\in \mathsf{\Theta }$. For $p\in \mathcal{G}$ and $\mathit{\theta }\in \mathsf{\Theta }$, let ${\lambda }_{\mathit{\theta }}\left(p\right)$ denote the smallest eigenvalue of the matrix ${\int }_S{\mathcal{H}}_{\mathit{\theta }}\left(x\right)p^{\frac{1}{2}}\left(x\right)dx.$ Assume that $inf\{{\lambda }_{\mathit{\theta }}\left(p\right):p\in \mathcal{G}\}\ge c>0$, where c is independent of $\mathit{\theta }$.

These hypotheses on the family $\mathfrak{F}$ are generally standard and are used to establish the asymptotic properties of the MHDE. Sufficient conditions on $\mathfrak{F}$ for the validity of these hypotheses are described in [3,16], and [17]. A remark on Hypothesis 4 is warranted here. When $p=g$, this assumption is related to the positive definiteness of the Fisher information matrix. If one assumes $\mathcal{G}=\mathfrak{F}$, then this hypothesis reduces to the condition that $inf\{{\lambda }_{\mathit{\theta }}:\mathit{\theta }\in \mathsf{\Theta }\}\ge c>0$, which is standard. Finally, we remark that we have not attempted to provide the weakest regularity conditions, and we do believe some of these conditions can possibly be relaxed.

Recall that the MHDE of $\mathit{\theta }$ can be obtained by solving the equation

$$\begin{array}{c}\hfill {\nabla \mathcal{A}}_n\left(\mathit{\theta }\right) :={\nabla }_{\mathit{\theta }}\mathcal{A}(f_{\mathit{\theta }},g_n)=\frac{1}{2}{\int }_{\mathbb{R}}{u}_{\mathit{\theta }}\left(x\right)s_{\mathit{\theta }}\left(x\right)g_n^{\frac{1}{2}}\left(x\right)dx=0,\end{array}$$

(11)

where $u_{\mathit{\theta }}\left(x\right)={\nabla }_{\mathit{\theta }}{f}_{\mathit{\theta }}\left(x\right){\left(f_{\mathit{\theta }}\left(x\right)\right)}^{-1}$ is the score function, which is obtained using ${\nabla }_{\mathit{\theta }}s(x;\mathit{\theta })=\frac{1}{2}u(x;\mathit{\theta })s(x;\mathit{\theta })$.

We begin by providing some heuristics for the case $d=1$. Let ${\dot{\mathcal{A}}}_n\left(\theta \right)$ denote the derivative of ${\mathcal{A}}_n\left(\theta \right)$ when $d=1$. Let ${\widehat{\theta }}_n$ denote the argzero of the function ${\dot{\mathcal{A}}}_n\left(\theta \right)$ obtained from (11) above. Let ${\widehat{\theta }}_{n,l}=inf\{\theta \in \mathsf{\Theta }:{\dot{\mathcal{A}}}_n\left(\theta \right)\le 0\}$ and ${\widehat{\theta }}_{n,u}=sup\{\theta \in \mathsf{\Theta }:{\dot{\mathcal{A}}}_n\left(\theta \right)\ge 0\}$. Since ${\widehat{\theta }}_{n,l}\le {\widehat{\theta }}_n\le {\widehat{\theta }}_{n,u}$, we obtain using Markov’s inequality that for any $ϵ>0$,

$$\begin{array}{c}\hfill {\mathit{P}}_g({\widehat{\theta }}_{n,l}\ge {\theta }_g+ϵ)\le {\mathit{P}}_g({\dot{\mathcal{A}}}_n({\theta }_g+ϵ)\ge 0)\le {\mathit{E}}_g[exp\left(n\alpha {\dot{\mathcal{A}}}_n({\theta }_g+ϵ)\right],\end{array}$$

(12)

where $\alpha >0$. Similarly, for $\alpha <0$, it can be seen that

$$\begin{array}{c}\hfill {\mathit{P}}_g({\widehat{\theta }}_{n,u}\le {\theta }_g-ϵ)\le {\mathit{P}}_g({\dot{\mathcal{A}}}_n({\theta }_g-ϵ)\le 0)\le {\mathit{E}}_g[exp\left(n\alpha {\dot{\mathcal{A}}}_n({\theta }_g-ϵ)\right].\end{array}$$

(13)

Thus, an evaluation of (9) will allow us to obtain the logarithmic upper bound for ${\widehat{\theta }}_{n,l}$ and ${\widehat{\theta }}_{n,u}$. Next, using the inequalities

$$\begin{array}{ccc}\hfill {\mathit{P}}_g({\widehat{\theta }}_{n,l}\ge {\theta }_g+ϵ)& \le & {\mathit{P}}_g({\dot{\mathcal{A}}}_n({\theta }_g+ϵ)\ge 0)\le {\mathit{P}}_g({\widehat{\theta }}_{n,u}\ge {\theta }_g+ϵ),\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill {\mathit{P}}_g({\widehat{\theta }}_{n,u}\le {\theta }_g-ϵ)& \le & {\mathit{P}}_g({\dot{\mathcal{A}}}_n({\theta }_g-ϵ)\le 0)\le {\mathit{P}}_g({\widehat{\theta }}_{n,l}\le {\theta }_g-ϵ),\hfill \end{array}$$

(15)

under additional hypotheses, one can derive large deviation lower bounds for ${\widehat{\theta }}_n$. Deriving these bounds for MLE and MCE is rather standard, since the objective functions and their derivatives are linear functionals of the empirical distribution, as stated in (10), but this is not the case for the Hellinger distance.

Observe that the probabilities in (12) and (13) represent rare-event probabilities since, under the hypotheses described previously, ${\widehat{\theta }}_n$ converges to ${\theta }_g$ almost surely as $n\to \infty$. The distributional results concerning ${\widehat{\theta }}_n$ rely on the continuity and differentiability properties of $\nabla {\mathcal{A}}_n\left(\theta \right)$, which depend nonlinearly on $g_n$, and the norm convergence of $g_n$ to g.

Let $\mathcal{G}$ denote the collection of all probability densities with support S. By Scheffe’s theorem, the pointwise convergence of $g_n$ to g implies $g_n\stackrel{L_1}{\to }g$ as $n\to \infty$. Additionally, when $g_n(·)$ is the kernel density estimator, then Glick’s Theorem guarantees that $g_n\stackrel{L_1}{\to }g$ almost surely as $n\to \infty$ when $b_n\searrow 0$ and $n\nearrow \infty$; cf. [18]. Since the MHDE are functionals of density estimators, it is natural to expect that the large deviations of density estimators will play a significant role in our analysis. For this reason, one is forced to consider the topological issues that arise in the large deviation analysis of density estimators. Interestingly, it turns out that the weak topology on $L_1\left(S\right)$ plays a prominent role. This, in turn, leads to the question of whether certain continuity properties, which were part of the traditional theory of MHD analysis, continue to hold if $\mathcal{G}$ were viewed as a subset of $L_1\left(S\right)$ equipped with weak topology. Expectedly, while the answer in general is no (cf. [19]), Proposition 1 provides sufficient conditions on the family $\mathcal{G}$ under which one additionally obtains norm convergence.

Before proceeding, we now introduce some further regularity conditions, as follows.

Hypothesis 5.

$u_{\mathit{\theta }}{s}_{\mathit{\theta }}\in L_2\left(S\right)$ and is an $L_2\left(S\right)$-continuous function of $\mathit{\theta }$.

Hypothesis 6.

The family $\mathfrak{F}$ consists of bounded equicontinuous densities.

Hypothesis 7.

The family $\mathcal{G}$ consists of bounded and equicontinuous densities.

Hypothesis 8.

$u_{\mathit{\theta }}g\in L_2\left(S\right)$ and is an $L_2\left(S\right)$-continuous function of $\mathit{\theta }$.

Here, we note that Hypotheses 6 and 7 are related. Furthermore, if one is willing to assume that $\mathcal{G}=\mathfrak{F}$, then one does not need Hypothesis 7. On the other hand, if one believes that parametric distributions are approximations to $\mathcal{G}$, then one needs to work with Hypothesis 7. For this reason, we have maintained both of these hypotheses in our main results. Hypotheses 5 and 8 are related to finiteness of the Fisher information and are standard in the statistical literature.

Before we state the first proposition, we recall the definition of weak topology on $L_1$ (cf. [19]). A sequence $\{h_n:n\ge 1\}$ is said to converge weakly in $L_1$ if ${\int }_S{h}_n\left(x\right)b\left(x\right)dx\to {\int }_{S}h\left(x\right)b\left(x\right)dx$ as $n\to \infty$ for every $b\in L_{\infty }\left(S\right)$, where $L_{\infty }\left(S\right)$ is a class of essentially bounded functions. We assume throughout the paper that the topology on $\mathsf{\Theta }$ is the standard topology generated by the Euclidean metric.

Proposition 1.

Let $\mathcal{G}$ denote the class of densities, equipped with the weak topology. Further assume that Hypotheses 1–7 hold. Let $\mathsf{\Theta }\otimes \mathcal{G}$ be equipped with the product topology. Then the mapping $\nabla \mathcal{A}:\mathsf{\Theta }\otimes \mathcal{G}\to {\mathbb{R}}^d$ defined by

$$\begin{array}{c}\hfill \nabla \mathcal{A}(\mathit{\theta },g){\int }_{\mathbb{R}}{u}_{\mathit{\theta }}\left(x\right)s_{\mathit{\theta }}\left(x\right)g^{\frac{1}{2}}\left(x\right)dx\end{array}$$

(16)

is jointly continuous in $(\mathit{\theta },g)$. Furthermore, if $g_n\stackrel{w}{\to }g$, then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}\left|\right|\nabla \mathcal{A}(\mathit{\theta },g_n)-\nabla \mathcal{A}(\mathit{\theta },g)\left|\right|=0.\end{array}$$

(17)

Finally, under Hypothesis 7, the family $\mathcal{G}$ is a weakly sequentially closed subset of $L_1\left(S\right)$.

Our next result is concerned with the limit behavior of the generating function of ${\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)$. In the following we use the notation $p\ll g$ to mean the probability measures associated with $p(·)$ and $g(·)$ are absolutely continuous.

Theorem 1.

Assume that Hypotheses 1–7 hold, and set

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{n,\mathit{\theta }}\left(\mathbf{\alpha }\right) := \frac{1}{n}log{\mathit{E}}_g[exp\left(n\langle \mathbf{\alpha },{\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)\rangle \right],\mathbf{\alpha }\in {\mathbb{R}}^d.\end{array}$$

(18)

Then ${\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right):={lim}_{n\to \infty }{\mathsf{\Lambda }}_{n,\mathit{\theta }}\left(\mathbf{\alpha }\right)$ exists and is a convex function given by

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)=\underset{\mathit{p}\in \mathcal{G}}{sup}\left\{{\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle s_{\mathit{\theta }}\left(x\right)p^{\frac{1}{2}}\left(x\right)dx-\mathrm{KL}(p,g)\right\},\end{array}$$

(19)

where

$$\begin{array}{c}\hfill \mathrm{KL}(p,g)=\left\{\begin{array}{cc}{\int }_{S}p\left(x\right)log\left(\frac{p\left(x\right)}{g\left(x\right)}\right)dx\hfill & \mathrm{if}p\ll g,\hfill \\ \infty \hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(20)

Remark 1.

Since ${\mathsf{\Lambda }}_{\mathit{\theta }}$ is defined via a limiting operation, it is hard to extract its qualitative properties. However, we can obtain a simple lower bound by observing that $\mathrm{KL}(p,g)=0$ if and only if $p=g$, and an upper bound using that the Kullback–Leibler information is nonnegative. This results in the following bounds:

$$\begin{array}{c}\hfill {\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle s_{\mathit{\theta }}\left(x\right)g^{\frac{1}{2}}\left(x\right)dx\le {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)\le \underset{p\in \mathcal{G}}{sup}\left[{\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle s_{\mathit{\theta }}\left(x\right)p^{\frac{1}{2}}\left(x\right)dx\right].\end{array}$$

(21)

Furthermore, if all densities in $\mathcal{G}$ are bounded by one, then $p^{\frac{1}{2}}(·)\ge p(·)$ implies

$$\begin{array}{ccc}\hfill {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)& \ge & \underset{p\in \mathcal{G}}{sup}\left\{{\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle s_{\mathit{\theta }}\left(x\right)p\left(x\right)dx-\mathrm{KL}(p,g)\right\}.\hfill \end{array}$$

(22)

Using a variational argument, it can be shown that the supremum on the right-hand side is attained at $p^{*}$ given by

$$\begin{array}{c}\hfill p^{*}\left(x\right) :=\frac{exp\left(\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle \right)s_{\mathit{\theta }}\left(x\right)}{{\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle s_{\mathit{\theta }}\left(x\right)g\left(x\right)dx};\end{array}$$

(23)

cf. [20]. Furthermore, the maximum that results from this choice of $p^{*}(·)$ is

$$log{\int }_{S}exp\left(\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle \right)s_{\mathit{\theta }}\left(x\right)g\left(x\right)dx,$$

yielding yet another lower bound for ${\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)$, although the comparison of these two lower bounds is not immediate.

Returning to our main discussion, recall from [21] that the convex conjugate of the function ${\mathsf{\Lambda }}_{\mathit{\theta }}$ is defined by

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{\mathit{\theta }}^{*}\left(\mathit{x}\right)=\underset{\mathbf{\alpha }\in {\mathbb{R}}^d}{sup}\left\{\langle \mathbf{\alpha },\mathit{x}\rangle -\mathsf{\Lambda }\left(\mathbf{\alpha }\right)\right\},\mathit{x}\in {\mathbb{R}}^d.\end{array}$$

(24)

Let ${\mathfrak{D}}_{\mathit{\theta }}$ denote the domain of ${\mathsf{\Lambda }}_{\mathit{\theta }}$; namely,

$$\begin{array}{c}\hfill {\mathfrak{D}}_{\mathit{\theta }}=\{\mathbf{\alpha }\in {\mathbb{R}}^d:{\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)<\infty \};\end{array}$$

(25)

and let ${\mathfrak{R}}_{\mathit{\theta }}$ denote the range of the gradient map $\nabla {\mathsf{\Lambda }}_{\mathit{\theta }}$; that is,

$${\mathfrak{R}}_{\mathit{\theta }}=\left\{\mathit{x}\in {\mathbb{R}}^d:\nabla {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\alpha \right)=\mathit{x},\mathrm{some}\mathbf{\alpha }\in {\mathbb{R}}^d\right\}.$$

We begin with the discussion of the case $d=1$. In this case, the generating function ${\mathsf{\Lambda }}_{\theta }$ reduces to

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{\theta }\left(\alpha \right)=\underset{p\in \mathcal{G}}{sup}\left\{\alpha {\int }_{S}exp(n\alpha {\dot{\mathcal{A}}}_n\left(\theta \right)s(x;\theta )p^{\frac{1}{2}}\left(x\right)dx-\mathrm{KL}(p,g)\right\}.\end{array}$$

(26)

By the convexity of ${\mathsf{\Lambda }}_{\theta }(·)$, this function is differentiable almost everywhere (cf. [21]), and in the proof, we would like to exploit the differentiability of this function at the point ${\alpha }_{\theta }^{*}$ where it attains its minimum value. If ${\mathsf{\Lambda }}_{\theta }$ is not differentiable at this point, it is helpful to consider the directional derivatives of ${\mathsf{\Lambda }}_{\theta }$. Specifically, let ${\mathsf{\Lambda }}_{\theta ,+}^{\prime }(·)$ and ${\mathsf{\Lambda }}_{\theta ,-}^{\prime }(·)$ denote the right and left derivatives of ${\mathsf{\Lambda }}_{\theta }(·)$, respectively. When $x\in \left({\mathsf{\Lambda }}_{\theta ,-}^{\prime }\left(\alpha \right),{\mathsf{\Lambda }}_{\theta ,+}^{\prime }\left(\alpha \right)\right)$, then it is well known that ${\mathsf{\Lambda }}_{\theta }^{*}\left(x\right)=\alpha x-{\mathsf{\Lambda }}_{\theta }\left(\alpha \right)$, but this observation will not be sufficient to obtain a proper lower bound. For that to hold, we need a stronger condition, namely that $0\in {\mathfrak{R}}_{\theta }$, which will only be true if ${\mathsf{\Lambda }}_{\theta }$ is differentiable at its point of minimum, ${\alpha }_{\theta }^{*}$. Otherwise, the expected lower bound turns out to be ${\mathsf{\Lambda }}_{\theta }^{*}\left(x\right)$, where $x={\mathsf{\Lambda }}_{\theta ,+}^{\prime }\left({\alpha }_{\theta }^{*}\right)$; cf. [13].

We now turn to our large deviation theorem in ${\mathbb{R}}^1$, where we study the rare-event probabilities ${\mathit{P}}_g({\widehat{\theta }}_n\in C)$ for sets C that are away from the true value ${\theta }_g$. Specifically, we establish an analogue of the LDP, but where a subtle difference arises in the lower bound in the absence of differentiability of ${\mathsf{\Lambda }}_{\theta }$.

We recall that ${\widehat{\theta }}_n$ is defined using the kernel density estimator $g_n(·)$ defined in (1), whose behavior is dictated by the bandwidth sequence $\left\{b_n\right\}$.

Theorem 2.

Assume $d=1$, Hypotheses 1–8 are satisfied, and ${\widehat{\theta }}_n$ is the unique zero of ${\dot{\mathcal{A}}}_n\left(\theta \right)=0$. Further assume that $b_n\searrow 0$ and $n{b}_n\nearrow \infty$ as $n\to \infty$. Then for any closed set F not containing ${\theta }_g$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\in F)\le -\underset{\theta \in F}{inf}{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right).\end{array}$$

(27)

Moreover, for any open set G not including ${\theta }_g$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\in G)\ge -\underset{\theta \in G}{inf}I\left(\theta \right),\end{array}$$

(28)

where

$$\begin{array}{c}\hfill I\left(\theta \right)=inf\left\{{\mathsf{\Lambda }}_{\theta }^{*}\left(x\right):x\in {\mathfrak{R}}_{\theta }\cap [0,\infty )\right\},\end{array}$$

(29)

and the infimum is taken to be infinity if the set ${\mathfrak{R}}_{\theta }\cap [0,\infty )$ is empty.

Remark 2.

If $F=[\theta ,\infty )$ where $\theta >{\theta }_g$, then in both the upper and lower bounds, it is sufficient to evaluate the infimum at the boundary point $\theta$. That is,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\in [\theta ,\infty ))\le -{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right).\end{array}$$

Similarly, if $G=(\theta ,\infty )$ where $\theta >{\theta }_g$, then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\in (\theta ,\infty ))\ge -I\left(\theta \right).\end{array}$$

Furthermore, if ${inf}_{\alpha }{\mathsf{\Lambda }}_{\theta }\left(\alpha \right)$ is achieved at a unique point ${\alpha }_{\theta }^{*}$ and ${\mathsf{\Lambda }}_{\theta }$ is differentiable at ${\alpha }_{\theta }^{*}$, then the right-hand side of (28) reduces to ${\mathsf{\Lambda }}_{\theta }^{*}\left(0\right)$, i.e., the upper and lower bounds coincide and the limits exist. Since the rate function appearing in the upper and lower bounds coincide in this case, we obtain a proper LDP if the resulting rate function has the required regularity properties, in particular, $I\left(\theta \right)={\mathsf{\Lambda }}_{\theta }^{*}\left(0\right)$ is lower semicontinuous and has compact level sets.

The proof of the above theorem relies on (14) and (15) combined with Theorem 1, together with a change of measure argument characteristic of large deviation analysis. The comparison inequalities in (14) and (15) are critical to obtaining the characterizations in the above theorem, but these are essentially one-dimensional results and their analogues in higher dimensions ($d\ge 2$) are not immediate. Consequently, when ${\mathsf{\Lambda }}_{\mathit{\theta }}$ is not differentiable, new complications arise, which lead to a slightly different, and less explicit, representation of the lower bound.

Next we establish a large deviation theorem for ${\mathbb{R}}^d$, generalizing the previous theorem to higher dimensions. In the following, let $\mathrm{dist}(x,G)={inf}_{y\in G}\left|\right|x-y\left|\right|$ denote the distance between a point $x\in {\mathbb{R}}^d$ and a set $G\subset {\mathbb{R}}^d$.

Theorem 3.

Assume Hypotheses 1–8 are satisfied, and assume that $b_n\searrow 0$ and $n{b}_n\nearrow \infty$ as $n\to \infty .$ Then for any closed set F not containing${\mathit{\theta }}_g$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in F)\le -\underset{\mathit{\theta }\in F}{inf}{\mathsf{\Lambda }}_{\mathit{\theta }}^{★}\left(\mathbf{0}\right).\end{array}$$

(30)

Moreover, for any open set G not including ${\mathit{\theta }}_g$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in G)\ge -\underset{\mathit{\theta }\in G}{inf}I\left(\mathit{\theta }\right),\end{array}$$

(31)

where $I\left(\mathit{\theta }\right)=inf\left\{{\mathsf{\Lambda }}_{\mathit{\theta }}^{*}\left(\mathit{x}\right):\mathit{x}\in {\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};c_{\mathit{\theta }})\right\}$ and $c_{\mathit{\theta }}=b\mathrm{dist}(\mathit{\theta },\mathsf{\Theta }-G)$ for some universal constant $b\in (0,\infty ),$ and the infimum is taken to be infinity if the set ${\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};c_{\mathit{\theta }})$ is empty.

Remark 3.

As we noted for the one-dimensional case in Remark 2, under a differentiability assumption on ${\mathsf{\Lambda }}_{\theta }$, the function $I\left(\mathit{\theta }\right)$ can be identified as ${\mathsf{\Lambda }}_{\theta }^{*}\left(\mathbf{0}\right)$, but in full generality, it is not immediately known that $I\left(\mathit{\theta }\right)$ is even nontrivial. Moreover, without differentiability, the infimum in the definition of $I\left(\mathit{\theta }\right)$ is more restrictive than what we encountered in the one-dimensional problem. However, if one assumes additional geometry on G, such as a translated cone structure, then one obtains improved estimates in the sense that one can take unbounded regions in the definition of $I\left(\mathit{\theta }\right)$, just as we saw in Theorem 2.2. For further remarks in this direction, see the discussion given after the proof of the theorem.

3. Proofs

We turn first to Proposition 1.

Proof of Proposition 1.

Since $\mathsf{\Theta }\otimes \mathcal{G}$ is equipped with product topology, it is sufficient to show that if ${\mathit{\theta }}_n\to \mathit{\theta }$ and $g_n\stackrel{w}{\to }g$, then ${\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)$ converges to $\nabla \mathcal{A}\left(\mathit{\theta }\right)$, where

$$\begin{array}{c}\hfill \nabla \mathcal{A}\left(\mathit{\theta }\right)={\int }_S{u}_{\mathit{\theta }}\left(x\right)s_{\mathit{\theta }}\left(x\right)g^{\frac{1}{2}}\left(x\right)dx.\end{array}$$

(32)

Let $r_{\mathit{\theta }}\left(x\right)=u_{\mathit{\theta }}\left(x\right)s_{\mathit{\theta }}\left(x\right)$, and observe that

$$\begin{array}{ccc}\hfill \left|\nabla \mathcal{A}({\mathit{\theta }}_n,g_n)-\nabla \mathcal{A}(\mathit{\theta },g)\right|& \le & {\int }_S|r_{{\mathit{\theta }}_n}\left(x\right)\left|\right|g_n^{\frac{1}{2}}\left(x\right)-g^{\frac{1}{2}}\left(x\right)|dx+{\int }_S|r_{{\mathit{\theta }}_n}\left(x\right)-r_{\mathit{\theta }}\left(x\right)|g^{\frac{1}{2}}\left(x\right)dx\hfill \\ & \le & \left|\right|r_{\mathit{\theta }}{\left|\right|}_2\mathrm{HD}(g_n,g)+{\int }_S|r_{{\mathit{\theta }}_n}\left(x\right)-r_{\mathit{\theta }}\left(x\right)|g^{\frac{1}{2}}\left(x\right)dx\hfill \\ & =& T_{n,1}+T_{n,2},\hfill \end{array}$$

(33)

where the penultimate equation follows by applying the Cauchy–Schwarz inequality. Then by the Cauchy–Schwarz inequality and Hypothesis 5, $T_{n,2}\to 0$. Since Hellinger distance is dominated by the $L_1$-distance, in order to complete the proof, it is sufficient to show that $\left|\right|g_n-g{\left|\right|}_1\to 0$. Now since $g_n\stackrel{w}{\to }g$, it follows that as $n\to \infty$,

$$\begin{array}{c}\hfill G_n\left(x\right) :={\int }_S{g}_n\left(y\right)I_{\{y\le x\}}dy\to {\int }_{S}g\left(y\right)I_{\{y\le x\}}dyG\left(x\right).\end{array}$$

(34)

Evidently, $G_n(·)$ and $G(·)$ are nondecreasing and right continuous. Furthermore, if $x_{*}=inf\{x:x\in S\}$ and $x^{*}=sup\{x:x\in S\}$, then $G_n\left(x_{*}\right)\to G\left(x_{*}\right)$ and $G_n\left(x^{*}\right)\to G\left(x^{*}\right)$, where $G_n\left(x_{*}\right)={lim}_{x\to x_{*}}{G}_n\left(x\right)$, $G_n\left(x^{*}\right)={lim}_{x\to x*}{G}_n\left(x\right)$, $G\left(x_{*}\right)={lim}_{x\to x_{*}}G\left(x\right)$, $G\left(x^{*}\right)={lim}_{x\to x*}G\left(x\right)$. Thus $G_n$ converges to G, which is a proper distribution function. Then by Lemma 1 of Boos [22], $g_n(·)$ converges to $g(·)$ uniformly on compact sets. This, in turn, implies the $L_1$ convergence of $g_n(·)$ to $g(·)$ (by Scheffe’s lemma), which establishes the convergence of $T_{n,1}$ to 0, thus completing the proof of the joint continuity of $\nabla \mathcal{A}(\mathit{\theta },g)$.

Next, the uniform convergence (17) follows by Hypothesis 5, since

$$\begin{array}{ccc}\hfill \underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}\left|\nabla \mathcal{A}(\mathit{\theta },g_n)-\nabla \mathcal{A}(\mathit{\theta },g)\right|& \le & {\int }_S|r_{\mathit{\theta }}\left(x\right)\left|\right|g_n^{\frac{1}{2}}\left(x\right)-g^{\frac{1}{2}}\left(x\right)|dx\hfill \\ & \le & \underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}\left|\right|r_{\mathit{\theta }}{\left|\right|}_2\mathrm{HD}(g_n,g)\to 0.\hfill \end{array}$$

Finally, to prove that $\mathcal{G}$ is weakly sequentially closed, note that convergence in weak topology implies pointwise convergence, yielding $g(·)\ge 0$. Noting that

$$\begin{array}{c}\hfill {\int }_{S}g\left(x\right)d\mu \left(x\right)=1+{\int }_S(g\left(x\right)-g_n\left(x\right))dx,\end{array}$$

(35)

it follows that $g(·)$ integrates to one, using $L_1$ convergence, thus completing the proof of the proposition. □

We now turn to the proof of Theorem 1. The proof relies on the large deviation theorem for the kernel density estimator $g_n(·)$ in the weak topology of $\mathcal{G}$. The next proposition is concerned with the LDP for $\left\{g_n\right\}$ in $\mathcal{G}$, equipped with the inherited weak topology from $L_1\left(S\right)$. This issue has received considerable attention recently (cf. [23,24]), where it is established that the full LDP may not hold for $\left\{g_n\right\}$ in norm topology, but does hold under the weak topology.

Proposition 2.

Assume Hypotheses 1–8 and that $b_n\searrow 0$ and $n{b}_n\nearrow \infty$ as $n\to \infty$. Then $\left\{g_n\right\}$ satisfies the LDP in the weak topology of $L_1\left(S\right)$ with good rate function I given by

$$\begin{array}{c}\hfill I\left(p\right)=\left\{\begin{array}{cc}{\int }_{S}p\left(x\right)log\left(\frac{p\left(x\right)}{g\left(x\right)}\right)dx\hfill & \mathrm{if}g\ll p,\hfill \\ \infty \hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(36)

Proof of Theorem 1.

As before, let $\mathcal{G}$ be equipped with the weak topology. Set $r_{\mathit{\theta }}\left(x\right)=u_{\mathit{\theta }}\left(x\right)s_{\mathit{\theta }}\left(x\right)$, and define $F:\mathcal{G}\to \mathbb{R}$ as follows:

$$\begin{array}{c}\hfill F\left(h\right)={\int }_S\langle \mathbf{\alpha },r_{\mathit{\theta }}\left(x\right)\rangle h^{\frac{1}{2}}\left(x\right)dx.\end{array}$$

(37)

By Hypothesis 5, $r_{\mathit{\theta }}\in L_2\left(S\right)$. To show that $F(·)$ is continuous, let $h_n\stackrel{w}{\to }h$ as $n\to \infty$. Then

$$\begin{array}{ccc}\hfill |F\left(h_n\right)-F\left(h\right)|& \le & {\int }_S{r}_{\mathit{\theta }}\left(x\right)|h_n^{\frac{1}{2}}\left(x\right)-h^{\frac{1}{2}}\left(x\right)|d\mu \left(x\right)\hfill \\ \hfill & \le & \left|\right|r_{\mathit{\theta }}{\left|\right|}_2\mathrm{HD}(h_n,h)\le \left|\right|r_{\mathit{\theta }}{\left|\right|}_2\left|\right|h_n-h{\left|\right|}_1\to 0\mathrm{as}n\to \infty ,\hfill \end{array}$$

(38)

where we have used the Cauchy–Schwarz inequality that the $L_1$ distance dominates the Hellinger distance in (38). Now by Hypothesis 7, as in the proof of Proposition 1, we have that $\left|\right|h_n-h{\left|\right|}_1\to 0$ as $n\to \infty$, establishing the continuity of $F(·)$. Next, to show that $F(·)$ is bounded, note that $sup\{F\left(p\right):p\in \mathcal{G}\}\le \left|\right|r_{\mathit{\theta }}{\left|\right|}_2$ by the Cauchy–Schwarz inequality. Then by Proposition 2, it follows by Varadhan’s integral lemma (see [10], Theorem 4.3.1) that

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log\mathit{E}\left[exp(nF\left(g_n\left(x\right)\right)\right]& =& \underset{n\to \infty }{lim}\frac{1}{n}log\mathit{E}\left[exp\left(n{\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)s_{\mathit{\theta }}\left(x\right)\rangle g_n^{\frac{1}{2}}\left(x\right)dx\right)\right]\hfill \\ & =& \underset{p\in \mathcal{G}}{sup}\left\{{\int }_S\langle \mathbf{\alpha },u_{\mathit{\theta }}\left(x\right)\rangle s_{\mathit{\theta }}\left(x\right)p^{\frac{1}{2}}\left(x\right)dx-\mathrm{KL}(p,g)\right\}\hfill \\ & := & {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right).\hfill \end{array}$$

(39)

This completes the proof of the theorem. □

The proofs of our main results will involve probability bounds on the modulus of continuity of ${\mathcal{A}}_n\left(\mathit{\theta }\right)$ and ${\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)$, respectively. Recall that the modulus of continuity $\omega (h;r)$ of a function $h:{\mathbb{R}}^d\to \mathbb{R}$ is given by

$$\begin{array}{c}\hfill \omega (h;r) := \underset{\left|\right|x_1-x_2\left|\right|\le r}{sup}|h\left(x_1\right)-h\left(x_2\right)|,r>0.\end{array}$$

(40)

Observe that when $h(·)$ is replaced by ${\mathcal{A}}_n\left(\mathit{\theta }\right)$ or ${\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)$, the modulus of continuity becomes a random quantity. Our next proposition summarizes the continuity properties of ${\mathcal{A}}_n\left(\mathit{\theta }\right)$ and ${\nabla \mathcal{A}}_n\left(\mathit{\theta }\right)$ via their modulus of continuity as real-valued functionals from $\mathcal{G}$ equipped with the weak topology.

Proposition 3.

Assume that Hypotheses 1–8 hold and that $b_n\searrow 0$ and $n{b}_n\nearrow \infty$ as $n\to \infty$. Then, with respect to $\left\{{\mathcal{A}}_n\right\}$ and $\mathcal{A}$, the modulus of continuity satisfies the following relations, each with probability one:

$$\begin{array}{c}\left(\mathrm{i}\right)\underset{n\to \infty }{lim}\omega ({\mathcal{A}}_n;r)=\omega (\mathcal{A},r);\left(\mathrm{ii}\right)\underset{r\to 0}{lim}\omega ({\mathcal{A}}_n;r)=0;\mathrm{and}\left(\mathrm{iii}\right)\underset{r\to 0}{lim}\omega (\mathcal{A};r)=0.\hfill \end{array}$$

Similarly, the sequence $\{\nabla {\mathcal{A}}_n\}$ and $\nabla \mathcal{A}$ satisfy the analogous relations with probability one; namely,

$$\begin{array}{c}\hfill \left(\mathrm{iv}\right)\underset{n\to \infty }{lim}\omega (\nabla {\mathcal{A}}_n;r)=\omega (\nabla \mathcal{A};r);\left(\mathrm{v}\right)\underset{r\to 0}{lim}\omega (\nabla {\mathcal{A}}_n;r)=0;\mathrm{and}\left(\mathrm{vi}\right)\underset{r\to 0}{lim}\omega (\nabla \mathcal{A};r)=0.\end{array}$$

Proof. 

First observe that ${\mathcal{A}}_n\left(\mathit{\theta }\right)$ converges uniformly to $\mathcal{A}\left(\mathit{\theta }\right)$. To see this, note that if $g_n\stackrel{w}{\to }g$, then by Proposition 1, it converges in $L_1$. Hence

$$\begin{array}{ccc}\hfill \underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}|{\mathcal{A}}_n\left(\mathit{\theta }\right)-\mathcal{A}\left(\mathit{\theta }\right)|& \le & \underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}{\int }_{\mathbb{R}}{s}_{\mathit{\theta }}\left(x\right)|g_n^{\frac{1}{2}}\left(x\right)-g^{\frac{1}{2}}\left(x\right)|dx\hfill \\ & \le & \left|\right|g_n^{\frac{1}{2}}\left(x\right)-g^{\frac{1}{2}}{\left(x\right)\left|\right|}_2\le \left|\right|g_n-g{\left|\right|}_1\to 0,\hfill \end{array}$$

(41)

where the last inequality follows using that the Hellinger distance is dominated by the $L_1$-distance. We now prove (i). For this we invoke the properties of the modulus of continuity. Observe that

$$\begin{array}{c}\hfill \omega ({\mathcal{A}}_n;r)=\omega ({\mathcal{A}}_n-\mathcal{A}+\mathcal{A};r)\le \omega ({\mathcal{A}}_n-\mathcal{A};r)+\omega (\mathcal{A};r),\end{array}$$

(42)

which yields

$$\begin{array}{c}\hfill |\omega ({\mathcal{A}}_n;r)-\omega (\mathcal{A};r)|\le \omega ({\mathcal{A}}_n-\mathcal{A};r).\end{array}$$

(43)

Next observe that

$$\begin{array}{ccc}\hfill \omega ({\mathcal{A}}_n-\mathcal{A};r)& =& \underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|\le r}{sup}|({\mathcal{A}}_n-\mathcal{A})\left({\mathit{\theta }}_1\right)-({\mathcal{A}}_n-\mathcal{A})\left({\mathit{\theta }}_2\right)|\hfill \\ & \le & 2\underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}|{\mathcal{A}}_n\left(\mathit{\theta }\right)-\mathcal{A}\left(\mathit{\theta }\right)|\to 0,\hfill \end{array}$$

(44)

where the last convergence follows from the uniform convergence of $({\mathcal{A}}_n-\mathcal{A})\left(\mathit{\theta }\right)$ to 0 as shown in (42). The proof of (iv) is similar, and specifically is obtained by using that

$$\begin{array}{c}\hfill \omega (\nabla ({\mathcal{A}}_n-\mathcal{A});r)\le 2\underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}\left|\right|\nabla {\mathcal{A}}_n\left(\mathit{\theta }\right)-\nabla \mathcal{A}\left(\mathit{\theta }\right)\left|\right|\to 0,\end{array}$$

(45)

where the above convergence follows from (17).

We now turn to the proof of (ii). Using the Cauchy–Schwarz inequality and the definition of Hellinger distance,

$$\begin{array}{ccc}\hfill \omega ({\mathcal{A}}_n;r)& =& \underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|\le r}{sup}|{\mathcal{A}}_n\left({\mathit{\theta }}_1\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_2\right)|\hfill \\ & =& \underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|\le r}{sup}\left|{\int }_{\mathbb{R}}(s_{{\mathit{\theta }}_1}\left(x\right)-s_{{\mathit{\theta }}_2}\left(x\right))g_n^{\frac{1}{2}}\left(x\right)dx\right|\le \mathrm{HD}(f_{{\mathit{\theta }}_1},f_{{\mathit{\theta }}_2})\hfill \\ \hfill & \le & \underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|\le r}{sup}\left|\right|f_{{\mathit{\theta }}_1}-f_{{\mathit{\theta }}_2}{\left|\right|}_1 := \omega (H;r),\hfill \end{array}$$

(46)

where $H:({\mathit{\theta }}_1,{\mathit{\theta }}_2)\to \left|\right|f_{{\mathit{\theta }}_1}-f_{{\mathit{\theta }}_2}{\left|\right|}_1$ is continuous since $\mathfrak{F}$ is continuous in $\mathit{\theta }$. Also, since $\mathsf{\Theta }\times \mathsf{\Theta }$ is compact, $H(·,·)$ is uniformly continuous. Since the modulus of continuity converges to 0 if and only if $H(·,·)$ is uniformly continuous, (ii) follows. Turning to (v), notice that, as before,

$$\begin{array}{c}\hfill \omega (\nabla {\mathcal{A}}_n;r)\le \underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|\le r}{sup}\left|\right|u_{{\mathit{\theta }}_1}{s}_{{\mathit{\theta }}_1}-u_{{\mathit{\theta }}_2}{s}_{{\mathit{\theta }}_2}{\left|\right|}_2.\end{array}$$

(47)

Now, since $u_{\mathit{\theta }}{s}_{\mathit{\theta }}$ is $L_2$ continuous, by Hypothesis 5, the proof follows as in (ii) due to to the compactness of $\mathsf{\Theta }$. The proofs of (iii) and (vi) are similar to (ii) and (v), respectively, and are therefore omitted. □

Proposition 4.

For any $0<M<\infty$ and $\delta >0$, there exists a positive number $r(M,\delta )$ such that

$$\begin{array}{c}\hfill {\mathit{P}}_g(\omega ({\mathcal{A}}_n;r)\ge \delta )\le e^{-Mn}and{\mathit{P}}_g(\omega (\nabla {\mathcal{A}}_n;r)\ge \delta )\le e^{-Mn}.\end{array}$$

(48)

Proof. 

By Markov’s inequality and (46), it follows that for any $\beta >0$,

$$\begin{array}{c}\hfill {\mathit{P}}_g(\omega ({\mathcal{A}}_n;r)\ge \delta )\le {\mathit{E}}_g\left[e^{n\beta \omega ({\mathcal{A}}_n;r)}\right]e^{-n\beta \delta }\le e^{-n\beta (\delta -\omega (H;r\left)\right)}.\end{array}$$

(49)

Since $\omega (H;r)\to 0$ as $r\searrow 0$, there exists an $r_0$ such that for all $r\le r_0$, $(\delta -\omega (H;r\left)\right)>0$. Since $\beta >0$ is arbitrary, the proposition follows by taking $\beta =M{(\delta -\omega (H;r))}^{-1}$, for some $r\le r_0$. The proof of the second inequality is similar, using (47). □

Proof of Theorem 2.

We begin with the proof of the upper bound. Since we assume that the equation ${\dot{\mathcal{A}}}_n\left(\theta \right)=0$ has a unique solution, it follows from the inequality in (12) that for any $\alpha >0$ and $\theta >{\theta }_g$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\ge \theta )\le \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{E}}_g[exp\left(n\alpha {\dot{\mathcal{A}}}_n\left(\theta \right)\right)]={\mathsf{\Lambda }}_{\theta }\left(\alpha \right),\end{array}$$

(50)

where the last equality follows by applying Theorem 1 with $d=1$. Since the inequality holds for every $\alpha >0$,

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\ge \theta )& \le & \underset{\alpha >0}{sup}{\mathsf{\Lambda }}_{\theta }\left(\alpha \right)\le \underset{\alpha \in \mathbb{R}}{sup}{\mathsf{\Lambda }}_{\theta }\left(\alpha \right).\hfill \end{array}$$

(51)

Now, noticing that ${sup}_{\alpha \in \mathbb{R}}{\mathsf{\Lambda }}_{\theta }\left(\alpha \right)=-{inf}_{\alpha \in \mathbb{R}}-{\mathsf{\Lambda }}_{\theta }\left(\alpha \right)=-{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right)$, we then obtain

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\ge \theta )\le -{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right).\end{array}$$

(52)

Similarly, for $\theta <{\theta }_g$, using (13), one can show by an analogous calculation that

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\le \theta )& \le & -{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right).\hfill \end{array}$$

(53)

Now let ${\theta }_1=inf\{\theta >{\theta }_g:\theta \in F\}$ and ${\theta }_2=sup\{\theta <{\theta }_g:\theta \in F\}$. Then

$$\begin{array}{c}\hfill {\mathit{P}}_g({\widehat{\theta }}_n\in F)\le {\mathit{P}}_g({\widehat{\theta }}_n\ge {\theta }_1)+{\mathit{P}}_g({\widehat{\theta }}_n\le {\theta }_2),\end{array}$$

(54)

and so by (52) and (53), it follows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\in F)\le -\underset{\theta \in \{{\theta }_1,{\theta }_2\}}{min}{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right)\le -\underset{\theta \in F}{inf}{\mathsf{\Lambda }}_{\theta }^{*}\left(0\right),\end{array}$$

(55)

where the last step follows since F closed implies $\{{\theta }_1,{\theta }_2\}\subset F$.

Next we turn now to the proof of the lower bound. Let G be an open set, and let $\theta \in G$. Then there exists an $ϵ>0$ (to be chosen) such that $I_{ϵ} := (\theta -ϵ,\theta +ϵ)⊊G$. Note that

$$\begin{array}{ccc}\hfill \{{\widehat{\theta }}_n\in I_{ϵ}\}& =& \{{\dot{\mathcal{A}}}_n\left({\widehat{\theta }}_n\right)=0,{\widehat{\theta }}_n\in I_{ϵ}\}\hfill \\ & \supset & \{{\dot{\mathcal{A}}}_n\left(\theta \right)-{\dot{\mathcal{A}}}_n\left({\widehat{\theta }}_n\right)\ge \delta \}\cup \{{\widehat{\theta }}_n\in I_{ϵ},\underset{{\theta }_1,{\theta }_2\in I_{ϵ}}{sup}|{\dot{\mathcal{A}}}_n\left({\theta }_1\right)-{\dot{\mathcal{A}}}_n\left({\theta }_2\right)|\le \delta \}.\hfill \end{array}$$

Thus,

$$\begin{array}{ccc}\hfill {\mathit{P}}_g({\widehat{\theta }}_n\in I_{ϵ})& \ge & {\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)-{\dot{\mathcal{A}}}_n\left({\widehat{\theta }}_n\right)\ge \delta )-{\mathit{P}}_g({\widehat{\theta }}_n\notin I_{ϵ},\underset{{\theta }_1,{\theta }_2\in I_{ϵ}}{sup}|{\dot{\mathcal{A}}}_n\left({\theta }_1\right)-{\dot{\mathcal{A}}}_n\left({\theta }_2\right|>\delta ))\hfill \\ \hfill & \ge & {\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)-{\dot{\mathcal{A}}}_n\left({\widehat{\theta }}_n\right)\ge \delta )-{\mathit{P}}_g(\underset{{\theta }_1,{\theta }_2\in I_{ϵ}}{sup}|{\dot{\mathcal{A}}}_n\left({\theta }_1\right)-{\dot{\mathcal{A}}}_n\left({\theta }_2\right)|>\delta )\hfill \\ \hfill & =& {\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)\ge \delta )-{\mathit{P}}_g(\underset{{\theta }_1,{\theta }_2\in I_{ϵ}}{sup}|{\dot{\mathcal{A}}}_n\left({\theta }_1\right)-{\dot{\mathcal{A}}}_n\left({\theta }_2\right)|>\delta )\hfill \\ & =& {\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)\ge \delta )-{\mathit{P}}_g(\omega ({\dot{\mathcal{A}}}_n;ϵ)>\delta ).\hfill \end{array}$$

(56)

We now investigate ${\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)\ge \delta )$. Let $Q_n$ denote the distribution of ${\dot{\mathcal{A}}}_n\left(\theta \right)$, and define $Q_{n,\alpha }$ as follows:

$$\begin{array}{c}\hfill Q_{n,\alpha }\left(B\right)=\frac{1}{{\mathsf{\Lambda }}_{n,\theta }\left(\alpha \right)}{\int }_B{e}^{-n\alpha y}d{Q}_n\left(y\right),B\in \mathcal{B}.\end{array}$$

(57)

Let $B=(x-\eta ,x+\eta )$, for some $\eta >0$, where $B\subset (\delta ,\infty )$ and $x\in {\mathfrak{R}}_{\theta }$. Then

$$\begin{array}{c}\hfill Q_n\left(B\right)\ge exp\left\{-n\alpha x-n\eta \left|\alpha \right|+n{\mathsf{\Lambda }}_{n,\theta }\left(\alpha \right)\right\}{Q}_{n,\alpha }\left(B\right).\end{array}$$

(58)

Taking the logarithm, dividing by n, and then taking the limit as $n\to \infty$, we obtain

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{Q}_n\left(B\right)\ge -\alpha x-\eta \left|\alpha \right|-{\mathsf{\Lambda }}_{\theta }\left(\alpha \right)+\underset{n\to \infty }{lim\; inf}\frac{1}{n}log{Q}_{n,\alpha }\left(B\right).\end{array}$$

(59)

Now since $x\in {\mathfrak{R}}_{\theta }$, we can apply Theorem IV.1 of [25] to obtain that the last term on the right-hand side of the previous equation converges to zero. Upon letting $\eta \to 0$, it follows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{Q}_n\left(B\right)\ge -{\mathsf{\Lambda }}_{\theta }^{*}\left(x\right).\end{array}$$

(60)

Since the above inequality holds for all $x\in {\mathfrak{R}}_{\theta }\cap (\delta ,\infty )$, we conclude that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log{\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)\ge \delta )\ge -I_{\delta }\left(\theta \right),\end{array}$$

(61)

where $I_{\delta }\left(\theta \right)={inf}_{x\in {\mathfrak{R}}_{\theta }\cap (\delta ,\infty )}{\mathsf{\Lambda }}_{\theta }^{*}\left(x\right)$.

By Proposition 4, choosing $M>I_{\delta }\left(\theta \right)$, one can find $ϵ>0$ such that

$$\begin{array}{c}\hfill {\mathit{P}}_g(\omega ({\dot{\mathcal{A}}}_n;ϵ)>\delta )\le e^{-Mn}.\end{array}$$

(62)

Since

$$\begin{array}{c}\hfill {\mathit{P}}_g({\widehat{\theta }}_n\in G)\ge {\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)\ge \delta )\left(1-\frac{{\mathit{P}}_g\left(\omega ({\dot{\mathcal{A}}}_n;ϵ)\right)}{{\mathit{P}}_g({\dot{\mathcal{A}}}_n\left(\theta \right)\ge \delta )}\right),\end{array}$$

(63)

by the choice of M, it follows from (61) that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{\mathit{P}}_g({\widehat{\theta }}_n\in G)\ge -I_{\delta }\left(\theta \right).\end{array}$$

(64)

Taking the supremum on left- and right-hand side over all $\delta >0$ yields the required lower bound. □

Turning to the higher dimensional case, we first need the following result, which provides a uniform bound on the Hessian of the objective function ${\mathcal{A}}_n\left(\mathit{\theta }\right)$.

Lemma 2.

Under Hypotheses 1–8, there exists a finite constant $0<C<\infty$ such that with probability one,

$$\begin{array}{c}\hfill \underset{n\ge 1}{sup}\underset{\mathit{\theta }\in \mathsf{\Theta }}{sup}\left|\right|H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right){\left|\right|}_2\le C.\end{array}$$

(65)

Proof. 

This is standard. Specifically, note that the ${(i,j)}^{\mathrm{th}}$ element of the matrix $H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right)$ is given by

$$\begin{array}{c}\hfill h_{n,ij}={\int }_S{\ddot{s}}_{\mathit{\theta }}^{ij}\left(x\right)g_n^{\frac{1}{2}}\left(x\right)dx.\end{array}$$

(66)

Next, writing down the expression for ${\ddot{s}}_{\mathit{\theta }}^{ij}$ in terms of the derivatives of the score function $u_{\mathit{\theta }}$, using the Cauchy–Schwarz inequality along with Hypotheses 3, 4, 6, and 8, and the definition of the matrix norm, the lemma follows. □

In the proof of the lower bound, we will take a somewhat different approach, involving the analysis of k constraints, and our strategy will be to reduce this to a problem involving a single constraint. Specifically, in (67) below, we establish that, instead of studying k constraints on a quantity ${\mathcal{D}}_n$ (which we are about to define), we can cast the problem in terms of a d-dimensional vector ${\mathit{Y}}_n$ (defined in (70) below) belonging to a ball centered at $\mathbf{0}$ and of appropriate radius.

To be more precise, let $G\subset {\mathbb{R}}^d$ be open, and consider the probability that we obtain an estimated value $\mathit{\theta }\in G$. Let $\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_k\}\subset \mathsf{\Theta }-G$, and for any $\delta >0$, set

$$d_n\left(j\right)={\mathcal{A}}_n\left(\mathit{\theta }\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)-\delta ,j=1,\dots ,k$$

and ${\mathcal{D}}_n\left(\mathit{\theta }\right)=(d_n\left(1\right),\cdots d_n\left(k\right))$. If $\mathit{\theta }$ is chosen as the estimate, then we must have ${\mathcal{A}}_n\left(\mathit{\theta }\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)\ge 0$ for all j, so, in particular,

$$\begin{array}{c}\hfill {\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in G)\ge {\mathit{P}}_g({\mathcal{D}}_n\left(\mathit{\theta }\right)\ge \mathbf{0})\end{array}$$

(67)

(by which we mean that $d_n\left(j\right)\ge 0$ for all j in this last probability).

To evaluate the latter probability, observe that by a second-order Taylor expansion,

$$\begin{array}{c}\hfill s_{\mathit{\theta }}\left(x\right)-s_{{\mathit{\theta }}_j}\left(x\right)=\langle \mathit{\theta }-{\mathit{\theta }}_j,\nabla s_{\mathit{\theta }}\left(x\right)\rangle +\frac{1}{2}(\mathit{\theta }-{\mathit{\theta }}_j)\mathcal{H}(x;{\mathit{\theta }}_j^{*}){(\mathit{\theta }-{\mathit{\theta }}_j)}^{\prime }.\end{array}$$

(68)

Using the positive definiteness and uniform boundedness of the matrix ${\int }_{\mathbb{R}}\mathcal{H}(x;\mathit{\theta })p^{\frac{1}{2}}\left(x\right)dx$, by Hypothesis 4, we have that for any unit vector $\mathit{v}\in {\mathbb{R}}^d$,

$$\underset{p\in \mathcal{G}}{sup}\underset{\mathbf{\eta }\in \mathsf{\Theta }}{inf}\left\{\mathit{v}\left({\int }_{\mathbb{R}}\mathcal{H}(x;\mathbf{\eta })p^{\frac{1}{2}}\left(x\right)dx\right){\mathit{v}}^{\prime }\right\}\ge c,$$

where c is a positive constant independent of $\mathit{v}$. Thus, for each j,

$$\begin{array}{c}\hfill \underset{p\in \mathcal{G}}{sup}\underset{\mathbf{\eta }\in \mathsf{\Theta }}{inf}\left\{(\mathit{\theta }-{\mathit{\theta }}_j)\left({\int }_{\mathbb{R}}\mathcal{H}(x;\mathbf{\eta })p^{\frac{1}{2}}\left(x\right)dx\right){(\mathit{\theta }-{\mathit{\theta }}_j)}^{\prime }\right\}\ge c{\parallel \mathit{\theta }-{\mathit{\theta }}_j\parallel }^2.\end{array}$$

(69)

Integrating with respect to $g_n^{\frac{1}{2}}(·)$ and using the definition of ${\mathcal{A}}_n(·)$, we then obtain that

$$\begin{array}{c}\hfill d_n\left(j\right)={\int }_{\mathbb{R}}\left[\langle \mathit{\theta }-{\mathit{\theta }}_j,\nabla s(x,\mathit{\theta })\rangle \right]g_n^{\frac{1}{2}}\left(x\right)dx+\mathcal{R}(\mathit{\theta },{\mathit{\theta }}_j),\end{array}$$

(70)

where

$$\mathcal{R}(\mathit{\theta },{\mathit{\theta }}_j)\ge c{\parallel \mathit{\theta }-{\mathit{\theta }}_j\parallel }^2-\delta .$$

Let ${\mathit{Y}}_n\left(\mathit{\theta }\right)=(Y_{n,1},\dots ,Y_{n,d})$, where for $s(x;\mathit{\theta }):=s_{\mathit{\theta }}\left(x\right)$:

$$\begin{array}{c}\hfill Y_{n,j}={\int }_S\frac{\partial }{\partial {\theta }_j}s(x;\mathit{\theta })g_n^{\frac{1}{2}}\left(x\right)dx,1\le j\le k.\end{array}$$

(71)

(We have suppressed $\mathit{\theta }$ in the notation for $Y_{n,j}$.) Then the inequality $d_n\left(j\right)\ge 0$ corresponds to an event ${\mathcal{E}}_{n,j}$ described by the occurrence of the inequality

$$〈\frac{\mathit{\theta }-{\mathit{\theta }}_j}{\left|\right|\mathit{\theta }-{\mathit{\theta }}_j\left|\right|},{\mathit{Y}}_n〉\ge -c\left|\right|\mathit{\theta }-{\mathit{\theta }}_j\left|\right|+\delta \left(\right||\mathit{\theta }-{\mathit{\theta }}_j{\left|\right|)}^{-1},$$

(72)

where the right-hand side is always negative for small $\delta$ (since $\mathrm{dist}(\mathit{\theta },\mathsf{\Theta }-G)>0$) and behaves like a constant multiple of $\mathrm{dist}(\mathit{\theta },\mathsf{\Theta }-G)$ as this distance tends to infinity. Thus, we can choose a positive constant $a_{\delta }$ such that

$$a_{\delta }\mathrm{dist}(\mathit{\theta },\mathsf{\Theta }-G)\le c\left|\right|\mathit{\theta }-{\mathit{\theta }}_j\left|\right|-\delta \left(\right||\mathit{\theta }-{\mathit{\theta }}_j{\left|\right|)}^{-1},j=1,\dots ,k,$$

and set $c_{\mathit{\theta }}\left(\delta \right):=a_{\delta }\mathrm{dist}(\mathit{\theta },\mathsf{\Theta }-G)$. Finally, let ${\tilde{\mathcal{E}}}_n$ denote the event that

$$\begin{array}{c}\hfill 〈\frac{\mathit{\theta }-{\mathit{\theta }}_j}{\left|\right|\mathit{\theta }-{\mathit{\theta }}_j\left|\right|},{\mathit{Y}}_n〉\ge -c_{\mathit{\theta }}\left(\delta \right).\end{array}$$

(73)

Then for all j, ${\mathcal{E}}_{n,j}\supset {\tilde{\mathcal{E}}}_n$, where we recall that ${\mathcal{E}}_{n,j}$ was defined via (72). Now, since the definition of the event ${\tilde{\mathcal{E}}}_n$ does not depend on any specific vector ${\mathit{\theta }}_j$, one can replace the vector $(\mathit{\theta }-{\mathit{\theta }}_j)\left(\right||\mathit{\theta }-{\mathit{\theta }}_j{\left|\right|)}^{-1}$ by any unit vector $\mathit{v}$ in ${\mathbb{R}}^d$. Hence

$$\begin{array}{c}\hfill {\mathit{P}}_g({\mathcal{D}}_n\ge \mathbf{0})\ge {\mathit{P}}_g(〈\mathit{v},{\mathit{Y}}_n〉\ge -c_{\mathit{\theta }}\left(\delta \right),\mathrm{for}\mathrm{all}\mathrm{unit}\mathrm{vectors}\mathit{v})={\mathit{P}}_g({\mathit{Y}}_n\in \overline{B}(\mathbf{0};c_{\mathit{\theta }}\left(\delta \right))),\end{array}$$

(74)

and we now derive a large deviation lower bound for the probability on the right-hand side.

Proposition 5.

Assume that Hypotheses 1–8 hold, and suppose that G is an open subset of ${\mathbb{R}}^d$. Assume that $b_n\searrow 0$ and $n{b}_n\nearrow \infty$ as $n\to \infty$. Then for any $\mathit{\theta }\in G$ and $r>0$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log{\mathit{P}}_g({\mathit{Y}}_n\in B(\mathbf{0};r))\ge -I_r\left(\mathit{\theta }\right),\end{array}$$

(75)

where $I_r\left(\mathit{\theta }\right)=inf\left\{{\mathsf{\Lambda }}_{\mathit{\theta }}^{*}\left(\mathit{x}\right):\mathit{x}\in {\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};r)\right\}$ and the infimum is taken to be infinity if the set ${\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};r)$ is empty.

Proof. 

We begin by studying the limiting generating function of ${\mathit{Y}}_n$. By Varadhan’s integral lemma, it follows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\mathsf{\Lambda }}_{n,\mathit{\theta }}\left(\mathbf{\alpha }\right) := \underset{n\to \infty }{lim}\frac{1}{n}log{\mathit{E}}_g[exp\left(n\langle \mathbf{\alpha },{\mathit{Y}}_n\rangle \right]={\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right),\end{array}$$

(76)

where

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)=\underset{p\in \mathcal{G}}{sup}\left[{\int }_S\langle \mathbf{\alpha },\nabla s_{\mathit{\theta }}\left(x\right)\rangle p^{\frac{1}{2}}\left(x\right)dx-\mathrm{KL}(p,g)\right].\end{array}$$

(77)

Define the $\mathbf{\alpha }$-shifted distribution by

$$\begin{array}{c}\hfill Q_{n,\mathbf{\alpha }}\left(B\right)=\frac{1}{{\mathsf{\Lambda }}_{n,\mathit{\theta }}\left(\mathbf{\alpha }\right)}{\int }_B{e}^{n\langle \mathbf{\alpha },\mathit{y}\rangle }d{Q}_n\left(\mathit{y}\right),\end{array}$$

(78)

where $Q_n$ denotes the distribution of ${\mathit{Y}}_n$. Note by the convexity of ${\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)$ that it is almost everywhere differentiable. Fix $x\in {\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};r)$ and choose $\mathbf{\alpha }$ such that $\nabla {\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)=\mathit{x}$. Let $\delta >0$ be such that $B(\mathit{x};\delta )⊊B(\mathbf{0};r)$. Then

$$\begin{array}{ccc}\hfill Q_n\left(B(\mathit{x};\delta )\right)& =& exp\left(n{\mathsf{\Lambda }}_{n,\mathit{\theta }}\left(\mathbf{\alpha }\right)\right){\int }_{B(\mathit{x};\delta )}exp(-n\langle \mathbf{\alpha },\mathit{y}\rangle )d{Q}_{n,\mathbf{\alpha }}\left(\mathit{y}\right)\hfill \\ \hfill & \ge & exp\left(n(-\langle \mathbf{\alpha },\mathit{x}\rangle +{\mathsf{\Lambda }}_{n,\mathit{\theta }}\left(\mathbf{\alpha }\right)+|\left|\mathbf{\alpha }\right|\left|\delta \right)\right)Q_{n,\mathbf{\alpha }}\left(B(\mathit{x};\delta )\right),\hfill \end{array}$$

(79)

implying

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{Q}_n\left(B(\mathit{x};\delta )\right)\ge -\langle \mathbf{\alpha },\mathit{x}\rangle +{\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)-\left|\right|\mathbf{\alpha }\left|\right|\delta +\underset{n\to \infty }{lim\; inf}\frac{1}{n}log{Q}_{n,\mathbf{\alpha }}\left(B(\mathit{x};\delta )\right).\end{array}$$

(80)

Now, notice that the limiting cumulant generating function of ${\mathit{Y}}_n$ under the measure $Q_{n,\mathbf{\alpha }}$ is given by

$$\begin{array}{c}\hfill {\tilde{\mathsf{\Lambda }}}_{\mathit{\theta }}\left(\mathit{\beta }\right)={\mathsf{\Lambda }}_{\mathit{\theta }}(\mathbf{\alpha }+\mathit{\beta })-{\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathit{\beta }\right).\end{array}$$

(81)

Since ${\tilde{\mathsf{\Lambda }}}_{\mathit{\theta }}$ is a proper convex function, it is continuous since ${\mathsf{\Lambda }}_{\mathit{\theta }}\left(\mathbf{\alpha }\right)$ is finite in the ${\mathbb{R}}^d$, and moreover, by the choice of $\mathit{x}$, it is differentiable at $\mathbf{0}$. Hence Condition II.1 of [25] is satisfied. Now, using Theorem IV.1 of [25], it follows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{Q}_{n,\mathbf{\alpha }}\left(B(\mathit{x};\delta )\right)=0.\end{array}$$

(82)

Substituting the above into (80), we obtain

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{\mathit{P}}_g({\mathit{Y}}_n\in B(\mathbf{0};r))\ge -{\mathsf{\Lambda }}_{\mathit{\theta }}^{*}\left(\mathit{x}\right).\end{array}$$

(83)

Taking the supremum in $\mathit{x}\in {\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};r)$, the proposition follows. □

Proof of Theorem 3: Upper Bound.

Let F be a closed subset of $\mathsf{\Theta }$. Note $\mathsf{\Theta }$ compact implies that F is compact. Let $\left\{B\right(\mathit{\theta };r):\mathit{\theta }\in \mathsf{\Theta }\}$ denote an open cover of F, and let $\{B({\mathit{\theta }}_1;r),\dots ,B({\mathit{\theta }}_k;r)\}$ denote the finite subcover. Using that $\nabla {\mathcal{A}}_n\left({\widehat{\mathit{\theta }}}_n\right)=\mathbf{0}$, we then obtain that for any $\mathbf{\alpha }\in {\mathbb{R}}^d$,

$$\begin{array}{ccc}\hfill {\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in F)& \le & \sum _{j=1}^k{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in B({\mathit{\theta }}_k;r))\hfill \\ & =& \sum _{j=1}^k{\mathit{E}}_g[exp\left(n\langle \mathbf{\alpha },{\dot{\mathcal{A}}}_n\left({\widehat{\mathit{\theta }}}_n\right)\rangle \right)I_{\left\{{\widehat{\mathit{\theta }}}_n\in B({\mathit{\theta }}_j;r)\right\}}] := \sum _{j=1}^k{T}_n\left(j\right).\hfill \end{array}$$

(84)

Adding and subtracting $\nabla {\mathcal{A}}_n\left({\mathit{\theta }}_j\right)$ to $\nabla {\mathcal{A}}_n\left(\mathit{\theta }\right)$ and then applying Hölder’s inequality yields $T_n\left(j\right)\le T_n(1,j,p)T_n(2,j,q)$, where

$$\begin{array}{ccc}\hfill log{T}_n(1,j,p)& =& \frac{1}{p}log{\mathit{E}}_g[exp(np{\langle \mathbf{\alpha },\nabla \mathcal{A}}_n\left({\mathit{\theta }}_j\right)\rangle )I_{\left\{\mathit{\theta }\in B({\mathit{\theta }}_j;r)\right\}}],\hfill \\ \hfill log{T}_n(2,j,q)& =& \frac{1}{q}log{\mathit{E}}_g[exp(nq\langle \mathbf{\alpha },{\nabla (\mathcal{A}}_n\left({\widehat{\mathit{\theta }}}_n\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_j\right))\rangle )I_{\{{\widehat{\mathit{\theta }}}_n\in B({\mathit{\theta }}_j;r)\}}].\hfill \end{array}$$

First we study $T_n(2,j,q)$. For ${\widehat{\mathit{\theta }}}_n\in B({\mathit{\theta }}_j,r_j)$ and ${\mathit{\theta }}_1,{\mathit{\theta }}_2\in \mathsf{\Theta }$, the Cauchy–Schwarz inequality gives

$$\begin{array}{ccc}\hfill |\langle \mathbf{\alpha },\nabla {\mathcal{A}}_n\left({\widehat{\mathit{\theta }}}_n\right)-\nabla {\mathcal{A}}_n\left({\mathit{\theta }}_j\right))\rangle |& \le & {\left|\right|\mathbf{\alpha }\left|\right|}_2\underset{{\mathit{\theta }}_1,{\mathit{\theta }}_2\in B({\mathit{\theta }}_j,r)}{sup}\left|\right|\nabla {\mathcal{A}}_n\left({\mathit{\theta }}_1\right)-\nabla {\mathcal{A}}_n\left({\mathit{\theta }}_2\right){\left)\right||}_2\hfill \\ & \le & {\left|\right|\mathbf{\alpha }\left|\right|}_2\left|r\right|\underset{\mathit{\theta }\in B({\mathit{\theta }}_j,r)}{sup}\left|\right|H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right){\left|\right|}_2\hfill \\ & \le & {\left|\right|\mathbf{\alpha }\left|\right|}_2\left|r\right|\underset{1\le j\le k}{max}\left[\underset{\mathit{\theta }\in B({\mathit{\theta }}_j,r_j)}{sup}\left|\right|H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right){\left|\right|}_2\right],\hfill \end{array}$$

where $H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right)$ is the Hessian matrix consisting of the second partial derivatives of ${\mathcal{A}}_n\left(\mathit{\theta }\right)$. Hence we obtain for any $1\le j\le k$ that

$$\begin{array}{ccc}\hfill \frac{1}{n}log{T}_n(2,j,q)& \le & r\frac{1}{nq}{\left(nq\right|\left|\mathbf{\alpha }\right||}_2)\underset{1\le j\le k}{max}\left\{\underset{\mathit{\theta }\in B({\mathit{\theta }}_j,r)}{sup}\left|\right|H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right){\left|\right|}_2\right\}\hfill \\ & =& {r\left|\right|\alpha \left|\right|}_2\underset{1\le j\le k}{max}\left\{\underset{\mathit{\theta }\in B({\mathit{\theta }}_j,r)}{sup}\left|\right|H_{{\mathcal{A}}_n}\left(\mathit{\theta }\right){\left|\right|}_2\right\}.\hfill \end{array}$$

(85)

Now by Lemma 2,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{T}_n(2,j,q)\le Cr.\end{array}$$

(86)

Also, for each $1\le j\le k$, Theorem 1 provides that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{T}_n(1,j,p)\le \frac{1}{p}{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(p\mathbf{\alpha }\right).\end{array}$$

(87)

Thus

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in F)& \le & \underset{1\le j\le k}{max}\underset{n\to \infty }{lim\; sup}\frac{1}{n}log{T}_n(1,j,p)+\underset{1\le j\le k}{max}\underset{n\to \infty }{lim\; sup}\frac{1}{n}log{T}_n(2,j,p)\hfill \\ & \le & \underset{1\le j\le k}{max}\frac{1}{p}{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(p\alpha \right)+Cr.\hfill \end{array}$$

(88)

Since the last inequality holds for all $p>1$,

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in F)& \le & \underset{1\le j\le k}{max}\frac{1}{p}{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(p\mathbf{\alpha }\right)+Cr\hfill \\ & \to & \underset{1\le j\le k}{max}{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(\mathbf{\alpha }\right)+Cr\mathrm{as}p\searrow 0.\hfill \end{array}$$

(89)

Moreover, for each j,

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(\mathbf{\alpha }\right)\le \underset{\mathbf{\alpha }\in {\mathbb{R}}^d}{sup}{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(\mathbf{\alpha }\right):=-{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}\left(\mathbf{0}\right).\end{array}$$

Hence

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in F)& \le & \underset{1\le j\le k}{max}-{\mathsf{\Lambda }}_{{\mathit{\theta }}_j}^{*}\left(\mathbf{0}\right)+Cr\hfill \\ & \le & -\underset{\mathit{\theta }\in F}{inf}{\mathsf{\Lambda }}_{\mathit{\theta }}^{*}\left(\mathbf{0}\right)+Cr.\hfill \end{array}$$

(90)

The upper bound follows by letting $r\searrow 0$. □

Proof of Theorem 3: Lower Bound.

Let G be an open subset of $\mathsf{\Theta }$, and let $\mathit{\theta }\in G$. Then $G^c=\mathsf{\Theta }-G$ is compact, and there exists a collection $\mathbb{T}=\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_k\}\subset G^c$ such that $B({\mathit{\theta }}_1;ϵ),\dots ,B({\mathit{\theta }}_k;ϵ)$ forms a finite subcover of $\mathsf{\Theta }-G$, where $ϵ>0.$ Since

$$\begin{array}{ccc}\hfill \left\{{\mathcal{A}}_n\left(\mathit{\theta }\right)\ge \underset{\mathit{t}\in \mathbb{T}}{sup}{\mathcal{A}}_n\left(\mathit{t}\right)\right\}& \supset & \left\{{\mathcal{A}}_n\left(\mathit{\theta }\right)\ge \underset{1\le j\le k}{max}{\mathcal{A}}_n\left({\mathit{\theta }}_j\right) + \underset{1\le j\le k}{max}\underset{\mathit{t}\in B({\mathit{\theta }}_j;ϵ)}{sup}[{\mathcal{A}}_n\left(t\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)]\right\}\hfill \\ & \supset & \left\{{\mathcal{A}}_n\left(\mathit{\theta }\right)\ge \underset{1\le j\le k}{max}{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)+\underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|<ϵ}{sup}[{\mathcal{A}}_n\left({\mathit{\theta }}_1\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_2\right)]\right\}\hfill \end{array}$$

(91)

it follows that

$$\begin{array}{ccc}\hfill {\mathit{P}}_g{\widehat{\mathit{\theta }}}_n\in G& \ge & {\mathit{P}}_g\left({\mathcal{A}}_n\left(\mathit{\theta }\right)>\underset{1\le j\le k}{max}{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)+\delta ,\underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|<ϵ}{sup}[{\mathcal{A}}_n\left({\mathit{\theta }}_1\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_2\right)]\le \delta \right)\hfill \\ & \ge & J_{n,1}-J_{n,2},\hfill \end{array}$$

(92)

where

$$\begin{array}{ccc}\hfill J_{n,1}& := & {\mathit{P}}_g\left({\mathcal{A}}_n\left(\mathit{\theta }\right)>\underset{1\le j\le k}{max}{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)+\delta \right),\hfill \\ \hfill J_{n,2}& := & {\mathit{P}}_g\left(\underset{\left|\right|{\mathit{\theta }}_1-{\mathit{\theta }}_2\left|\right|\le r}{sup}[{\mathcal{A}}_n\left({\mathit{\theta }}_1\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_2\right)]\ge \delta \right):={\mathit{P}}_g\left(\omega ({\mathcal{A}}_n;ϵ)\ge \delta \right).\hfill \end{array}$$

We now investigate the behavior of $J_{n,1}$ and $J_{n,2}$. Starting with $J_{n,1}$, note that

$$\begin{array}{c}\hfill J_{n,1}\ge {\mathit{P}}_g\left(\underset{1\le j\le k}{min}({\mathcal{A}}_n\left(\mathit{\theta }\right)-{\mathcal{A}}_n\left({\mathit{\theta }}_j\right)-\delta )\ge 0\right)={\mathit{P}}_g({\mathcal{D}}_n\ge \mathbf{0}).\end{array}$$

(93)

Now by (74), it follows that

$$\begin{array}{c}\hfill J_{n,1}\ge {\mathit{P}}_g({\mathit{Y}}_n\in B(\mathbf{0};r)),\end{array}$$

(94)

where $Y_n$ is as in (71) and $r=c_{\mathit{\theta }}\left(\delta \right)$. Applying Proposition 3.4, we obtain

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log{J}_{n,1}\ge -I_r\left(\mathit{\theta }\right),\end{array}$$

(95)

where $I_r\left(\mathit{\theta }\right)=inf\left\{{\mathsf{\Lambda }}_{\mathit{\theta }}^{*}\left(\mathit{x}\right):\mathit{x}\in {\mathfrak{R}}_{\mathit{\theta }}\cap B(\mathbf{0};r)\right\}$, and we now observe that r may be chosen to be $c_{\mathit{\theta }}:={lim}_{\delta \downarrow 0}{c}_{\mathit{\theta }}\left(\delta \right)>0$, where $c_{\mathit{\theta }}\left(\delta \right)$ is given as in (73). Hence we may replace $I_r(·)$ with $I(·)$ on the right-hand side of the previous equation. Next, using Proposition 4 yields that

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n}log{\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in G)\ge \underset{n\to \infty }{lim\; inf}\frac{1}{n}log{J}_{n,1}+\underset{n\to \infty }{lim}log\left(1-\frac{J_{n,2}}{J_{n,1}}\right)\ge -I\left(\mathit{\theta }\right).$$

(96)

Finally, the required lower bound is obtained by maximizing the right-hand side over all $\mathit{\theta }\in G$. □

In the proof of the lower bound, it is clear that the choice of $\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_k\}$ plays a central role, and the rate function $I\left(\mathit{\theta }\right)$ will be minimized when k is small. As a simple example, suppose that our goal is to obtain a lower bound for ${\mathit{P}}_g({\widehat{\mathit{\theta }}}_n\in G)$, where

$$G=\left\{({\theta }_1,{\theta }_2):{\theta }_1>a_1\mathrm{or}{\theta }_2>a_2\right\}\subset {\mathbb{R}}^2,{\mathit{\theta }}_g\notin G,$$

which is a union of two halfspaces, This can be expressed as $\mathit{a}+\mathcal{C}$, where $\mathit{a}=(a_1,a_2)$ and $\mathcal{C}=\{({\theta }_1,{\theta }_2):{\theta }_1>0\mathrm{or}{\theta }_2>0\}$, which is an example of a translated cone. Now if $\mathit{\theta }\in G$, then we can find two elements which generate the entire set $\mathsf{\Theta }-G$, in the sense that all other normalized differences lie between these two unit vectors. These two representative points are the unit vectors ${\mathbf{e}}_1=(-1,0)$ and ${\mathbf{e}}_2=(0,-1)$, and all other normalized differences $(\mathit{\theta }-\tilde{\mathit{\theta }}/\parallel \mathit{\theta }-\tilde{\mathit{\theta }}\parallel$ lie between these vectors for all $\tilde{\mathit{\theta }}\in \mathsf{\Theta }-G.$ Now going back to (73), we see that this equation again holds. Furthermore, (74) holds with $B(\mathbf{0};c_{\delta }\left(\mathit{\theta }\right))$ now replaced by an intersection of two halfspaces rather than of all halfspaces, yielding an unbounded region in the definition of $I\left(\mathit{\theta }\right)$. This potentially improves the quality of the lower bound compared with what is presented in the statement of Theorem 3. This idea can be potentially generalized to other sets, such as other unions of halfspaces, and so from a practical perspective, could apply somewhat generally.

4. Concluding Remarks

In this article, we have derived large deviation results for the minimum Hellinger distance estimators of a family of continuous distributions satisfying an equicontinuity condition. These results extend large deviation asymptotics for M-estimators given, e.g., in [6,9]. In contrast to the case for M-estimators, our setting is complicated due to its inherent nonlinearity, leading to complications in the proofs of both the upper and lower bounds, and an unexpected subtlety in the form of the rate function for the lower bound. Our results suggest that one can, under additional hypotheses, establish saddlepoint approximations to the density of MHDE, which would enable one to sharpen inference for small samples.

Similar results are expected to hold for discrete distributions. However, the equicontinuity condition is not required in that case, since ${\ell }_1$, unlike $L_1\left(S\right)$, possesses the Schur property. Hence the LDP in the weak topology of ${\ell }_1$ can be derived (more easily) using a standard Gärtner–Ellis argument, and utilizing this, one can, in principle, repeat all of the arguments above to derive results analogous to Theorems 2 and 3. Large deviations for other divergences under weak family regularity (such as noncompactness of the parameter space $\mathsf{\Theta }$)—and their connections to estimation and test efficiency—are interesting open problems requiring new techniques beyond those described in this article.