On Falsifiable Statistical Hypotheses

Abstract

Popper argued that a statistical falsification required a prior methodological decision to regard sufficiently improbable events as ruled out. That suggestion has generated a number of fruitful approaches, but also a number of apparent paradoxes and ultimately, no clear consensus. It is still commonly claimed that, since random samples are logically consistent with all the statistical hypotheses on the table, falsification simply does not apply in realistic statistical settings. We claim that the situation is considerably improved if we ask a conceptually prior question: when should a statistical hypothesis be regarded as falsifiable. To that end we propose several different notions of statistical falsifiability and prove that, whichever definition we prefer, the same hypotheses turn out to be falsifiable. That shows that statistical falsifiability enjoys a kind of conceptual robustness. These notions of statistical falsifiability are arrived at by proposing statistical analogues to intuitive properties enjoyed by exemplary falsifiable hypotheses familiar from classical philosophy of science. That demonstrates that, to a large extent, this philosophical tradition was on the right conceptual track. Finally, we demonstrate that, under weak assumptions, the statistically falsifiable hypotheses correspond precisely to the closed sets in a standard topology on probability measures. That means that standard techniques from statistics and measure theory can be used to determine exactly which hypotheses are statistically falsifiable. In other words: the proposed notion of statistical falsifiability both answers to our conceptual demands and submits to standard mathematical techniques.

1. Introduction

Probabilistic theories posed a challenge for Popper’s falsificationism. But, on first glance, there are strong analogies between “naive” falsificationism, and standard practice in frequentist hypothesis testing. For example, a standard frequentist test of a sharp null hypothesis rejects upon observing an event that would be highly improbable if the null hypothesis were true. Such a test has a very low chance of erroneously rejecting the null hypothesis. That falls short of, but is closely analogous to, the infallibility of rejecting a universal law upon observing a countervailing instance. Alternatively, a standard frequentist test of a sharp null hypothesis has a high chance of erroneously failing to reject if the null hypothesis is subtly false. That is similar to the fallibility of inferring a universal hypothesis from finitely many instances. Such analogies are natural and sometimes made explicitly by statisticians. Here are Gelman and Shalizi [1]:

Extreme p-Values indicate that the data violate regularities implied by the model, or approach doing so. If these were strict violations of deterministic implications, we could just apply modus tollens …as it is, we nonetheless have evidence and probabilities. Our view of model checking, then, is firmly in the long hypothetico-deductive tradition, running from Popper (1934/1959) back through Bernard (1865/1927) and beyond (Laudan, 1981).

Statistical falsification, Gelman and Shalizi suggest, is all but deductive.1 But how extremal, exactly, does a p-Value have to be for a test to count as a falsification? Popper was loathe to draw the line at any particular value: “It is fairly clear that this ‘practical falsification’ can be obtained only through a methodological decision to regard highly improbable events as ruled out …But with what right can they be so regarded? Where are we to draw the line? Where does this ‘high improbability’ begin?” ([3], p. 182) The problem of when to consider a statistical hypothesis to be falsified has engendered a significant literature [4,5,6] but no universally accepted answer.

A natural idea, due to Fisher [7] and Gillies [4], is to pick some canonical event that is highly improbable if the hypothesis is true. The problem with this idea, pointed out by Neyman [8] and given dramatic expression by Redhead [9], is that there is no unique methodological convention satisfying this property; indeed there may be competing conventions giving conflicting verdicts on every sample.2 Neyman and Pearson [11] attempt to solve this uniqueness problem by requiring the falsifying event to have maximum probability if the hypothesis is false. Falsificationists like Gillies [4] object that this strategy involves restricting the set of alternatives in artifical ways. But even if we grant the basics of the Neyman–Pearson framework, there is typically no way to choose a single falsifying event with “uniformly” maximum probability in all the alternative possibilities in which the hypothesis is false: one must always favor some alternative over others (See Casella and Berger [12], p. 393). In short, the question of when to consider an event falsifying seems to have no answer entirely free of arbitrariness.

Our response to this situation is two-fold. The first response is nicely summarized by Redhead [9]: “Popper demands in science refutability, not refutation”. The question of which hypotheses are refutable is at least as important as the question of when data should be taken to have refuted a hypothesis.3 The existence of a univocal methodological convention is not necessary for refutability; indeed, if there were a univocal answer, the problem would no longer have any conventional or methodological aspect. The results of this work allow us to be fairly ecumenical about which methodology of falsification should be adopted without losing any clarity about which hypotheses are falsifiable: Theorem 2 says that a variety of different methodologies of falsification—ranging from the permissive to the stringent—give rise to exactly the same collection of falsifiable hypotheses. Moreover, it provides a relatively simple mathematical criterion for identifying the falsifiable hypotheses. That means scientific controversies about the testability of hypotheses can be adjudicated without awaiting a precise resolution of the problem of statistical falsification.

Although the existence of a univocal methodological convention is not necessary for refutability, what is important is the existence of a coherent set of methodological decisions exhibiting some desirable properties. Statistical tests are implicit proposals for such a convention, proposing a falsifying event (rejection region) for every sample size. To the extent possible, these tests should exhibit the synchronic virtues sought after by Fisher, Neyman, Pearson and others: at every sample size, if the hypothesis is true, the chance that it is rejected should be small; and if it is false, the chance that it is rejected should be large. But excessive focus on these synchronic virtues tends to obscure the social dimensions of these decisions. Our second response is a plea to consider another axis on which competing methodological proposals can be compared. If it were adopted by researchers conducting independent investigations of the same hypothesis, the method should also exhibit certain diachronic virtues: if the hypothesis is true, the chance that it is falsely rejected should shrink as monotonically as possible as the investigation is replicated at larger sample sizes; and similarly, if the hypothesis is false, the chance that it is correctly rejected should grow as monotonically as possible. In other words: a good methodological convention would, if adopted, support a pattern of successful replication by independent investigators. Crucially, possessing the synchronic virtues at every sample size is not sufficient to ensure the diachronic virtues: care must be taken such that the methodological decisions for different sample sizes cohere in a particular way.4 Finally, falsifiable hypotheses would be those for which a methodological convention exhibiting all of these virtues exists.

We end this introductory section with a few comments on routes not taken in the following. The work of Mayo and Spanos [6] is in its own way a mathematical elaboration of Popperian falsifiability. It is natural to equate falsifiability with Mayo and Spanos’ severe testability. Although severe testability is an important notion of independent interest, we do not adopt this identification—it would be too radical a revision of falsifiability. Roughly, a hypothesis is severely tested by some evidence if, were the hypothesis false, then with high probability evidence less favorable to the hypothesis would have been observed. On that stringent notion, hypotheses like the coin is precisely fair are not severely testable, since whatever sequence of flips has been observed, a sequence equally favorable to the precisely fair hypothesis could have been generated by a subtly biased coin. But we take sharp null hypotheses like the coin is precisely fair as archetypical falsifiable hypotheses. Finally, it might be justly said that any contemporary philosophical discussion of statistics should say something about the Bayesian point of view. We do not attempt to fully satisfy a philosophical Bayesian. But although this work is inspired by a rival philosophical tradition, it should not be understood as incompatible with Bayesianism. On any of a number of reasonable correspondence principles, any hypothesis falsifiable by a Bayesian method will be falsifiable in our sense. Although the question of whether the converse is true is an interesting one, we do not pursue it here.

2. In Search of Statistical Falsifiability

Accordingly, instead of asking ‘when should a statistical hypothesis be regarded as falsified?’, we are guided in the following by the question: ‘when should a statistical hypothesis be regarded falsifiable?’. We suggest that statistical falsifiability will be found by analogy with exemplars from philosophy of science and the theory of computation. We have in mind universal hypotheses like ‘all ravens are black’, or co-semidecidable formal propositions like ‘this program will not halt’. Although there is no a priori bound on the amount of observation, computation, or proof search required, these hypotheses may be falsified by suspending judgement until the hypothesis is decisively refuted by the provision of a non-black raven or a halting event. We wish to call the reader’s attention to several paradigmatic properties of these falsification methods. We do not claim that these properties are typically achievable in empirical inquiry; rather, they should be seen as regulative ideals that falsification methods ought to approximate.

• Error Avoidance: Output conclusions are true.

By suspending judgement until the hypothesis is logically incompatible with the evidence, falsification methods never have to “stick their neck out" by making a conjecture that might be false.5

• Monotonicity: Logically stronger inputs yield logically stronger conclusions.

Exemplary falsification methods never have to retract their previous conclusions; their conjecture at any later time always entails their conjecture at any previous time. In the ornithological context, conjectures made on the basis of more observations always entail conjectures made on the basis of fewer; once a non-black raven has been observed, the hypothesis is decisively falsified. (We appeal here to an idealization that we later discharge in the statistical setting: an exemplary method never mistakes a black raven for a non-black raven). In the computational context, conjectures made on the basis of more computation always entail conjectures made on the basis of less; once the program has entered a halting state, it will never exit again.

• Limiting Convergence: The method converges to ¬H iff H is false.

If all ravens are black, the falsification method may suspend judgement forever; but if there is some non-black raven, diligent observation will turn up a falsifying instance eventually. Similarly, if the program eventually halts, the patient observer will notice.6

We propose that statistical falsifiability will be found if we look for the minimal weakening of these paradigmatic properties that is feasible in statistical contexts. First, some definitions.7 Inference methods output conjectures on the basis of input information. Here “information" is understood broadly. One conception of information, articulated explicitly by Bar-Hillel and Carnap [19] and championed by Floridi [20,21], is true, propositional semantic content, logically entailing certain relevant possibilities, and logically refuting others. This notion is prevalent in epistemic logic and many related formal fields—call it the propositional notion of information. A second conception, ubiquitous in the natural sciences, is random samples, typically independent and identically distributed, and logically consistent with all relevant possibilities, although more probable under some, and less probable under others. Call that the statistical notion of information. Of course, statistical information can be expressed propositionally. The trouble is that—at least so far as the relevant possibilities are concerned—the proposition is always the same, since every proposition about the observed data is logically consistent with every probabilistic proposition.

Statistical methods cannot be expected to be infallible. We liberalize that requirement as follows:

• α-Error Avoidance: For every sample size, the objective chance that the output conclusion is false is not higher than $\alpha$.

The $\alpha$-error avoidance property is closely related to frequentist statistical inference. A confidence interval with coverage probability $1-\alpha$ is straightforwardly $\alpha$-error avoiding: the chance that the interval excludes the true parameter is bounded by $\alpha$. A hypothesis test with significance level $\alpha$ is also $\alpha$-error avoiding, so long as one understands failure to reject the null hypothesis as recommending suspension of judgment, rather than concluding that the null hypothesis is true. The chance of falsely rejecting the null is bounded by $\alpha$, and failing to reject outputs only the trivially true, or tautological, hypothesis.

We first define a weaker notion. A method is a function from information to conjectures. A method falsifies hypothesis H in the limit by converging, on increasing information, to not-H iff H is false. In statistical settings, that means that in all and only the worlds in which H is false, the chance that $L(·)$ outputs not-H converges to 1 as sample size increases.8 A method verifies H in the limit if it falsifies not-H in the limit. A method decides H in the limit if it verifies H and not-H in the limit. Hypothesis H is verifiable, refutable, or decidable in the limit iff there exists a method that verifies, refutes or decides it in the limit.

Falsification in the limit is a relatively undemanding concept of success—it is consistent with any finite number of errors and volte-faces prior to convergence. Falsification is a stronger success notion. Consider the following cycle of definitions.9

F1.

Hypothesis H is falsifiable iff there is a monotonic, error avoiding method M that falsifies H in the limit;

F1.5.

Hypothesis H is falsifiable iff there is a method M that falsifies H in the limit, and for every $\alpha >0$, M is $\alpha$-error avoiding;

F2.

Hypothesis H is falsifiable iff for every $\alpha >0$ there is an $\alpha$-error evoiding method that falsifies H in the limit.

Concept F1 is the familiar one from epistemology, the philosophy of science, and the theory of computation. Our motivating examples are all of this type. These may be falsified by suspending judgement until the relevant hypothesis is logically refuted by information. Although there is no a priori bound on the amount of information (or computation) required, the outputs of a falsifier are guaranteed to be true, without qualification.

Concept F1.5 weakens concept F1 by requiring only that there exist a method that avoids error with probability one. Hypotheses of type F1.5 are less frequently encountered in the wild.10 Concept F1.5 is introduced here to smooth the transition to F2.

F2 weakens F1.5 by requiring only that for every bound on the chance of error, there exists a method that achieves the bound. Hypotheses of this type are ubiquitous in statistical settings. The problem of falsifying that a coin is fair by flipping it indefinitely is an archetypical problem of the third kind. For any $\alpha >0$, there is a consistent hypothesis test with significance level $\alpha$ that falsifies, in the third sense, that the coin is fair. Moreover, it is hard to imagine a more stringent notion of falsification that could actually be implemented in digital circuitry. Electronics operating outside the protective cover of Earth’s atmosphere are often disturbed by space radiation—energetic ions can flip bits or change the state of memory cells and registers [22]. Even routine computations performed in space are subject to non-trivial probabilities of error, although the error rate can be made arbitrarily small by redundant circuitry, error-correcting codes, or simply by repeating the calculation many times and taking the modal result. Electronics operating on Earth are less vulnerable, but are still not immune to these effects.

Since issues of monotonicity are ignored, F2 provides only a partial statistical analogue for F1. A statistical analogue of monotonicity is suggested by considerations of replication. Suppose that researchers write a grant to study whether NewDrug is better at treating migraine than OldDrug. They perform a pre-trial analysis of their methodology and conclude that if NewDrug is indeed better OldDrug, the objective chance that they will reject the null hypothesis of “no improvement” is greater than 50%. The funding agency is satisfied, providing enough funding to perform a pilot study with N = 100. Elated, the researchers perform the study, and correctly reject the null hypothesis. Now suppose that a replication study is proposed at sample size 150, but the chance of rejecting has decreased to 40%. The chance of rejecting correctly, and thereby replicating successfully, has gone down, even though the first study was correct. Nevertheless, investigators propose going to the trouble and expense of collecting a larger sample! Such methods are epistemically defective. They may even be immoral: why expose more patients to potential side effects for no epistemic gain? Accordingly, consider the following statistical norm:

• Monotonicity in chance: If H is false, then the objective chance of rejecting H is strictly increasing with sample size.

Unfortunately, strict monotonicity is often infeasible (see Lemma 1). Nevertheless, it should be a regulative ideal that we strive to approximate. The following principle expresses that aspiration:

• α-Monotonicity in chance: If H is false, then for any sample sizes $n_1<n_2$, the objective chance of rejecting H decreases by no more than $\alpha$.

That property ensures that collecting a larger sample is never a disastrously bad idea. Equipped with a notion of statistical monotonicity, we state the final definition in our cycle:

F3.

Hypothesis H is falsifiable iff for every $\alpha >0$ there is an $\alpha$-error avoiding and $\alpha$-monotonic method that falsifies H in the limit.

F3 seems like a rather modest strengthening of F2. Surprisingly, many standard frequentist methods satisfy F2, but not F3. Chernick and Liu [23] noticed non-monotonic behavior in the power function of standard hypothesis tests of the binomial proportion, and proposed heuristic software solutions. That defect would have precisely the bad consequences that inspired our statistical notion of monotonicity: attempting replication with a larger sample might actually be a bad idea. That issue has been raised in consumer safety regulation, vaccine studies, and agronomy [24,25,26]. But Chernick and Liu [23] have noticed only the tip of the iceberg—similar considerations attend all statistical inference methods. One of the results of this paper (Theorem A1) is that F3 is feasible whenever F2 is. That suggests that stasticial feasibility is a robust notion; many different formulations yield the same collection of falsifiable hypotheses.

It is sometimes more natural to speak not in terms of falsifiability but its dual: verifiability. The falsifiability notions F1, F2, F3 give rise to corresponding notions of verifiability V1, V2, V3 by letting H be verifiable (in the relevant sense) iff its logical complement ¬H is falsifiable (in the relevant sense). Then, such archetypical hypotheses as, ‘not all ravens are black’, or semidecidable formal proposition like ‘this program will halt’ are verifiable in the sense of V1. The statistical hypotheses that typically serve as composite alternatives to sharp null hypotheses turn out to be verifiable in the sense of V2 and V3.

Both verifiability and falsifiability are one-sided notions. They can be symmetrized by defining H to be decidable (in the relevant sense) iff H is both verifiable and falsifiable (in the relevant sense). That gives rise to the three corresponding decidability concepts D1, D2 and D3. Finite-horizon empirical hypotheses such as ‘the first hundred observed ravens will be black’ and formal propositions like ‘this program will halt in under a hundred steps’ are decidable in the sense of D1. Typically, simple vs. simple hypothesis testing problems turn out to be decidable in the sense of D3. However, some more interesting problems also turn out to be statistically decidable (see Genin and Mayo-Wilson [27]).

3. The Topology of Inquiry

A central insight of Abramsky [28], Vickers [29], Kelly [30] is that falsifiable propositions of type F1 enjoy the following properties:

C1.

If $H_1,H_2$ are falsifiable, then so is their disjunction, $H_1\cup H_2$;

C2.

If $\mathcal{H}$ is a (potentially infinite) collection of falsifiable propositions, then their conjunction, $\cap \mathcal{H}$, is also falsifiable.

Together, C1 and C2 say that falsifiable propositions of type F1 are closed under conjunction, and finite disjunction. Why the asymmetry? For the same reason that it is possible to falsify that bread will cease to nourish sometime next week, but not possible to falsify that it will cease to nourish on some day in the future. It is also important to notice what C1 and C2 do not say: if H is falsifiable, its negation may not be. To convince yourself of this it suffices to notice that it is possible to falsify that bread will always nourish, but not that it will cease to nourish on some day in the future.

For their part, verifiable propositions of type V1 satisfy the dual properties:

O1.

If $H_1$ and $H_2$ are verifiable, then so is their conjunction, $H_1\cap H_2$;

O2.

If $\mathcal{H}$ is a (potentially infinite) collection of verifiable propositions, then their disjunction, $\cup \mathcal{H}$, is also verifiable.

O1 and O2 express the characteristic asymmetries of verifiability. Although it is possible to verify that bread nourishes for any finite number of days, it is not possible to verify that bread will continue to nourish into the indefinite future. Moreover, if H is verifiable, its negation may not be: if bread will cease to nourish, we will read about it in the newspaper; but nothing we can observe about bread will ever rule out the possibility of a future dereliction of duty.

Jointly, C1 and C2 ensure that the collection of all propositions of type F1 constitute the closed sets of a topological space.11 The collection of all propositions of type V1 constitute the open sets of the topology; and the propositions of type D1 constitute the clopen sets of the topology. Sets of greater topological complexity are formed by set-theoretic operations on open and closed sets. The central point of Kelly [30] is that degrees of methodological accessibility correspond exactly to increasingly ramified levels of topological complexity, corresponding to elements of the Borel hierarchy. Roughly speaking, the Borel complexity of a hypothesis measures how complex it is to construct the hypothesis out of logical combinations of verifiable and falsifiable propositions. Higher levels of Borel complexity correspond to inductive notions of methodological success, where by inductive we mean any success notion where the chance of error is unbounded in the short run (see Figure 1).

Taking falsifiability of type F1 as the fundamental notion, the view sketched above was worked out in its essentials by Kelly [30] and further generalized by de Brecht and Yamamoto [31], Genin and Kelly [32] and Baltag et al. [33]. Genin and Kelly [34] exhibit a topology on probability measures in which the closed sets are exactly the propositions falsifiable in the sense of F2. That shows that the structure of statistical verifiability is also topological, at least when issues of monotonicity are ignored. The characteristic asymmetries are all present: while it is possible to falsify that the coin is fair, it is not possible to falsify that it is biased.

In this work, we show that hypotheses of type F3 also enjoy a topological structure; in fact, under a weak assumption, every hypothesis falsifiable in the sense of F2 is also falsifiable in the sense of F3 (Theorem A1). Imposing the demands of monotonicity does not make any fewer hypotheses falsifiable. That result provides a kind of cross-check on our notion of statistical falsifiability: it enjoys the same algebraic closure properties as the traditional notion F1, familiar from classical philosophy of science.

4. The Statistical Setting

4.1. Models, Measures and Samples

A model characterizes the contextually relevant features of a data-generating process. Models may be quite simple, as when they specify the bias of a coin. Models may also be elaborate structural hypotheses, as when they specify a set of structural equations expressing the causal processes by which data are generated. Inquiry typically begins with the specification of a set of possible models $\mathcal{M}$, any one of which, for all the inquirers know, may characterize the true data-generating process. Each model $\theta \in \mathcal{M}$ determines a probability measure,${\mu }_{\theta }$ over a common sample space $\mathfrak{S}=(\mathsf{\Omega },\mathcal{F})$. A sample space is a set of possible random samples $\mathsf{\Omega }$, together with a $\sigma$-algebra of events $\mathcal{F}\subseteq \mathcal{P}\left(\mathsf{\Omega }\right)$ over the set of samples. Each probability measure ${\mu }_{\theta }$ is a possible assignments of probabilities to events in $\mathcal{F}$ that is consistent with the axioms of probability and with the constraints specified by the model $\theta$. We let W be the set $\{{\mu }_{\theta }:\theta \in \mathcal{M}\},$ i.e., the set of all probability measures on the sample space $\mathfrak{S}$ generated by possibilities in $\mathcal{M}$.

A set of models $\mathcal{M}$ is said to be identified if the map $\theta \mapsto {\mu }_{\theta }$ is one-to-one, i.e., if no two models generate the same probability measure over the space of observable outcomes $\mathfrak{S}.$ If two models $\theta ,{\theta }^{\prime }$ generate the same probability measure over the observables, then no amount of random sampling can possibly distinguish them and the question of whether $\theta$ or ${\theta }^{\prime }$ is the generating mechanism is, in a sense, hopeless from a statistical perspective.12 In the following, we will assume that the set of models is identified. Since each probability measure uniquely determines a model, we can identify the model with its probabilistic consequences and drop the subscript $\theta$ from our notation.

Call the triple $(W,\mathsf{\Omega },\mathcal{F}),$ consisting of a set of probability measures W on a sample space $(\mathsf{\Omega },\mathcal{F})$, a statistical setup. The inquirer observes events in $\mathcal{F}$, but the hypotheses she is interested in are typically propositions over W. In paradigmatic cases, an event A in $\mathcal{F}$ is logically consistent with all probability measures in W.

Although measures in W assign probabilities to all events in $\mathcal{F},$ not all these events are the kinds of events than an agent can observe. No one can tell whether a real-valued sample point is rational or irrational, or if it is exactly $\pi .$ For another example, suppose that region A is the closed interval $[1/2,\infty ]$, and that the sample $\omega$ happens to land right on the boundry-point $1/2$ of A. Suppose, furthermore, that given enough time and computational power, the sample $\omega$ can be specified to arbitrary, finite precision. No finite degree of precision: $\omega \approx 0.50$; $\omega \approx 0.500$; $\omega \approx 0.5000$; … suffices to determine that $\omega$ is truly in A. But the mere possibility of a sample hitting the boundary of A does not matter statistically, if the chance of obtaining such a sample is zero, as it typically is, unless there is discrete probability mass on the point $\frac{1}{2}$. Both these examples hinge on an implicit underlying topology $\mathcal{T}$ on the sample space $\mathsf{\Omega }$.

The topology $\mathcal{T}$ the sample space reflects what is formally verifiable about the sample itself. For example, if $\mathsf{\Omega }$ is the real line, then it is typically verifiable whether a sample point $\omega$ is greater or less than $r\in \mathbb{R}$; and whether it lies in the interval $(r,s).$ It is typically not verifiable whether the sample $\omega$ is rational or irrational, or whether it is exactly $\pi .$ We assume that these formally verifiable propositions over $\mathsf{\Omega }$ are closed under finite conjunctions and arbitrary disjunctions and therefore, form the open sets of a topological space. Then, the formally decidable proposition in $\mathcal{F}$ form the clopen sets of the same topological space. Furthermore, we assume that the $\sigma$-algebra $\mathcal{F}$ contains all the verifiable proposition $\mathcal{T},$ and in fact that it is the least such $\sigma$-algebra, i.e., that it is generated by closing the verifiable propositions under countable unions and negations. A $\sigma$-algebra that arises from a topology in this way is called a Borel algebra and its members are called Borel sets.

A Borel set A for which $\mu \left(\mathsf{bdry}A\right)=0$ is said to be almost surely clopen (decidable) in μ.13 Say that a collection of Borel sets $\mathcal{S}$ is almost surely clopen in $\mu$ iff every element of $\mathcal{S}$ is almost surely clopen in $\mu$. We say that a Borel set A is almost surely decidable iff it is almost surely decidable in every $\mu$ in W. Similarly, we say that a collection of Borel sets $\mathcal{S}$ is almost surely clopen iff every element of $\mathcal{S}$ is almost surely clopen.

A topological basis $\mathcal{I}$ on $\mathsf{\Omega }$ is a collection of subsets of $\mathsf{\Omega }$ such that (1) the elements of $\mathcal{I}$ cover $\mathsf{\Omega }$; and (2) if $E,F$ are elements of $\mathcal{I},$ then for each $\omega \in E\cap F,$ there is $G\in \mathcal{I}$ containing $\omega$ and contained in $E\cap F.$ Closing a topological basis under unions generates a topology, and every topology is generated by some basis. In the following, we will assume that the topology $\mathcal{T}$ on $\mathsf{\Omega }$ is generated by a basis $\mathcal{I}$ that is at most countably infinite and almost surely clopen. Say that a statistical setup $(W,\mathsf{\Omega },\mathcal{F})$ is feasibly based if $\mathcal{F}$ is a Borel $\sigma$-algebra arising from a countable, almost surely clopen basis. That assumption is satisfied, for example, in the standard case in which the worlds in W are Borel measures on ${\mathbb{R}}^n$, and all measures are absolutely continuous with respect to Lebesgue measure, i.e., when all measures have probability density functions, which includes normal, chi-square, exponential, Poisson, and beta distributions. It is also satisfied for discrete distributions like the binomial, for which the topology on the sample space is the discrete (power set) topology, so every region in the sample space is clopen. It is satisfied in the particular cases of Examples 1 and 2.

Example 1.

Consider the outcome of a single coin flip. The set Ω of possible outcomes is $\{H,T\}$. Since every outcome is decidable, the appropriate topology on the sample space is $\mathcal{T}=\{⌀,\{H\},\{T\},\{H,T\left\}\right\}$, the discrete topology on Ω. Let W be the set of all probability measures assigning a bias to the coin. Since every element of $\mathcal{T}$ is clopen, every element is also almost surely clopen.

Example 2.

Consider the outcome of a continuous measurement. Then, the sample space Ω is the set of real numbers. Let the basis $\mathcal{I}$ of the sample space topology be the usual interval basis on the real numbers. That captures the intuition that it is verifiable that the sample landed in some open interval, but it is not verifiable that it landed exactly on the boundary of an open interval. There are no non-trivial decidable (clopen) propositions in that topology. However, in typical statistical applications, W contains only probability measures μ that assign zero probability to the boundary of an arbitrary open interval. Therefore, every open interval E is almost surely decidable, i.e., $\mu \left(\mathsf{bdry}\right(E\left)\right)=0$.

Product spaces represent the outcomes of repeated sampling. Let I be an index set, possibly infinite. Let ${({\mathsf{\Omega }}_i,{\mathcal{T}}_i)}_{i\in I}$ be sample spaces, each with basis ${\mathcal{I}}_i$. Define the product$(\mathsf{\Omega },\mathcal{T})$ of the $({\mathsf{\Omega }}_i,{\mathcal{T}}_i)$ as follows: let $\mathsf{\Omega }$ be the Cartesian product of the ${\mathsf{\Omega }}_i$; let $\mathcal{T}$ be the product topology, i.e., the topology in which the open sets are unions of Cartesian products ${\times }_{i\in I}{O}_i$, where each $O_i$ is an element of ${\mathcal{T}}_i$, and all but finitely many $O_i$ are equal to ${\mathsf{\Omega }}_i$. When I is finite, the Cartesian products of basis elements in ${\mathcal{I}}_i$ are the intended basis for $\mathcal{T}$. Let $\mathcal{B}$ be the $\sigma$-algebra generated by $\mathcal{T}$. Let ${\mu }_i$ be a probability measure on ${\mathcal{B}}_i$, the Borel $\sigma$-algebra generated by the ${\mathcal{T}}_i$. The product measure$\mu ={\times }_{i\in I}{\mu }_i$ is the unique measure on $\mathcal{B}$ such that, for each $B\in \mathcal{B}$ expressible as a Cartesian product of $B_i\in {\mathcal{B}}_i$, where all but finitely many of the $B_i$ are equal to ${\mathsf{\Omega }}_i$, $\mu \left(B\right)=\prod {\mu }_i\left(B_i\right)$. (For a simple proof of the existence of the infinite product measure, see Saeki [37].) Let ${\mu }^{\left|I\right|}$ denote the $\left|I\right|$-fold product of $\mu$ with itself. If W is a set of measures, let $W^{\left|I\right|}$ denote the set $\{{\mu }^{\left|I\right|}:\mu \in W\}.$ If $\mathcal{B}$ is a Borel $\sigma$-algebra generated by $\mathcal{T}$, let ${\mathcal{B}}^{\otimes n}$ be the Borel $\sigma$-algebra generated by n-fold product of $\mathcal{T}$ with itself.

4.2. Statistical Tests

A statistical method is a measurable function $\varphi$ from random samples to propositions over W, i.e., for every A in the range of $\varphi ,$ its preimage ${\varphi }^{-1}\left(A\right)$ is an element of $\mathcal{F}$. A test of a statistical hypothesis $H\subseteq W$ is a statistical method $\psi :\mathsf{\Omega }\to \{W,H^{\mathsf{c}}\}$. Call ${\psi }^{-1}\left(W\right)$ the acceptance region, and ${\psi }^{-1}\left(H^{\mathsf{c}}\right)$ the rejection region of the test.14 The power of test $\psi (·)$ is the worst-case probability that it rejects truly, i.e., ${inf}_{\mu \in H^{\mathsf{c}}}\mu \left[{\psi }^{-1}\left(H^{\mathsf{c}}\right)\right]$. The significance level of a test is the worst-case probability that it rejects falsely, i.e., ${sup}_{\mu \in H}\mu \left[{\psi }^{-1}\left(H^{\mathsf{c}}\right)\right]$.

A test is feasible in μ iff its acceptance region is almost surely decidable in $\mu$. Say that a test is feasible iff it is feasible in every world in W. More generally, say that a method is feasible iff the preimage of every element of its range is almost surely decidable in every world in W. Tests that are not feasible in $\mu$ are impossible to implement—as described above, if the acceptance region is not almost surely clopen in $\mu$, then with non-zero probability, the sample lands on the boundary of the acceptance region, where one cannot decide whether to accept or reject. If one were to draw a conclusion at some finite stage, that conclusion might be reversed in light of further computation. Tests are supposed to solve inductive problems, not to generate new ones.

Considerations of feasibility provide a new perspective on the assumption that appears throughout this work: that the basis $\mathcal{I}$ is almost surely clopen. If that assumption fails, then it is not an a priori matter whether geometrically simple zones are suitable acceptance zones for statistical methods. But if that is not determined a priori, then presumably it must be investigated by statistical means. That suggests a methodological regress in which we must use statistical methods to decide which statistical methods are feasible to use. Therefore, we consider only feasible methods in the following development.

4.3. The Weak Topology

A sequence of measures $\left({\mu }_n\right)$converges weakly to $\mu$, written ${\mu }_n\Rightarrow \mu$, iff ${\mu }_n\left(A\right)\to \mu \left(A\right),$ for every A almost surely clopen in $\mu$. It is immediate that ${\mu }_n\Rightarrow \mu$ iff for every $\mu$-feasible test $\psi (·)$, ${\mu }_n\left(\psi \mathrm{rejects}\right)\to \mu \left(\psi \mathrm{rejects}\right)$. It follows that no feasible test of $H=\left\{\mu \right\}$ achieves power strictly greater than its significance level. Furthermore, every feasible method that correctly infers H with high chance in $\mu$, exposes itself to a high chance of error in “nearby” ${\mu }_n$.

It is a standard fact that one can topologize W in such a way that weak convergence is exactly convergence in the topology.15 That topology is called the weak topology, n.b.: the weak topology is a topology on probability measures, whereas all previously mentioned topologies were topologies on random samples. In other words, the open sets of the weak topology are propositions over W, not over $\mathsf{\Omega }.$ If we interpret the closure operator in terms of the weak topology, $\mu \in \mathsf{cl}A$ iff there is a sequence $\left({\mu }_n\right)$ lying in A such that ${\mu }_n\Rightarrow \mu$.

If A is an almost surely decidable event in the appropriate $\sigma$-algebra then it is immediate from the definition of weak convergence that $\{{\mu }^n:{\mu }^n\left(A\right)>r\}$ and $\{{\mu }^n:{\mu }^n\left(A\right)<r\}$ are open in the weak topology over $W^n$. In this case, it is also true that $\{\mu :{\mu }^n\left(A\right)>r\}$ and $\{\mu :{\mu }^n\left(A\right)<r\}$ are open in the weak topology over W. That observation will often be appealed to in the following. For a proof, see Theorem 2.8 in Billingsley [38].

5. Statistical Falsifiability

Hypothesis H was said to be falsifiable (in the sense of $\mathsf{F}\mathsf{1})$ if there is an error avoiding method that converges on increasing information to not-H iff H is false. That condition implies that there is a method that achieves every bound on chance of error, and converges to not-H iff H is false. In statistical settings, one cannot insist on such a high standard of infallibility. Instead, we say that H is falsifiable in chance iff for every bound on error, there is a method that achieves it, and that converges in probability to not-H iff H is false. The reversal of quantifiers expresses the fundamental difference between statistical and propositional falsifiability. The central point in this section is encapsulated in Theorem 1: for feasibly based statistical setups, the statistically falsifiable propositions (in the sense of F2) are exactly the closed sets in the weak topology.

Say that a family ${\left({\lambda }_n\right)}_{n\in \mathbb{N}}$ of feasible tests of H is an α-falsifier in chance of $H\subseteq W$ iff for all $n\in \mathbb{N}$:

• BndErr. ${\mu }^n\left[{\lambda }_n^{-1}\left(W\right)\right]\ge 1-\alpha$, if $\mu \in H$;
• LimCon. ${\mu }^n\left[{\lambda }_n^{-1}\left(H^{\mathsf{c}}\right)\right]\stackrel{n}{⟶}1$, if $\mu \in H^{\mathsf{c}}$.

Say that H is α-falsifiable in chance iff there is an $\alpha$-falsifier in chance of H. Say that H is falsifiable in chance iff H is $\alpha$-falsifiable in chance for every $\alpha >0$.

Several strengthenings of falsifiability in chance immediately suggest themselves. One could demand that, in addition to BndErr, the chance of error vanishes to zero:

• VanErr. ${\mu }^n\left[{\lambda }_n^{-1}\left(W\right)\right]\stackrel{n}{⟶}1$, if $\mu \in H$.

A strengthening of this requirement will be taken up in the following section.

Defining statistical verifiability requires no new ideas. Say that H is α-verifiable in chance iff there is an $\alpha$-falsifier in chance of $H^{\mathsf{c}}$. Say that H is verifiable in chance iff $H^{\mathsf{c}}$ is $\alpha$-falsifiable in chance for every $\alpha >0$.

The central theorem of Genin and Kelly [34] states that, for feasibly based statistical setups, falsifiability in chance is equivalent to being closed in the weak topology.

Theorem 1.

(Fundamental Characterization Theorem). Suppose that the statistical setup $(W,\mathsf{\Omega },\mathcal{F})$ is feasibly based. Then, for $H\subseteq W,$ the following are equivalent:

• H is α-falsifiable in chance for some $\alpha >0;$
• H is falsifiable in chance;
• H is closed in the weak topology on W.

Since a hypothesis is verifiable iff its complement is refutable, the characterization of statistical verifiability follows immediately.

Since a topological space is determined uniquely by its closed sets, Theorem 1 implies that the weak topology is the unique topology that characterizes statistical falsifiability (in the sense of $\mathsf{F}\mathsf{2}$), at least under the weak conditions stated in the antecedent of the theorem. Thus, under those conditions, the weak topology is not merely a convenient formal tool; it is the fundamental topology for statistical inquiry.

6. Monotonic Falsifiability

In Section 5, we said that $\left({\lambda }_n\right)$ is a statistical falsifier of H if it converges to not-H if H is false, and otherwise has a small chance of drawing an erroneous conclusion. But that standard is consistent with a wild see-sawing in the chance of producing the informative conclusion not-H as sample sizes increase, even if H is false. Of course, it is desirable that the chance of correctly rejecting H increases with the sample size, i.e., that for all $\mu \in H^{\mathsf{c}}$ and $n_1<n_2$,

• Mon. ${\mu }^{n_2}\left[{\lambda }_{n_2}^{-1}\left(H^{\mathsf{c}}\right)\right]>{\mu }^{n_1}\left[{\lambda }_{n_1}^{-1}\left(H^{\mathsf{c}}\right)\right].$

Failing to satisfy Mon has the perverse consequence that collecting a larger sample might be a bad idea. Researchers would have to worry whether a failure of replication was due merely to a clumsily designed statistical method that converges to the truth along a needlessly circuitous route. Unfortunately, Mon is infeasible in typical cases, so long as we demand that verifiers satisfy VanErr. Lemma 1 expresses that misfortune.

Say that a statistical setup $(W,\mathsf{\Omega },\mathcal{I})$ is purely statistical iff for all $\mu ,\nu \in W$ and all events $B\in {\mathcal{B}}^{\otimes n}$ such that ${\mu }^n\left(\mathsf{bdry}B\right)=0,$${\mu }^n\left(B\right)>0\mathrm{iff}{\nu }^n\left(B\right)>0.$ This is the formal expression of the idea that almost surely clopen sample events have no logical bearing on statistical hypotheses and is easily seen to be a weakening of mutual absolute continuity.

Lemma 1.

Suppose that the statistical setup $(W,\mathsf{\Omega },\mathcal{F})$ is feasibly based and purely statistical. Let H be closed, but not open in the weak topology. If $\left({\lambda }_n\right)$ is an α-falsifier in chance of H, then, $\left({\lambda }_n\right)$ satisfiesVanErronly if it does not satisfy Mon.

Proof of Lemma 1.

Suppose for a contradiction that $\left({\lambda }_n\right)$ is an $\alpha$-falsifier in chance of H satisfying VanErr. Since H is not open, there is $\nu \in H\cap \mathsf{cl}\left(H^{\mathsf{c}}\right).$ Since the statistical setup is purely statistical and $\left({\lambda }_n\right)$ satisfies LimCon, there must be a sample size $n_1$ such that $0<{\alpha }_{n_1}:={\nu }^{n_1}\left[{\lambda }_{n_1}^{-1}\left(H^{\mathsf{c}}\right)\right].$ Let $O^{\prime }=\{\mu :{\alpha }_{n_1}-ϵ<{\mu }^{n_1}\left[{\lambda }_{n_1}^{-1}\left(H^{\mathsf{c}}\right)\right]\}.$ By construction, $\nu \in O^{\prime }.$ Since $\left({\lambda }_n\right)$ satisfies VanErr, there is $n_2>n_1$ such that ${\nu }^{n_2}\left[{\lambda }_{n_2}^{-1}\left(H^{\mathsf{c}}\right)\right]<{\alpha }_{n_1}-ϵ.$ Let $O^{″}=\{\mu :{\alpha }_{n_1}-ϵ>{\mu }^{n_2}\left[{\lambda }_{n_2}^{-1}\left(H^{\mathsf{c}}\right)\right]\}.$ By construction, $\nu \in O:=O^{\prime }\cap O^{″}.$ Since $\left({\lambda }_n\right)$ is feasible, both $O^{\prime },O^{″}$ are open in the weak topology (see the observation at the end of Section 4.3). Since open sets are closed under finite conjunction, O is open in the weak topology. But since $\nu \in \mathsf{cl}\left(H^{\mathsf{c}}\right),$ there is $\mu \in H^{\mathsf{c}}\cap O.$ But then, ${\mu }^{n_1}\left[{\lambda }_{n_1}^{-1}\left(H^{\mathsf{c}}\right)\right]>{\alpha }_{n_1}-ϵ,$ whereas ${\mu }^{n_2}\left[{\lambda }_{n_2}^{-1}\left(H^{\mathsf{c}}\right)\right]<{\alpha }_{n_1}-ϵ.$ Therefore, $\left({\lambda }_n\right)$ does not satisfy Mon. □

But even if strict monotonicity of power is infeasible, it ought to be our regulative ideal. Say that an $\alpha$-falsifier ${\left({\lambda }_n\right)}_{n\in \mathbb{N}}$ of H, whether in chance, or almost sure, is α-monotonic iff for all $\mu \in H^{\mathsf{c}}$ and $n_1<n_2$:

• $\alpha$-Mon${\mu }^{n_2}\left[{\lambda }_{n_2}^{-1}\left(H^{\mathsf{c}}\right)\right]+\alpha >{\mu }^{n_1}\left[{\lambda }_{n_1}^{-1}\left(H^{\mathsf{c}}\right)\right]$.

Satisfying $\alpha$-Mon ensures that collecting a larger sample is not a disastrously bad idea. Surprisingly, some standard hypothesis tests fail to satisfy even this weak requirement. Chernick and Liu [23] noticed non-monotonic behavior in the power function of textbook tests of the binomial proportion, and proposed heuristic software solutions. The test exhibited in the Genin and Kelly’s proof of Theorem 1 also displays dramatic non-monotonicity (Figure 2). Others have raised worries of non-monotonicity in consumer safety regulation, vaccine studies, and agronomy [24,25,26].

We now articulate a notion of statistical falsifiability that requires $\alpha$-monotonicity. Write $a_n\downarrow 0$ if the sequence $\left(a_n\right)$ converges monotonically to zero. Say that a family ${\left({\lambda }_n\right)}_{n\in \mathbb{N}}$ of feasible tests of $H\subseteq W$ is an α-monotonic falsifier of H iff

• MVanEee. For all $\mu \in H$, there exists a sequence $\left({\alpha }_n\right)$ such that each ${\alpha }_n\le \alpha$, ${\alpha }_n\downarrow 0,$ and ${\mu }^n\left[{\lambda }_n^{-1}\left(H^{\mathsf{c}}\right)\right]\le {\alpha }_n$;
• LimCon.$\mu \in H^{\mathsf{c}}$, ${\mu }^n\left[{\lambda }_n^{-1}\left(H^{\mathsf{c}}\right)\right]\stackrel{n}{⟶}1$;
• α-Mon.$\mu \in W$, ${\mu }^{n_2}\left[{\lambda }_{n_2}^{-1}\left(H^{\mathsf{c}}\right)\right]+\alpha >{\mu }^{n_1}\left[{\lambda }_{n_1}^{-1}\left(H^{\mathsf{c}}\right)\right]$, if $n_1<n_2$.

Say that H is α-monotonically falsifiable iff there is an $\alpha$-monotonic falsifier of H. Say that H is monotonically falsifiable iff H is $\alpha$-monotonically falsifiable, for every $\alpha >0$. It is clear that every $\alpha$-monotonic falsifier of H is also an $\alpha$-falsifier in chance. However, not every $\alpha$-monotonic falsifier of H is an almost sure $\alpha$-falsifier. The converse also does not hold.

Defining monotonic verifiability requires no new ideas. Say that H is α-monotonically verifiable in chance iff there is an $\alpha$-monotonic falsifier of $H^{\mathsf{c}}$. Say that H is montonically verifiable iff $H^{\mathsf{c}}$ is $\alpha$-monotonically refutable for every $\alpha >0$.

The central theorem of this section states that every statistically falsifiable hypothesis is also monotonically falsifiable. The proof is provided in the Appendix A.

Theorem 2.

Suppose that the statistical setup $(W,\mathsf{\Omega },\mathcal{F})$ is feasibly based. Then, for $H\subseteq W,$ the following are equivalent:

• H is α-falsifiable in chance for some $\alpha >0$;
• H is statistically falsifiable;
• H is monotonically falsifiable;
• H is closed in the weak topology on W.

The characterization of monotonic verifiability follows immediately.

7. Conclusions: Falsifiability and Induction

This work is inspired by Popper’s falsificationism and its difficulties with statistical hypotheses. We have proposed several different notions of statistical falsifiability and proven that, whichever definition we prefer, the same hypotheses turn out to be falsifiable. That shows that the notion enjoys a kind of conceptual robustness. Finally, we have demonstrated that, under weak assumptions, the statistically falsifiable hypotheses correspond precisely to the closed sets in a standard topology on probability measures. That means that standard techniques from statistics and measure theory can be used to determine exactly which hypotheses are statistically falsifiable. Hopefully, this result will be a boon to statistical practice by providing a simple diagnostic for statistical falsifiability—controversies about testability may be resolved while maintaining an ecumenical attitude about what, precisely, statistical falsification consists in.

However, this work should not be taken as a wholesale endorsement of Popperian falsificationism. It is easy to generate respectable statistical hypotheses that are not falsifiable or, for that matter, verifiable. Indeed, many pressing scientific hypotheses are of this nature, especially when researchers are interested in answering causal questions (see [39]). These should by no means be regarded as unscientific. The hallmark of these kinds of hypotheses is that, although it is possible to converge to the truth as sample sizes increase, it is not possible to do so with finite sample bounds on the probability of error. This fact should be faced squarely—and not necessarily by searching for stronger assumptions that make it possible to provide finite sample guarantees.

A large class of scientific problems can be reconstructed in the following way: we enumerate a collection of disjoint hypothesis $H_1,H_2,\dots$ in such a way that that ${\cup }_{i=1}^n{H}_i$ is falsifiable no matter how large n is.16 Then, we test longer and longer initial segments, conjecturing the first hypothesis that we fail to reject. Since the procedure involves nesting tests, we cannot expect finite sample bounds on the probability of conjecturing a false hypothesis. Nevertheless, this procedure answers pretty closely to a Popperian methodology of conjectures and refutations. Unlike Popper, we have no problem calling the outcome of such a procedure—belief in, or acceptance of, the first unrejected hypothesis in the enumeration—an induction. But is there anything to be said in favor of this natural and commonplace scientific procedure? Popper hoped that it would produce theories of greater and greater truthlikeness or verisimilitude. That notion has had a troubled and fascinating career, which we do not review here.17 There is, so far as we know, no demonstration that this procedure must produce theories of increasing truthlikeness.18

Let us briefly consider a different idea. Call such a procedure progressive if (1) it converges in chance to the true $H_i$ (if any is true) and (2) if $H_i$ is true, then the objective chance of conjecturing $H_i$ increases monotonically with sample size. It should come as no surprise that this kind of strict monotonicity is not usually feasible. However, it should be a regulative ideal: say that such a procedure is $\alpha$-progressive if (1’) it converges in chance to the true $H_i$ (if any is true) and (2’) if $H_i$ is true, then the objective chance of conjecturing $H_i$ never decreases by more than $\alpha$ as sample sizes increase. The latter is a natural generalization of $\alpha$-Mon to problems of theory choice and precludes egregious backsliding. It should be intuitive that such a procedure can only be $\alpha$-progressive if the constituent test are themselves $\alpha$-monotonic in chance. Genin [45] Theorem 3.6.3, shows that, for all $\alpha >0$ it is possible to produce an $\alpha$-progressive solution to this problem, so long as the consitutent tests are chosen to be sufficiently monotonic in chance. The existence of such tests is guaranteed by Theorem 2. That means that a carefully calibrated methodology of conjectures and refutations is conducive to progress—it converges to the right answer (if any candidate is right) with an arbitrarily low degree of backsliding. We hope that these remarks suggest to some degree the relevance of this work to a positive theory of induction.