A Note on Blind Techniques for New Physics Searches in Particle Physics

Abstract

This paper attempts to classify various blinding strategies used in particle physics. It argues that the blinding technique is not used consistently throughout searches for new physics. More importantly, the blinding technique, in its traditional sense, cannot be applicable for many current and future searches when the statistical precision of data significantly exceeds the current level of our understanding of Standard Model (SM) backgrounds.

1. Introduction

A blind analysis is a technique based on measurements of event signatures in “signal" regions (i.e., where a signal is expected to show up above some background level) using selection cuts developed with the help of theoretical predictions or data control regions, without looking at signal regions directly (see, for example, Refs. [1,2]). The goal of such a technique is to avoid unintended biases that may influence a measurement toward desirable results. On the technical side, blinding can be applied to shapes of distributions or to normalizations of distributions.

Some variations of the blinding technique have been widely used at the Large Hadron Collider (LHC) and other high-energy particle (HEP) experiments. Often, this technique is considered an official policy in dealing with preparations of physics analyses for publications. However, published articles often lack a proper description of the criteria that define the level of rigor of “blindness” to signal regions within a broad range of possible blinding methods.

It is interesting to note that, historically, no unexpected discoveries beyond the Standard Model (SM) have been made in recent decades using the blinding technique in its traditional definition (see the discussion below); it is more difficult to say how many “false-positive” results have been avoided when using this technique since such studies often do not merit publication, and are usually dismissed after sufficient scrutiny by collaborations.The observation of the Higgs boson was a special case since its properties were well known prior to its observation, and the expected SM background was well understood and modeled by Monte Carlo (MC) simulations. The mass of this boson was unknown on the theory side, but the experimental limits of previous experiments pointed to the expected mass region for the LHC searches. It is easy to argue that the discovery of the Higgs boson could easily have been made even without the blinding method after collecting a sufficient amount of data for SM measurements of invariant mass distributions (such as $\gamma \gamma$).

In this article, we will discuss conceptual limitations of the blinding techniques and why this approach in its traditional sense may not be an appropriate method for many searches beyond the Standard Model (BSM). It is not unreasonable to think that these techniques may slow down the pace of discoveries compared to previous decades where such techniques (in combination with MC simulations) were not widely used. Support for this point of view can be drawn from our analyses of the history of particle physics, which will be briefly discussed.

Some specific techniques used for data blinding are discussed in Ref. [2]. As correctly pointed out in Ref. [2], there is no single blinding technique. Still, we think it is possible to characterize such techniques using broad conceptual terms, without giving exact technical details on how the blinding is achieved. In the following, we will attempt to define different classes of blinding procedures used in the past and, more recently, in the LHC experiments.

2. Classic Case—Type A

The most classical case of a blinding technique is when theoretical predictions for background and signal distributions are well established beyond the statistical uncertainties expected for signal regions of data.

Alternatively, theoretical simulations can be replaced by a control region derived from data. It is expected that the control region has statistics as high as those in the signal region itself, has the same physics menu of SM background processes, and uses the same data reconstruction procedures.

In this method, an analysis strategy was developed using the predictions, but while hiding the signal region from the analysis teams that develop selection cuts. Then, this strategy is applied to data after unblinding. It is expected that several teams (analyzers) work on developing selection cuts independently and, preferably, use independent techniques and unblind the signal region at the same time (without biasing conclusions of other analysis teams).

The results of the unblinding should be published independently of actual observations. No additional manipulations with data are expected prior to publication. More specifically, a reduction in discrepancies with the SM, if they are observed, is not allowed. Examples of such blinding can often be found in particle spectroscopy where a region of invariant masses is removed while keeping “side-bands” of real data. In high-energy physics, the discovery of the Higgs boson is a classical example of the Class A blinding [3,4]. The selection procedures were formally approved and fixed before the results from data in the signal region were examined. This was possible since reliable predictions for both background and expected signal rates were available, the signal mass region was known from exclusion limits of previous experiments, and two independent experiments agreed on the strategy for releasing their positive results.

3. Type B

Although the method described above is very straightforward, it should be noted that the immediate publication of a discovery by a single experiment (or by a single analysis group) is unlikely to occur without extensive post-unblinding checks. An application of Sagan’s standard of “extraordinary claims require extraordinary evidence” implies that it is very unlikely unblinded results from a single experiment (and, to a more extreme, by a single group within the same experiment) can be published without an extensive evaluation of systematic effects that may cause the unexpected features. All such “post-unblinding” checks do not fall under the blinding strategy of type A since data in signal regions can easily be manipulated. It is not uncommon to adopt a safer approach of reducing discrepancies with expectation by increasing systematics in the cases when there is no full confidence in the size of systematic uncertainties (in which case, the most conservative assumption is used).

As a result, this leads to a “semi-blinded” approach in a soft understanding of the blinding strategy, i.e., a blinding element is used initially, but further post-blinding manipulations with data are still allowed. This is particularly relevant for the cases when there are no independent analysis teams involved in the analysis. A recent example of type B blinding can be found in [5], where an observation of a near-threshold structure in the $K^{+}$ recoil-mass spectra in an $e^{+}{e}^{-}$ collision was reported by the BES III collaboration.

4. Type C

In practice, a good theoretical understanding of signal regions in terms of predictions may not be possible. The type C blinding deals with the following situations:

• Theoretical predictions have significant uncertainties, i.e., larger than uncertainties expected for the signal region.
• MC simulations used for the description of the background have significantly lower statistics than the data.
• Control regions in data are not expected to catch all the kinematic details of the signal region. For example, they have a different physics menu or contain some reconstruction biases.

To overcome the above problems, a small fraction of some “unblinded” signal region can be used (typically, this fraction is determined by looking at previously published low-statistics data). As a result, generally, no strong requirement of “not to look” at the data can be imposed.

The type C blinding can be used as a guiding principle to perform some basic checks before looking at a signal region of data. Strictly speaking, type C is a method used to make “an educational guess” about background behavior in a signal region, but it cannot give full confidence in our understanding of the background (i.e., its shape and event rates). None of the above studies at the pre-unblinding steps guarantee that the background for the signal region is sufficiently well understood at the level required for a proper blinding procedure, type A or B. Therefore, analysis team(s) should take a certain risk during opening the signal region and should be prepared to see deviations in the signal region from the established background hypothesis. Surely, such deviations do not need to be related to new physics. As a result, extensive cross checks have to be carried out with unblinded data to convince the community of the observation of genuine new physics. All such checks do not fall under blinding principles since data can be manipulated one way or the other.

Type C blinding in searches can be found in Refs. [6,7] and many other similar publications.

5. Type D

This technique does not assume blinding using quantitative estimates of shapes or normalizations of SM backgrounds. This type of blinding is appropriate when no well-understood theoretical predictions exist, or a data control region. All object selections are standard and there is no need to design complex phase-space regions to enhance the signal-over-background ratio.

Generally, analyzers should have some qualitative expectations of how the SM backgrounds should look like, but they do not have precise quantitative predictions for the SM background or for the BSM signal events. For example, when searching for BSM signals in invariant masses (or jet masses), it is expected that the background is a smoothly falling distribution above the Sudakov peak, while signals can be seen as bell-shaped enhancements on top of smoothly falling data spectra. One can argue that type D means “no-blinding”, but we still prefer to call it a variation of the blinding technique since qualitative predictions are typically known from general kinematic arguments or previous low-statistic observations. Expectations for a smoothly falling background can be included in some analytic functions with unknown parameters.

In scenario D, extensive posterior checks are expected before claiming a discovery. Therefore, pre-unblinding preparation can be significantly reduced, or not used at all. The analyzers can look directly at the data using established performance selection cuts for all the objects used in the analysis. It is assumed that no modifications of such selection criteria must be performed. In this sense, analyzers blindly follow the recommended object selections (jets, leptons, and photons) provided by performance groups that are not directly involved in such searches.

Typical examples of type D blinding are searches in dijet invariant masses [8], angular distributions derived from the rapidity of the two jets [9], jet masses, etc. (here, we give only one reference per measurement type). In all such measurements, QCD predictions are not at the same level of precision as required for BSM searches in the signal region. However, qualitatively, we expect that the background shape is a falling function, while the signal has a bell-shaped form. Typically, type D searches are combined with SM measurements. One striking example of the type D is an evidence for the top quarks [10] at the Tevatron. The analyzers knew about possible signatures of top quarks and made efforts to estimate the SM backgrounds using MC simulations and control regions of data. What comes out was an excess of events near 174 GeV above the estimated background. Multiple posterior tests could not reduce this excess. The observation of hadronic $W/Z$ decays in two-jet invariant masses by the UA2 [11] also used an assumption on the approximate shape of the SM background, without any detailed knowledge of the SM predictions (and without applying blinding). Similarly, the observation of exotic structures in the $J/\mathsf{\Psi }p$ channel [12], which can be interpreted as a pentaquark, was made following the general knowledge of how this exotic state can decay, and what reconstruction steps should be undertaken to find it.

6. Conclusions

Although all the above types of blind analysis are expected to be well applied to real-world LHC studies, it is unlikely that types A and B are the best representations of high-precision searches for BSM physics at the LHC. The reason for this is as follows: It is a rare case when there are several independent groups performing the same analysis using different methods, thus “post”-blinding checks with real data are going to happen anyway in the case of unusual observations. The price of making mistakes in claims of extraordinary discoveries is too high. In addition, LHC searches in inclusive events (such as dijets and di-leptons) will deal with levels of statistical precision that is often significantly larger than theoretical uncertainties (or the statistical precision of MC simulations). Thus, blinding A (or B) cannot be used in such situations.

As mentioned before, blinding A or B is most effective in situations with several independent analysis teams that are responsible for processing data and final analysis. For example, a technical team blinds the signal region while other teams define the analysis strategy based on the data blinded by the technical team. If such separation is impossible, the blinding strategy could be affected by psychological effects that are not easy to overcome by small analysis teams with easy access to the data since the signal region can be looked at (intentionally or unintentionally). The most common situation at the LHC is when the same analysis team performs many levels of data processing, including the reconstruction of signal regions.

It is not unreasonable to think that, in the case of inclusive observables, the type D approach, which does not elevate blinding to an absolute necessity, is the most sensible approach. It does not require a precise understanding of theory or SM backgrounds. However, it heavily relies on recommendations for object reconstructions that are typically developed by the teams not directly involved in searches. Another requirement is to have a simple kinematic phase space that does not require complex selection cuts. For example, diphoton and dijet masses are typical examples because no special selection is required to enhance signal regions. In this case, the blinding concerns “blindingly follow” analysis recommendations of performance groups to reconstruct and identify objects and build final observables for searches with a clear understanding of how unusual BSM events may look like.

Even in more extreme scenarios, searches for unusual kinematic features in events where precise theoretical calculations are missing can be prioritized over other methods. Many major discoveries in the past, such as the observation of W [13], the discovery of gluon [14], and the unusually higher rate of diffracting events in $ep$ [15], observations of the top quarks [16,17] were performed without blinding techniques A, B, and C. Observations of unusual events as a byproduct of measurements with significant posterior checks to avoid false positives is a fully justifiable path for new discoveries. Many such studies can be a part of SM measurements, or searches that use SM measurements in combination with qualitative assumptions on how unusual BSM events should look like.

Directly looking at high-precision data when searching for unusual features, and performing extensive posterior checks if such features are found, can be more appropriate and faster than semi-blinding methods (B, C). The latter methods may significantly delay analyses while performing studies of various phenomenological models, dealing with low-precision simulations, or developing systematic uncertainties on statistical limits for theoretical models even before seeing actual data. For example, when it comes to searches of bumps in dijet masses above the QCD background in inclusive events, our understanding of QCD processes (which are dominant backgrounds for many searches) is at the level of a few percent [18], while typical searches for enhancements in dijet masses are performed with relative precision below a few permille [19]. Instrumental effects are also larger than the precision with which data are probed when looking for new physics in high-statistics LHC data. Other examples include the usage of various machine learning methods based on anomaly detection (see the recent LHC article [20]), where the knowledge of the SM background shape in the anomaly regions is difficult to obtain using MC simulations. In such situations, blinding methods cannot reduce the risks of observations of spurious signals, while posterior checks have significantly more value in reducing false positives.

The “eureka moment” is often the result of a careful examination of data and explanation of unusual effects, rather than blinding strategies based on models that “lock” the attention of analyzers to a restrictive parameter domain of some narrowly designed BSM physics models. This is especially true for the LHC where only a few percent of exclusive event categories have been explored so far using predefined BSM models [21]. As argued before, even when using the blinding method B and C for observing unusual features above a background level, a significant effort must be invested in exploring such new features, i.e., by modifying selection cuts and by looking for possible systematic effects. Such checks do not fall into the paradigm of the strict blind strategy A. Therefore, the entire blinding procedure will be put into the doubt and may even lose its merit.

It is advisable that analyzers agree about what type of blinding should be used prior to searches and describe the type of blinding in final publications, which may reduce confusion and possible misinterpretations by the readers.