New Directions in Intelligence Research: Avoiding the Mistakes of the Past

Abstract

This brief commentary considers the potential for new directions in intelligence research, as well as possible pitfalls associated with these approaches. Specifically, this commentary focuses on the use of big data in intelligence research, the study of genes and gene-environment interactions, the interpretation of neuroscience evidence, and the effectiveness of intelligence interventions. The major pitfalls identified include methodological and data analytic limitations, as well as concerns regarding the communication of findings to other scientists and the lay public.

1. Introduction

We applaud the authors of the two target articles and the editorial board for pointing to critical issues, promising developments, and new directions for intelligence research, while simultaneously identifying methodological and data interpretation concerns as well as areas of potentially less interest and scientific value [1,2,3]. We agree particularly with the importance of delineating what intelligence is and what is not, carefully measuring related cognitive constructs, considering gene-environment interactions, taking advantage of big data, and integrating across different levels of analysis from neural (and even molecular) up to social and environmental. Here we identify a set of methodological concerns and data interpretation problems (especially with respect to public policy), many of which are of historic importance in the field of intelligence and are perhaps especially salient to us, given our interest in research on statistical and scientific reasoning and on cognitive training interventions. We believe these concerns should be kept in mind even as we move forward with future work. After all, the phrase “Those who cannot remember the past are condemned to repeat it,” is not just a famous quote from Santayana (a man who had a few things to say about intelligence himself), it is also an acceptable answer to a question in one of the most widely used modern intelligence assessments [4].

2. Big Data

The potential of big data is a recurring theme throughout the editorials and commentaries. Big data may be unimaginably big, both in scope and implications for future research—there are now datasets that arguably reflect the actual population of entire countries [5]. While the potential of big data is great, the insights we have so far achieved with it are not quite as sizable. We point to four concerns. First, the quality of the data and understanding of measurement can be negatively correlated with the size of the dataset. Potential shortcuts like using short forms of measures or data collected for many different purposes may not include exactly the right constructs for a particular analysis or question. Second, drawing conclusions about mechanism is fraught with difficulty when variables are highly inter-correlated, especially when datasets include large numbers of measures not necessarily selected for theoretical reasons or for a particular study. Third, different researchers use the same data set to ask different and related questions but there is no method for correcting for multiple comparisons across research groups. Finally, given that large datasets can yield statistically significant findings of small effect size, scientists and the public may overvalue their meaning. Overuse of these results may undermine individual success with respect to use of published data for workforce or educational selection (by non-researchers). For example, if a factor has a small but significant predictive value, it may be used to make decisions about candidates regardless of its relatively low importance. Based on these concerns we suggest that researchers always communicate effect size information and that we begin to develop approaches to dealing with multiple analyses of the same data in different laboratories.

3. Genes, Gene-Environment Interactions

We note with enthusiasm the movement away from a focus on behavioral genetics alone to a more nuanced approach inclusive of epigenetics and gene-environment interactions. While promising, it is important to remember that these interactions in other contexts often only account for a very, very small amount of the variance [6]; this is quite possibly true for intelligence as well. We take this opportunity to briefly discuss behavioral genetics. Despite the appearance of consensus amongst the commentators that behavioral genetics work is of reduced importance today, it is a large field and numerical estimates of “percentage of variance explained” by genetics versus environment are still frequently discussed [7]. In particular, though estimates of heritability in many models cannot take into account shared environmental factors (i.e., there is absolutely no reduction in heritability estimates in twin models even when environmental factors are explicitly added that dramatically affect both twins), large heritability estimates are sometimes used to make claims about the likelihood of intervention success. The lay public, and perhaps even some scientists, may value numbers such as “percent of variance explained” more than is warranted. In our lab, we have dubbed this effect “number absolutism” [8]. One concern given our interest in cognitive interventions is that the underestimation of environmental effects may negatively impact motivation to improve.

4. Neuroscience

The integration of neuroimaging data across levels of analysis is an exciting development in intelligence research. As with numbers and equations, one caveat is the concern that the lay public seems to “over-value” data from neuroscience [9,10]. It is important that intelligence researchers who use neuroscience data communicate with care and, in particular, avoid overselling claims especially in the media. Unfortunately, “neuromyths” based on misinterpretations of neuroscience data may influence the public’s decisions. Individuals who believe, for example, that they are right-brained and therefore cannot learn left-brained tasks, or who believe that because there is a neural basis to ADHD or a learning disability and therefore no possibility of improvement, may not be motivated to achieve academically or put forth the effort needed to learn certain cognitive skills. On the other hand, individuals may positively and inappropriately respond to under supported marketing claims for energy drinks or brain-training interventions developed by “neuroscientists”.

5. Interventions to Improve Intelligence

The editors and commentators generally express cautious optimism regarding attempts to improve intelligence. Research in this domain (broadly construed to include everything from interventions that focus on training of small numbers of specific cognitive skills to large scale full family interventions) is notoriously difficult. Small, well-controlled studies often find effects that disappear at scale, partially because of difficulty maintaining high levels of quality control. At the same time, individual differences in ability and environment are likely to be important moderators and cannot be adequately addressed with small-scale studies [11]. Further, small-scale studies cannot adequately address questions related to spacing, dose-response effects, and motivational factors that may influence training. In our recent work we find compelling evidence that such factors may significantly influence training and transfer outcomes, and that these factors frequently interact with one another [12,13,14]. High quality, methodologically sound studies that include large enough samples to adequately answer such questions must be completed before strong conclusions can be made regarding intelligence interventions. In this respect certain aspects of big-data approaches and new internet-based research techniques offer exciting possibilities.

6. Conclusions

Above, we have highlighted a smattering of concerns that have been a focus of our thinking in recent months and years. Together they may seem to provide daunting roadblocks to the progress and dissemination of intelligence research, but we share Conway’s assertion that our colleagues should have “nothing to fear” from this area of study. Rather we wish to highlight these issues to ensure that this progress takes place smoothly. We are cautiously optimistic about new directions of research. For example, one reviewer of this commentary pointed to the fact that some research teams are now combining big data, neuroscience, and genetics into single large-scale studies. Such approaches have the advantage of understanding intelligence at multiple levels of analysis. At the same time, they may also inherit the pitfalls of all the individual levels.

Although we quoted Santayana’s most-used aphorism earlier, perhaps intelligence researchers will be better served by a lesser-known line immediately before it in his Life of Reason [15]. “Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual”. Thus, we look forward to watching intelligence research “grow up”. This journal is an important step in that process, and we thank the editor for the opportunity to share our perspective.