什么值得买荣登360手机助手2017“闪耀”App年度榜

Whenever people speak of a “crisis” in any enterprise that has been around for a very long time—like experimental psychology (or science in general)—a measure of skepticism is probably a very sensible reaction. Is the present flurry of concern about replicability and replication—the development that prompted the current special issue of Perspectives on Psychological Science—overblown? In this article, we explore the three arguments that we have heard most often from scientists who see the current outpouring of concern over replicability as greatly overblown. These scientists, some of whom are prominent and accomplished researchers, view efforts to change scientific practices as unnecessary. We contend that these arguments are misguided in instructive ways. The first argument focuses on the rate of false positives (a topic aficionados of statistics and methodology are well familiar with).

Thanks to the work of Ioannidis (2005b) and other statisticians, the essential problems with this view are already familiar to many. Nonetheless, it is our sense that a fairly large number of scientists in psychology and other fields are not familiar with this work and still tend (erroneously) to assume that alpha levels represent upper bounds on the rate at which errors can accumulate in a literature.

So what is the truth of the matter? To put it simply, adopting an alpha level of, say, 5% means that about 5% of the time when researchers test a null hypothesis that is true (i.e., when they look for a difference that does not exist), they will end up with a statistically significant difference (a Type 1 error or false positive.)¹ Whereas some have argued that 5% would be too many mistakes to tolerate, it certainly would not constitute a flood of error. So what is the problem?

Unfortunately, the problem is that the alpha level does not provide even a rough estimate, much less a true upper bound, on the likelihood that any given positive finding appearing in a scientific literature will be erroneous. To estimate what the literature-wide false positive likelihood is, several additional values, which can only be guessed at, need to be specified. We begin by considering some highly simplified scenarios. Although artificial, these have enough plausibility to provide some eye-opening conclusions.

For the following example, let us suppose that 10% of the effects that researchers look for actually exist, which will be referred to here as the prior probability of an effect (i.e., the null hypothesis is true 90% of the time). Given an alpha of 5%, Type 1 errors will occur in 4.5% of the studies performed (90% × 5%). If one assumes that studies all have a power of, say, 80% to detect those effects that do exist, correct rejections of the null hypothesis will occur in 8% of the time (80% × 10%). If one further imagines that all positive results are published then this would mean that the probability any given published positive result is erroneous would be equal to the proportion of false positives divided by the sum of the proportion of false positives plus the proportion of correct rejections. Given the proportions specified above, then, we see that more than one third of published positive findings would be false positives [4.5% / (4.5% + 8%) = 36%]. In this example, the errors occur at a rate approximately seven times the nominal alpha level (row 1 of Table 1).

Table 1 shows a few more hypothetical examples of how the frequency of false positives in the literature would depend upon the assumed probability of null hypothesis being false and the statistical power. An 80% power likely exceeds any realistic assumptions about psychology studies in general. For example, Bakker, van Dijk, and Wikkerts, (2012, this issue) estimate .35 as a typical power level in the psychological literature. If one modifies the previous example to assume a more plausible power level of 35%, the likelihood of positive results being false rises to 56% (second row of the table). John Ioannidis (2005b) did pioneering work to analyze (much more carefully and realistically than we do here) the proportion of results that are likely to be false, and he concluded that it could very easily be a majority of all reported effects.

For the probability that any given positive result in the published literature is wrong to occur as infrequently as the 5% alpha level, assuming 35% power, one would have to assume that the differences looked for exist about 75% of the time (fourth row in the table).

So, what is a reasonable estimate for the prior probability of effects that are tested by psychologists? Ioannidis (2005b) considers many domains in which the probability seems quite certain to be extremely low—for example, epidemiological studies examining very long lists of dietary factors and relating them to cancer or genome-wide association studies (GWAS) examining tens of thousands of genetic results. Experimental psychologists may think, “Well, that may be the case for massive exploratory studies in biomedicine, but I typically have some theoretically motivated predictions for which I have pretty high confidence.” Thus, the reader may want to argue that a prior probability of 75% is not necessarily too high. However, in considering the likely credibility of the psychological literature as a whole, the issue is not whether investigators sometimes test hypotheses that they view as having a reasonable likelihood of yielding a positive result. The issue, rather, is the number of effects that are tested and for which, given a positive result, investigators would proceed to publish the result and devise some theoretical interpretation. We suspect that if readers reflect on this question, they will conclude, as we have, that this number is often quite large even in research that is, in a broad sense, theoretically motivated. As Kerr (1998) and Wagenmakers, Wetzels, Borsboom, and van der Maas (2011) describe, experimental research is normally at least partly exploratory in nature even when it is presented in a confirmatory template. Thus, we would argue that the second row of the table probably comes closer to the situation in experimental psychology than we might like to imagine—implying that Ioannidis’s devastating surmise (“Most published results are false”) could very easily be the case throughout our field. (Of course, this likelihood is amplified if journals are willing to publish surprising results based on a single positive finding, and we would argue that such a willingness is quite common, although certainly not universal.)

Thus, the fact that psychology and similar fields usually insist upon a reassuringly low alpha level (typically 5%) does not by any means imply that no more than 5% of the positive findings in the published literature are likely to be errors. Moreover, the situation is surely much worse than what the discussion above would suggest because in addition to testing many hypotheses with a low likelihood of effects, investigators often exploit hidden flexibility in their data analysis strategies, allowing the true alpha level to rise well above the nominal alpha level (Simmons, Nelson, & Simonsohn, 2011; see also discussion by Ioannidis, 2005b, of “bias” and Wagenmakers et al., 2012, this issue, on “fairy tale factors”). Moreover, the highest impact journals famously tend to favor highly surprising results; this makes it easy to see how the proportion of false positive findings could be even higher in such journals than it would be in less career-enhancing outlets (Ioannidis, 2005a; Munafò, Stothart, & Flint, 2009). Naturally, those areas within psychology that lend themselves to performing a great number of tests on a variety of variables in any given study, as well as areas in which underpowered studies are more common, are likely more prone to false findings than are other areas.

In summary, our standard statistical practices provide no assurance that erroneous findings will occur in the literature at rates even close to the nominal alpha level (see Wagenmakers, Wetzels, Borsboom, van der Maas, and Kievit, 2012, this issue, on the remarkably low diagnosticity of results meeting the standard criteria for rejecting the null hypothesis). Given that errors are sure to be published in many cases, it seems to us that the most critical question is what happens to these errors after they are published. If they are often corrected, then the initial publication may not cause much harm.

Unfortunately, as many of the articles in the current special issue document and discuss (e.g., Makel et al., 2012, this issue), the sort of direct replications needed for identifying erroneous findings are disturbingly rare. Moreover, even when such data are collected, the results are hard to publish, regardless of whether they confirm or disconfirm the finding. This brings us then to the second, and probably the most popular, response from defenders of the status quo.

This view was advocated, for example, by senior psychologists quoted by Carpenter (2012). In our opinion, this is a seductive but profoundly misleading argument. We contend that in any field where it is rare for people to conduct direct replications, and common to undertake conceptual replications, the field can be grossly misled about the reality of phenomena; and misled even more gravely than would happen based on the issues discussed above. The reason, it seems to us, is that conceptual replication attempts (especially when such studies are numerous and low in statistical power) interact in an insidious fashion with publication bias and also with the natural tendency for results perceived as “interesting” to circulate among scientists through informal channels.

To shed light on this, consider the question: When investigators undertake direct replications and they fail to obtain an effect, what are they likely to do with their results? In the ideal world, of course, they would publish these outcomes in journals or make them public through other mechanisms such as websites or scholarly meetings. In fact, published nonreplications are rare (e.g., Makel et al., 2012; Sterling, Rosenbaum, & Weinkam, 1995). Nonetheless, we conjecture, when a respected investigator obtains but does not publish a negative result, the fact of the failure often achieves some limited degree of dissemination through informal channels. A failure to confirm a result based on a serious direct replication attempt is interesting gossip, and the fact is likely to circulate at least among a narrow group of interested parties. At a minimum, the investigator and his or her immediate colleagues will have reduced confidence in the effect.

By contrast, consider what happens when a scientific community undertakes only conceptual replication attempts. If a conceptual replication attempt fails, what happens next? Rarely, it seems to us, would the investigators themselves believe they have learned much of anything. We conjecture that the typical response of an investigator in this (not uncommon) situation is to think something like “I should have tried an experiment closer to the original procedure—my mistake.” Whereas the investigator may conclude that the underlying effect is not as robust or generalizable as had been hoped, he or she is not likely to question the veracity of the original report. As with direct replication failures, the likelihood of being able to publish a conceptual replication failure in a journal is very low. But here, the failure will likely generate no gossip—there is nothing interesting enough to talk about here. The upshot, then, is that a great many failures of conceptual replication attempts can take place without triggering any general skepticism of the phenomenon at issue (see also Nosek, 2012, this issue, for a similar point).

On the other hand, what happens when investigators undertake a conceptual replication and succeed? Without a doubt, such a success will be seen as interesting, and the researchers will seek to publish it, present it at meetings, and otherwise promote it. When they do so, they will likely receive encouragement from the original investigators (who are plausible reviewers for the work). Because the conceptual replication attempt differs from the original study in procedure it will often be seen as novel enough to warrant publication. In short, a conceptual replication success is much more publishable than would be a direct replication success, which is of course precisely why investigators are tempted to skip the direct replications and focus their efforts on conceptual replications.

The inevitable conclusion, it seems to us, is that a scientific culture and an incentive scheme that promotes and rewards conceptual rather than direct replications amplifies the publication bias problem in a rather insidious fashion. Such a scheme currently exists in every area of research of which we are aware.

We would go further and speculate that entire communities of researchers in any field where direct replication attempts are nonexistent and conceptual replication attempts are common can easily be led to believe beyond question in phenomena that simply do not exist. What conditions are needed to promote such pathological results? The key element would seem to be that a pseudoresult appears both exciting and easy—the kind of result that would tempt hundreds of researchers to undertake small low-powered conceptual replication attempts (see Bakker et al., 2012; Ioannidis, 2005b). Enough of these will “work” just by chance to generate the strong impression in the community that successful confirmations are plentiful (and if investigators exploit the hidden degrees of freedom pointed out by Simmons et al., 2011, there will be more of these than one would expect from the nominal alpha level).