I often hang out with scientists from different disciplines. Over the years, I have found that there exist profound differences in how people evaluate their own scientific field. Physicists, for instance, tend to believe that physics is the perfect science, and sometimes seem to think that the only reason that other fields lack strong scientific laws is that the physicists haven’t gotten involved yet. Medical doctors, in contrast, generally acknowledge their lack of understanding in many respects, but are almost always optimistic, and convinced that rapid progress will quickly unravel the mysteries of the human body. Economists invariably state that their science can’t predict anything and never will, but at the same time won’t hesitate to use that same science to explain everything that happened in the past, while philosophers seem convinced that real progress is not a concept that applies to their field anyway and will happily make fun of their own discipline.
Psychologists also tend to strongly relativize their field (browsing the about eighty columns I have written about psychology for Spiegeloog, it is evident that I am no exception to that rule). Psychologists continuously stress the preliminary nature of their insights and the shaky foundation of their knowledge base. Suppose, for instance, that in a multidisciplinary discussion a biologist states that psychotherapeutic interventions have been shown to be effective for the treatment of depression. Psychologists are unlikely to respond by just acknowledging this. It is more likely that a psychologist will note that these interventions are not effective in roughly half of the cases, that despite the effectiveness of psychotherapy relapse rates are high, or that nobody knows what depression really is anyway. This is quite interesting. The same points could be made with respect to many medical interventions, but I have rarely seen physicians relativize their field in this way.
Is it a coincidence that the ‘reproducibility crisis’ that we face in science was first identified in psychology? Many people seem to think so. For example, it is not uncommon to find people responding to the crisis as if it is specific to psychology (‘I always knew that line of research in psychology was no good’). However, it is now evident that reproducibility crises are widespread and affect many distinct fields. For instance, reproducibility projects in biomedical fields report comparable results to the now famous Reproducibility Project: Psychology, which concluded that less than half of hundred replication attempts was successful. Through the grapevine, I have heard that even informatics suffers from replication problems (i.e., performance of computer programs being irreproducible), and I suspect that my own field of methodology (which is largely based on computer simulations) is likely to be no exception either. Perhaps the only reason that we haven’t seen a replication crisis in physics yet is that nobody has systematically looked into it.
If this is true, the standard picture of the reproducibility crisis may be inaccurate. Usually, people single out psychology as a problem area, and portray the situation as if scientists have now found out that many of the results in that particular area don’t replicate. Perhaps, however, scientific findings face replication problems as a rule rather than an exception, and psychologists were simply the first to find out.
I often hang out with scientists from different disciplines. Over the years, I have found that there exist profound differences in how people evaluate their own scientific field. Physicists, for instance, tend to believe that physics is the perfect science, and sometimes seem to think that the only reason that other fields lack strong scientific laws is that the physicists haven’t gotten involved yet. Medical doctors, in contrast, generally acknowledge their lack of understanding in many respects, but are almost always optimistic, and convinced that rapid progress will quickly unravel the mysteries of the human body. Economists invariably state that their science can’t predict anything and never will, but at the same time won’t hesitate to use that same science to explain everything that happened in the past, while philosophers seem convinced that real progress is not a concept that applies to their field anyway and will happily make fun of their own discipline.
Psychologists also tend to strongly relativize their field (browsing the about eighty columns I have written about psychology for Spiegeloog, it is evident that I am no exception to that rule). Psychologists continuously stress the preliminary nature of their insights and the shaky foundation of their knowledge base. Suppose, for instance, that in a multidisciplinary discussion a biologist states that psychotherapeutic interventions have been shown to be effective for the treatment of depression. Psychologists are unlikely to respond by just acknowledging this. It is more likely that a psychologist will note that these interventions are not effective in roughly half of the cases, that despite the effectiveness of psychotherapy relapse rates are high, or that nobody knows what depression really is anyway. This is quite interesting. The same points could be made with respect to many medical interventions, but I have rarely seen physicians relativize their field in this way.
Is it a coincidence that the ‘reproducibility crisis’ that we face in science was first identified in psychology? Many people seem to think so. For example, it is not uncommon to find people responding to the crisis as if it is specific to psychology (‘I always knew that line of research in psychology was no good’). However, it is now evident that reproducibility crises are widespread and affect many distinct fields. For instance, reproducibility projects in biomedical fields report comparable results to the now famous Reproducibility Project: Psychology, which concluded that less than half of hundred replication attempts was successful. Through the grapevine, I have heard that even informatics suffers from replication problems (i.e., performance of computer programs being irreproducible), and I suspect that my own field of methodology (which is largely based on computer simulations) is likely to be no exception either. Perhaps the only reason that we haven’t seen a replication crisis in physics yet is that nobody has systematically looked into it.
If this is true, the standard picture of the reproducibility crisis may be inaccurate. Usually, people single out psychology as a problem area, and portray the situation as if scientists have now found out that many of the results in that particular area don’t replicate. Perhaps, however, scientific findings face replication problems as a rule rather than an exception, and psychologists were simply the first to find out.
