The field of Psychology has long been taunted about the our grand replication crisis as the biggest scam of our lives; but there’s another fire that’s screaming for our attention and deservedly so.
The field of Psychology has long been taunted about the our grand replication crisis as the biggest scam of our lives; but there’s another fire that’s screaming for our attention and deservedly so.
Photo by Ante Hamersmit
Photo by Ante Hamersmit
From time to time, we’re reminded of the state of our field, it being an ostracised science, the dubiousness of our research methods, all peppered with snarky references to the big old replication crisis. Although our courses have made us cognisant of these burning issues in psychological research, which is well and good, it often throws my dear friends and I into these personal crisis situations that usually result in angsty fuming into the wee hours of the night or tense discussions on whether academia is the path ahead for us. What particularly shook me in the spring of 2021 is my first encounter with the term ‘theory crisis’. This absolutely wonderful course (I’m an unabashed and unpaid advertiser of this course), Fundamentals of Methodology, introduced us students to a new fire that’s been burning and needs our immediate attention. While writing papers in the initial years of the bachelor, I sometimes wondered if some random theory I picked up online for the paper was even good or not. Since the focus of the assignment was the writing, I didn’t bother too much about it, but it hit me one day that papers build on theories that may hardly have any evidence, have been criticised, or the like. And as this chain goes on, we begin building a science equivalent to a Jenga game. However, one that I hope we can rescue before it all falls down.
Oberauer and Lewandowsky (2019) note that failures to find effects are usually attributed to methodological issues, but the actual theory is not questioned. There has been progress to improve methodological issues; namely, open science being a major movement. However, it is becoming evident that the theory crisis needs more attention and amendments. They also point out how the focus has been on the empirical level, wherein we move from hypothesis to data, but the theoretical level, where we go from theories to hypothesis, needs more attention as well. Paul Meehl (1978) was particularly vocal about this issue back in the day, as he’d said that theories don’t establish themselves in psychology: “Theories are neither decisively refuted nor accepted as part of established knowledge; they simply hang around until they are abandoned or forgotten.” (Eronen & Bringmann, 2021). The shakiness of our foundations are already reflected by Meehl’s pioneering thought from almost 40-50 years ago. An amusing analogy of this theory crisis concern is captured by Mischel (2008) in what he calls the toothbrush problem: “psychologists treat other peoples’ theories like toothbrushes—no self- respecting person wants to use anyone else’s” (Mischel, 2008 as cited in Borsboom et al., 2021). Essentially, everyone has gone out and about coming up with their own theories, so the issue is largely pertaining to the quality of the theories as opposed to the quantity.
“Everyone has gone out and about coming up with their own theories, so the issue is largely pertaining to the quality of theories as opposed to the quantity.”
Several researchers and groups have proposed formal modelling as a solution to the crisis (Robinaugh et al., 2021; Borsboom et al., 2021). Oude Maatman (2021) suggests though that it is first important to understand where the crisis stems from before moving on to addressing these causes. Eronen and Bringmann (2021) point out that the inherent nature of the psychological field is one of the key reasons that makes theory formation a rocky road.
In his comprehensive piece on the crisis, Freek Oude Maatman (2021) neatly explains a few points that I find fundamental to this discussion since they throw light on some issues faced when we go from theories to hypotheses. He first explains the underdetermination of theory by data, which is essentially not being fully sure of what theoretical conclusions to make from empirical findings. One cause of this is that findings that match the predicted hypotheses do not necessarily imply that a particular theory is true; there could be alternative explanations. More the alternative explanations available, higher the underdetermination. Another cause is that when testing a hypothesis, we always require additional assumptions (known as auxiliary assumptions) and these are inherently tested with the main hypothesis, thus preventing us from testing a hypothesis (and consequently, theory). More the number of auxiliary assumptions falsified, post-hoc, higher the underdetermination. Oude Maatman (2021) also brings to our notice the fact that our worldview (which he calls ontological commitments) lurks in the background and influences whatever research we conduct, methodology we use and conclusions we make. Importantly, he shows us how these ontological commitments also influence the statistical tools we choose to apply to our experimentation, giving the example of whether we look at individuals or the population. One of the main ways to deal with this, as suggested by Oude Maatman and others in the field, is to increase the specifications of the theory and the assumptions that are accompanied with testing the theory.
“Unlike statistics and methodology, theory construction or skills for doing so are often absent from the Psychology curricula.”
As mentioned earlier, one of the main solutions that has been proposed to this crisis is formal modelling, a language which uses math, programming or logic instead of a verbal language. Robinaugh et al. (2021) and Borsboom et al. (2021) introduce formal modelling as an essential toolkit to revamp the theory construction scene. They argue that formal models allow us to derive more testable predictions, force us to be more specific about our variables and predictions, and more transparent about our assumptions. Additionally, Robinaugh et al. (2021) speak of formal theories as being expressed in a language that transgresses a specific sub-field, thus it being potentially easier to facilitate the integration of theories due to this shared language (i.e., formal models). Borsboom et al. (2021) take a more pedagogical and overarching approach of this issue as well, as they note that our field lacks a “theory construction program”. They remark that unlike statistics and methodology, theory construction or skills for doing so are often absent from the Psychology curricula. This was also something that personally made me realise how essential this segment of the Fundamentals of Methodology course is for each student studying Psychology and should not solely be taught to students specialising in Psychological Methods. Theory forms the basis of research that ultimately guides clinical work and other applications of psychology, which makes this extremely relevant for students of all specialisations within Psychology to know. Borsboom et al. (2021) echo the views of Oberauer and Lewandowsky (2019) when they speak of how the focus is on hypothesis testing; moreover, they acknowledge how the culture in academia is one where the emphasis is such. They formalise their formal modelling solution as they propose a practical five-step ‘Theory Construction Methodology’ (TCM) in which they essentially portray theory construction as a skill that has to be developed. Briefly speaking, TCM consists of firstly identifying empirical phenomena, finding a theory from another field akin to the one being developed, developing a formal theory, and evaluating the theory.
While formal modelling is being seen as something that can swoop in and start mending the battered state of our theories, a few in the field don’t believe this is the ultimate solution. Eronen and Bringmann (2021) pointed out issues that pertained to the nature of our science as a constraining factor in theory development, and these cannot be solved by formal modelling. Despite this, they believe that clearly and precisely defined constructs that go into our theories is a crucial step that should not be undermined even if it’s not really a quantitative formalisation. Oude Maatman (2021) echoes this sentiment and also emphasises on strengthening the conceptual basis of our verbal theories since those would provide a strong base. <<
References
– Borsboom, D., van der Maas, H. L. J., Dalege, J., Kievit, R. A., & Haig, B. D. (2021). Theory Construction Methodology: A Practical Framework for Building Theories in Psychology. Perspectives on Psychological Science, 16(4), 756–766.
– Eronen, M. I., & Bringmann, L. F. (2021). The Theory Crisis in Psychology: How to Move Forward. Perspectives on Psychological Science, 16(4), 779–788.
– Oberauer, K., & Lewandowsky, S. (2019). Addressing the theory crisis in psychology. Psychonomic Bulletin & Review, 26(5), 1596–1618.
– Oude Maatman, F. (2021). Psychology’s Theory Crisis, and Why Formal Modelling Cannot Solve It. PsyArXiv.
– Robinaugh, D. J., Haslbeck, J. M. B., Ryan, O., Fried, E. I., & Waldorp, L. J. (2021). Invisible Hands and Fine Calipers: A Call to Use Formal Theory as a Toolkit for Theory Construction. Perspectives on Psychological Science, 16(4), 725–743.
From time to time, we’re reminded of the state of our field, it being an ostracised science, the dubiousness of our research methods, all peppered with snarky references to the big old replication crisis. Although our courses have made us cognisant of these burning issues in psychological research, which is well and good, it often throws my dear friends and I into these personal crisis situations that usually result in angsty fuming into the wee hours of the night or tense discussions on whether academia is the path ahead for us. What particularly shook me in the spring of 2021 is my first encounter with the term ‘theory crisis’. This absolutely wonderful course (I’m an unabashed and unpaid advertiser of this course), Fundamentals of Methodology, introduced us students to a new fire that’s been burning and needs our immediate attention. While writing papers in the initial years of the bachelor, I sometimes wondered if some random theory I picked up online for the paper was even good or not. Since the focus of the assignment was the writing, I didn’t bother too much about it, but it hit me one day that papers build on theories that may hardly have any evidence, have been criticised, or the like. And as this chain goes on, we begin building a science equivalent to a Jenga game. However, one that I hope we can rescue before it all falls down.
Oberauer and Lewandowsky (2019) note that failures to find effects are usually attributed to methodological issues, but the actual theory is not questioned. There has been progress to improve methodological issues; namely, open science being a major movement. However, it is becoming evident that the theory crisis needs more attention and amendments. They also point out how the focus has been on the empirical level, wherein we move from hypothesis to data, but the theoretical level, where we go from theories to hypothesis, needs more attention as well. Paul Meehl (1978) was particularly vocal about this issue back in the day, as he’d said that theories don’t establish themselves in psychology: “Theories are neither decisively refuted nor accepted as part of established knowledge; they simply hang around until they are abandoned or forgotten.” (Eronen & Bringmann, 2021). The shakiness of our foundations are already reflected by Meehl’s pioneering thought from almost 40-50 years ago. An amusing analogy of this theory crisis concern is captured by Mischel (2008) in what he calls the toothbrush problem: “psychologists treat other peoples’ theories like toothbrushes—no self- respecting person wants to use anyone else’s” (Mischel, 2008 as cited in Borsboom et al., 2021). Essentially, everyone has gone out and about coming up with their own theories, so the issue is largely pertaining to the quality of the theories as opposed to the quantity.
“Everyone has gone out and about coming up with their own theories, so the issue is largely pertaining to the quality of theories as opposed to the quantity.”
Several researchers and groups have proposed formal modelling as a solution to the crisis (Robinaugh et al., 2021; Borsboom et al., 2021). Maatman (2021) suggests though that it is first important to understand where the crisis stems from before moving on to addressing these causes. Eronen & Bringmann (2021) point out that the inherent nature of the psychological field is one of the key reasons that makes theory formation a rocky road. (Add brief information from Eronen & Bringmann, 2021)
In his comprehensive piece on the crisis, Freek Oude Maatman (2021) neatly explains a few points that I find fundamental to this discussion since they throw light on some issues faced when we go from theories to hypotheses. He first explains the underdetermination of theory by data, which is essentially not being fully sure of what theoretical conclusions to make from empirical findings. One cause of this is that findings that match the predicted hypotheses do not necessarily imply that a particular theory is true; there could be alternative explanations. More the alternative explanations available, higher the underdetermination. Another cause is that when testing a hypothesis, we always require additional assumptions (known as auxiliary assumptions) and these are inherently tested with the main hypothesis, thus preventing us from testing a hypothesis (and consequently, theory). More the number of auxiliary assumptions falsified, post-hoc, higher the underdetermination. Maatman (2021) also brings to our notice the fact that our worldview (which he calls ontological commitments) lurks in the background and influences whatever research we conduct, methodology we use and conclusions we make. Importantly, he shows us how these ontological commitments also influence the statistical tools we choose to apply to our experimentation, giving the example of whether we look at individuals or the population. One of the main ways to deal with this, as suggested by Maatman and others in the field, is to increase the specifications of the theory and the assumptions that are accompanied with testing the theory.
“Unlike statistics and methodology, theory construction or skills for doing so are often absent from the Psychology curricula.”
As mentioned earlier, one of the main solutions that has been proposed to this crisis is formal modelling, a language which uses math, programming or logic instead of a verbal language. Robinaugh et al. (2021) and Borsboom et al. (2021) introduce formal modelling as an essential toolkit to revamp the theory construction scene. They argue that formal models allow us to derive more testable predictions, force us to be more specific about our variables and predictions, and more transparent about our assumptions. Additionally, Robinaugh et al. (2021) speak of formal theories as being expressed in a language that transgresses a specific sub-field, thus it being potentially easier to facilitate the integration of theories due to this shared language (i.e., formal models). Borsboom et al. (2021) take a more pedagogical and overarching approach of this issue as well, as they note that our field lacks a “theory construction program”. They remark that unlike statistics and methodology, theory construction or skills for doing so are often absent from the Psychology curricula. This was also something that personally made me realise how essential this segment of the Fundamentals of Methodology course is for each student studying Psychology and should not solely be taught to students specialising in Psychological Methods. Theory forms the basis of research that ultimately guides clinical work and other applications of psychology, which makes this extremely relevant for students of all specialisations within Psychology to know. Borsboom et al. (2021) echo the views of Oberauer and Lewandowsky (2019) when they speak of how the focus is on hypothesis testing; moreover, they acknowledge how the culture in academia is one where the emphasis is such. They formalise their formal modelling solution as they propose a practical five-step ‘Theory Construction Methodology’ (TCM) in which they essentially portray theory construction as a skill that has to be developed. Briefly speaking, TCM consists of firstly identifying empirical phenomena, finding a theory from another field akin to the one being developed, developing a formal theory, and evaluating the theory.
While formal modelling is being seen as something that can swoop in and start mending the battered state of our theories, a few in the field don’t believe this is the ultimate solution. Eronen and Bringmann (2021) pointed out issues that pertained to the nature of our science as a constraining factor in theory development, and these cannot be solved by formal modelling. Despite this, they believe that clearly and precisely defined constructs that go into our theories is a crucial step that should not be undermined even if it’s not really a quantitative formalisation. Oude Maatman (2021) echoes this sentiment and also emphasises on strengthening the conceptual basis of our verbal theories since those would provide a strong base. <<
