BUT FIRST!
Hello Everyone! Let’s Catch some knowledge.
I want to talk to you a little bit about the different kinds of scientific studies, and how they rank in terms of reliability.
I get so worked up whenever I’m arguing with someone about an issue related to to health and fitness, and they start saying things like
“research shows that XY and Z” or “I read a study that proves This that and the other thing”. However,
1-they don’t actually provide the source.
2-They don’t say what kind of study it was
3-They don’t acknowledge the limitations of the study
So to be clear, aside from it being good manners, addressing those 3 points can help move the argument along and closer to the truth.
And when you don’t do that it immediately signals to me that you aren’t interested in the truth as much as you are winning an argument (but that’s a topic for a different time). That said
1-If you’re going to make a claim, provide some source of support.
2-Understand that there are different kinds of studies and they are not all created equal. The information from some studies is much more reliable that others.
3-Get used to knowing the limitations of the study. It keeps you humble, and also tells you how far you can go with your claims.
Humility is really important when searching for truth, and you actually have to do the work, you can’t just say “I’m humble”. Now that’s out of the way, let’s get to work.
WHAT’S IN A STUDY?
Not all studies are created equal, and the information that you get from some studies is a lot less reliable than others.
Some studies are limited by resources and funding.
Others are limited by time (subjects have places to go and people to meet).
And sometimes the biases of both the researchers or the subjects can impact the quality of the data.
One of the ideas behind the scientific Method is that other scientists and researchers must be able to replicated the research and the findings.
When a study is replicated over and over by different researchers, the findings are compared, and certain patterns emerge: we can then see the similarities, differences, limitations etc. And get the best possible answer.
And at the end of the day science us the best answer (and not necessarily the RIGHT answer).
1-Meta-Analysis: This is the most reliable kind of research because it is the combined statistical data from lots of different research studies, from which definitive conclusions are drawn.
This is important because it looks at what are the most salient and consistent findings in many studies across time.
It would be incredibly difficult to do one major, giant study on a subject and take into account people from different backgrounds, countries, ethnicities, age groups, health and fitness profiles and so on. So gathering many different studies from different people and places and combining the findings brings us pretty damn close.
2-Systematic Review and Randomized Controlled Trials
Both of these are close behind at number 2:
Systematic reviews are critical assessments and evaluations of all the clinical literature on a subject. The findings from different scientists and researchers are compiled and synthesized both quantitatively and qualitatively
In a systematic review you take a summary of all current literature that pertains to a specific research questions. Something like “What age group is most affected by schizophrenia?” Then you look at all the scientific databases and look for research papers that address this question in some way, shape or form, but you want to be as specific as possible.
Randomized Controlled Trials are controlled clinical trials where people are randomly assigned to either an experimental group or a controlled group.
Researchers testing the efficacy of a new drug for example will give it to the experimental group, while the control group would get a placebo or nothing at all.
These kinds of studies can also be single blind, where the groups don’t know what they are getting, or double blind, where the neither the groups or the researchers know what is being given, in order to eliminate as much biases as possible.
3-Cohort Observational studies are third:
This is where subjects are observed over time and outcomes are either recorded or self reported, and the they are compared.
There can be many variables at play here, which can definitely impact the quality of the findings here. Remember, the data is only as reliable as the instrument that is used to obtain it, and if that instrument is the human brain you’re gonna have all sorts of biases affecting the outcomes.
The case is particularly challenging when people are self-reporting because people can sometime lie, cheat, and steal.
This does not make them bad studies, but it adds to the list of limitations, and definitely gives scientists ideas on how to better control and refine future research.
4-Case Control Studies and Case Reports:
This is where they take one group of people who have a certain condition (like diabetes for example) and compare them to a group of people that does not have the condition.
These studies are useful in some sense because they are much cheaper to conduct when compared to the other types of research. They can usually be done by small teams or even a single person.
But because they rely so heavily on observation, the quality of evidence, and the reliability of the information obtained from them is much lower that the other studies.
That being said, they can be very useful in helping to point the way towards what researchers should focus on more deeply.
5-Animal Research:
these studies are at the bottom of the pyramid, and for good reason. Specifically, we don’t know how the findings and results will look like on humans.
Researchers can definitely extrapolate some very useful and helpful data, in that they usually test on animals to see whether or not a hypothesis is valid or worth pursuing.
One great example that many people are familiar with is the effect of Aspartame on the organism. It is widely believed that aspartame is harmful, toxic, and poisonous etc.
These attitudes come from the findings of a study that was done on lab rats who were given copious amounts of aspartame (not everyone knows about this bit). Obviously, the rats got sick, suffered, and died.
But here’s what people don’t know, the rats were given the equivalent of what a person would consume in diet pop every day for a year (about 17-19 cans/day)
Oh and the other really important bit…rats don’t have the digestive capability to synthesize and breakdown aspartame…so of course it’s going to build up in the system and kill them!
Anyway, got sidetracked for a bit there…
FINAL THOUGHTS:
And there you have it. Hopefully you can see why I get so irritated and why it’s always a good idea to get critical whenever someone starts saying stuff like “Research shows” and “Studies prove”.
What kind of research? Was it done one animals? Because that brings the reliability factor way down.
Was it a cohort observational study? Because my first concern is going to be around the bias of both researchers and subjects.
The problem is that research with poor reliability can make for very sensational claims, as for example in the case of aspartame being poisonous. Now people are afraid to drink diet pop because rats died from being given copious amounts of aspartame.
Do you know the ethical implications you would have to consider if you were to do this kind of research on humans? Not to mention resources and funding!?
You would have to take so many different groups of people from different backgrounds, split them up, give one group nothing but diet pop for a given time, to eliminate any cross contamination and other variables.
take a different group and give them diet pop and something else to test it’s effect when combined with something else.
Take another group and lie to them by telling them they’re getting diet pop but really give them sugar.
Take another group and lie to them and tell them they are getting sugar, but in fact give them diet pop.
And you would still not be able to come to any definitive conclusions because there are so many other factors and variables to consider!
Sigh…….
At the end of the day my biggest issue is when people try to reorganize their whole belief system, and try to convince me that I should do the same, based on incomplete or very poor research.
So the next time someone wants to tell you about research they hear about or a study they’ve read, ask them
-What’s the source?
-What kind of study was it?
-What were the limitations?
Think about it. PEACE!