Follow us 261.1k
Quick Summary tl;dr
Although some studies rank lower on the scientific rigor scale than others, that does not mean that you just discount them.
Whenever a new study is published or discussed online, make sure you do your homework and find out exactly what type of study they are talking about. Once you know the type of study, you will be able to understand the limitations that come with that study and what that means for the results.
As a general rule of thumb, any study based on RCT or systematic review principles will have robust results. However, there are still limitations and biases that occur in these studies, so again interpret these with caution.
When reading a science-based article, again take time to look at what type of study they are meaning. If the author has based all of their argument on an animal or a one-person case study — you can confidently say the article is not scientifically robust enough. However, if they have a mixture of studies including observational, RCTs and even one systematic review, their writing is scientifically robust enough.
No matter what type of evidence is being used — no study should ever have a definitive answer and should always acknowledge their limitations.
Clinical research lays the groundwork for progress within the field of medicine and the prerequisite for ‘evidence based medicine.’ Evidence based medicine is what all researchers and clinicians should focus upon when talking about a treatment or disease.
Not All Research is Equal
By the time the media or an ‘expert’ gets a hold of a new study, they often forget about the scientific rigor. What we can see happening is a cherry picking of the data or the conclusion of a study only presented.
However, as with the world of science, not all research is equal. Even though a conclusion of a study may show promising results or validate a particular belief, without understanding the type of study, we do not actually know if these results have any particular meaning.
That's why you should always interpret statements such as "studies say..." with caution. Always ask yourself a few questions. What type of study is it? Who is the author? Are there any conflicts of interest? Are there any new and/or conflicting studies available?
This article aims to provide an overview of the different types of study designs that are available — looking at the strengths and limitations of each, how and when they should be used, including the strength of the study. In a follow up article we will focus on other aspects that affect the strength of scientific evidence.
Not all research is equal. That's why you should always interpret studies with caution. What type of study is it? Who is the author? Are there any conflicts of interest? Are there any new and/or conflicting studies available?
Free Download
Trialed & tested for best results
Optimized for nutrition
Never feel hungry
Track all macros including net carbs
Scan products
Create your own meals
Track ketones, blood glucose & lipids
Stay hydrated with water tracking
Track your mood & energy levels
Calculate your ideal fat, protein & carb intake
Set any goal: weight loss, maintenance or weight gain
Your macros update based on your progress
Monitor your macros, water intake, mood & energy
Body weight, body fat and body measurements
Ketones, blood glucose & lipids
Expert articles to help you make informed choices
Guides & free diet plans
New daily content
Complete Keto Diet guide
Integrated shopping basket
Restaurants & guide to eating out
Free Download
1. Systematic Reviews
Strength of evidence: HIGH
These types of studies are the highest level of scientific evidence. They are classed as secondary research articles, including meta-analyses, and review papers. A systematic review is a critical assessment and evaluation of all research studies that address a particular treatment or disease area. The review should be based on a peer-reviewed protocol so that they can be replicated if necessary. A high quality systematic review will aim to:
- Identify all published and unpublished evidence within a particular area
- Select studies or reports that are of high quality evidence
- Assess each study or report based on its quality
Once all of the studies are collected, the quality of the studies and the results undergoes rigorous examination. Interpretation of the findings within a systematic review should be balanced and impartial. Accurate presentation of all errors and bias contained within the studies must be displayed.
Many systematic reviews also include a quantitative pooling of data, which is termed a meta-analysis. A meta-analysis combines the results of many studies using statistical methods to calculate an overall or ‘absolute’ effect of a treatment or intervention. For example, a meta-analysis detailing all studies looking at low fat vs low carb diets — pooling the results to establish which has the better treatment effect.
Not all published systematic reviews are produced with proper care; therefore, the findings should still be interrogated before accepting the conclusion of a paper.
2. Randomized Controlled Trials (RCTs)
Strength of evidence: HIGH
These types of studies are the ‘gold standard’ of clinical research. The study randomly places participants in two or more groups to test the effect of a particular treatment vs placebo or a standard treatment.
Information from the control group allows the researchers to see whether the new treatment(s) are more or less effective than the current standard treatment. The randomization ensures that the results of the trials are not biased.
Blinding of subjects will also help to mitigate any potential bias. An RCT can be double blind, single blind or open. A double-blind study means neither patient nor study researcher know which treatment the patient has been assigned.
Although this is the gold standard of clinical research, they can carry certain limitations. Often the sample sizes of these studies are small (less than 20 people). This can skew the results of the study, making a treatment option appear better than it actually is. In order for the results of an RCT to be translated into the real world, a large sample size with a mix population group is required to show a meaningful clinically significant effect.
3. Cohort Studies
Strength of evidence: MEDIUM
These types of studies typically observe large groups of individuals over time. Their exposure to certain risk factors are recorded, helping to highlight possible causes of disease. They can be prospective (where they look forward and collect data) or retrospective (where they look back at already existing data).
Although similar to case-controlled studies, cohorts are considered as strong observational studies because they contain larger amounts of people and follow them for a longer period. However, these studies again cannot determine causation so interpretation must be with caution.
4. Case-Controlled Studies
Strength of evidence: LOW to MEDIUM
These types of studies are retrospective involving two groups of subjects, one with a particular condition (“cases”) and one without (“control”). The aim of the study is to look back and determine an attribute or exposure that caused the problem or made it worse/better, compared to the control group.
For example, researchers may take a group of people with lung disease and those without. An exposure such as smoking is then applied and the rate of smoking in each group is determined. Researchers use this to quantify the risk of smoking vs non-smoking based on the results of both of the groups.
Again, like case-reports, case-controlled studies can help to inform further studies and evidence. However, as they are observational, there can be no identification of confounding factors and causality.
5. Cross-Sectional Studies
Strength of evidence: LOW
This study is an example of an epidemiological study — designed to examine the relationship between a disease and a particular population and point in time. For example, a sample of males between the ages of 50-55 years have their prevalence of heart disease assessed.
Unlike cohort or case controlled studies, cross-sectional studies cannot determine any causality or effect. The design intention is to give prevalence insights into population groups. They often carry a lot of bias depending on how the data collection methods. The role of these types of studies is for descriptive statistics and hypothesis generating only.
6. Case Reports
Strength of evidence: LOW
This study usually features a written record on a particular subject. The report will focus on observations of one or two patients and the impact of a particular intervention on them.
This type of study can help with insights into a disease or condition, informing any other research that is required. However, it cannot determine causation, only correlation — meaning the conclusions are weak about why a certain intervention helped or did not help.
Since the reports only focus on 1-2 people, reproduction of the results have to be in a larger population before firm conclusions are drawn.
7. Animal and Cell Studies
Strength of evidence: LOW
These studies focus on the investigation of how a particular intervention behaves in animals or particular cells. This is basic primary research intended to be the starting point or rationale for carrying out work within humans.
Although often discounted, these studies can be useful at predicting effects also seen in humans. However, how an animal responds will be different to how a human will respond, therefore the results cannot be directly translated.
The intention should be to take the findings from these studies and use it to inform clinical studies in humans. Combining these types of studies with more robust studies will add more weight to a scientific argument or claim.
8. Expert Opinions and Expert Reviews
Strength of evidence: LOW
This type of information usually means one person’s opinion on a particular topic. This can be anything from an article through to an editorial piece.
Generally ranked as the lowest in terms of being scientifically robust. However, if an author of an opinion piece is an expert within a field, then their words may be of value.
The sign of a good expert review will contain multiple references within their writing, backed by credible science whilst pointing out the limitations of their own research/writing.
Not all research is equal. Here's a comprehensive guide to help you read, understand and trust studies. Learn all about the different grades of scientific evidence.
Do you like this post? Share it with your friends!
Let us know what you think, rate this post!
