Becoming critical consumers - research and the media

Becoming Critical Consumers: Research and the
Media
Elizabeth Blackmore, M.A. and Wendi Rockert, M.Ed.
Over the past year, newspaper and magazine columns have headlined the “effectiveness”
of the newest generation of weight loss fads – the low carbohydrate diet. A recent study,
however, showed that both low fat diets and low carbohydrate diets produced comparable
rates of weight loss in the longer term. In May, a Toronto research team reported that
strict dieting, dangerous weight control practices and eating disorders are occurring at
alarming rates amongst grade school children. We are presented with information about
food, weight and eating disorders in the media every day, and often make important
decisions about our own lives based on what is communicated. At times the information
reported in one study is in conflict with information from another study. How do we
determine the validity of what we read and hear?

Develop a critical stance
We rely on the media to inform us of current trends and to transmit the latest research
results, despite the fact that we are not formally trained to evaluate the information that
appears in newspapers, magazines, on the radio, TV or internet. This is especially true in
the case of articles that cite research evidence to support their position. Because most of
us are not scientists or statisticians, we often believe what is written in print without
questioning some basic assumptions about the research.
As consumers of information, we need to adopt a critical stance when we read about
research findings. Although the interpretation of complex statistical analyses is well
beyond the scope of this article, there are some basic questions/issues to consider which
can help you to decide how much emphasis to place on the results that are being reported.
This will make you a more informed consumer and ultimately make you more able to
arrive at personal decisions which may incorporate what you learn from the media.
The research process
The general purpose of research is to seek answers to a specific question or problem of
interest, known as an ‘hypothesis’. Generally, researchers attempt to answer two main
types of questions: 1) What is happening?; and 2) Why is it happening? To explore a
particular research question, the researcher chooses a group of interest that they wish to
investigate, known as a ‘population’ (e.g., women with an eating disorder). Ideally, the
researcher would like to study the entire population of interest, but normally this is not
possible due to factors such as geography, time constraints and financial resources.
Instead, the researcher chooses a subset of people to represent that larger population,
National Eating Disorder Information Centre ~ 200 Elizabeth St. 7ES-421, Toronto.ON. M5G 2C4 www.nedic.ca known as a ‘sample’ (e.g., 100 women from the Toronto area with an eating disorder).
The hope is that the sample will be a true representation of the larger population of
interest.
In all research, there are certain procedures that can be followed to try to ensure that the
sample is representative of the larger population. These procedures give us confidence
that the study is valid (i.e., measures who and what it is supposed to measure) and
reliable (i.e., results can be replicated consistently within your own study, or by other
researchers). The onus is on the researcher to design and carry out a valid and reliable
research study.
Research studies can take several different forms such as observation, an individual case
study, or the most common form, surveys/questionnaires. Upon completion of a study,
the researcher compiles the findings, or ‘results’, for presentation to a larger audience.
This larger audience can include other research professionals or the general public, and
involves the communication of results of research through various media channels.
Usually, research findings are presented in the form of statistical percentages. For
example, research has reported that 60% of high school-aged females are dissatisfied
with their bodies. This is a very straightforward way of transmitting information to the
public, but the interpretation of such percentages is influenced by a number of factors
such as the following:
1. Sample size – 60% can be calculated as 6/10 people, 60/100 people, 600/1000 people,
and so on. As consumers, it is important that we are aware of how many people took part in the research study (called the ‘sample size’). The results from a survey of 1000 people are much stronger than from a survey of either 10 or even 100 people because, as the number of people who take part increases, there is a greater chance that the sample will better represent the larger population of people whom the sample was chosen to represent.
2. Source of information – In understanding the meaning of a particular percentage
reported, we also need to consider where the information originated from, or its source. Is the information being cited from a scientific journal or a university publication, or is it from a ‘popular-culture’ magazine such as a women’s health and beauty magazine? Often information is presented in a sensationalized way in popular magazines in an attempt to catch your attention or grab headlines. Moreover, these magazines often conduct surveys on their readers who may be a very select group of people (sample). The results of questionnaires administered to this group may not be readily generalizable to the population. If the research is from a scientific or academic source, look at the specifics of their sample too – e.g.: is it exclusively young college students, or is it a community sample of young adults?
3. Impact of research – Researchers expose their findings to specific statistical tests to
National Eating Disorder Information Centre ~ 200 Elizabeth St. 7ES-421, Toronto.ON. M5G 2C4 www.nedic.ca ensure that the results did not occur solely by chance (significance testing). However, even when a research finding is considered statistically significant, this does not always mean that it will have an immediate clinical impact, such as affecting the treatment or management of an illness. For example, results of a research study testing a new medication for the treatment of anorexia were recently summarized in major newspapers and on television. The study reported an improvement in patients’ eating disorder symptoms, and this is what the media reported to the general public. However, this study had a very small sample size of 14 patients, so at this point in time it is difficult to draw strong conclusions from the results. This is an example of a significant research finding that must be reduced, or replicated with more patients before conclusions about its effectiveness can be made.
It is also important to be aware of the language that is used to describe research findings.
Specifically, there is a very important difference between saying there is a ‘relationship’
(or correlation) between what the researcher has measured, versus saying that one thing
‘caused’ another. For example, research shows that physical activity is often ‘related’ to
anorexia. Many individuals with anorexia engage in large amounts of physical activity,
but we don’t know if anorexia causes one to be active or if the activity can play a role in
developing anorexia. This is very different from results that have found that a restricted
diet can ‘cause’ binge-eating behaviour.
Be an active consumer of information

It is our job as “conscious consumers” to examine information that is presented to us.
Often an article actually prompts more questions than it answers. Use your critical
thinking skills to ask yourself what the implications are of a particular finding. For
example, some years ago the media reported some studies that found that the medication
Prozac was effective in reducing bingeing and purging. Before running out immediately
to get a prescription for this drug, there are a number of questions that one should ask.
How much of a decrease in symptoms did the Prozac group experience? Are there any
side effects to taking Prozac? What are the long-term effects? Did participants with
particular characteristics benefit more or less than others?
Follow up on your questions by talking to people who may know more about the
particular subject matter. Your family doctor or psychotherapist, the local mental health
clinic or hospital, and of course NEDIC, are often good sources of accurate information
about research on weight and eating disorders. Don’t be afraid to ask questions and
check out other sources to get the information you need. Those who are familiar with
computer databases such as Medline or Psychlit can look up related research themselves.
Often the main findings are summarized in the newspaper article, and if you are
interested it its useful to get a copy of the original article to review yourself.
In this age of technology, people often rely on the Internet to provide information to National Eating Disorder Information Centre ~ 200 Elizabeth St. 7ES-421, Toronto.ON. M5G 2C4 www.nedic.ca questions they have. Although the Internet can be a valuable resource there is no quality
control over what appears online. Many of the websites contain individual personal
accounts or partially substantiated research reports with little of the information included
that would help you to decipher the validity of the report. For example, many individuals
suffering from eating disorders post their personal stories on the Internet or cite
percentages without any references. It is important not to interpret a personal account as
indicative of the disorder in general – rigorous research is needed in order to be able to
make statements about a larger sample of individuals.
The final recommendation is to recognize that all research studies contain a margin of
error. To simplify a fundamental principal of statistics, even when a study reports that a
particular finding is statistically significant, there is the possibility that this finding
occurred by chance – typically this error rate is 5 percent.
In sum, there is much to learn from research studies that investigate issues related to
dieting, weight and eating disorders. As you adopt a more informed and critical stance,
you will be more able to meaningfully decipher the take home message of the research.
Questions to ask yourself when reading a research finding
Ÿ Is this article from a reputable source?
Ÿ How many people participated in the study?
Ÿ Who is in the sample studied?
Ÿ What are the characteristics of the sample?
Ÿ Are the questions asked (the hypotheses) useful and/or reasonable?
Ÿ Has this study been replicated?
Ÿ Do the conclusions seem reasonable?
Ÿ Did the researchers account for other factors that may have influenced the findings?
Ÿ Remember - Don’t make a significant personal decision based on the results of a

Where to go for more information
Ÿ Medical or community personnel who work in the area
Ÿ Family doctor or psychotherapist
Ÿ Databases such as Psychlit or Medline
Ÿ Internet – But be careful to use interpret this information with caution and a critical
Ÿ An introductory text on psychology & statistics

NEDIC 2004 www.nedic.ca
National Eating Disorder Information Centre ~ 200 Elizabeth St. 7ES-421, Toronto.ON. M5G 2C4 www.nedic.ca

Source: http://nedic.ca/sites/default/files/becoming-critical-consumers-research-and-media.pdf