/ Health

Ask for evidence: don’t let scientific claims go unchallenged

What if we could challenge the scientific and medical claims made on products, adverts and websites? That’s exactly what the charity Sense About Science wants us to do – will you join them in asking for evidence?

If we don’t ask for evidence, people and companies will get away with making misleading claims.

At Sense About Science, we know the cost of false claims. Take the husband of a multiple sclerosis sufferer who told us that he wished he’d known the questions to ask a clinic offering unproven stem cell treatments. He might have spent the thousands of pounds, and the last few months of his wife’s life, on a holiday instead of chasing false hope.

We’ve launched an Ask for Evidence campaign to encourage everyone, every consumer, citizen and patient, to ask for evidence for every claim they encounter.

We’re bombarded with claims

Scientific and medical claims appear everywhere: on advertising material, product websites, advice columns, campaign statements, celebrity health fads and policy announcements.

Where there is regulation, in advertising or trading standards for example, regulators are overworked and overwhelmed. Organisations get around adjudications with small tweaks, and the same claims crop up again and again.

What are groups like patient support forums supposed to do when people pose as members to promote unproven treatments – police every post made on their forums? That probably wouldn’t work, even if we wanted it to. The only solution is to give everybody the questions to ask for themselves.

You don’t need to be a scientist

This doesn’t mean you have to go back to school and become an expert in everything. You don’t have to become an epidemiologist to ask penetrating questions about claims made about mobile phones and cancer, for example. You can ask whether evidence exists, how conclusions are reached, whether there have been fair tests, whether results have been peer reviewed, replicated or challenged.

Asking questions in whatever way we can is a conversation that society needs to start having. People are already doing this – organisations like Which? scrutinise the evidence for product claims, medical research charities make it their business to take on claims that hit the headlines – but this is fragile, fragmented work. It would be much less fragile if everybody joined in.

There are stories of people’s evidence hunting on our website. Tamlyn asked companies for evidence on supplements his mum uses. Sarah asked a company to back up its claims that head massages would improve childrens’ academic performance.

These evidence hunters have described how surprised they were that companies didn’t expect to be asked and often don’t have evidence to hand – when Jennifer asked Marks & Spencer’s for the evidence behind their “MRSA resistant” pyjamas it took an age to get an answer (the answer was that tests are underway).

The more of us who ask for evidence, the more every company, politician, exaggerating scientist, advertiser and journalist will expect to be asked about any claim they make. You don’t need to be a scientist to scrutinise claims and hold people making them to account.

Comments
Guest
John Symons says:
25 October 2011

We CAN challenge, through the Advertising Standards Agency, but ambiguous words and statements are problematic. Essential oils are not essential like vitamins, they are just an “essence” that you can squeeze out of something. “Reduces the appearance of fine lines”. Does that mean it hides them with some human Polyfilla, or that it stops them appearing so often? I once challenged a car advert that claimed they paid the VAT, because they didn’t as it would have been illegal. They actually reduced the price, by 15 115ths at the time. Unsuccessful.

Guest

Good article.

Do you think that we can do the same for spurious environmental claims aswell? But not just for companies selling their wares but for governmental taxing policies such as CO2 emissions and why cars should be taxed that way?

Guest

As a scientist, pseudo-scientific claims annoy me especially when made to provide headlines, scare the public or sell a product.
The ASA is toothless – a request not to use the same advert made several months later is useless and hey cant enforce the request anyway.

Maybe Which? could set an example by expanding on some of its testing methods.
In this months magazine it claims that some electric heaters used 50% more energy than others to heat the test room to 15C. As this is scientifically impossible Which? must mean “heat the room until some test points reach 15C” rather than heat the whole room uniformly.

And then there is the Which statement that cars travelling faster produce less pollution as they are on the road for a shorter time.

So Which? lead the way with scientifically accurately and well researched statements and make your test methods easily accessible.

Guest
Neel says:
25 October 2011

“So Which? lead the way with scientifically accurately and well researched statements and make your test methods easily accessible.”

I second that… too often it seems that Which? allows journalists with little scientific/statistical training to jazz up the articles… presumably for greater exposure in the national media and to raise the profile of Which? which I proudly proclaims they are funded by subscribers like myself but are not above using cheap PR stunts to get into the national press.

Great post raraar

Guest

I had spotted the curious claim about electric heaters, mentioned by rarrar. Some explanation is definitely needed or I will start to doubt what I learned as a schoolkid.

I would be very grateful if additional information about testing methods on the Which? website, so that those who want more information can access it easily, though I appreciate that giving more detail in the magazine is probably not warranted. As a scientist I frequently wonder how many samples of a product were tested for reports and whether findings are statistically significant.

Guest

Hello Rarrar, Neel and Wavechange,

Thanks for your comments about how we present our test results in the magazine. As we’re a consumer magazine we attempt to write our research in a way that everyone can understand, but of course, we also make sure that this isn’t in anyway misleading.

Our tests are not intended to be large scale statistical analysis of the performance of products, they are meant to be a fair comparative test of products doing what might quite reasonably be expected of them.

But I was disappointed to read that our article on portable heaters did not give you all of the information you needed, to know that we were measuring the room air temperature, not the fabric of the room itself. Unfortunately we didn’t have enough space in the article or footnotes to tell you how our measurements were made.

However now that I know you are keen to hear the detail I can tell you that we used 16 temperatureprobes laid out in a 4×4 grid, 1.35 metres above the ground (the height of a sitting person) to measure the air temperature. The mean temperature, recorded every few seconds by these probes was used as the mean room temperature. The testing was conducted in accordance with the conditions set out in BS EN 60675.

We also compared the temperature 5, 10 and 15 cm off the ground just 0.5 metres in front of the grill.

How we test is also expanded on which.co.uk here:
http://www.which.co.uk/home-and-garden/heating-water-and-electricity/guides/how-we-test-electric-heaters/
so I hope that’s of use!

We also try our best to present our test results and research as accurately as possible, though we can’t control how other publications interpret and cover it.

Guest

As a person who is not a scientist but who has studies scientific subjects in the distant past (as well as being naturallly picky) I should point out that it is possible for one heater to use more electricity/energy to heat a room than another, assuming that there is a net heat gain from (heat input minus heat lost through walls, ceiling and floor). The net heat gain per minute should dictate how long an appliance takes to heat the room and the rate of heat loss by the room will impact the overall efficiency.

However, given that the room is claimed to be well insulated the difference in rate of heat output would have to be pretty big to give a 50% difference in energy useage.

I can see that a device could take 50% longer to heat a room if the heat output was lower, but take 50% more energy? Where is all the lost energy going, if it is not being output as heat?

Is it perhaps the requirement to get a stable, even temperature?
In that case a more efficient and more powerful fan could raise the room temperature much more evenly that an inefficient fan heater which is acting more as a source of radiant heat.

[I have written to Which? directly over this article, which I found deplorably lacking in useful information.
I also note also that you did not follow the methodology outlined in the linked article as you specifically did not measure how well the heater maintained the room temperature despite the article claiming “We also measure how well the heater can keep the room temperature stable by varying the temperature of the ‘outside’ to stimulate mild, cold and very cold weather conditions – the best heaters will keep the room temperature stable, whether they have to work a little or a lot to do it.”]

Guest

Oh, go on, while I am being picky 🙂

“And then there is the Which statement that cars travelling faster produce less pollution as they are on the road for a shorter time.”

True in some circumstances – there is an optimum speed for greatest fuel efficiency (often around 55mph?) and a car cruising at that speed will cause less polution than one crawling along in a traffic jam.

However the implication is that if you are travelling at 140mph you cause less pollution than if you are travelling at 70mph.
If this is true, can I have one of those cars, please? 🙂
Sadly, I don’t think it is.