/ Health

Ask for evidence: don’t let scientific claims go unchallenged

What if we could challenge the scientific and medical claims made on products, adverts and websites? That’s exactly what the charity Sense About Science wants us to do – will you join them in asking for evidence?

If we don’t ask for evidence, people and companies will get away with making misleading claims.

At Sense About Science, we know the cost of false claims. Take the husband of a multiple sclerosis sufferer who told us that he wished he’d known the questions to ask a clinic offering unproven stem cell treatments. He might have spent the thousands of pounds, and the last few months of his wife’s life, on a holiday instead of chasing false hope.

We’ve launched an Ask for Evidence campaign to encourage everyone, every consumer, citizen and patient, to ask for evidence for every claim they encounter.

We’re bombarded with claims

Scientific and medical claims appear everywhere: on advertising material, product websites, advice columns, campaign statements, celebrity health fads and policy announcements.

Where there is regulation, in advertising or trading standards for example, regulators are overworked and overwhelmed. Organisations get around adjudications with small tweaks, and the same claims crop up again and again.

What are groups like patient support forums supposed to do when people pose as members to promote unproven treatments – police every post made on their forums? That probably wouldn’t work, even if we wanted it to. The only solution is to give everybody the questions to ask for themselves.

You don’t need to be a scientist

This doesn’t mean you have to go back to school and become an expert in everything. You don’t have to become an epidemiologist to ask penetrating questions about claims made about mobile phones and cancer, for example. You can ask whether evidence exists, how conclusions are reached, whether there have been fair tests, whether results have been peer reviewed, replicated or challenged.

Asking questions in whatever way we can is a conversation that society needs to start having. People are already doing this – organisations like Which? scrutinise the evidence for product claims, medical research charities make it their business to take on claims that hit the headlines – but this is fragile, fragmented work. It would be much less fragile if everybody joined in.

There are stories of people’s evidence hunting on our website. Tamlyn asked companies for evidence on supplements his mum uses. Sarah asked a company to back up its claims that head massages would improve childrens’ academic performance.

These evidence hunters have described how surprised they were that companies didn’t expect to be asked and often don’t have evidence to hand – when Jennifer asked Marks & Spencer’s for the evidence behind their “MRSA resistant” pyjamas it took an age to get an answer (the answer was that tests are underway).

The more of us who ask for evidence, the more every company, politician, exaggerating scientist, advertiser and journalist will expect to be asked about any claim they make. You don’t need to be a scientist to scrutinise claims and hold people making them to account.

John Symons says:
25 October 2011

We CAN challenge, through the Advertising Standards Agency, but ambiguous words and statements are problematic. Essential oils are not essential like vitamins, they are just an “essence” that you can squeeze out of something. “Reduces the appearance of fine lines”. Does that mean it hides them with some human Polyfilla, or that it stops them appearing so often? I once challenged a car advert that claimed they paid the VAT, because they didn’t as it would have been illegal. They actually reduced the price, by 15 115ths at the time. Unsuccessful.

Profile photo of dean

Good article.

Do you think that we can do the same for spurious environmental claims aswell? But not just for companies selling their wares but for governmental taxing policies such as CO2 emissions and why cars should be taxed that way?

Profile photo of rarrar

As a scientist, pseudo-scientific claims annoy me especially when made to provide headlines, scare the public or sell a product.
The ASA is toothless – a request not to use the same advert made several months later is useless and hey cant enforce the request anyway.

Maybe Which? could set an example by expanding on some of its testing methods.
In this months magazine it claims that some electric heaters used 50% more energy than others to heat the test room to 15C. As this is scientifically impossible Which? must mean “heat the room until some test points reach 15C” rather than heat the whole room uniformly.

And then there is the Which statement that cars travelling faster produce less pollution as they are on the road for a shorter time.

So Which? lead the way with scientifically accurately and well researched statements and make your test methods easily accessible.

Neel says:
25 October 2011

“So Which? lead the way with scientifically accurately and well researched statements and make your test methods easily accessible.”

I second that… too often it seems that Which? allows journalists with little scientific/statistical training to jazz up the articles… presumably for greater exposure in the national media and to raise the profile of Which? which I proudly proclaims they are funded by subscribers like myself but are not above using cheap PR stunts to get into the national press.

Great post raraar

Profile photo of wavechange

I had spotted the curious claim about electric heaters, mentioned by rarrar. Some explanation is definitely needed or I will start to doubt what I learned as a schoolkid.

I would be very grateful if additional information about testing methods on the Which? website, so that those who want more information can access it easily, though I appreciate that giving more detail in the magazine is probably not warranted. As a scientist I frequently wonder how many samples of a product were tested for reports and whether findings are statistically significant.

Profile photo of Victoria Pearson

Hello Rarrar, Neel and Wavechange,

Thanks for your comments about how we present our test results in the magazine. As we’re a consumer magazine we attempt to write our research in a way that everyone can understand, but of course, we also make sure that this isn’t in anyway misleading.

Our tests are not intended to be large scale statistical analysis of the performance of products, they are meant to be a fair comparative test of products doing what might quite reasonably be expected of them.

But I was disappointed to read that our article on portable heaters did not give you all of the information you needed, to know that we were measuring the room air temperature, not the fabric of the room itself. Unfortunately we didn’t have enough space in the article or footnotes to tell you how our measurements were made.

However now that I know you are keen to hear the detail I can tell you that we used 16 temperatureprobes laid out in a 4×4 grid, 1.35 metres above the ground (the height of a sitting person) to measure the air temperature. The mean temperature, recorded every few seconds by these probes was used as the mean room temperature. The testing was conducted in accordance with the conditions set out in BS EN 60675.

We also compared the temperature 5, 10 and 15 cm off the ground just 0.5 metres in front of the grill.

How we test is also expanded on which.co.uk here:
so I hope that’s of use!

We also try our best to present our test results and research as accurately as possible, though we can’t control how other publications interpret and cover it.

Profile photo of littlegreycat

As a person who is not a scientist but who has studies scientific subjects in the distant past (as well as being naturallly picky) I should point out that it is possible for one heater to use more electricity/energy to heat a room than another, assuming that there is a net heat gain from (heat input minus heat lost through walls, ceiling and floor). The net heat gain per minute should dictate how long an appliance takes to heat the room and the rate of heat loss by the room will impact the overall efficiency.

However, given that the room is claimed to be well insulated the difference in rate of heat output would have to be pretty big to give a 50% difference in energy useage.

I can see that a device could take 50% longer to heat a room if the heat output was lower, but take 50% more energy? Where is all the lost energy going, if it is not being output as heat?

Is it perhaps the requirement to get a stable, even temperature?
In that case a more efficient and more powerful fan could raise the room temperature much more evenly that an inefficient fan heater which is acting more as a source of radiant heat.

[I have written to Which? directly over this article, which I found deplorably lacking in useful information.
I also note also that you did not follow the methodology outlined in the linked article as you specifically did not measure how well the heater maintained the room temperature despite the article claiming “We also measure how well the heater can keep the room temperature stable by varying the temperature of the ‘outside’ to stimulate mild, cold and very cold weather conditions – the best heaters will keep the room temperature stable, whether they have to work a little or a lot to do it.”]

Profile photo of littlegreycat

Oh, go on, while I am being picky 🙂

“And then there is the Which statement that cars travelling faster produce less pollution as they are on the road for a shorter time.”

True in some circumstances – there is an optimum speed for greatest fuel efficiency (often around 55mph?) and a car cruising at that speed will cause less polution than one crawling along in a traffic jam.

However the implication is that if you are travelling at 140mph you cause less pollution than if you are travelling at 70mph.
If this is true, can I have one of those cars, please? 🙂
Sadly, I don’t think it is.

Profile photo of rarrar

Just worked out that the header for this Conversation was actually written by “Sense about Science” and not Which?

The “We” does not refer to Which? !!!
Now that is misleading, I though for a moment that Which? were claiming that the “Ask for Evidence ” Campaign was a Which? one.

Shouldnt all conversations on a Which? site be hosted by a Which? employer .

Profile photo of Ian

I second rarrar’s posts (above); Which? should make it abundantly clear when inviting those not from the organisation to write an article. The sentiments might be laudable, but the lack of transparency by Which? is rather worrying.

Profile photo of Patrick Steen

Hello Rarrar and Ian, I’m sorry that you felt it wasn’t clear that this was a guest post and not written by Which?

We try to make it as clear as possible, with the author’s name and where they work at the top of the article, as well as in the second sentence of the introduction “That’s exactly what the charity Sense About Science wants us to do” and in the text “At Sense About Science, we know the cost of false claims” which includes a link to their website. You can also click on the author’s name so that you can find more information about them: https://conversation.which.co.uk/author/slane/

We’ve had over 20 guest Conversations, including Martin Lewis, Vince Cable, Nicky Campbell, the FSA, Greenpeace UK and Chris Huhne. We try to find people who are experts in their field to write guest Conversations in order to open up a debate. They add to the variety of the website and they also give you a chance to talk directly to companies and politicians.

You can find all of our guest Conversations here, since every guest post has the tag ‘guest’ on it: https://conversation.which.co.uk/tag/guest/ Thanks.

Profile photo of rarrar

Hi Patrick

The authors name, employer and tags are in smaller and fainter type than anything else !
It is not till the 3rd para that there is any indication that this isnt Which? speaking:

….”At Sense About Science, we know the cost of false claims. “….

I am sure that many people skim through articles without carefully reading every word and analysing the grammar !

I didnt realise there had been so many guest contributors, maybe it would be clearer if Guest contributions were enclosed within a Which? authored wrapper to make it clear that this isn’t Which? talking .
It would also allow details of Which?’s involvement in the specific campaign to be highlighted.

Dont get me wrong I fully support this campaign (200% !!!), maybe Which? magazine could include the odd para on inappropriate scientific claims in their reviews and reports where appropriate. The reports often list the factors to consider when buying a product so maybe a list of factors or claims to ignore ?

Profile photo of wavechange

I agree with rarrar. I was aware of the guest contributors but did not realise how much expertise is being brought in.

Profile photo of rarrar

Maybe we could have a Conversation about the format and content of Which? Conversations ?
There is a real danger that the debate on these matters on this and other conversations is masking the more “on topic” contributions.

Profile photo of Patrick Steen

Thanks again for your comments – the bolded wrap at the top of these Conversations are written by us here at Which? Convo in order to introduce the post and the author – other websites inviting guest posters do not flag them in this way. However, we’ll look into ways of making it even clearer for those of you who are confused by guest posts.

Also, the reason why we have guest posts here on Which? Convo is so that we can invite experts in their fields to discuss topics that we think will interest you, but Which? might not be working on at this time.

However, this is getting off-topic from Sense About Science’s Ask for Evidence campaign. So if you’d like to talk about this further or any other thoughts about how the site works or looks, please email us: https://conversation.which.co.uk/contact-us/ Thanks guys.

Profile photo of wavechange

I’m a scientist too, and I am disgusted by the way that so many companies make unfounded scientific claims about their products. Advertising seems to revolve around dishonesty and misrepresentation, so I start by assuming that claims are false!

The Sense about Science website contains useful information and avoids unnecessary technical terms, so much of it is useful for the general public. It might be useful to refer to this site next time someone posts a bogus but common ‘scientific’ information on this forum.

Profile photo of wavechange

Thanks Victoria. I have just spotted your reply.

I don’t think it’s particularly relevant here but it would be good to know how many samples of a product are tested. That could make a big difference in assessing reliability. CFLs are an example of one product where some people where many have experienced poor reliability, despite the fact that your tests and the experience of many users (including me) is that they are reliable.

Internet John says:
9 December 2011

Supermarkets clearly don’t think there’s anything at all in the research to suggest a link between cigarettes and cancer.
Thant’s why they have a little collection box for cancer research in front of 5,000 packs of cigarettes.

Eric Gendle says:
18 December 2011

Can I recommend that everybody should read Ben Goldacre’s book “Bad Science” for a sensible and humorous overview of “Detoxing” and other examples of pseudo science and its practitioners.

Profile photo of wavechange

You can read some pages of this book on the Amazon website.

I trust my own judgement to recognise bad or dubious science, having a personal interest in public understanding of science and how it is misrepresented. I was initially disappointed that the book is focused on a narrow range of issues, but realise that it would not be practical to cover much more and the topics chosen are ones that the public can relate to.

I found it interesting to read the positive and negative reviews of the book and some of the critical comments on the reviews (both positive and negative) are fascinating.

Craig says:
27 August 2012

I feel to ask staff in a store is abit unfair, as they can not be expected to have evidence to back up all the claims there are, on all the millions of products they store stocks. But I do feel that any claims of a product maybe, should be required by law to be actually true, it maybe should be the manafacture(s) to prove the claims, not the staff in a store.

Profile photo of wavechange

I agree, but I don’t believe that anyone has suggested that staff in a store should be able to provide any evidence to back up claims.