Monday, 30 November 2015

A Mum's Guide To Dodgy Science: From Petri Dishes To People

Time for the second instalment of my "how to spot a dodgy science/health story" series. This one is more well known than Publication By Press Release and is one of the first questions to ask of any article claiming a new discovery kills, cures or otherwise alters your health:

Was this study done on humans like me?

All too often the answer to that question is no. The most notorious examples are probably the "Miracle Drug/Food Cures Cancer" headlines, where it turns out that the cancer being destroyed isn't in a person, it's just a bunch of cells in a dish. Certainly killing the contents of a petri dish may be an important first step in discovering a new treatment but the vast majority of stuff that can do that never makes it much further. Lots of things can kill cancer cells in a dish, domestic bleach works well, but I wouldn't recommend it as a medication.

Cartoon from the wonderful xkcd

But even if your new wonder drug does turn out to be safe, that doesn't mean it'll work in people. Firstly there is the issue of quantity. Cells can be given huge doses but humans might need to consume impossible amounts for it to do anything. Plus human beings are wonderfully, confusingly, unpredictably complex. Far more so than a single type of cell in a carefully controlled plastic environment. There are an almost infinite number of reasons why something that works "ex-vivo" (in the dish) will be a total flop in a full blown person.

It's not just cells in dishes that are the problem though. Moving on to whole animals isn't a guarantee things will work in humans. Fruit flies, Zebra fish and nematode worms are all popular "model organisms" but quite clearly very different to us. Even calling in our mammal cousins can't tell us everything.

I was reminded of this today when I read this particularly groan worthy story which claims that being born by Cesarean section could affect brain development. It draws this conclusion from a study done in mice and says that those mice born by C section showed increased brain cell death after birth and this may be linked to these mice having a different microbiome to those who got a good dose of mum's bugs on the way out.

(I'm not quite sure which is the most glaring error in this story, although the fact that the abstract it's based on says exactly the opposite, that cell death decreased not increased in the CS group, is probably a good place to start. I was seriously tempted to just write a whole blog pulling this story apart but I'm going to try to restrain myself and get back to the point....)

Fairly obviously mice aren't people. Laboratory mice definitely aren't the equivalent of a diverse human population. Neither the article or the abstract tell us what type of mice were used in this study but they were most likely one of many specifically bred laboratory strains with carefully selected and controlled characteristics. So something that is true in one strain of mice might not even be true of all of them.

This becomes even more important when the story starts speculating about the involvement of the microbiome (the bacteria that live in and on an individual). Different strains of mice have different microbiomes. Even identical strains raised in different labs will vary thanks to diet, environment and how strictly they are isolated from other germs etc. The microbiome is a fascinating field for study and will probably open up a whole new world of information about our bodies, but at the moment it is very much in it's infancy. We can't even accurately identify all the bacteria that are there, never mind how they interact. So it's a whopping great Olympic triple jump of a leap to suggest that something seen in baby lab mice, which may or may not be linked to their microbiomes, can tell us anything about the affect of differing microbiomes in humans.

Oh and it gets worse. Even humans aren't always a good model for other humans. Especially if the humans that need medical treatment are female. Historically, far fewer women have been included in clinical trials. There are lots of diseases that affect men and women differently so treatments tested mostly on males don't always work as well or in the same way in females. Our bodies also change more and more often. Hormone levels change throughout life and even over the course of a month. If you're pregnant, forget it! Since the horrors of Thalidamide almost nothing gets tested on pregnant women, (although there are some calls to change this).

Then there are the confounding variables of culture, diet, geography etc. etc. These could be subtle, biological interactions or something more obvious: Say a study claims that eating Kale makes you healthier because people who eat loads of it have less heart disease - does that mean there is something miraculous in the green stuff or could it just be that, in the experiment, the people who ate lots of Kale all lived in communities where everyone had a great diet and was very physically active?

Salad, the bringer of joy and laughter (in stock photo land)

None of this is to say that we should throw up our hands and ignore all scientific advice. But if you're going to make an alteration to your life based on it, then it's worth checking how likely it is to apply to you as an individual. Cells in a petri dish or mice in a lab are interesting, and certainly useful in early research and safety testing, but they aren't as complex as you. other humans are more comparable but if they are a different sex and/or living a totally different lifestyle you might want to think about what else could be going on.

Finally (I'm beginning to think I'll be ending every one of these posts like this), just don't put too much faith in health advice from newspapers, facebook shares and, of course, random people's blogs.


Ok I couldn't leave it alone, here's some other stuff I got annoyed with in that story about C section mice:
1- It's just a meeting abstract, nothing has been peer reviewed yet (see post on Publication By Press Release!) 
2- Even if there is an increase / decrease in brain cell death - how do we know the C section mice are worse off? This doesn't seem to have been tested, it's assumed natural is better but it's theoretically possible that the opposite is true.
3-It's one single (unpublished) study, many if not most individual papers turn out to be flawed so it should be taken with a big 'ole pile of salt until it's repeated by others.
4- The whole microbiome bit just seems to have been chucked in because it's a bit of a buzzword at the moment, especially with anything related to C sections. I'm amazed they didn't get epigenetics in there too.
5- The article speculates that brain differences may also be caused by C section babies not getting the "rush of hormones" during normal labour. Which may be true of lab mice having planned C sections but a large number of human c sections happen well after labour began *coughs* sometimes 34 hours after *coughs*
6- Wonderfully, for an article so full of extrapolation, speculation and a few baffling errors, it ends with the phrase: "having the full facts available before a choice is made is important."

Well yes it is.

*face palm*

Wednesday, 25 November 2015

Don't Fear Sunday's Child - A Look At Today's Headlines On Weekend Births

If  I was pregnant at the moment I suspect I'd find today's headlines pretty scary. They report that giving birth at the weekend, something most women can not control or predict, puts babies lives at risk.

It would be great if we could dismiss this as yet another case of media hype, a rouge press release or even (as has been suggested) a conspiracy theory aiming to turn the public against junior doctors.  But having read the stories, the press release and the paper they are all based on, I think it's all a bit more complicated than that.

The first thing I want to say though, to any readers who are pregnant and scared  by this news is please, don't be.

The headline story that there is a binary difference between weekend and weekdays simply isn't true. Take a look at the data below, taken from the BMJ paper this is all based on. The death rate is highest for babies born on a Thursday. In fact the most notable trend isn't a peak at the weekend but a drop on Mondays and Tuesdays. This trough drags down the average across all weekdays, making the weekend average seem much higher  by comparison.  This data doesn't tell us that lives would be saved if weekends were more like weekdays. It shows we'd be better off if every day was more like Tuesday (this may be the only time in all of human history when that sentence has been true!)

Plot taken (axis adapted for clarity) from the BMJ paper 

So should we instead  be terrified of giving birth on a Thursday?  The paper doesn't really address this issue. It merely speculates  (unconvincingly in my opinion) that babies born on a Thursday may be more likely to die a few days later because of the care they receive over the weekend.

This study also doesn't prove that weekends are more dangerous because the consultants are all off playing golf or something. It found no difference in death rates between units that met the recommended consultant levels and those that didn't. In fact of all the outcomes looked at, including injuries, infections and hospital re-admissions, the only one where weekend consultant numbers were important was the number of perineal tears. Yet even this isn't necessarily an indicator of poorer care when consultants aren't around. It is quite widely believed that obstetricians are more risk averse than midwives and more likely to recommend emergency caesareans. The decreased number of tears when there are more consultants may be due to an increase in possibly avoidable C sections. I expect many mums will have an opinion on which is the lesser of those particular evils!

I am, by the way, just speculating on the fewer tears means more C-sections thing. But that brings me to the fundamental flaws in this paper and how it's being reported to the public. The news stories and (unusually) the paper itself all seem determined that there is indeed a "weekend effect". They gloss over or ignore the data showing that this effect starts on Wednesday, and, having shown that consultant cover isn't an issue, offer only speculation on potential causes. Much of these seems to have been thought up to make the data fit the hypothesis. For example, while it may be the case that outcomes for babies born on Thursday are influenced by their care the following weekend, there is no real evidence for that. The data doesn't tell us how long babies were in hospital after the birth. Similarly it doesn't tell us how long the mother was in labour (and in the hospital) prior to birth, a period where good care is crucial. A large proportion of the women giving birth on a Monday and even Tuesday will have been receiving weekend care during labour, yet these are actually the days with the lowest death rates. Why? We don't know, but perhaps the question of why babies born at the weekend fare worst should be re-framed so we're asking what we can learn from the Monday and Tuesday births.

This study is fascinating and will certainly be a valuable contribution to the academic literature on the subject, but it generates more questions than it answers.

For example, the birth rate is 21% lower at weekends. Even if elective c sections (which are very low risk and almost always done on weekdays) are removed, it is still 11% lower. Why? Could a reduction in the number of Inductions (IOL) at the weekend be involved? There is increasing evidence that IOL for overdue babies improves a number of outcomes, but this isn't covered int he paper so we don't know.

If consultant cover makes no difference what about the number of midwives, junior doctors, pediatricians etc.? What about their work patterns and moral?

What about dozens of other factors that we just haven't thought of? The simple answer is we don't know.

What we do know, if we actually look at the data not the headlines, is that there is no reason for women to be afraid of going into labour on the weekend. I've heard several health care professionals today express concern that women who start to get contractions, who's waters break or who feel a reduction in their baby's movements could delay going to hospital to avoid the weekend effect and in so doing risk something worse. The prospect of giving birth can be scary enough as it is, women need real evidence not hype. I find myself saying this over and over again, we are grown up adult human beings - why are we so rarely treated like it when there are column inches up for grabs (and perhaps in this case academic impact factors)?

Clearly more research is needed on the questions raised by this paper, it could be the starting point for something really useful, maybe lifesaving. But right now, it and the media response to it, only risk causing yet more fear and confusion.

If you are pregnant and reading this, just take this one thing away: The NHS is flawed and under funded, sometimes it makes mistakes, sometimes it just doesn't do things as well as it should, but if things get difficult in childbirth it has your back. The vast majority of women will take home wonderful, healthy babies, whatever day of the week they decide to turn up on. I, and my babies, wouldn't be here without it.


In case you missed the other links - the original paper including the full version of the diagram above can be found HERE

EDITED TO ADD (26.11.15) I've just seen this response  from the RCOG it raises a few additional and important points that I didn't include in this post and is well worth a read (I'm neither an obstetrician or a statistician and didn't want to include anything I wasn't sure I had right!) An important one is that for 86% of still births the baby dies before birth, often several days before so the day they are actually born doesn't tell us what happened or when.

Friday, 6 November 2015

A Mum's Guide To Dodgy Science: Publication By Press Release

Sometimes, it sucks being a grown up. I've actually had enough time to write some blog posts recently but other events have left me with a lack of motivation for anything other than eating cake and reading Tudor crime novels under a blanket (it's a fairly niche slump). Anyway, in an attempt to drag myself out of it, before any more Lutheran priests meet a grisly end, I've decided to revisit some old blog posts as a way of sharing tips on how to spot dodgy science and health stories. The reproductive lives of the Tudor queens may have been subject to much debate and gossip, but hey, we modern Mums have to deal with medical advice from the Daily Mail!

First up in what will hopefully be a series of posts - Publication By Press Release:

Note on stock photos: anything featuring mobile phones gets very dated, very quickly.

What is supposed to happen when a scientist completes a study or set of experiments it that she/he writes it all up and sends it to a scientific journal who get other experts in the field to check it. They then usually ask for a few corrections and maybe some extra data and decide if the work is good enough to be published. It's a process known as peer review and there are any number of flaws in it but it is the best available system for scientific publishing and the one all scientists are supposed to endure  engage with.

Increasingly though I see cases where this isn't happening. A good example came almost exactly a year ago when a whole raft of news outlets got all excited over initial results from a study hoping to increase breast feeding rates by giving breast feeding Mums shopping vouchers. You can read my post from the time here but essentially this wasn't a story about a published paper, it was based on the author planing to give a presentation at a meeting. As I said at the time, you could in theory get up and say any old nonsense you wanted at a scientific meeting so simply planning to speak at one tells us nothing about the quality of what will be said. Of course if you do talk rot you'll face some pretty stiff questioning from the audience afterwards and I hope that happened in this case, I certainly had a lot of queries about the quality of the study! Interestingly, a year on, I still can't find any published results. To be fair the peer review process can be achingly slow but so far all that seems to have come out of this much touted study is two papers about how the study team sought opinions from a small number of women and health care professionals. Not very impressive, and really not worthy of repeat slots on breakfast news!

Breast feeding is totally absolutely always just like this *snort*

Instead of waiting until the study had been assessed and the results checked, someone, somewhere (I could probably take a guess at who), decided to get some media attention and put out a press release and because the media just luurve stories about health/ breast feeding/ women's bodies/ judging poor people that press release was gratefully picked up.

So who's to blame for this kind of non-story? It's tempting (presumably unless you are one) to just blame the journalists, looking for sensational stories and to hell with the truth. Sometimes that's fair, In an ideal world a journalist receiving a press release about some scientific research would check the published data and seek out alternative expert opinions. But we're not talking about getting this done in time for tonight's print run or the 10pm news bulletin anymore. If it's news, it needs to be up online and on social media in hours or perhaps minutes and it needs to compete in an almost infinite fight for attention. So perhaps it's not surprising that journalists often depend on a press release from a respectable institution being accurate. In the immortal words of Whitney Huston: It's not right, but it's ok.

Sing it Whitney

Actually no it's not ok, but I had that song in my head for some reason, anywaaay,

Sometimes the blame for this sort of thing needs to be shared out. If the journalists are relying on university press releases to be accurate, then university press officers need to make sure they are. But again there is a conflict. Not putting out a press release because something isn't published yet isn't going to get you any publicity, so there is an understandable temptation to take any opportunity to get your institution talked about. The scientists themselves should be checking that press releases are an accurate reflection of their work but this also doesn't always happen. Scientists are ordinary people they're busy, they may be under pressure to publish or at least get some publicity and some ordinary people just really like their own opinions and/or their 15 minutes or fame.

Which brings me on to another old blog post. In this case there was a published paper, but it bears little resemblance to the the press coverage it generated.  According to the Telegraph and (oh yes) The Daily Mail, we should all co-sleep/bedshare with our children until they are three. Not because they like it or it's just the only way for anyone to get some sleep sometimes, but because not doing so results in lasting damage to children's hearts and harms brain development causing behavioral problems later in life. Pretty damn scary news if, like me, your kid was actually pretty happy in her own cot most of the time. But the paper says absolutely nothing about these dreadful long term problems, it looked at the heart rate of a few newborns being held by their mother and compared that to their heart rate when they were in a crib. Presumably one of the authors got rather carried away and extrapolated wildly from that to the three years or brain damage headlines. The media were happy to go along with it because hey look! Another opportunity to tell Mum's we're doing it wrong!

Actually if Mum's are so dreadful at this baby sleep thing perhaps Dad's should be in charge of it, all of it *evil cackle*

All this bugs me. There is loads of good, interesting, important science going on out there. A small proportion of which offers real, useful advice to improve people's health and well being. But when small, unchecked, unproven studies start fighting for column inches or the stories that reach the public are wild fantasies based only vaguely on the evidence, then how can anyone know what to believe and act on? The valuable stuff gets crowded out or people simply feel drowned in information and give up on all of it.

Clearly most parents have one or two other things to do and don't have time to conduct their own personal scientific literature review for every single bit of health advice so here are a few things to look out for in a news story which may suggest you're looking at Publication By Press Release rather than a properly checked over bit of research.

The Spotters Guide To:

Publication By Press Release

If a health news story meets any of these criteria, it may be a bit dodgy:
  • There is no mention of or reference to a scientific paper
  • The story says the information will be presented at a meeting
  • The story says the information came from an interview or other media article
  • The only quotes and opinions come from people who worked on the research

Basically just don't ever put too much faith in one off bits of information and advice in the media and online.

Like for example information and advice on random blogs.


Next up - more exaggerations, From Petri Dishes To People.