It’s not just me. Really, it’s not. An article in the October 2010 issue of Australasian Psychiatry By Christopher Ryan, Olav Nielssen, Michael Paton, and Matthew Large has put very nicely the case against risk assessment. The authors use an analogy with the insurance industry (where the practice of risk assessment originated) to illustrate the problems with risk assessment in psychiatry, and I think it’s an excellent article.
Really it boils down, for me, to individuals versus groups. I discussed this in my very first Midweek Medicine post, drawing an analogy with road traffic fatalities: we can identify very clearly people who fit into higher-risk groups for crashing on the road, but we simply cannot identify which individuals from those groups will be the ones who crash and die.
Their insurance analogy is related but somewhat more comprehensive:
“Insurance policies share risk among policy holders. Each policy holder pays a small defined cost, in the form of a premium, in return for a larger compensatory payout in the event of an adverse event. Insurance companies manage the sharing of risk by setting different premiums for different categories of policy holder. These categories are determined according to the calculated probability of the adverse events occurring and the magnitude of potential loss.
…
“In conducting their business however, insurers make simple, reliable estimations of the probable total number of crashes or thefts within each group sharing the same risk. They do not attempt to predict and identify which individual policy-holder will make a claim, and insurance is not a way of reducing car accidents or thefts.“
(emphasis mine)
The second point there is the same as mine: that we can make predictions about populations and groups much much better than we can about individuals. When a clinician “does a risk assessment” we do not predict the likelihood that patient will cause or come to harm. In fact we are simply assigning them to categories (low, medium, or high risk). No matter how much we “might feel, intuitively,” that we are predicting something, we are most assuredly not.
Still, what’s the harm, right? Better than nothing, right? Well … no. The bit that this insurance analogy adds to mine of road traffic crashes/fatalities is that of individuals paying a premium, which in psychiatry, is:
“…the personal and financial cost of additional treatment experienced by the patient, including coercive treatment imposed on ‘high-risk’ patients, whether or not they will actually cause or experience harm.”
(again, emphasis is mine)
To illustrate their points the authors chose to look at the MacArthur Violence Risk Assessment Study, and the risk assessment tool derived therefrom. They noted that there are numerous actuarial tools – which have been shown to be better than clinical assessment – for both violence and suicide. They chose this particular one as it is “supported by a substantial body of research and is perhaps the most proven risk assessment instrument devised for use in general psychiatric settings.” They examined the data supporting this tool:
157 patients – 55 categorised as high risk and 102 as low risk.
Observed for 20 weeks, during which 27 committed an act of violence.
71% classified correctly as being high or low risk.
Area under the “receiver operator curve” was 63%, which they say translates to “a 63% chance that a randomly picked patient who went on to commit an act of violence would have a higher score than a randomly picked patient who did not…”
The instrument had a sensitivity of 67.8% and specificity of 72.1%
Of the 157 patients there were 19 true-positive categorisations, 36 false positives, 9 false-negatives, and 93 true-negatives.
The authors of this paper go on to discuss in turn the effects of true-positives, false-positives, false-negatives, and true-negatives:
The first group (true-positives; those who were identified as being high-risk, and did go on to commit a violent act) pay a higher “premium” but do potentially gain in terms of more assertive treatment and additional input, which could potentially improve the course of their illness and reduce (not eliminate) the likelihood of adverse events.
The second group (false-positives) are those who have been assessed as at high-risk of violence, but do not go on to commit any violent act. They pay the same high premium as the former, but with no benefit in terms of avoiding harm. Possibly it could be argued they benefit in terms of more assertive treatment, but (1) this has to be weighed up against the more restrictive and coercive care, possibly higher doses of medication than they need, and (2) it can also be argued that directing extra resources to this group deprives patients who are clinically similar but not categorised as “high-risk”.
The third group, false negatives, are those who were categorised as low risk but did go on to commit some violence. They pay a low premium, but missed out thereby on treatments which might have improved their conditions and/or prevented an adverse event (possibly)
The fourth group is the true-negatives: those who were categorised as low-risk and did nothing violent. They pay a low premium and get no benefit.
Already it appears to me to be pretty clear that there’s definite potential harm from risk assessment, for little to no gain. To ram the point home further, the authors plug in some numbers (the 1 in 10,000 annual incidence of homicide by patients with treated schizophrenia) to the sensitivity and specificity values for the MacArthur tool. Basically the result is:
“In other words, 4117 patients would have to be detained or otherwise managed in a homicide-proof fashion for a year to try to prevent just one of those patients committing a homicide, and yet one in every 22,421 patients assessed to be ‘low-risk’ would commit a homicide in that period.”
An NNT of 4117 is pretty piss-poor, frankly. Especially when the treatment at issue is restrictive and coercive, and carries its own risks.
We have to stop doing this. Politicians and health departments have to stop mandating it, but we as a profession have to state clearly that it doesn’t work, it’s wasting scarce resources and is completely unjust, and it has the potential to cause harm by overtreating some and missing danger in others. As a profession we all have to realise this, and then help politicians and wider society to realise it.
All too often I hear that someone can or cannot “guarantee their safety” – after they’ve been assessed say following deliberate self-harm, or presenting with depression or suicidal thoughts … or pretty much anything, actually. The notion that anyone can “guarantee their safety” is quite frankly, stark raving bonkers (says the shrink). 
In Which Heavy Metal is Evil, Because Suicide!!1!1
Oh we have ourselves another fool, smothering a plate of idiocy in moron sauce. This time it’s Dr Katrina McFerran, from the University of Melbourne (who, as Guitar World noted, is not Tipper Gore, though you’d be forgiven for thinking so) who is trumpeting the dangers of teh heavy metalz – with a beautifully ironic URL there …
According to Dr McFerran, teenagers who listen to heavy metal might be placing themselves at risk for depression, anxiety, and suicide:
Ummm… Haven’t we been here before?
Oh noes! The metal brootalz! Won’t someone think of the children?!
Ok, having convincingly declared my bias…
In none of the “articles” about this (all of which – other than the above-linked from Guitar World – are essentially reprinting the press release, so I won’t even bother linking to them) is there any link to, or even mention of, a paper – published, in press, or even submitted. So, in similar fashion to the moronic “study” about caffeine-induced hallucinations that, incidentally, also came from a university in Melbourne, I cannot look at the actual methods, results, or analysis engaged in by Dr McFerran.
The lack of logic, and the alarmist tone, displayed in the press release however, do not give me cause to think the study would be particularly rigorous, or itself logical.
… She said. She did. She said.
Let’s deconstruct this and see if it means anything. “Not uncommon” for “some”… to me is saying that “some” people (however many that is) but not others, will “not uncommon”ly (however often that is) listen to music for 7-8 hours a day. It doesn’t tell me who make up that “some”, or what music they’re listening to, or why, or the context … Or indeed anything useful.
And really, what it is designed to say, is not what I take from it; what it is meant to do is create the image of the children (“won’t someone please think of the children?!”) spending too much time listening to music (which is after all the devil’s pastime – unless it’s whatever music Dr McFerran is keen on in the Melbourne Conservatory of Music.
Probably not the metal brootalz.
So … Hang on. Back up the metal-bashing bus just a minute.
You’ve just suggested, have you not, that the arrow of causation is the other way? That depressed/anxious/suicidal kids might listen to metal more than do the non-depressed/anxious/suicidal kids? … not that the heavy metal is what’s making them depressed, anxious, and suicidal. They are perhaps seeking solace, support, expression, whatever, in that music.
But that’s a bad thing if it’s heavy metal.
Because, y’know … the brootalz.
Or perhaps it’s just a spurious association. I wonder, how did Dr McFerran arrive at her startling conclusions? Why, by:
Well hold the phone. That’ll do it for me. Especially without any substantive details (who they selected, how they selected, how they controlled, what they asked…); they might just spoil a good story.
You know, I’m almost surprised there was no mention of satanism and back-masked messages.
I hate, hate, hate this sort of crap: putting out melodramatic press releases with no data, no mention of evidence to back up the sensationalist claims made – and then dutiful churnalists just regurgitate it without a scrap of critical thought.
So. Unless Dr McFerran does produce a good-quality peer-reviewed paper that shows solid evidence to back up what she says, I say: enjoy your metal brootalz. \m/