Should AI systems in mental health settings have a duty to warn?
A brief item that I could not resist leaving a quick comment on. The Atlantic posted a fascinating story last week on a machine learning program that could help make more accurate psychiatric diagnoses. The system currently in place is a “schizophrenia screener” that analyzes primary care patients’ speech patterns for some of the tell-tale verbal ‘tics’ that can be a predictor of psychosis. For now, as the author points out, there are many weaknesses with widespread deployment of such a system because there are so many cultural, ethnic, and other differences in speech and behavior that could throw the system off. But still, the prospect of an AI system playing a role in determining whether a person has a mental disorder raises some intriguing questions.
The lawyer in me immediately thought “could the Tarasoff rule apply to AI systems?” For those of you who are normal, well-adjusted human beings (i.e., not lawyers), Tarasoff was a case where the California Supreme Court held that a psychiatrist could be held liable if the psychiatrist knows that a patient under his or her care poses a physical danger to someone and fails to take protective measures (e.g., by calling the police or warning the potential victim(s)).
Now granted, predicting violence is probably a much more difficult task than determining whether someone has a specific mental disorder. But it’s certainly not out of the realm of possibility that a psychiatric AI system could be designed that analyzes a patient’s history, the tone and content of a patient’s speech, etc, and comes up with a probability that the patient will commit a violent act in the near future.
Let’s say that such a violence-predicting AI system is designed for use in medical and psychiatric settings. The system is programmed to report to a psychiatrist when it determines that the probability of violence is above a certain threshold–say 40%. The designers set up the system so that once makes its report, its job is done; it’s ultimately up to the psychiatrist to determine whether a real threat of violence exists and, if so, what protective measures to take.
But let’s say that the AI system determines that there is a 95% probability of violence, and that studies have shown that the system does better than even experienced human psychiatrists in predicting violence. Should the system still be designed so it can do nothing except report the probability of violence to a psychiatrist, despite the risk that the psychiatrist may not take appropriate action? Or should AI systems have a freestanding Tarasoff-like duty to warn police?
Given that psychiatry is one of the more subjective fields of medicine, it will be interesting to see how the integration of AI in the mental health sector plays out. If AI systems prove to be, on average, better than humans at making psychiatric diagnoses and assessing risks of violence, would we still want a human psychiatrist to have the final say–even though it might mean worse decisions on balance? I have a feeling we’ll have to confront that question some day.
