Suppose we’ve trained a system to make medical diagnoses.
The system, perfectly trained, has specific recall and accuracy. (recall: of all the diseases that have been submitted, what percentage did I recognize? precision: of all those that I identified as diseases, in what percentage were they really?)
Accuracy and recall are a short blanket. it does not depend on a “calibration” of the machine, it is not an inherent error. it is a matter of statistical properties.
We know the system, perfectly trained, will make incorrect predictions.
Of course, even aspecialist doctor makes incorrect predictions. For this reason his predictions are according to science and conscience, follow a decision-making path and sometimes he might be called to motivate his decisions.
Now suppose we provide the diagnostic machine to the doctor.
Machines, in fact, can scale in speed and quantity to levels that a human cannot reach. But when they make a prediction, they cannot explain why.
Let’s now suppose that the machine makes a prediction that differs from the doctor’s opinion.
What will the doctor do?
In my opinion, the incentive for the doctor is to confirm the prediction of the machine, even if it contradicts his own opinion.
One could argue that we could just legally recognize the probabilistic nature of the machine’s outcomes, thus providing doctors with a way to cover themselves if they contradict the machine. But would it really be an incentive for an in depth analysis by the doctor ? I don’t think so. it would just be less troublesome and quicker to agree with the machine.
Now suppose there is a patient with a serious acute illness.
- Suppose the machine makes the correct prediction, the doctor makes a mistake and his decision is the final one. The patient gets worse and the doctor is called to justify himself. Why has the doctor stubbornly taken his position? Even the machine indicated that there was a pathology!
- Suppose the machine makes the wrong prediction and the doctor sees it right but conforms to the prediction of the machine. The patient gets worse and the doctor is called to justify himself. Why did the doctor confirm the prediction of the machine? Didn’t he recognize the symptoms? He can always argue that in science and conscience he didn’t consider them sufficient to contradict the machine.
In both cases, the doctor’s position is more defensible if it conforms to the prediction of the machine, even if he thinks otherwise, even if the doctor knows that the machine produces false predictions. In any case, he can lift his hands and say “also the machine confirmed this”.
The obvious conclusions are that doctors are redundant, that a certain level of risk (due to recall/precision of AIs) is inevitable and we should live with it.
Now let’s suppose we knowingly alter some of the predictions of the machine (keeping track of them) and inform the doctor that some of the predictions that are communicated to him, are actually the opposite of what the machine really determined (of course he doesn’t know which ones). We call these predictions “poisoned”. (and those that are not, “sincere”).
The machine makes a prediction, but the doctor is told a poisoned prediction. The machine believes that there is a pathology but the doctor is told that the machine believes that there is no pathology.
What can the doctor do now? When he believes that the prediction of the machine is wrong, will he confirm it ? And what if the machine’s diagnose was poisoned ? He must tell what he truly thinks, in science and coscience.
At this point we have a situation in which
- the prediction of the machine is sincere and that of the doctor is in agreement. in this case the diagnosis is confirmed.
- the prediction of the machine is sincere and that of the doctor is discordant. the case is highlighted and another opinion is required.
- the prediction of the machine is poisoned and that of the doctor is discordant. in this case the diagnosis of the doctor is confirmed.
- the prediction of the machine is poisoned and that of the doctor is in agreement. the case is highlighted and another opinion is required.
Such “another opinion” could be by a doctor or it could be a second machine, with different algorithm and training, absolutely non poisoned.
This is a simple example of how the concept of redress by design could be implemented with prediction poisoning.
A single doctor is a “one opinion system”. We invented the idea of science and conscience and all audits on correct behaviors, requiring explanations, to build incentives to avoid negligence and guilt.
A doctor with a machine is a “two opinions system” where (I believe) the machine’s opinion, as argued above, would tend to prevail. Lab technicians can already confirm this tendency. Furthermore, inspections and audits might not be effective on systems that cannot explain their motivations (like humans do).
The redress idea exposed above is that when the two opinions (machine and doctor) are in disagreement, a third one needs to be involved to resolve the dispute
The act of “poisoning” only serves the purpose of keeping doctors attentive, so that they avoid just conforming to the machine thereby being able to spot some incorrect predictions. It is a mean to improve effectiveness in the detection of incorrect predictions.
Someone could object that such a procedure would eliminate the benefit of speed and scale increase allowed by the application of AI. As a matter of fact, since the physician knows that his predictions are subject to an AI-assisted examination procedure, he can operate at a higher speed because his activity is augmented with an AI based error control system. In this case a desirable result is obtained: the functions of the doctor are augmented by the AI and not replaced.
Arguably, such a procedure will be costly. In some states some variations could be envisioned, for example by involving patients in the decision. Let’s say the machine’s prediction is 0.65 confident. Before going for a third opinion, the doctor and the patient could have a conversation about the costs of listening or not listening to the machine and about any reasons the doctor thinks the ML might be wrong and let the patient decide after having received all due informations.
One final comment about the “poisoning” word I’ve chosen to express this concept. It may appear somewhat ungraceful (even more when used in an example about healthcare) but I think the term is quite appropriate as the term is already used with a comparable meaning in network security (ARP poisoning, DNS cache poisoning) .
