OrthoSense™: Designing Clinical AI That Clinicians Will Actually Overrule
A decision-support tool that is never questioned is a decision-making tool wearing a disguise. Designing for productive disagreement.
The most dangerous clinical AI is not the one that is wrong. It is the one that is right often enough that nobody checks the times it is not.
Automation bias is well documented and it does not respond to training reminders. It responds to interface design, to workload, and to whether the system makes its own uncertainty legible at the moment of decision.
Design commitments
- Show the uncertainty, not just the answer. A confidence band that changes the reading is worth more than a number in a footer.
- Surface the basis. Which features drove this output, in clinical language.
- Make disagreement cheap. One click to override, with the reason captured as structured data.
- Feed overrides back. Override patterns are the highest-signal monitoring data a clinical system produces.
Under the EU AI Act, clinical decision support in this category sits squarely in high-risk territory, and Article 14 expects oversight to be effective rather than nominal. Designing for productive disagreement is not just good clinical practice; it is the cheapest route to demonstrating the obligation is met.