How to Think About Explainability in Surgical AI

Deck: An explanation is useful only when it is faithful to the model, understandable to its user, and connected to a real review task.

Evidence status: Evergreen educational analysis based on established Surgical Data Science principles. Not clinical guidance.

Estimated reading time: 9–12 minutes.

Tags: explainability, interpretability, interfaces.

The issue beneath the headline

Heatmaps, examples, concept scores, and language rationales reveal different aspects of behavior. None automatically proves that the model used clinically appropriate evidence.

Surgical AI discussions often compress several different questions into one: whether a signal can be predicted, whether the prediction transfers, whether a user can interpret it, and whether acting on it improves anything that matters. Separating those questions is not a technicality. It prevents a benchmark result from being mistaken for evidence of clinical readiness.

The operating room is a particularly demanding setting for this reasoning. Procedures unfold over time, visibility changes, anatomy varies, and team decisions depend on information that may never appear in the video. Data are also produced within institutions that have different equipment, governance, documentation, and case mixes. A model can be technically competent and still be inappropriate for a proposed use.[1,2]

A concrete way to see the problem

A heatmap over Calot's triangle may reassure a viewer, but it should be tested by changing that region and observing whether the prediction responds coherently.

This example shows why the unit of analysis matters. Frames from one operation are highly related. Labels may be operational definitions rather than direct observations of a clinical state. A model may use a visually convenient proxy, and a strong average score may hide the specific conditions in which that proxy breaks.

The first question should therefore be: what evidence is actually available in the input? The second is: how was the target defined? The third is: where was the relationship tested? Only after those questions should architecture and performance improvements dominate the discussion.

Why this matters

The distinction matters for both research efficiency and clinical responsibility. Teams waste time when they optimize a benchmark before confirming that its target maps to a useful question. They also create avoidable risk when they present a retrospective output as though it were validated decision support.

Clinicians can improve projects by defining events, edge cases, consequences, and workflow constraints. Engineers can improve them by exposing assumptions, testing shift, measuring calibration, and designing systems that recognize unsupported inputs. Neither group can replace the other. Surgical AI is strongest when the clinical definition, data representation, model, interface, and evaluation are designed as one connected system.

What to watch for

Watch for explanations selected only because they look plausible, no user study, unstable outputs, and claims that explanation compensates for weak validation.

Also examine what the study does not report. Common omissions include independent case counts, excluded recordings, annotation disagreement, center-specific performance, false alerts per procedure, prospective latency, and the exact role of the user. Missing detail does not prove poor work, but it limits what a reader can conclude.

Be cautious with words such as understands, assists, prevents, autonomous, and clinically useful. Each implies evidence beyond simple label prediction. A careful article can still describe promising technical progress while stating that external validation, prospective testing, human-factors work, or regulatory review remains necessary.

What better evidence would look like

Better evidence starts with a precise intended use and a dataset that reflects the variation relevant to it. It separates patients and procedures across splits, documents provenance, reports difficult cases, and uses metrics connected to the consequences of error. External validation should cross a named boundary such as hospital, device, surgeon group, or time period.

For systems intended to operate live, silent prospective evaluation can test data flow, latency, missing inputs, and domain shift before users rely on outputs. Human-factors studies can then examine whether information is understood, whether it changes behavior, and whether it creates alert burden or automation bias. Clinical benefit requires an appropriately designed comparative study; it cannot be inferred from retrospective accuracy alone.

Practical question for readers

What decision can the user make more safely because this explanation is present?

Try answering the question without using the words AI or accuracy. Describe the user, the moment, the input, the output, the possible response, and the consequence of an error. If those elements remain vague, the project probably needs a clearer intended use before it needs a larger model.

Closing perspective

The goal is not to demand a clinical trial for every exploratory model. Early research needs room to test ideas. The goal is to label evidence honestly so that feasibility, validation, integration, and impact are not confused. A prototype can be valuable because it reveals a measurable signal or a better experimental method. It becomes more credible, not less, when its limits are explicit.

Surgical AI will advance through accumulation of well-defined questions, representative data, reproducible methods, external tests, and careful integration. Hype shortens that chain in language. Good editorial work restores the missing steps.

References

1. Surgical Data Science — from Concepts toward Clinical Translation. https://doi.org/10.1016/j.media.2021.102306
2. Surgical Data Science: A Consensus Perspective. https://arxiv.org/abs/1806.03184

One additional test is to ask how the conclusion would change if the easiest ten percent of cases were removed. If performance depends heavily on routine, well-framed examples, the result may still be scientifically useful, but the intended claim should narrow. Error distributions usually teach more than a single average.

The same caution applies to newer foundation and multimodal models. Their broad pretraining may improve adaptation, yet scale does not define clinical labels, obtain consent, prevent leakage, or establish safe use. Those responsibilities remain with the people designing and evaluating the system.

One additional test is to ask how the conclusion would change if the easiest ten percent of cases were removed. If performance depends heavily on routine, well-framed examples, the result may still be scientifically useful, but the intended claim should narrow. Error distributions usually teach more than a single average.

The same caution applies to newer foundation and multimodal models. Their broad pretraining may improve adaptation, yet scale does not define clinical labels, obtain consent, prevent leakage, or establish safe use. Those responsibilities remain with the people designing and evaluating the system.