How surgeons learn with AI

AI may change how operative skills are taught, assessed, and maintained. Education systems need fairness, construct validity, useful feedback, and clear boundaries around surveillance.

In this edition

• 01 — A haptic trainer must subtract the robot before it can reveal the tissue
• 02 — An AI that answers questions about minutes of surgery, not one frozen frame
• 03 — Teaching a robotic laparoscope the hidden strategies of a camera assistant

01 — A haptic trainer must subtract the robot before it can reveal the tissue

When a force sensor sits at a robotic instrument’s wrist rather than its tip, it measures much more than tissue contact. It also feels the weight of the instrument, the angle of gravity, sensor bias, mounting offsets and the torque created because the mass is not centred on the sensor. Feed that raw signal back to a trainee and the “tissue” may partly be the robot itself.

This preprint tackles that unglamorous but essential problem for RoboScope, the authors’ low-cost robotic-surgery training platform. Wrist mounting avoids the size, sterilization and durability challenges of a tip sensor. The trade-off is a compensation problem that changes as the instrument rotates.

Calibration while the robot moves

The proposed algorithm estimates force bias and the gravity vector first, then estimates the centre-of-gravity offset and torque bias. It updates those parameters recursively during non-contact movement and rejects measurements that look like external contact. Unlike methods that require a carefully scripted set of robot poses, the estimator is designed to converge during ordinary motion and then keep producing compensated contact force and torque in real time.

In the no-contact experiment, mean absolute force errors fell by more than 95% on all three axes: for example, x-axis error dropped from 5.055 N to 0.113 N. Torque errors fell by more than 91%. Worst-case errors were below about 0.58 N for force and 0.05 N·m for torque, tighter than the earlier method used for comparison.

The authors then pressed the instrument into a silicone phantom and tested a known 104 g mass, approximately 1.02 N, while rotating the instrument through 360 degrees at four speeds. The compensated magnitude averaged 1.014 N. At the fastest rotation, 72 degrees per second, mean absolute error reached 0.04 N. Those tests show that contact can be separated from changing gravitational load in this bench setup.

No trainees or surgeons were studied. There is no comparison of skill acquisition, force control, tissue handling or perceived realism with and without feedback. The silicone and reference-mass experiments validate an engineering layer; they do not show that the resulting haptic trainer improves performance or transfers to an operation. Rapid acceleration, tool exchange, sensor drift over long sessions and more complex multi-axis contact deserve further stress testing.

Still, this is exactly the kind of work a credible training device needs. Haptics is not valuable because a handle vibrates; it is valuable when the cue corresponds reliably to external interaction. The next study should pair this compensation with novice and expert users, define a force-control task, and test whether better measurement produces better learning rather than simply a cleaner graph.

Open the original source

02 — An AI that answers questions about minutes of surgery, not one frozen frame

Ask whether the critical view of safety has been achieved and one frame may be insufficient. The relevant evidence can be scattered across dissection, retraction and changing exposure. SurgTEMP is built around that temporal problem: answering open-ended questions from surgical video while keeping both fine detail and the longer procedural arc available.

Two memories for two kinds of question

The model uses the text question to select a small set of visually detailed frames for a spatial memory. In parallel, it compresses every sampled frame into a temporal memory that preserves the sequence. A Qwen2-7B language model receives both. Training progresses from perception—tools, actions, anatomy—to assessment and then reasoning, mirroring the authors’ view that higher-level judgement depends on first recognising the scene.

The accompanying CholeVidQA-32K dataset contains about 32,000 question-answer pairs across 3,855 laparoscopic-cholecystectomy segments, roughly 128 hours of video. Its 11 tasks include tool and anatomy perception, critical-view assessment, difficulty, adverse events, skill, scene description and planning. The material repurposes structured labels from CholecT50, Endoscapes and CholeScore; descriptive and reasoning answers are enriched or generated with large language models. That provides scale, but also means the benchmark partly measures agreement with a constructed answer style rather than an independent surgeon’s live judgement.

On the long CholeScore subset, where segments average 710 seconds, SurgTEMP’s LLM-judged correctness was 66.28 versus 48.34 for the strongest fine-tuned baseline. The model also led on the shorter caption and critical-view subsets. Five different LLM judges largely agreed on the ranking, with overall Kendall’s W of 0.852.

A compelling answer still needs the right to abstain

The entire study concerns laparoscopic cholecystectomy. Scores are dominated by automated language-model grading and reference-answer overlap, not blinded surgeons making decisions from model output. The paper itself notes the absence of calibrated uncertainty and abstention. That is not a minor feature: an assistant asked whether clipping is safe must be able to say that the video is inadequate.

For education, this system could become a powerful way to query a recorded case: find the frames supporting a critical-view judgement, compare an answer with expert annotation, or revisit how difficulty evolved. Intraoperative use is a much higher bar. It would require prospective evaluation, source-linked answers, explicit uncertainty, latency measurement and testing of whether the model changes a surgeon’s decision correctly. SurgTEMP shows that long-video memory materially improves a surgical QA benchmark. It does not yet show that a fluent answer is a safe second opinion.

Open the original source

03 — Teaching a robotic laparoscope the hidden strategies of a camera assistant

Good laparoscopic camera work is anticipatory. The assistant recentres before the instrument disappears, adjusts depth without magnifying tremor, tolerates a momentary fog and withdraws for cleaning when visibility truly fails. These are strategies, not isolated left-right commands.

This preprint tries to extract those strategies from expert video and use them to supervise an autonomous camera holder. Rather than train one black box to convert pixels directly into motor velocity, the authors split the problem into interpretable stages.

From camera events to reusable behaviour

Offline, the pipeline detects camera-relevant events—tool interaction, depth deviation, visibility loss and lens contamination—and connects them in attributed graphs. Clustering those graphs yields 12 recurring strategy categories, including recentering, working-distance control and visibility recovery. Three surgeons annotated a 94-minute subset of 162 segments; event detection averaged 0.86 F1, and the mined clusters aligned with expert strategy labels at a normalized mutual information of 0.77.

A Qwen2.5-VL 7B model is then trained to identify the current strategy and issue a discrete image-plane or depth direction. Importantly, it does not send unrestricted motor velocities. A conventional image-based controller executes the command while enforcing the trocar’s remote-centre-of-motion constraint. Optional speech lets the surgeon refine the desired view.

Training used 109 laparoscopic-cholecystectomy videos—50 private expert cases and 59 selected Cholec80 cases. Physical evaluation remained ex vivo: silicone stitching, porcine-intestine stitching and porcine-stomach dissection across nine trials. Compared with manual camera control in that setup, the autonomous controller reduced field-centering error by 35.26% and image shaking by 62.33%. It also demonstrated a contamination workflow; average recovery took 25 seconds, including 10 seconds for an assistant or nurse to perform the actual cleaning.

The most interesting design choice is the separation

A vision-language model can misunderstand a scene. Keeping its output at the level of discrete intent, then passing motion through a constrained controller, creates an inspectable boundary between interpretation and actuation. The event vocabulary also gives a surgeon a meaningful way to query why the camera moved.

The evidence does not yet cover breathing, bleeding, smoke, unexpected anatomy or live patient motion. Training centres on one procedure, the physical study is small, and a human still performs lens cleaning. Manual comparison in nine ex-vivo trials cannot establish workflow benefit or safety in vivo.

Even so, the paper reframes camera autonomy well. The goal is not to imitate every motor twitch; it is to learn a small library of expert viewing strategies and execute them under geometric constraints. The next study should test prospective in-vivo cases with override frequency, unsafe-view duration, task interruption and surgeon workload—not only a steadier image.

Open the original source