Foundation models enter surgery

Foundation and multimodal models promise reusable representations across tasks. The surgical question is whether that reuse survives domain shift and improves a defined workflow.

In this edition

• 01 — Giving an AI agent a memory of tissue motion
• 02 — Training on synthetic surgical captions without trusting them
• 03 — More surgical video is not enough: the mix matters

01 — Giving an AI agent a memory of tissue motion

From a flat video to an explorable scene

Ask a surgeon where traction was applied and the answer is not just a pixel. It is a point on a moving surface, at a particular moment, with a direction. This preprint tries to give an AI agent that same kind of coordinate system. From a single laparoscopic camera, the pipeline combines estimated depth, point tracking and semantic segmentation into a tracked 4D representation: 3D geometry changing over time. A multimodal language model does not absorb that representation during training. Instead, it calls tools that return distances, overlap, relative motion and trajectories, then uses those measurements to answer a question.

That design choice matters. The agent can inspect geometry rather than asking a vision-language model to infer every spatial relation from appearance alone. The authors tested the idea on 134 surgeon-annotated questions across 25 short Cholec80 clips, divided into spatial, temporal and directional tasks. For spatial localization, the tool-using agent reduced mean projected error from 90.76 pixels for the plain multimodal model to 46.54 pixels. For direction, its mean absolute error was 0.18 versus 0.83 for the strongest 2D baseline. These are not clinical endpoints, but they make a concrete case for an explicit representation when the question is about how tissue moved.

The weak point is also clinically revealing

The method did not win everywhere. On temporal intervals, the 4D agent reached an intersection-over-union of 0.39 while the 2D baselines reached 0.51. All approaches struggled with temporal questions, and variability was high. The reconstruction is assembled from monocular video without known camera parameters; specular highlights, occlusion and poor camera motion can corrupt depth or tracking. The authors also run the vision components offline with access to the full clip, so this is not an intraoperative assistant operating in real time.

For surgeons, the useful idea is architectural rather than product-ready: separate fluent reasoning from measurable scene state. A future system could show the trajectory or 3D point behind an answer, making disagreement inspectable. But before that becomes dependable, it needs metric ground truth, longer and more diverse procedures, prospective timing tests, and a clear way to expose reconstruction uncertainty rather than letting a precise coordinate disguise a bad estimate. This is a preprint and has not completed peer review.

Open the original source

02 — Training on synthetic surgical captions without trusting them

The annotation bargain

Surgical video is abundant compared with surgeon-written descriptions of what each clip contains. SurgLIME asks whether an AI can generate those descriptions cheaply without teaching the next model its mistakes. The authors start with 4,194 open surgical videos from LEMON, split them into roughly 54,000 five-second clips, filter blurred material, and ask Gemini 2.5 Flash to describe instruments, anatomy, actions, view and lighting. No clinician verifies those narratives. That is the point of the experiment—and its central risk.

A standard vision-language model treats every video-caption pair as a reliable match. If a caption confidently names the wrong structure or action, the training signal pulls the visual representation in the wrong direction. SurgLIME instead keeps the established PL-Stitch surgical vision encoder largely anchored, adds low-rank adapters to the vision and text encoders, and estimates confidence for each generated narrative. Uncertain text contributes less to the contrastive loss. In other words, synthetic prose is allowed to add a language interface, but not to rewrite the model's visual foundation without resistance.

The clearest evidence comes from the ablation. On AutoLaparo zero-shot phase recognition, an unweighted version reached 13.4% accuracy and 10.1% F1; confidence weighting raised those figures to 18.1% and 11.2%. SurgLIME also beat the expert-narration-based SurgVLP by 8.1 percentage points in zero-shot accuracy on AutoLaparo, although SurgVLP remained stronger on Cholec80 accuracy. With a linear probe, the adapted visual features retained PL-Stitch's performance and gained about one accuracy point on AutoLaparo.

Useful supervision is not verified knowledge

This is a smart answer to label scarcity, not permission to remove clinicians from the loop. The evaluation is phase recognition on two established datasets, not instrument safety, anatomy recognition or action generation. Confidence is estimated automatically; it does not prove that a caption is clinically correct, and the paper itself calls the narratives unverified. The authors also manually reviewed clips after automatic blur filtering, so the pipeline is not entirely human-free even if the text generation is.

For surgical teams, the distinction is important: synthetic captions may be useful as a weak training signal when the system is designed to distrust them. They are a way to organize visual patterns at scale, not a source of operative truth. Before such representations support clinical-facing reasoning, they need expert error audits at the level of anatomy and action, tests beyond phase labels, and evidence that the confidence mechanism catches the errors surgeons actually care about. This work is a preprint awaiting peer review.

Open the original source

03 — More surgical video is not enough: the mix matters

SurgRec begins with a familiar foundation-model promise: train on much more surgical video, then reuse the representation across many tasks. The corpus is genuinely broad—10,535 videos and 214.5 million frames spanning laparoscopy, endoscopy, cataract and robotic surgery. The authors train two self-supervised variants, one based on masked reconstruction and one on predictive latent modeling, and evaluate them with consistent video-level splits across 16 downstream datasets.

The important experiment is hidden behind the scale. Surgical video collections are not naturally balanced. Common endoscopic and laparoscopic footage can dominate, while cataract or robotic examples become statistical footnotes. SurgRec clusters clip embeddings and samples across that space, then mixes the balanced unlabeled pool with a clinical core from 32 public datasets. With the masked-autoencoder model, this raised macro accuracy by 5.43 points over generic VideoMAE and improved the weakest domain score from 19.09% to 22.23%. Compared with the same surgical model trained without balancing, the balanced version gained 24.57 points in the robotic domain and 14.76 in endoscopy, while laparoscopy fell by 2.39 points. That trade-off is more informative than a single average: distribution design decides who benefits from scale.

The paper also tests general-purpose vision-language models in zero-shot fine-grained recognition. Across the 16 datasets, the fine-tuned SurgRec-MAE reference outperformed the tested VLMs, and small prompt changes moved VLM results in inconsistent directions. For surgeons, that is a useful reminder that an articulate general model is not automatically a reliable temporal observer. Recognizing a phase or gesture often depends on subtle evolution across frames, not on naming what is visible in one image.

This remains a preprint and a representation study, not evidence for deployment. Many source videos are public educational material that may favor clean highlights and omit complications. Some benchmark subsets are small; the downstream models are fine-tuned and evaluated retrospectively; and no prospective multi-centre workflow is tested. The results nevertheless sharpen a practical question for anyone building surgical AI: not only “How many hours did the model see?” but “Which procedures, sites, difficult moments and minority patterns were allowed to shape it?” A larger corpus can amplify its blind spots. SurgRec’s contribution is showing that curation and sampling are part of the model, not housekeeping around it.

Open the original source