Seeing the operation

Surgical video is becoming a structured source of workflow, anatomy, action, and skill information. This edition separates useful capabilities from benchmark performance alone.

In this edition

• 01 — Open surgery finally enters the training set
• 02 — Before AI marks a safe zone, it must know where the operation is
• 03 — An error detector improves when it knows the intended gesture

01 — Open surgery finally enters the training set

Surgical video AI has a camera problem. Most public datasets look through an endoscope: a contained field, a small instrument set, a familiar visual grammar. Open surgery looks different - wider views, exposed anatomy, several hands, changing light and the surrounding team. Yet it has barely existed in the datasets used to teach multimodal models.

SurgAtlas changes the scale of that conversation. The preprint assembles 15,291 public YouTube videos, totalling 2,391 hours across 18 specialties and more than 5,000 procedure types. Of those videos, 6,182 show open surgery, 4,913 laparoscopy and 3,051 robotic procedures. That makes open surgery 40.4% of the corpus rather than a footnote.

One corpus, three kinds of evidence

The authors do not pretend every video contains the same quality of supervision. Narrated lectures are transcribed and refined into time-linked captions. Videos with useful on-screen text contribute phases and steps. Metadata-only videos contribute coarse procedure-level summaries. A separate pipeline turns local captions into question-answer pairs about actions, sequence, risk anatomy and operative reasoning.

Scale comes partly from automation: speech recognition, optical character recognition and language models clean or generate annotations. Seventeen human annotators verified that retained videos contained operative content. For a stricter evaluation set, two surgical residents independently reviewed 2,960 generated question-answer pairs; 2,890 were unanimously accepted across groundedness, timing, answerability, usefulness and absence of unsupported clinical claims. That high agreement supports the held-out subset, not every one of the roughly 400,000 generated pairs.

The team then fine-tuned an eight-billion-parameter Qwen model. On established benchmarks, it reported a macro F1 of 68.2% for Cholec80 phase recognition and 77.7% average accuracy across three Critical View of Safety criteria. On the expert SurgAtlas questions, overall judged accuracy was 42.9% for minimally invasive clips and 39.3% for open clips. Operative reasoning remained a particular weakness relative to larger general models.

Diversity is not the same as clinical representativeness

YouTube unlocks breadth but also determines what the model sees: cases selected for teaching or publication, edited footage, uneven camera quality and unknown institutional or patient sampling. The annotation tiers also carry different evidential weight. A sentence grounded in narration is not equivalent to a summary inferred from metadata, and an LLM-verified answer is not equivalent to surgeon-authored truth.

The most revealing ablation may be the regime gap. A model trained only on minimally invasive footage scored 29.1% on open-surgery questions; an open-only model scored 32.2% on minimally invasive questions. Even inside 'surgical video,' visual experience did not transfer cleanly.

SurgAtlas should therefore be read as infrastructure, not a clinical model. It gives researchers a much larger and more diverse starting point, especially for open surgery. It also makes the next audit unavoidable: which hospitals, operations, complications, camera positions and patient groups are still absent when public internet video becomes the atlas?

Open the original source

02 — Before AI marks a safe zone, it must know where the operation is

A coloured box over Calot's triangle can look authoritative even when the system has misunderstood the operation. SurGo-R1 is built around that failure. It asks the model to identify the phase first, then locate and explain the current 'Go Zone' - the region where the next safe manoeuvre should occur under the paper's expert-defined rules.

That order is the paper's best idea. A safe region is not a static object like an instrument. During preparation, it may be the anatomy that still needs exposure; during dissection, it depends on the operative field and the next intended action. If the phase is wrong, a geometrically neat prediction can be clinically meaningless. The benchmark therefore sets grounding scores to failure whenever phase recognition fails.

The annotations contain a miniature operative plan

ResGo is built from 21 laparoscopic cholecystectomy videos. From 6,138 sampled frames, the team retained 2,686 clinically relevant frames spanning preparation, Calot's triangle dissection, clip-and-divide and gallbladder-bed dissection. Three senior surgeons annotated phase, a Go Zone bounding box, exposure quality, next action and a risk reminder. Three chief surgeons with more than 15 years' experience then reviewed every sample; inclusion required unanimous agreement.

Seventeen videos, containing 2,191 samples, were used for training. Four videos and 495 samples formed the held-out test set. The proposed model first answers a phase multiple-choice question. It then retrieves that phase's operative definition and generates the box plus a structured explanation. Reinforcement learning rewards phase accuracy, overlap with the expert box, proximity to it and inclusion of key clinical concepts.

On the held-out cases, SurGo-R1 reached 76.6% phase accuracy and a mean intersection-over-union of 32.7% for the predicted Go Zone. Its end-to-end 'hardcore' accuracy at the paper's overlap threshold was 54.8%, compared with 8.28% for the strongest listed general model. A three-clinician blind review preferred the full model's reasoning over ablated versions, but its reported factual-accuracy rating was 52.5% - a reminder that better than a weak baseline is not the same as dependable guidance.

This is a benchmark for a dangerous idea

The dataset covers one operation in 21 videos collected at collaborating institutions. A bounding box simplifies a deforming three-dimensional operative field, and unanimous expert labels can hide legitimate differences in technique rather than eliminate them. Four test videos cannot establish robustness to acute inflammation, variant anatomy, bleeding, smoke, difficult exposure or unfamiliar equipment.

Nothing in this study evaluates live use, surgeon behaviour or bile duct injury. The authors explicitly state that rigorous validation is required before development and deployment.

That boundary makes the work more, not less, useful. SurGo-R1 supplies a tougher test for systems that claim to reason about operative safety: first prove that the context is correct; then show where; then explain why. A future copilot should not earn credit for drawing the right-looking box for the wrong operation.

Open the original source

03 — An error detector improves when it knows the intended gesture

A needle slips, the grasp is too close to its tip, an instrument leaves the view. These are not necessarily dramatic visual events. They are deviations from what should happen during a particular gesture. The same motion may be normal in one part of suturing and wrong in another.

This preprint treats error detection as a contextual problem. Its model combines a two-second video window with short text descriptions of the current activity and its known errors. In one configuration it also receives the robot's kinematics - position, velocity, rotation and grasper angle - so the visual scene can be checked against physical motion.

The prompts are not free-form questions to a chatbot. They encode structured knowledge: the gesture being performed, which grasper should contact or hold the needle, thread or tissue, and the executional errors associated with that interaction. A contrastive text encoder turns those descriptions into features and a small fusion model predicts whether the current window contains an error.

The team tested the approach on two settings. JIGSAWS contributed 67 dry-lab suturing and needle-passing trials with synchronized video and kinematics. SAR-RARP50 contributed 48 videos of robot-assisted prostatectomy suturing, with 40 for training and eight for testing, but no kinematics. On JIGSAWS, video plus gesture-and-error prompts reached an F1 score of 0.745; adding kinematics increased it to 0.760. On SAR-RARP50, interaction-and-error prompts reached 0.746 F1, compared with 0.640 for the cited SEDMamba baseline.

The model's measured end-to-end computation took about 36 milliseconds per window, fast enough for near-30 Hz processing on the test hardware. But 'real time' here describes computation after a two-second input window sampled every 1.2 seconds. It is not a measured warning-to-intervention time in an operating room. The authors' own example shows an early error whose probability rose but did not cross the detection threshold.

The approach works because someone specifies the relevant activities and errors. JIGSAWS uses established gesture-specific error definitions. For SAR-RARP50, where those descriptions were not supplied, the researchers used ChatGPT to draft gesture-specific errors and adapted definitions from the similar suturing task. Prompt quality therefore becomes part of model quality and requires expert governance.

The strongest multimodal result also depends on synchronized robot kinematics, which are absent from many surgical datasets and may not be accessible across platforms. Validation covers structured suturing and needle passing; performance on dissection, unexpected anatomy, bleeding or a new robot is unknown. Error windows are imbalanced, and an F1 score does not tell us which clinically serious errors are missed or how many false alarms a surgical team would tolerate.

The paper offers a useful engineering direction: do not ask pixels to carry the whole meaning of a technical error. Combine what the camera sees with what the robot moved and what the gesture was supposed to achieve. The next study must move from labelled windows to prospective, procedure-level detection - measuring alert timing, error severity, false alarms and whether feedback actually helps rather than distracts.

Open the original source