The state of surgical AI

A field map for surgeons: what Surgical AI can currently observe, predict, or support, and where the evidence still stops short of clinical use.

In this edition

• 01 — A surgical video becomes a map, not a recording
• 02 — A lung pathology model meets the people who may rely on it
• 03 — A 235-billion-parameter model misses the instruments

01 — A surgical video becomes a map, not a recording

A one-hour operation is rich in information and almost impossible to review at scale. The Halsted project asks a practical question: what if a video could be converted into a timeline that a surgeon can search, compare and revisit?

This is not a chatbot watching an operation. Halsted is a vision-language model that receives short video clips plus an instruction - identify the step, name the instrument, assess a suturing component - and returns structured tags. One shared model is trained across 16 procedures, eight specialties and 104 categories. Its training library contains more than 650,000 annotated clips, expanded through one round of model-assisted labelling.

The paper does more than report a model. Its web platform lets a surgeon upload a procedure and receive a temporal map: where an action happened, how long it lasted and which clips were classified as proficient or not. The authors report that a one-hour video takes about 15 minutes to process through the platform. That makes postoperative review conceivable without asking someone to annotate the whole case by hand.

Performance is uneven in an instructive way. On held-out atlas clips, coarse labels were easier: step recognition reached 99% accuracy and procedure recognition 91%. Fine actions were harder, at 70% accuracy. Suturing proficiency also varied by specialty, with reported AUROC ranging from 0.71 in general surgery to 0.85 in urology. On the external RARP-50 action benchmark, Halsted reached 68.6% accuracy across eight actions.

The lightweight decoder is another useful result. Replacing its 44-million-parameter decoder with a one-billion-parameter Llama decoder did not produce a statistically significant improvement on the tested micro-activity tasks. Better surgical representations may matter more than simply adding a larger language model.

The atlas does not cover neurosurgery, orthopaedics or plastic and reconstructive surgery. Some training labels were produced by the earlier model, and the authors acknowledge that incorrect labels and erroneous mappings remain possible. The platform is commercially operated, while only the 27,000-clip HSA-27k subset is released for benchmarking.

Most importantly, this study evaluates mapping accuracy, not whether showing these maps improves a surgeon's technique, education or patient outcomes. Those prospective trials are described as future work. For now, Halsted is best understood as a serious attempt to make operative video navigable - a new review instrument whose influence on practice still has to be measured.

Open the original source

02 — A lung pathology model meets the people who may rely on it

Foundation models in pathology are often introduced as versatile image encoders, then tested task by task on retrospective slides. PulmoFoundation takes a much longer route. It adapts a general pathology model specifically to lung tissue, evaluates it across the diagnostic pathway and then places its output in front of pathologists.

The model was continually pretrained on more than 88 million image tiles from about 40,000 whole-slide images. The broader study contains 66,146 slides from 16 medical centres and public sources. Evaluation spans 32 tasks: biopsy diagnosis, intraoperative frozen section, resection findings, biomarkers and prognosis. That breadth matters because lung pathology is not one classification problem; a specimen triggers a sequence of linked decisions.

First came retrospective internal and external testing, including 21 external cohorts from eight independent institutions. Then the authors registered a prospective study of 1,357 consecutive patients at Nanfang Hospital. Across 11 diagnostic tasks, the model's average AUC was 0.923. Performance was not uniform: prospective lymph-node-metastasis prediction fell to 0.849 from 0.975 internally, with false-positive node predictions contributing to the drop.

The most consequential experiment was a crossover randomized reader study. Eight pathologists reviewed 658 cases across four tasks, once with AI assistance and once without it, separated by a four-week washout. Across 5,264 matched case-reader pairs, accuracy was 91.7% with AI versus 83.2% without it. Median diagnostic time fell by 18.3%, and inter-rater agreement rose from a kappa of 0.55 to 0.76. Junior pathologists showed larger absolute gains than senior colleagues.

The warning is inside the success

When the displayed model prediction was wrong - 120 AI-assisted observations - the final diagnosis matched that wrong answer in 93 cases. In 12 observations, a previously correct unaided answer became incorrect after AI assistance. Overall harm was uncommon because the model was usually correct, but this pattern is the part a deployment team cannot ignore: a high-performing model can still create concentrated automation bias.

The prospective cohort and reader trial were conducted at one institution, and only eight pathologists participated. The interface exposed a task-specific prediction; different confidence displays, uncertainty cues or heat maps might change trust. The model also operates on H&E morphology alone and does not yet combine the molecular, immunohistochemical and clinical information used in routine decisions.

This remains a preprint, not a deployed standard of care. Yet it offers something rare in foundation-model research: evidence about both the algorithm and the human response to it. For surgeons, the key lesson is not simply that AI may accelerate pathology. It is that clinical benefit depends on designing the moment when a specialist decides whether to believe the machine.

Open the original source

03 — A 235-billion-parameter model misses the instruments

Ask a modern vision-language model to describe a photograph and it can sound remarkably capable. Ask it to name every instrument in a neurosurgical frame and the illusion can collapse.

This preprint tests 20 open-weight vision-language models, from 2 billion to 235 billion parameters, on tool presence in endoscopic endonasal surgery. The primary SDSC-EEA dataset contains 67,634 annotated frames from procedures contributed by 10 surgeons. Splitting by procedure - rather than mixing neighbouring frames from the same case - creates the harder and more useful question: can the model recognise tools in operations it did not train on?

Scale met a 13.4% baseline

The majority-class strategy, which predicts the same common tool set every time, achieved 13.41% exact-match accuracy on the validation set. The best zero-shot open model reached 14.52%; most did worse. A medicine-tuned model also underperformed its general sibling. Failures were not mainly malformed responses: recent models usually produced valid output, but often named the wrong tools or hallucinated tools that were not present.

Task-specific training helped. A 27-billion-parameter Gemma model with a dedicated classification head reached 51.08% exact-match accuracy. Yet increasing the LoRA adapter rank from 2 to 1,024 drove training accuracy up to 98.6% while validation accuracy stayed below 40% in the controlled rank experiment. More capacity made the model remember the training distribution; it did not remove the shift between procedures.

Then a 26-million-parameter YOLO model reached 54.73% exact-match accuracy - better than the best vision-language approach with roughly one-thousandth the parameters. The broad pattern repeated on three public datasets covering laparoscopic cholecystectomy, endoscopic pituitary surgery and robotic training on porcine tissue. Fine-tuned models and the specialist vision model consistently beat zero-shot frontier systems.

The lesson is narrower than the headline

This is a tool-presence study, not a verdict on every possible role for general models. A vision-language model may add value when a task needs flexible language or broad reasoning. The specialist may dominate when the target is narrow, visual and precisely defined. Even the paper's strongest model was far from perfect, and one dataset's labels described tools installed on robot arms rather than necessarily visible in each frame.

The specialist earns the hand-off

The useful design idea is therefore a division of labour: let a general model coordinate or explain, but give high-stakes perception to modules trained and validated for that exact signal. For a hospital evaluating surgical AI, parameter count and performance on generic medical leaderboards are weak evidence. A small model tested on local procedural variation may be the more credible technology - and it will be cheaper to run.

This paper does not say scaling never works. It shows that, for a simple-looking operative perception task, the missing ingredient was not another order of magnitude of parameters. It was surgical specificity.

Open the original source