The Signal · Datasets and benchmarks
Datasets that shape the field
Datasets and benchmarks quietly define what the field can measure. Their sampling, labels, access rules, and evaluation design shape every downstream claim.
Datasets and benchmarks quietly define what the field can measure. Their sampling, labels, access rules, and evaluation design shape every downstream claim.
In this edition
- 01 — Can a skill-scoring model survive a change of operating room?
- 02 — A polyp detector designed for blur, glare and small hardware
- 03 — Before a robot acts, it needs a map of where action is possible
01 — Can a skill-scoring model survive a change of operating room?
A score built in one room often stays there
A video model may rank suturing skill convincingly in the dataset on which it was trained and then lose the plot when the camera, instruments or procedure changes. CoRe-DA addresses that failure directly. The authors ask whether labelled videos from one setting, plus unlabelled videos from a new setting, can produce useful scores in the new setting.
They built two demanding transfers. One moves from 314 open-surgery dry-lab suturing videos to 103 robotic dry-lab videos. The other moves between two clinical robotic datasets: hysterectomy vaginal-cuff closure and prostatectomy dorsal-venous-complex suturing. Those are not cosmetic changes. The visual field, task and equipment all shift while the output remains a continuous OSATS or M-GEARS-style score.
Learning the difference before the number
CoRe-DA does not rely only on predicting an absolute score. During training it compares pairs of source videos and learns their relative score difference. It then pairs an unlabelled target video with scored exemplars and uses agreement between relative and absolute predictions as a self-training signal. In plain language: deciding which of two performances is stronger may be a more portable lesson than memorising what “23 points” looks like in one dataset.
Across four runs, the method reached Spearman correlations of 0.46 in the dry-lab transfer and 0.41 in the clinical transfer, with mean absolute errors of 4.29 and 1.93 points respectively. Every comparator was weaker in both settings; several source-only or domain-specific models were barely correlated with the target scores. A ten-labelled-video variant improved the dry-lab correlation to 0.52.
Useful benchmark, unfinished assessor
For surgical education, the important result is not that an algorithm can replace an expert rater. It cannot—not on this evidence. The correlations are moderate, the clinical target sets are small, and the model saw all target videos without labels during adaptation before those same target cases were scored for evaluation. There is also no test of whether its feedback changes a trainee’s technique or whether errors are distributed fairly across experience groups.
The real advance is a better test for portability. Any hospital considering automated video assessment should demand a local, procedure-specific validation and inspect score errors—not accept performance from another task. CoRe-DA makes that domain shift measurable and shows that unlabelled local video can help, but it remains a preprint benchmark rather than a credentialing instrument.
02 — A polyp detector designed for blur, glare and small hardware
A segmentation mask is useful only if it arrives while the lesion is still on screen. That makes DepthPolyp interesting for a reason that has little to do with model size contests: it tries to preserve the polyp outline under motion blur, defocus and specular glare while running on inexpensive hardware.
The model was trained on Kvasir-SEG and tested across CVC-ClinicDB, CVC-ColonDB and PolypGen. The authors also built a four-way robustness experiment: train on clean or synthetically degraded images, then test on clean or noisy images. This separates two questions that are often mixed together—whether a model can tolerate bad images, and whether training for bad images damages its performance on good ones.
DepthPolyp adds an auxiliary pseudo-depth task. A depth estimator supplies relative structural cues during training, encouraging the network to preserve shape when colour and texture are unreliable. Efficient feature blocks keep the model to 3.57 million parameters and 0.86 GMACs. In the ablation, removing depth guidance reduced average Dice from 0.784 to 0.759; removing the uncertainty-weighted balancing between segmentation and depth hurt more, dropping Dice to 0.605.
On PolypGen sequences 18–22—real colonoscopy video with authentic blur, defocus and reflections—the model recorded a Dice score of 0.679. That was higher than the compared lightweight models and slightly above much larger systems in the same table. At batch size one, the authors measured 181.54 frames per second on an iPhone 15 and 4.05 on a Raspberry Pi 4. These are engineering measurements, not proof that a phone-based clinical device is ready.
A Dice score of 0.679 also means the boundary is far from perfect. The real-video evaluation covers five PolypGen sequences, while much of the robustness training uses synthetic degradation. The paper does not test adenoma detection rate, missed lesions, withdrawal workflow, latency through a full endoscopy stack or prospective use by endoscopists. “Real time” describes inference under the authors’ setup, not a regulated clinical system.
Still, the design priority is right. A deployable endoscopy model must be judged during camera motion, poor preparation, glare and fleeting views—not only on carefully selected still frames. DepthPolyp, accepted to ICPR 2026, offers a compact research model and a more realistic robustness test. The next decisive experiment is prospective video from different centres and processors, with lesion-level misses and false prompts measured in the procedure rather than pixel overlap alone.
03 — Before a robot acts, it needs a map of where action is possible
A segmentation model can tell a robot, “this is lung” or “this is a vessel.” That is not the same as telling it where to retract safely or where a clip should go. SurgAM explores the missing layer between perception and action: an affordance map, a heatmap of locations where a particular surgical action appears possible.
Labels built from the moment before contact
The authors assembled 1,915 video sequences from seven public surgical datasets, spanning procedures including cholecystectomy, prostatectomy and coronary bypass. They extracted the frame immediately before a tool touched tissue, marked the intended contact point, then used the Segment Anything Model to expand that point into a candidate tissue-region mask. The final collection covers three actions: 904 retraction clips, 780 aspiration clips and 231 clipping clips.
SurgAM combines features from DINOv2, which is used for semantic content, and Stable Diffusion, which contributes spatially coherent features. Text prompts encode the requested action and procedural context; an attention decoder turns the fused representation into a heatmap. On the held-out dataset it led five comparison methods across all four reported affordance metrics.
That benchmark becomes more tangible in the robot experiments. The team connected the predictor to a visual-servoing controller on a da Vinci Research Kit. For each phantom task they first collected targeted teleoperated demonstrations and fine-tuned a task-specific model. The system completed lung retraction on a silicone torso phantom in 50 of 50 trials and vessel clipping on a prostate phantom in 49 of 50. The single clipping failure followed camera occlusion; prediction recovered after visibility returned.
The action map is not yet a safety map
The strongest claim stops at two controlled phantom tasks. The ground-truth regions begin with a single human-selected point and a general segmentation model, so the labels encode intended contact—not a full map of tissue fragility, alternative safe actions or “do not touch” anatomy. Clipping is also under-represented in the dataset. In the physical study, experts exposed the target vessel before autonomous clipping, and the models were fine-tuned on demonstrations from the target setup.
For surgeons, the conceptual move is valuable: autonomy needs an explicit representation of action opportunities, not only object names. It could also make a robot’s next move easier to inspect than an opaque end-to-end policy. But no live tissue, bleeding, smoke, unexpected anatomy or multi-step recovery was tested. SurgAM is an ICRA 2026 research result showing that predicted action regions can drive a robot on phantoms—not evidence for autonomous clinical retraction or clipping.