Robotics, autonomy and navigation

Robotics, navigation, and autonomy span very different levels of assistance. This edition keeps sensing, guidance, execution, and clinical responsibility distinct.

In this edition

• 01 — A knee robot that turns anatomy and live tracking into actions
• 02 — A robot learns retraction by watching the tissue edge
• 03 — The surgical robot dataset where every stream shares a clock

01 — A knee robot that turns anatomy and live tracking into actions

Autonomy becomes a synchronization problem

A knee robot already knows the planned resection planes. What it usually lacks is a learned policy that can connect those plans to what the camera, tracker and robot are reporting now. ArthroCut treats that connection as a sequence of tokens. Preoperative Imaging Tokens encode CT/MR anatomy and planned planes; Time-Aligned Surgical Tokens combine RGB-D video, tracked bone and end-effector poses, robot state and the previous action on a shared timeline. A Qwen2.5-VL-based policy then emits constrained commands such as move, align and cut.

The training set came from 21 complete knee arthroplasty procedures and includes 23,205 synchronized RGB-D pairs plus optical tracking and robot logs. The evaluation, however, was not performed in those patients. It used a physical knee prosthesis and bone analogs, with seven independent runs through six standard resections. A plane counted as successful when mean surface deviation from the plan stayed within 1.5 mm. Averaged across the six planes, ArthroCut achieved a reported success rate of 0.86 and success weighted by path length of 0.75. Four planes succeeded in all seven runs, while tibial and posterior chamfer cuts succeeded in four of seven.

The ablation makes the system easier to interpret. Preoperative tokens alone moved the vanilla policy only from 0.05 to 0.12 success. Time-aligned intraoperative tokens raised it to 0.68. Combining both reached 0.86. The live stream—not simply the preoperative scan—did most of the work, while anatomy supplied the stabilizing reference. Grammar and safety masks also narrowed what the language model could command, an important distinction from asking a general model to generate unconstrained robot motion.

Read the misses, not just the average

The weakest planes expose how far this is from an autonomous knee operation. The authors report instability after minimal resection as a problem for tibial and chamfer performance; their qualitative fit check still showed a small tibial interface gap. The dataset comes from one collection pipeline, the bench uses surrogates, and seven runs are too few to characterize rare errors. Patient anatomy, bleeding, registration drift, tool wear and intraoperative plan changes remain outside this test.

ArthroCut is therefore best read as a systems experiment: synchronizing anatomy, geometry and robot state can make learned actions far more coherent than vision alone. The translational question is whether that coherence survives cadaveric and clinical conditions with independent safety monitoring, failure recovery and surgeon override. Until then, the 86% figure describes a constrained benchtop sequence, not readiness for autonomous arthroplasty. The paper is a preprint and has not completed peer review.

Open the original source

02 — A robot learns retraction by watching the tissue edge

Exposure is an active negotiation with deformable tissue. A fixed robot trajectory cannot assume that the same pull produces the same opening in a sponge, a liver or another patient. This preprint takes a more surgical approach: observe the visible tissue boundary, make a small movement, estimate how the boundary responded, and adapt the next movement until the target region is uncovered.

The controller works in a low-dimensional description of that boundary. A deformation network, trained on 100,000 simulated samples, supplies an initial estimate of how grasping and pulling will reshape the tissue. During execution, the robot updates that relationship from camera feedback rather than following the simulation blindly. The same learned model also helps choose a grasp point and impose a predicted deformation-force constraint. In physical tests on an MP550 system, the authors used both PVA sponge and ex vivo porcine liver. A stereo camera and force sensor supported measurement, while SAM2 extracted the laparoscopic boundary from a few offline-annotated point prompts.

There are two encouraging results. Across 1,000 simulation and 100 physical deformation tests, prediction error increased outside simulation but remained below 1 cm for pulls up to 10 cm, according to the paper. In autonomous retraction trials, the method completed most tasks and approached 100% success for configurations judged physically feasible. Its selected grasp points were not globally optimal, but consistently fell in relatively low-force regions when compared with candidates spaced along the boundary. That is a sensible objective: find a workable, less forceful pull rather than chase an abstract optimum.

The experimental details prevent a clinical reading of those percentages. The physical specimens are two controlled materials, not perfused tissue in an operation. The grasp point was computed offline and then manually set. SAM2 needed annotated prompts, and controller parameters were adjusted between simulation and the physical setup. The region to expose was represented by a configurable marker; the system was not discovering a partly hidden anatomical target such as Calot's triangle. Force under the platform was also a surrogate for internal tissue deformation.

Even with those boundaries, the paper offers a valuable engineering direction. High-level AI may suggest what exposure is needed, but a task-specific controller still has to feel its way through the mechanics. Here, “intelligence” is the repeated correction between what the tissue was expected to do and what its boundary actually did. Progress toward an operative assistant will depend on closing the remaining loops too: automatic grasp acquisition, anatomical target recognition, direct force sensing, recovery from slips and validation across living tissue. This is an unreviewed preprint, not evidence of clinical safety.

Open the original source

03 — The surgical robot dataset where every stream shares a clock

The unglamorous bottleneck

A robotic surgery dataset can contain beautiful stereo video, precise kinematics and a contact signal—and still be misleading if those streams are out of time. A few milliseconds can decide whether a model learns that a motion caused tissue contact or merely happened nearby. SurgSync treats synchronization as infrastructure, not post-processing trivia. Its open framework records endoscopic and side-view video alongside dVRK arm state, setpoints, jaw motion and an optional capacitive contact sensor on a common time base.

The team built two modes because real-time inference and clean archival data have different needs. Online matching produces data that can be used immediately, with mean latency of 6.36 ms in the reported sample and an average recording rate of 4.04 Hz. Offline matching requires interpolation after collection but reduced mean latency to 1.35 ms and ran at 10 Hz on the primary setup. The framework was deployed at Johns Hopkins and the University of British Columbia, although the second site used a different endoscope, omitted the contact sensor and allowed a wider online timing tolerance.

The accompanying dataset contains 214 recordings from 13 participants: four novices, five experienced dVRK users and four surgeons. They performed peg transfer, suturing and knot tying, tissue manipulation and dissection, mainly on ex vivo tissue with some phantoms. The modalities extend beyond video and kinematics to camera calibration, contact, task events and processing tools for depth, optical flow and kinematic reprojection. This is the kind of dataset from which several questions can be asked without rebuilding the acquisition stack each time.

To show that the recordings are usable, the authors trained a multimodal skill-assessment model on 43 instances from eight users. Across four folds, predicted ratings had mean Spearman correlations of 0.803 for suturing and 0.765 for knot tying. Those figures demonstrate a learnable signal, not a validated assessment product: one experienced rater supplied the scores, the sample is small, and participant identity and task conditions remain tightly bounded.

SurgSync matters because synchrony is a form of ground truth. If images and robot states are misaligned, a larger model may learn the wrong causal sequence more efficiently. But the current release is still a research dataset built on dVRK systems, phantoms and ex vivo tasks; only nine recordings came from the second institution, and collection rates depend on workstation load. Wider hardware, in vivo conditions, more instruments and independent annotations are still needed. For surgeons evaluating robotic AI, a useful question now sits beside dataset size and accuracy: how precisely did the camera, tool and tissue event agree on when “now” was?

Open the original source