SurgTAP-3D

Understanding 3D scene dynamics in minimally invasive surgery (MIS) is a fundamental step toward enabling physically intelligent and autonomous surgical systems. However, existing public datasets are largely confined to ex-vivo or simulated environments, offering limited scalability and diversity in real-world clinical scenarios. Moreover, current annotations typically focus on either tissue or instruments alone, overlooking the critical contact dynamics that govern tool–tissue interactions. This gap restricts the development of downstream applications such as augmented reality guidance and autonomous surgery driven by vision-language-action (VLA) models or world models, which demand large-scale, high-quality 3D supervision.

In this work, we construct SurgTAP-3D, a large-scale in-vivo pseudo-3D surgical motion tracking dataset and benchmark, comprising 520K frames, 4,549 action clips, and 1.4M 2D/3D point trajectories from diverse robotic and manual procedures under unified action categories. To make such annotation scalable, we design the first unified framework for Surgical 3D Point Tracking that converts uncalibrated monocular surgical videos into structured 4D annotations through contact-region prompting, active-tool segmentation, dynamic bundle adjustment, and global 4D refinement. This pipeline adapts foundation priors to surgical videos and produces temporally consistent trajectories, pseudo-scale geometry, camera poses, and instrument masks. The dataset captures realistic challenges inherent to surgical environments, including occlusions, smoke, bleeding, and rapid instrument motion.

Furthermore, we establish a comprehensive benchmark to evaluate 3D point tracking performance across the SurgPose, SurgT, and STIR datasets, covering both instrument and tissue tracking scenarios. We also demonstrate downstream applications such as instrument displacement estimation, deformable surface reconstruction, and fine-tuning foundation models, highlighting the practical utility of our dataset and framework. Extensive experiments and analysis show the strong robustness and effectiveness of our dataset and annotation framework, for 3D surgical point tracking, reconstruction applications, and surgical action understanding, addressing an important limitation in real-world 3D surgical motion data.