Several key technologies for AIAugSurgery, an image guided surgery with artificial intelligenceaugmented reality, have been developed. For AIAugSurgery system, we develop a marker based flexible 6-DoF high precision camera pose estimation technique. We also propose an end-to-end deep model to perform single-image blind inpainting to remove the specular highlights for endoscope imaging. Moreover, for the computing platform of AIAugSurgery, we have developed processors for super-resolutionimage deblurring used for AR-aided surgical auxiliary systems. A high-throughput, energy-area-efficient CIM for neural network circuits for surgical information processing is also developed.

Several break-through technologies for AIAugSurgery, an image guided surgery with artificial intelligenceaugmented reality, have been developed. For AIAugSurgery system, we develop the most flexible marker based 6-DoF camera pose estimation technique with 0.5mm in accuracyan end-to-end deep model for removing specular highlights in endoscope imaging. The normalized area efficiencyenergy efficiency of the proposed super-resolution processor chip is 3x6.5x more than the previous work. The proposed CIM SRAM design achieves 2.26×, 1.12×,2.89× performance improvements in throughput, energy,area efficiency, respectively, compared to the state-of-the-art designs.

Several technologies for AIAugSurgery, an image guided surgery with artificial intelligenceaugmented reality, have been developed in this project. The development of high-performance image processing ICs, computing-in-memory circuits, 6-DoF marker based pose estimation, AI based image processing for endoscopic images, human-computer-interface with haptics enable to lay the foundation of mobile augmented reality. Not only be used for AR surgery systems, for the coming metaverse era, these technologies can also be employed as the key technologies for consumer metaversemedical metaverse.