IMAGE SEGMENTATION USING TRANSFER LEARNING WITH DEEPLABV3 TO FACILITATE PHOTOGRAMMETRIC LIMB SCANNING

Abstract

In this paper, we explore the use of deep learning (DL) in conjunction with photogrammetry for scanning amputated limbs. Combining these two technologies can expand the scope of prosthetic telemedicine by facilitating low-cost limb scanning using cell phones. Previous research identified image segmentation as one of the main limitations of using photogrammetry for limb scanning. Based on those limitations, this work sought to answer two main research questions: (1) Can a neural network be trained to identify and segment an amputated limb automatically? (2) Will segmenting 2D limb images using neural networks impact the accuracy of 3D models generated via photogrammetry? To answer the first question, transfer learning was applied to a neural network with the DeepLabv3 architecture. After training, the model was able to successfully identify and segment limb images with an IoU of 79.9%. To answer the second question, the fine-tuned DL model was applied to a dataset of 22 scans comprising 6312 limb images, then 3D models were rendered utilizing Agisoft Metashape. The Mean Absolute Error (MAE) of models rendered from images segmented with DL was 0.57 mm ± 0.63 mm when compared to models rendered from ground truth images. These results are important because segmentation with DL makes photogrammetry for limb scanning feasible on a large clinical scale. Future work should focus on generalizing the segmentation model for different types of amputations and imaging conditions.

1. INTRODUCTION

Rehabilitative care for persons with limb loss is rapidly evolving due to advances in digital healthcare technologies. Novel digital workflows are empowering clinicians with tools for visualizing patient anatomy and physiology, designing custom fitting prostheses via computer aided design (CAD), building assistive devices with computer aided manufacturing (CAM), and tracking patient response in environments such as virtual reality (VR) Cabrera et al. Segmentation is an important problem in medical imaging and involves separating regions of interest (ROIs) from the rest of an acquired image. Convolutional neural networks (CNNs) are regarded as the dominant state-of-the-art approach for medical image segmentation in applications requir-



(2021). Medical imaging technologies are fundamental to every digital workflow because they inform clinicians of limb geometry, surface and/or sub-surface features, plus pathology of amputated limbs Paxton et al. (2022). Systematic reviews by Cabrera et al. (2021) and Paxton et al. (2022) identified photogrammetry as a promising technology for capturing patient surface anatomy. The main advantage of photogrammetric scanning is that models can be rendered using photographs captured via smartphones Cabrera et al. (2020); Barbero-García et al. (2018); De Vivo Nicoloso et al. (2021); R. B. Taqriban et al. (2019); Ismail et al. (2020); Barbero-García et al. (2020; 2021). Scanning with smartphones is significantly cheaper than other medical imaging modalities Cabrera et al. (2021); Paxton et al. (2022) and results in reliable and robust surface accuracy on par with existing clinical gold standard technologies Nightingale et al. (2020; 2021). Unfortunately, photogrammetry workflows often require extensive image segmentation, at the expense of human operator time and effort, in order to render 3D models Cabrera et al. (2021).

