Cystoscopy is an essential diagnostic procedure for bladder tumors. However, the effectiveness of the examination is greatly influenced by the characteristics of the lesion and the proficiency of the examiner, leading to possible misdiagnosis and missed diagnosis. The application of deep learning models can help improve the identification of tumors under cystoscopy. This study explores the performance of deep learning models in segmenting tumors of different sizes and clarity under cystoscopy.

A retrospective study included 198 patients who underwent transurethral bladder tumor resection in Peking Union Medical College Hospital, Beijing Sixth Hospital, and Beijing Longfu Hospital from June 1, 2022, to June 1, 2024. Semantic segmentation models Mask2Former and DeepLabV3+ were used to identify bladder tumors. Video data from 171 patients and 27 patients were used as the training set and test set, respectively. Observational indicators included intersection over union (IOU), accuracy, recall rate, and mean Dice coefficient. Analysis was performed using R 4.0.2 software, and all count data were expressed as numbers or percentages.

The DeepLabV3+ model (with 21.75 Mparameters) achieved an accuracy and recall of 83.3% for segmenting low-clarity bladder tumors. The Mask2Former model (with 197 M parameters) achieved an accuracy and recall rate of 92.1% and 88.9%, respectively, for low-clarity bladder tumors. Both models achieved an accuracy and recall rate of over 98.0% for high-clarity bladder tumors. When applying the Mask2Former model, as the number of cases in the training set increased from 10 to 171, the model's accuracy rate for segmenting bladder tumors increased from 94.0% to 97.7%, recall rate increased from 73.3% to 93.5%, and mDice increased from 62.3% to 85.5%, with a particularly noticeable improvement in the low-clarity group. When using the Mask2Former model to segment bladder tumors of different sizes, the accuracy rate for tumors occupying less than 0.02 of the screen and more than 0.1 was 90.9% and 98.1%, respectively; the recall rate was 83.3% and 99.8%, respectively.

The deep learning model Mask2Former demonstrates excellent performance in the segmentation of bladder tumors under cystoscopy, particularly exhibiting robust segmentation capabilities for low-clarity images and tumors occupying a significant portion of the visual field.