BUS-Set: A benchmark for quantitative evaluation of breast ultrasound segmentation networks with public datasets

Cory Thomas*, Michal Byra, Robert Marti, Moi Hoon Yap, Reyer Zwiggelaar

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)
126 Downloads (Pure)

Abstract

Purpose
BUS-Set is a reproducible benchmark for breast ultrasound (BUS) lesion segmentation, comprising of publicly available images with the aim of improving future comparisons between machine learning models within the field of BUS.

Method
Four publicly available datasets were compiled creating an overall set of 1154 BUS images, from five different scanner types. Full dataset details have been provided, which include clinical labels and detailed annotations. Furthermore, nine state-of-the-art deep learning architectures were selected to form the initial benchmark segmentation result, tested using five-fold cross-validation and MANOVA/ANOVA with Tukey statistical significance test with a threshold of 0.01. Additional evaluation of these architectures was conducted, exploring possible training bias, and lesion size and type effects.

Results
Of the nine state-of-the-art benchmarked architectures, Mask R-CNN obtained the highest overall results, with the following mean metric scores: Dice score of 0.851, intersection over union of 0.786 and pixel accuracy of 0.975. MANOVA/ANOVA and Tukey test results showed Mask R-CNN to be statistically significant better compared to all other benchmarked models with a p-value >0.01. Moreover, Mask R-CNN achieved the highest mean Dice score of 0.839 on an additional 16 image dataset, that contained multiple lesions per image. Further analysis on regions of interest was conducted, assessing Hamming distance, depth-to-width ratio (DWR), circularity, and elongation, which showed that the Mask R-CNN's segmentations maintained the most morphological features with correlation coefficients of 0.888, 0.532, 0.876 for DWR, circularity, and elongation, respectively. Based on the correlation coefficients, statistical test indicated that Mask R-CNN was only significantly different to Sk-U-Net.

Conclusions
BUS-Set is a fully reproducible benchmark for BUS lesion segmentation obtained through the use of public datasets and GitHub. Of the state-of-the-art convolution neural network (CNN)-based architectures, Mask R-CNN achieved the highest performance overall, further analysis indicated that a training bias may have occurred due to the lesion size variation in the dataset. All dataset and architecture details are available at GitHub: https://github.com/corcor27/BUS-Set, which allows for a fully reproducible benchmark.
Original languageEnglish
Pages (from-to)3223-3243
Number of pages21
JournalMedical Physics
Volume50
Issue number5
Early online date28 Feb 2023
DOIs
Publication statusPublished - 09 May 2023

Keywords

  • MEDICAL PHYSICS DATASET ARTICLE
  • MEDICAL PHYSICS DATASET ARTICLES
  • breast segmentation
  • public datasets
  • ultrasound
  • Neural Networks, Computer
  • Humans
  • Ultrasonography, Mammary
  • Image Processing, Computer-Assisted/methods
  • Benchmarking
  • Female
  • Machine Learning

Fingerprint

Dive into the research topics of 'BUS-Set: A benchmark for quantitative evaluation of breast ultrasound segmentation networks with public datasets'. Together they form a unique fingerprint.

Cite this