A deep learning screening model of esophageal endoscopic images can reduce the burden on endoscopists. However, most deep learning methods require many labeled data with balanced categories, and their ability to deal with new diseases not appearing in the training set is limited. This study elaborated a semi-supervised anomaly detection model for the initial screening of esophageal endoscopic images, requiring a single class of samples as a training set. The reconstruction-based method was used for anomaly detection. The model’s framework was a variational auto-encoder, with two memory modules added in latent space to restrain its generalization ability. In the memory module, a clustering operation was introduced to provide a better distribution of memory vectors, promoting their compactness with encoded features and separation from each other. A detailed description and theoretical substantiation of the proposed model were presented. A dataset containing 7989 esophageal endoscopic images labeled by experienced endoscopists was used for numerical experiments. The proposed model results were compared with those of other auto-encoder-based anomaly detection methods, outperforming them and achieving an area under the curve of 0.8212. The ablation study was also conducted to validate the effectiveness of each model’s part, and new data were successfully incorporated to assess the model feasibility and applicability range.
