Self-prompting Large Vision Models for Few-Shot Medical Image Segmentation

Abstract

Recent advancements in large foundation models have shown promising potential in the medical industry due to their flexible prompting capability. One such model, the Segment Anything Model (SAM), a prompt-driven segmentation model, has shown remarkable performance improvements, surpassing state-of-the-art approaches in medical image segmentation. However, existing methods primarily rely on tuning strategies that require extensive data or prior prompts tailored to the specific task, making it particularly challenging when only a limited number of data samples are available. In this paper, we propose a novel perspective on self-prompting in medical vision applications. Specifically, we harness the embedding space of SAM to prompt itself through a simple yet effective linear pixel-wise classifier. By preserving the encoding capabilities of the large model, the contextual information from its decoder, and leveraging its interactive promptability, we achieve competitive results on multiple datasets (i.e. improvement of more than 15% compared to fine-tuning the mask decoder using a few images). Our code is available at https://github.com/PeterYYZhang/few-shot-self-prompt-SAM

Q. Wu and Y. Zhang—Co-first authors.

A Supplementary Materials

1.1 A.1 Limitation

Multi-instance Segmentation. The first limitation of our method lies in the segmentation task that has multiple instances. One can blame the problem on the plugged-in linear classifier, the simple classifier cannot know the number of instances accurately, so the spatial information passed to the prompt encoder is not complete, thus leading to the limited performance. More advanced training and prompting techniques need to be explored in the future.

Limitation of Modality Knowledge in Decoder. We also tested our method on datasets of other modalities. For example, we tested it on an ultrasound dataset, Pubic Symphysis-Fetal Head Segmentation and Angle of Progression [13], which includes lots of high-frequency features. We test head segmentation using this dataset, see Table 3 We found that the SAM decoder will lead to degrading performance. To keep the fairness for both of our methods and other fine-tuning methods, we use k=20 for testing. The result in Table 3 shows that our method has a lower performance compared to MedSAM and SAMed. Surprisingly, in the examples in Fig. 3, we found that the performance is better when just using the linear classifier and then upscaling. The reason is that the decoder of SAM does not have the capability to predict the accurate mask from the interference of high-frequency features. This reflects that although the size and position of the instance are important, the classifier needs to know the basic knowledge of the modality. To solve it, one may combine our methods together with other fine-tuned decoder.

Table 3. The results of our method on the ultrasound dataset: Symphysis-Fetel, in a 20-shot setting. “Linear” means the coarse mask generated by the linear pixel-wise classifier, “point+box” means both of the self-generated point and box are used for the final output.

Fig. 3.

Some examples of the result of our method on the Pubic Symphysis-Fetel dataset. The segmentation result is not satisfactory in ultrasound images, although the score is high. Also, the linear classifier even outperforms our method in some case. The result show that original SAM is sensitive to high-frequency perturbations (i.e. edges or noise in ultrasound).

1.2 A.2 Ablation Study Table

Table 4. Results of the ablation study on Kvasir-SEG. “Linear” means the coarse mask from the linear pixel-wise classifier; “point” means only using the point as prompt; “box” represents only using the bounding box as prompt; “point+box” means using both of bounding box and point as prompt. The scores here are dice scores.