In medical image analysis, segmentation is crucial as it delineates images into specific anatomical structures or lesions, providing valuable information for diagnosis and treatment. Recently, many automatic models have been proposed for segmentation, but the effectiveness of these models is often compromised by variations across different medical centers and subjective interpretations by raters, namely the multi-center and multi-rater scenarios. These adaptation challenges are intensified by the high annotation costs and the difficulty in accessing source medical data due to strict security protections. To bridge this gap, we propose a versatile Source-Free Active Domain Adaptation (SFADA) method (a fusion paradigm of Active Learning and Source-Free Domain Adaptation) for medical image segmentation, designed to address variations in multiple medical centers and raters assessments with limited annotations. Specifically, the proposed method leverages only a source-trained model and the ReCal-HS selective strategy to recommend a limited number of representative and calibration target samples for annotation, and then further fine-tunes the model for enhanced performance. To evaluate the effectiveness, we retrospectively collected two large datasets for model development and evaluation, encompassing 1057 scans from five hospitals and 680 scan annotations from four radiation oncologists (ROs). Our method demonstrates superior performance compared to existing domain adaptation and active learning techniques, offering a practical solution to the prevalent issue of data variability in clinical analyses. We will construct a new benchmark including multi-center and multi-rater datasets and release code to support further advancements in the field: https://github.com/whq-xxh/Versatile-SFADA.