Extracellular vesicles (EVs), including exosomes, microvesicles, and apoptotic bodies, are nanosized membrane vesicles derived from most cell types. Carrying diverse biomolecules from their parent cells, EVs are important mediators of intercellular communication and thus play significant roles in physiological and pathological processes. Owing to their natural biogenesis process, EVs are generated with high biocompatibility, enhanced stability, and limited immunogenicity, which provide multiple advantages as drug delivery systems (DDSs) over traditional synthetic delivery vehicles. EVs have been reported to be used for the delivery of siRNAs, miRNAs, protein, small molecule drugs, nanoparticles, and CRISPR/Cas9 in the treatment of various diseases. As a natural drug delivery vectors, EVs can penetrate into the tissues and be bioengineered to enhance the targetability. Although EVs' characteristics make them ideal for drug delivery, EV-based drug delivery remains challenging, due to lack of standardized isolation and purification methods, limited drug loading efficiency, and insufficient clinical grade production. In this review, we summarized the current knowledge on the application of EVs as DDS from the perspective of different cell origin and weighted the advantages and bottlenecks of EV-based DDS.