The emerging regulatory role of deubiquitinases (DUBs) has been implicated in various fundamental processes and pathogenesis. To determine the pivotal role that DUBs play in mediating tumorigenesis, we have performed a non-biased screen of 67 human DUBs based on a mammary cell transformation assay. This led to the identification of USP11 as a critical determinant of mammary tumor initiation and progression. Using an approach of protein complex purification coupled with mass spectrometry, we further identified XIAP to be a target for USP11. We demonstrated that, while depletion of XIAP attenuates cell transformation, elevated USP11 significantly promotes the tumor colony formation through stabilization of XIAP. Molecular modeling coupled with mutagenesis analyses further revealed that Leu207 on the BIR2 domain of XIAP facilitates its interaction with USP11. Stabilization of XIAP due to its deubiquitylation by USP11 leads to the inhibition of cell anoikis and apoptosis, which in turn promotes tumorigenesis. Finally, immunohistochemical staining revealed that aberrant accumulation of USP11 correlates with elevated levels of XIAP in breast cancer tissues. We therefore propose that aberrant USP11, via stabilization of XIAP, promotes tumor initiation and progression. Copyright © 2016. Published by Elsevier B.V.