The Yukawa-potential self-interacting dark matter (YIDM) particles are assumed to interact through a Yukawa potential, leading to a velocity-dependent cross section. The YIDM is a modified dark matter model that can remedy the problems of the collisionless cold dark matter on small scales. We explored the black hole growth mechanism through accretion of the YIDM. With some reasonable assumptions, the accretion of the YIDM particles onto initial stellar mass black holes could help to explain the formation of the known $z>6$ quasars.