So on the wiki I contribute to has way to many Buercrats I was wondering is there a way to remove that right from the inactive ones and those who left a wiki and a way to remove that right from them?
Indeed only Fandom Staff (and some related global groups) can remove bureaucrats. However, Long total inactivity (say, a year) is a valid reason for requesting a removal. Contact your Wiki Representative in case you have one, and if not, contact Fandom via https://support.fandom.com