Accused should not continue as office bearer: Sports minister