Minister won’t resign, off on Umrah