Peeved BSY to keep off party affairs