‘PM knows Left can’t join his govt’