Pak inaction won’t affect talks: Antony