Top Ajit associate may quit, go to SP or BJP