XP: food allergy help

If your LO had a food allergy how long did it take before you noticed a difference when you stopped eating the food causing the allergy?

Long story short....DS2 is not gaining enough weight. Doc thought it was a supply issue, but I saw the lactation specialist and we suspect it might be a food allergy. Some of the things DS2 is doing are not finishing nursing, he goes off the nipple and pulls away from me. Sometime he "pushes"me away with his hands. So he is not getting enough BM throughtout the day. He usually nurses well during the night or if I feed him right after he wakes up and is still sleepy. He also has cradle cap that has never gone away and I read that might be related to allergies. Other things too..but I just dont know what is related to a potential allergy or just his personality.

So, basically I am eating just fruits, veggies, and meat. If this helps I plan to figure it out from there. The LC said I should notice a difference in about a week.