Is it definitely an allergy? It sounds more like a metabolic (thyroid, endocrine) reaction.
Your body will naturally feel more tired after a carb meal, because while carbs are a great fast source of energy, they require more energy to break down initially. Do you get less tired if you have a carb/protein combo, like cheese and crackers, or a sandwich with meat and/or cheese? The protein will require more time to break down, but will also slow the absorption of the carbs, which could ameliorate your symptoms.
I would recommend seeing an endocrinologist, who can better diagnose you.
Best of luck!