If your doing BB movements (bilateral movements), you could have one side compensating more than the other that could lead to size imbalances. I would suggest more DB exercises (unilateral movements), if your not already. If the smaller size muscle groups are all the same side and you do use unilateral exercises for your delts, bis, and tris then you could have nerve damage. This would lead to less muscle myofibril stimulation leading to atrophy of some fibers which you see as smaller size comparing to the other side. This would be an extreme case, but it's not uncommon especially since you were a football player and took many hits.