Is it a betrayal for my husband not to want as much because I gained weight?

We dated 4 yrs, married for 5 yrs, in our mid-40s, no kids. He has known about my weight issues from the start. I am 5'5", was 140lbs when we dated, 175lbs when we married, and 220lbs now. My husband does not want to have with me as often as he used to. He said that he is not as attracted to my body as much as he was. He said "for better or worse" is about things like having a terrible illness, not gaining weight. We have talked and been to counselors. I know marriage takes work. But I believe some things just aren't okay. He should have told me, before we married, that he had a problem with my weight. I feel betrayed. I need to know his affection is constant. I need to know he is "fighting alongside me" not on the sidelines judging me. Are my expectations unreal? Is this not a good enough reason to want a divorce? “Fatty haters" need not respond.