How should I approach my neighbor?

Ok I live in an upstairs apartment with my husband and 2 small children and we have a family that lives downstairs. they didnt really like us @ 1st when we moved in because I guess thay wanted the yard to themselves. However we have managed to get along just fine now that we have been here for almost a 2 yrs. OK well summer started and the husband downstairs started making perverted remarks towards me. Mind you I am a mommy of 2 girls and I wear appropriate attire to set an example for my children.... My cheeks and never hang out of my clothing and I take into well consideration how I carry myself. This afternoon I was uncovering my pool and he asks me if I could swim in a bikini? WTF? Then one time he said if my husband and I ever get into an arguement to tell my husband "by the way dont you think our youngest baby has Mark's eyes?" (the name of the guy downstairs)??? again WTF??? On another occasion He was asking about something in the trunk of our suburu and I told him it was a cover to hide the junk in our trunk....lol.... and of course he starts talking about the junk i9n MY trunk....... he has also told me that I'm the hottest one out of all of my sisters and numerous other things! Too numerous to mention. I dont even like taking my kids out back to play cause his wife works all day and I know he's gunna come out! They are constantly fighting downstairs and on one occasion I know it was because of something he said to another female neighbor! I dont want to tell my husband because He's going to flip out and possibly hurt this man. I love my husband and dont want him to end up in Jail or into any kind of trouble over this. && thats what keeps me from telling him! I have beat around the bush but never told my husband EXACTLY whats this man says to me! This is an every day thing so Im used to it now but it really makes me uncomfortable especially in the presence of my children. When he says these things I NEVER respond I just roll my eyes and ignore him..... I know if I respond then there is going to be a whole lot of arguing between everyone and I dont want or need the conflict!!! He drinks a lot and sometimes I cant tell if he is sober or not and other times he says "you know I'm just joking right?" and thats when I say ok well its not really funny!!! But thats all, I thought he would get a clue when I ignore him and roll my eyes but apparently he doesnt care! I know he needs a good *** woopin (pardon my french) but I cant allow my husband to get into it... I just want to keep the peace. Should I keep doing what Im doing and deal with it? Or is there any other advice anyone has for me? Please let me know your thoughts and opinions and keep in mind that things are easier said then done and when you post your reply I dont wanna hear unrealistic answers that you yourself wouldnt even do! Thank You for reading this!