Would your feelings be hurt if ...?

I have a boyfriend of three months. In this short time, we are together a lot. We communicate throughout the day during the week, even while we are both at work. We typically spend each weeknight together. He spends a lot of time with my children. When we are all together, it feels perfect. He has told me he loves me, asked about the possibility of all of us living together, and is very liberal about bringing his two daughters over to spend time with us. On the weekends, however, it is a completely different story. Sometimes, he has side jobs to earn extra money, which I think is great and I have never once begrudged him for that. But, he will go back to his house (that the ex-wife rents) and stay there for the weekend. I simply can't accept that. He assures me he goes back there to maintain the house and has no interest in the ex-wife. His communication with me over the weekends wanes significantly. Without jumping to any conclusions, I spoke calmly with him about this, indicating my hurt feelings and greater concern that he is still very much connected to this woman. He assures me that is not the case. Well, it happened again last weekend, so I finally broke up with him. He went on about how happy I make him and that he does not feel he ignores me. He also told me that he didn't stay at her house, that he stayed with one of his guy friends. His reason for staying there for the weekend is because there were apparently several fallen trees in the backyard and he was trying to clean that up. Admirable ... but I highly doubt that is the whole truth. I explained to him that our ideas of being in a relationship were very different from each other and that I would never be second to another woman. I opened my entire life to this man, gave him my love, cooked for him, did his laundry, and tried my best to make him happy. I know in my heart that he was taking advantage of me and that getting rid of him was the right thing to do. But it sure would help my feelings of loneliness to hear what others think ... hopefully, you will bolster my confidence in my decision to get rid of this man ... I can't lie ... I miss his terribly.