Let's put it this way. If you leave him, it won't hurt his heart because it's unlikely he has one.
His ego most probably, because, right now, you're his rug!
So, HE may hurt you PHYSICALLY if you try to leave him. All the more reasons to do so and stay awya from him for good.
I read not long ago, about a girl who left her family to move in with her young boy friend that everybody was warning her of.
She wouldn't listen, had two children with him, and ended up confiding ther fears of him to her sister.
And guess what, the sister ended up killing the guy in self defense.
I'm telling you, he's putting you down because, he thinks that that's were you belong. He has no respect for you as he doesn't think you deserve any and by standing by him, you are telling him that he's right.
So, do yourself and your children a favour and get out.
Surely, you wouldn't want your children to be using the same type of behaviour towards you later on? then, you'd better start showing them that it's not right!