No, they shouldn't HAVE to go, but be sure you are NOTguilty of playing your kids against your x. It's tempting.
You need to check the decree to see if you would be in contempt of court to change it without going to court. Most judges would be on your side.
You might try playing nice with x.
Examine yourself that this is BEST for THEM, not you. What is best for them? Answer that, that then STICK to your GUNS. Help your x to see that is the bottom line for both of YOU.
Go back to court and tell them you want your children to be with their DAD, not his girlfriend, when on vacation. SHE doesn't have visitation! (don't be nasty-but be firm.)
He needs to be on VACATION (at least part of the time) when they come. Or with other relatives, not girlfirends or sitters.
Tell the judge he takes the phone away, is keeping secrets, and tries to undermine your authority and your security. He could fool around and lose any visitation unless he behaves himself.
Ask him to change the DATES of his visitation to let your girls go to the party. Or change the party date. You can do this.
They only have one daddy. Unless he is a pedophile or a addict, make easy easy for them to have a relationship with him or you and they will all pay the price.
He can't be a prince or he would still be home, but he is daddy, warts and all. They need to learn forgiveness, and compassion. They can learn it from him. (If you show them how.) Or they can be divorce VICTIMS their whole lives.
What are you teaching them about boundaries, respect for themselves, dealing with difficult people, and situations, and independence?
Good luck, honey.