Demand he either stop the affair and all contact with the other woman, or he get the heck out!
Your his wife and you have rights. He needs to get his priorities straight! No, he does not have the right to hurt you and just do as he pleases. If he wants to act single, well, then he might end up being that way. If you don't confront him and make your demands known, he will take it as acceptance. Nothing more screwed up as the self delusion of a cheater!
Be "honest", you will not live like this, it is hurtful to you.
I suggest that you find a strong support group with others who are going through this kind of thing. Whether or not your marriage survives, you need support.
http://www.lifesaviors.com/SI/
http://www.survivinginfidelity.com/
A yahoo group site with lots of helpful information. Look in FILES. Easy to join, then check out the links and files. Not an active support group.
http://groups.yahoo.com/group/AffairsTalk/