Your husband seems to be having a fairly flagrant affair. Even your four-year old knows about it.
In some ways, the ability of a WS (wayward spouse) to tell bare-faced lies is just SHOCKING. But it's all too common.
Your husband has been lying to you for quite some time it would appear. Men don't leave without a reason the way he did. The excuses he gave you were just too flimsy.
Unfortunately, some guys don't believe they've cheated until they've had actual intercourse. They won't recognize the
Emotional Affair as infidelity in the marriage. They'll often stick with the "just friends" story. Once committed to it, they'll continue on sticking with their story even after the affair becomes physical.
It's a common tactic really.

The idea is to keep the wife from finding about the affair so as not to endanger the divorce settlement. After the divorce, the affair couple will act like they've only just now become involved. That's the plan....and he'll say whatever is necessary in order to make it work.
Among the out-and-out lies are the
rationalizations and
justifications. This is how he gives himself permission to proceed with an illicit relationship. He'll sell them well, because he's convinced of them himself. In his foggy brain, he believes these are good enough reasons. But they're only excuse....mountains that he's made out of molehills. Don't buy into it. Because he's going to turn those weak-ass excuses back on YOU in an effort to clear his guilty conscience and make YOU at fault.
"You don't trust me." is a rationalization. If his wife doesn't trust him anyway, then why not go ahead and have the affair? If he's already 'doing the time', then why not 'do the crime'?
"You're just the mother of my children, not my wife" is a justification. This gives him permission to cheat. You're not really a
Wife to him, so why not?
Do you see through his
Affair Babble yet?
Really, it's time to stop being nice to him.

He's just doing what he has to in order to keep you calm. He doesn't want you to make a big fuss while he moves on with his life.
I don't know if you've checked out marriagebuilders.com yet, but what you need right now is "Plan B". The basic synopsis is that in Plan A, while your husband is 'on the fence', you behave sweetly to him. You identify and fulfill as many ENs (emotional needs) as you can. And you continue to call him back to the family and into accountablility.
Plan B is best executed after a really good Plan A. You want your spouse to see what he's going to be missing afterall. It does no good to get into a No Contact situation with someone if they're just going to be glad of it.
I think you've probably put in a pretty good Plan A already though, based on what you've written. Plan B is for the cheater who is 'off the fence' and in the OW's yard. Clearly, he is.
It's time to make him see what he's going to be missing by taking it away from him. You have children together, so No Contact will be nearly impossible. But you can do Minimal Contact. The best case scenario would be that you never discuss anything with him aside from the children, and only briefly as possible.
It's time for you to get a lawyer, and to consider filing for divorce. You need to protect yourself and your family. You need full custody of your children, and you need to protect your home and assets. Right now, he can take whatever he wants.
Quite frankly, if my four-year old came home telling me that "Daddy's girlfriend says she loves me soooo much".....I'd go off like a BALLISTIC MISSILE.

He wouldn't get ANY visitation until I had a full explanation of why some stranger was permitted intimate contact with my children.
If it's at all possible, you need to hire a PI and get the details on the affair. You need to know who this woman is. If she's married or significantly involved, you need to EXPOSE her to her S/O. You need to know exactly what your husband is doing and expose him to the key people in his life. The bubble of
fantasy is burst when the ugliness of the affair is dragged out into the light.
Part of that "fantasy" is holding onto all that you had before. It's not possible. He needs to start dealing with that. There's a price to be paid for abandoning your family. It's time for him to start paying it. This also will make the affair less fun. Don't you think he'd like to have both his OW and his WIFE being cooperative with him? Don't you think he wants you to be his "Friend" after he's done deserting you? Wouldn't that all be so much easier for him?
You don't have to let that fantasy continue. You can give him a
peek through the window of divorce BEFORE it becomes an actuality. You can show him the consequences of his actions NOW, while there's still time for him to change.
He will put tremendous pressure on you when you start messing with his affair. He'll tell you that YOU are the one ruining any chance of reconcilliation. He'll say horrible things to you when you expose him and the OW. He'll continue to try to make this all about you....and not about his affair.
You don't fall for that.

These are common tactics, and the bottom line is that your marriage has NO CHANCE at reconcilliation unless this affair ends.
Further, you don't take him back until he PROVES to you that the affair is over. Change the door locks. Refuse his calls. See a lawyer and make him accountable for his actions. Above all, protect your children and your assets.
If he ever does come back to the family, he'll be glad that you preserved all this for him. If he doesn't, then why would you even begin to care if he's mad at you?
You have to consider that there's a REASON that he's keeping you in the dark. Clearly, he's afraid that you could ruin his plans if you had all the details of what's really going on in his life. He's keeping his distance. He's unapproachable, and keeping you at arm's length with his moodiness. These are indicators that he's worried about what you can do.
So....get the details and then DO IT! Make what he's been avoiding something that has to be faced.
It's extremely difficult to enjoy your new romance while your wife's lawyer is bending you over the settlement table.

The added bonus is that the more stressed out he is....the more stress he'll bring into his relationship with OW. It won't be long before he'll have trouble on EVERY front.