We can't tell you what to do but if you're unhappy with the situation then why not just leave?
He's happy with a FWB situation. Men who can't say yes, also can't say no. Don't think for a second that if he fell for another woman he wouldn't go after her without giving you another thought.
IMO, you need to let him go and move on but that is only because I couldn't be happy in that kind of situation. If you can be then great, hang out, have some fun and when one of you meets other people it will just end. If you're posting here about your feelings for him then it doesn't sound like you can handle the latter.
Best of luck