First make sure it is passive aggression. The silence doesn't necessarily mean he is passive aggressive. If he is PA he will act out his feelings particularly when he knows you are looking. LOL. Like slamming doors, stomping around, distressed sighs, well, you get the picture. All it amounts to is a calculated tantrum that he saves for only you. If someone else drops by or calls on the phone it will usually subside until you two are alone again.
Some people are quiet because they have a certain process. They need time to think about a situation and they can go into a sort of emotional cave to do it. It might seem like they withdraw because their mind is on other things,but it is not what they mean to do --it is just their process. But aside from that, they do not treat you badly, or act aggressively.
If it is passive aggression, he is not going to stop doing it until you stop enabling him to do it. If you let it affect you to the point of begging what is wrong, changing your routine, walking on eggshells and so on, then he is going to keep doing it because he knows it bothers you.
If you want it to stop then leave him to it. Do not change your own behavior; go about your business as if it wasn't happening. I know it is hard, but it is the same way you have to deal with a child who is in a temper tantrum. You have to ignore it, and eventually they they will stop.