I have a similar arrangement with my partner, except his son is much younger (nearly 4).
Your partner will be feeling pressure to give his son his entire attention when he does get to see him given it's not all that often. My guy does this too, but I hear ya honey, and it is difficult to deal with isn't it when suddenly he doesn't seem to feel he can treat you the same as he always does. He may well be afraid of making his son jealous.
Also, you consider their feelings by giving them space on the first day of he visit, and he thanks you for it by shutting you off! Where is the appreciation for the respect you show them? Does your partner actually think how blimmin difficult the situation is on you? Probably not - he's too wrapped up in how it affects his son.
I can only really tell you how I cope with all these things: when his son is around, I do not expect my partner's full attention, and I see it as being one of the sacrifices I have to make if I chose to be with someone who has a child from a prev relationship. I do not give him a hard time about it because I feel sorry for his son and for my partner not getting to see him much. I know he misses him terribly. Also I try to get involved with some of what they do (my partner encourages this which is nice for me), so sometimes I'll come too if they're off to cinema or the park, but I am also careful to allow them time just those two together.
I wonder how long have you and your partner lived together? I remember in the beginning I found it really hard and had very similar feelings to you, but in time his son and me got used to eachother and got to know eachother (it was a big adjustment for both of us!).
These days all three of us get on really really well together, I look forward to his son's visits and I miss him too when he isn't around. We talk about him and make plans together for things he might like for Christmas, what new disney Dvd's he might like etc.
So I would say give it time, and try to gradually build a relationship with his son too, just be nice, be fun, ask him about things in his life. Talk with your partner about how you feel, that you're not trying to attack his son but just would like to alter the dynamic a little so that you aren't left feeling like a wall flower. It can work out :)
Good luck x