What is the matter with my horse?

I had a similar problem a few years ago with my gelding to the point of him not even wanting me to brush him. I had a lady who did equine mage out a few times and It did improve alot. I'd say theres definately something bothering him, does his saddle fit ok/ has he changed shape since it was last fitted? it could be something really simple bothering him lol