Here in Missouri, we paint our fence posts PURPLE, or at least the tops of them. If I were you I'd paint them purple where this guy is looking onto your land. PURPLE paint means NO HUNTING/NO TREASPASSING. Also, is he bow hunting or gun? If it's bow, he might be waiting for deer to come on his place, but if you know it's gun, I would definately try to talk to the guy and/or talk to the game warden if he is non-compliant. I would also invest in some "no hunting" signs, that way if this guy does "hunt your property", he'll be in more serious trouble.
Don't know about everyone else, but I can't stand people who don't respect others' land. Some people think they can hunt anywhere and that fences don't matter. This is how people end up hurt, or dead on hunting trips.