You always have to be careful with grunt calls. It sounds to me like your call was probably too loud and he was looking around for the buck and saw nothing. He more than likely knew something was up and decided the safest alternative would be to 180 and go back the way he came. Remember, anytime you SEE a buck and grunt at them, do it softly at first and if you don't get his attention you can increase the volume. You don't want him to think "man that was LOUD, where's that buck at that I just walked right up on?" you want him to think "what was that? That sounded like a grunt coming from 200yds that way"
...and on that day when my strength is failing, the end draws near and my times has come, still my soul will sing Your praise unending, ten thousand years and then forevermore!