I had a similar confrontation in the woods a few years back. And, like you did not cause a scene.
Ended up this guy was nothing but trouble and the poaching habits were observed by many others. The officials stopped me once and thought I was this poacher. We talked things over and cleared my name and he informed me they (the DNR) were doing a huge investigation.
They caught him eventually, drugs, stolen guns, poaching...lots of bad news with this guy.
Clean up your area, it feels good to know jerks like this are behind bars.