first off welcome to the club
i wouldn't worry about what your buddies think.
i also, if the area has to many does, have no problem taking a couple for the freezer within the law. but to kill just to kill is an issue. crop damage culling, feeding friends and neighbors, are good reasons to take few out.
if the does are there the older bucks will find them. wiser older bucks will lots of times avoid bait piles while the does and young bucks won't leave them alone.
just remember if you shoot that little buck he won't grow up to be a big buck, its your call if it with in the rules.