Nothing wrong with trying something new. That said, I'm not sure there's enough difference between a 150 and a 165gr out of an '06 to make all that much difference on deer. A like bullet (meaning the same kind) in 165gr would actually more than likely overcome any velocity advantage the 150gr had at the muzzle at longer ranges due to it's better B.C.
Either weight bullet will work very well for deer in an '06 and I'd likely choose the one that showed better accuracy as a good reason to use it, if the gun showed a preference one way or another. Good Luck!