I've experienced this since very early in the game, and am still experiencing it with a Sneak skill in the low 50s and Personality in the mid-70s. When I was walking around with Kim with the same-ish stats, almost everything would attack her before me. There seems to be kind of an order to it, though: rats and crabs never attack me, diseased deer initially attacked me but don't anymore, human enemies vary in their responses (often I would find a group of three bandits, two of whom wouldn't attack until the third saw me and drew his weapon) but seem to have become less likely to attack me over time, and most types of beasts and undead attack me either often or always. I also stumbled across a cave, the Enclave/Vault of Jenada, where none of the druids, spriggans, and spider spriggans attacked except for the "boss", but that might have been intentional.
DJ_McLovin: You might also try drawing your weapon when you see enemies. Everything that doesn't attack me will attack when I do so.