I had an old save with 17 unused memory points, so I did some experimenting and found out some interesting things. First, it is possible to have 2 affinities at the same time. I had "Well-travelled one", boosted elementalist + Vandal, and added the affinity Battle Mage.
Another test, I started with Arcane Archer (shown as arcane arrow on the P screen), boosted Vagrant and added "Well-travelled one." Then, I boosted Vandal to 10 points and got the message that I could replace Well-travelled one with a new affinity Battle Mage. 3 options: Maintain old, accept new, or maintain old and stop showing this message. I chose maintain, but then even if I went back and added points to Elementalist & Vandal, when I returned to the normal world, there was no repeat of the message offering to change affinities; so it seems that choosing "maintain" has the same effect as choosing "maintain and stop showing this message."
Did all this again and "accepted" Battle Mage to replace WTO (still had arcane archer, too). Then I boosted Thaumaturge to 10 points, and since I already had 10+ points in Vandal, I should have been offered to switch to Seraph, but no such message. I even increased Thaumaturge to more points than I had in Elementalist, but still no offer of Seraph. So it seems that there is some system of prioritizing affinities when multiple affinities are possible. That is, Elem + Vandal = Battlemage, Thaum + Vandal = Seraph; but if I have 10+ points in Elem, Thaum, and Vandal, I can only get Battlemage, never Seraph, even if I have more points in Thaum than in Elem.
Did it again, adding WTO to Arcane Archer. Then added points to Sinistrope and got the offer to change WTO to Black Mage. Accepted that. Boosted Infiltrator to 10 and got offer to swap Black Mage to Shadow Dancer. Accepted. Then I boosted Elementalist 1 point, but still got no offer to switch again back to Black Mage.
So, whenever I had arcane archer first and WTO second, and then boosted some class to receive an offer to swap affinities, it was always an offer to swap WTO and never an offer to swap arcane archer. Perhaps that is because I had arcane archer first, or perhaps there is some other system of prioritizing affinities at work.
I tried once more, a save with the WTO affinity. I had 10+ points in all 3 rogue classes. Added 11 points to Elem so I should have had Arcane Archer added, but no. Likewise, added 10 points to Thaum but still no arcane archer. Then reloaded the same save and added 10 points in Blade Dancer and immediately received a new affinity Assassin (while keeping WTO). Go figure, I give up.