The loading times are so long because every quarter is an own worldspace with it's own LOD, not an interior cell like dungeons. We couln't do it another way because each quarter needs many ressources.
You can use the sign posts to fast travel between the different quarters immediately. Also there are telport scrolls to the suntemple and the marketplace. Maybe check out the "mark" and "return" spells - some merchants sell it.
Having an SSD helps a lot with the loading times.