My specs are similar to OP's, except I have a GTX 860M (won't make much difference), and I get playable fps the whole way through. Some places in the game do lag a little bit, but it's rarely something that can't be fixed if you don't mind changing some settings occasionally. Regardless, at 1080p I generally get 30-60fps almost all of the time.
If you aren't getting this, first... first I would try putting all the settings right down, and seeing if that makes any difference (low grass distance, no grass shadows, etc, are usually the big things). If this does not do anything, then delete the Oblivion ini to reset all the settings to default, just in case there is some other problem (the one in your user folder, Documents\Games\Oblivion or whatever, NOT the oblivion_default.ini in your Nehrim folder). If you still aren't getting a playable framerate after that, then there must be some other issue (Drivers, DirectX, etc).
Also something I found was, when I used OBSE, even without mods or plugins, it seemed to make the game run a lot less smoothly.
But to repeat what Caleb said: in some areas, you will get bad fps. The big cities (Erothin, Cahbaet, Ostian), do lag a bit, so do some bleak and stormy places as well; The Dark Woods/Forest (near Treomar) and The Crater (in Southrealm) are good examples (forgive me if the place names are a bit wrong, I played the German version). Usually I find that turning off distant terrain can improve things a LOT, but, changing the view distance after that doesn't really have much more of an effect (in my experience).