I actually decided to try having a bounty for the hell of it by sneakily assaulting a guard in Duneville with my bow and I got caught with a 40 pennies bounty. In Duneville, Sun Temple, and Ark Noble area, each guard have their own dialogue that leads to the obligatory fine/jail/attack options.
Just quickly put your weapon away and the guards and everybody should calm down.
For the hell of it, I decided to click jail and in prison itself, you lose 2 skill points and there is no way to escape unless you use a fireball on a nearby guard to open the door while running away without grabbing stuff (I don't think there's a chest with your items?) and it was pretty funny.
However, if you pay your fine, be prepared to lose your stolen goods regardless. I would recommend you put your stolen goods in a secure chest before having to face a guard. Also, facing a guard happens automatically if you pass one or exit from a place.
Duneville dialogue: Alright, let's make it short: you might think you can do whatever the f--- you want down here, but you're wrong. There are rules, and you will respect them if you ever want to see the light of (the) day again. Pay your fine or live with the consequences.
Keeper at Sun Temple dialogue: You bloody fool! Your demeanor desecrates this holy place. We got two ways we can do this: you either pay a fine right now, or I'll take you down to the prison. Make your choice, and make it quick.
City Guard of Ark: I don't know who you think you are, but no one breaks the law and gets away with it. There are two ways we can do this: you either pay your fine right now, or you go straight to jail. And believe me, I'm serious.