At the moment you need to teleport to your house, wait for the game to load, than wait 10 seconds and teleport to the shrine that you want to go to.
Please make it so you can teleport to any shrine from anywhere but if you are not at a shrine/in city than a option will come up that will use or pay for removing the cooldown of your house. This will save time having to teleport to a house first, but still keep the original transfer cost.