The Republican Party could take three courses as far as their future goes, but I only see one viable course they could take that could make them a governing party again which includes the White House. You are not a governing party when you only control the lower chamber of Congress with a small divided majority at that. What you are is an opposition party with some power, but where you tend to be on the defensive a lot. Because the President is from the other party and you can't even pass anything out of Congress on your own and force a veto because the other party controls the Senate.
I'll start with the two dead bad courses the GOP can take. Stick with the establishment Neoconservatives who believe America should police the world and security and virtue before freedom. Who when it comes to the general election try to appeal to everyone while offending everyone at the same time. Mitt Romney case in point in 2011-12. Go back with the Christian Right and try to tell educated adult Americans "we know best how you should live your lives. And if you don't live your life this way, we'll put you in jail".
Or the only viable option being the Rand Paul Conservative Libertarians where I believe Glen Beck and Andrew Napolitano are part of. That basically says "live and let live when it comes to society and culture as long as you take responsibility for your own decisions and aren't hurting innocent people". When it comes to economic policy "get as much government as possible out of the economy. With the safety net, we can't end it or destroy it, but what we can do is turn it over to the states and locals to run it. And give people who need these services choice in how they get them". Foreign policy, "defend America first and always and be strong enough to defend America. And help our allies who need it, but don't try to police the world".
The last option is the only viable option for the Republican Party because that is where the country is. We are still not that social democracy hoping for some type of socialist utopia. But this is also not 1955 anymore and the country has changed so much politically and culturally and Americans want the freedom to live their own lives and not be directed by government. The GOP should embrace that as they counter Democrats when it comes to economic and foreign policy if they want to govern America again.