What happens to your Oyster card when you do this?

There's nothing to stop you at the end of the line, unlike a ticket barrier on the tube, so you can travel beyond the zones you are covered for. But, as you've stated, you'll need to pay on the way back (or walk back to zone 4!) and if this is a regular journey it wont work out to be worthwhile, a travel card will be much cheaper. And naturally, if you get caught then the fine will definitely not make this worth your while!