I often opt to leave one or more default routes configured with low priority (lower than BGP). The thinking is that if there is a fault with BGP, the router will still operate and the fault can be corrected remotely (in-band). The downside is that I might pass traffic for non-existing destinations an additional hop and put the load of generating an ICMP unreachable on someone else's router.
--Blake