The intrinsic problem is that the routing table is growing faster than
we can develop bigger routers. It's not just us. It's growing faster
One should differentiate between the table needed by a router to route
and the table needed to calculate routes. As has been discussed, the
former isn't a problem - you can route 16M different routes with 16M
of memory by simple storing the interface number in a table indexed by
the address. If your router has 15 or fewer interfaces, you can route
16M routes with 8MB (one interface per nibble). That's every /24
address there is. There is no size problem here.
Calculating routes takes more memory and does eventually need to be
limited, but machines that can do that with 256MB + virtual memory are
readily available. So this problem is a ways off.
IMO, this "routing table size problem" has started to create too much
momentum towards "reduce the size" and too little towards "how can we
handle the size". Perhaps it is because the people deciding some of
these things aren't the ones who have to spend thousands of dollars
renumbering thousands of machines on an end user network and/or don't
need their small network to be dual homed.