What's the danger in raising the gas limit even further?

What bad things could actually happen if we had a higher gas limit (say 20,000,000)? Storage requirements wouldn't be all that bad, and would be plenty easy for retail businesses to host nodes still. 5 terabyte hard drives aren't even that expensive, and can be speed up quite easily by using a smaller ssd as a cache.

Is CPU power a big bottleneck? That ones a little harder to analyze cause there isn't any definitive data on it.

I don't think bandwidth is an issue. Block on average are currently 52 kb source. The FCC defined a broadband connection as one with 25 Mbps download and 3 Mbps upload, which can easily download and upload 52 kilobytes every 13 seconds, and it has the capacity for much more, even doubling the block size would still allow it to run on a broadband connection, which is very slow compared to most modern internet connections.

Spam attacks wouldn't really be possible, since people would still have to pay pretty high fees, at least above 20 gwei I'd imagine.

So what's stopping miners from wanting to raise the gas limit?