I think I have one of the worst ISPs in the country (and speedtest.net tends to agree). We get 1.5mbps download speed on a good day, but sometimes it's well below that point. I've had entire weeks where I missed 56k speeds, because they seriously would have been faster. Once, we got a maximum of .08mpbs for most of the day.
I have two roommates, and sometimes it's difficult for all three of us to use the internet at the same time, even if we're browsing primarily text-based sites that don't eat much bandwidth.
I have no idea how this happens when I live in the middle of a city that is known for its tech sector. I mean, the Googleplex is only a few minutes away, and the free wifi they provide downtown is more reliable than my home connection.
First world problems, man.