Google wants your website to be faster. Matt Cutts recent interview at PubCon clearly indicates this. So if your website is slow, pull-up your sleeves and start working to make it faster. It would not only make your visitors happy, but can also improve your search rankings.

• Identify garbage codes on your pages and remove it. Html, css, javascript, any junk that has piled up there should go. 
• Follow web standards. Check w3c validators to remove and correct your codes.
• Happy talk should die. Check, edit and remove unnecessary content that does not create value.
• Reduce graphic images and file sizes or other media which is creating overload without creating much design and usability benefits.
• Be real stingy

Website load speed is a usability issue. By making your site faster you are providing users a pleasure equivalent to driving on the freeways.

Matt Cutts Interview Video on Page Load Speed