The background image in the Hello World portlet is a part of the Classic Theme. If you create your own theme or use another theme, the mountain image won't be there.

Ok I think what I need to do is edit the hello world portlet. Is that possible? I'm guessing I'd have to edit the source and redeploy it? Ie, not gunna happen?

It's not expected to have the Hello World portlet on any production page - you can just use the content management system (e.g. Web Content Display Portlet) and create an article displaying an image of your taste on any page. This way you can even change it at runtime, matching the seasons (granted, there's rumors that the seasons in Florida are "summer" and "23 February").

No need for patching the source or even deploying anything to your server.