Sorry about the performance lately. The slow down is coming from the server itself. I've been working with my host and it appears there has been some severe spam from other sites hosted on the same server which is taking up most of the resources. That's why the site is slow at times, but then fine a few minutes later.
They're trying to work it out, but have also offered to move us to another server. Unfortunately that's not a simple thing and could result in some downtime and unforseen fires I'd have to put out from my end. So I'm leaving it where it is for now with hopes they'll find and ban the spammers which will help speed the site back up.
To throw another wrinkle into it...as BroadwayBlue mentioned the site is getting more popular which factors into it also. It probably won't be long before I need to look at upgrading the hosting plan which means moving to a new server.